2024-04-15T15:03:38.665 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-15T15:03:38.665 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657120 --verbose --name leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi --owner scheduled_leonidus@teuthology --description fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/quiesce} -- /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657120/orig.config.yaml 2024-04-15T15:03:38.709 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-15T15:03:38.820 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657120 branch: wip-lusov-quiesce-xlock description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/quiesce} email: null first_in_suite: false job_id: '7657120' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-quiesce-xlock ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow request sha1: 9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 subvolume_version: 1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-quiesce-xlock sha1: 9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 2288 sha1: 9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 sleep_before_teardown: 0 suite: fs suite_branch: wip-lusov-quiesce-xlock suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi072.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG+jvD3lC11LrXoo70wa+dUFmGv6veuLETc7MspcK85RTHe7zimRkbrxwOjw8TurdnsOI7szwqxlzsmbcgrpAUA= smithi175.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGpS+fS1n7SHFRiy96aTxyNYq1ubnETNBoHqo+uqNcC3llhjvSVSFpQM/r0JijXdeTsCN9v8KOD5Q2Fkkvx/hlQ= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - ceph-fuse: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_quiesce teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-15_14:46:31 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-15T15:03:38.820 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa; will attempt to use it 2024-04-15T15:03:38.821 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks 2024-04-15T15:03:38.821 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-15T15:03:38.823 INFO:teuthology.task.internal:Checking packages... 2024-04-15T15:03:38.847 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14' 2024-04-15T15:03:38.847 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-15T15:03:38.847 INFO:teuthology.packaging:ref: None 2024-04-15T15:03:38.847 INFO:teuthology.packaging:tag: None 2024-04-15T15:03:38.847 INFO:teuthology.packaging:branch: wip-lusov-quiesce-xlock 2024-04-15T15:03:38.847 INFO:teuthology.packaging:sha1: 9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 2024-04-15T15:03:38.847 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-quiesce-xlock 2024-04-15T15:03:38.990 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2861.g9bf1c4ba 2024-04-15T15:03:38.992 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-15T15:03:38.999 INFO:teuthology.task.internal:no buildpackages task found 2024-04-15T15:03:38.999 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-15T15:03:39.017 INFO:teuthology.task.internal:Saving configuration 2024-04-15T15:03:39.034 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-15T15:03:39.039 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-15T15:03:39.065 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi072.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657120', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 14:58:17.161213', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG+jvD3lC11LrXoo70wa+dUFmGv6veuLETc7MspcK85RTHe7zimRkbrxwOjw8TurdnsOI7szwqxlzsmbcgrpAUA='} 2024-04-15T15:03:39.091 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi175.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657120', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 14:58:17.163025', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGpS+fS1n7SHFRiy96aTxyNYq1ubnETNBoHqo+uqNcC3llhjvSVSFpQM/r0JijXdeTsCN9v8KOD5Q2Fkkvx/hlQ='} 2024-04-15T15:03:39.091 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-15T15:03:39.097 INFO:teuthology.task.internal:roles: ubuntu@smithi072.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-04-15T15:03:39.097 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-04-15T15:03:39.097 INFO:teuthology.run_tasks:Running task console_log... 2024-04-15T15:03:39.171 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f93f876e0d0>, signals=[15]) 2024-04-15T15:03:39.171 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-15T15:03:39.177 INFO:teuthology.task.internal:Opening connections... 2024-04-15T15:03:39.178 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi072.front.sepia.ceph.com 2024-04-15T15:03:39.179 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:03:39.269 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi175.front.sepia.ceph.com 2024-04-15T15:03:39.269 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:03:39.359 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-15T15:03:39.372 DEBUG:teuthology.orchestra.run.smithi072:> uname -m 2024-04-15T15:03:39.389 INFO:teuthology.orchestra.run.smithi072.stdout:x86_64 2024-04-15T15:03:39.389 DEBUG:teuthology.orchestra.run.smithi072:> cat /etc/os-release 2024-04-15T15:03:39.445 INFO:teuthology.orchestra.run.smithi072.stdout:NAME="CentOS Stream" 2024-04-15T15:03:39.446 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION="9" 2024-04-15T15:03:39.446 INFO:teuthology.orchestra.run.smithi072.stdout:ID="centos" 2024-04-15T15:03:39.446 INFO:teuthology.orchestra.run.smithi072.stdout:ID_LIKE="rhel fedora" 2024-04-15T15:03:39.446 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION_ID="9" 2024-04-15T15:03:39.446 INFO:teuthology.orchestra.run.smithi072.stdout:PLATFORM_ID="platform:el9" 2024-04-15T15:03:39.446 INFO:teuthology.orchestra.run.smithi072.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-15T15:03:39.446 INFO:teuthology.orchestra.run.smithi072.stdout:ANSI_COLOR="0;31" 2024-04-15T15:03:39.446 INFO:teuthology.orchestra.run.smithi072.stdout:LOGO="fedora-logo-icon" 2024-04-15T15:03:39.446 INFO:teuthology.orchestra.run.smithi072.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-15T15:03:39.447 INFO:teuthology.orchestra.run.smithi072.stdout:HOME_URL="https://centos.org/" 2024-04-15T15:03:39.447 INFO:teuthology.orchestra.run.smithi072.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-15T15:03:39.447 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-15T15:03:39.447 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-15T15:03:39.447 INFO:teuthology.lock.ops:Updating smithi072.front.sepia.ceph.com on lock server 2024-04-15T15:03:39.474 DEBUG:teuthology.orchestra.run.smithi175:> uname -m 2024-04-15T15:03:39.492 INFO:teuthology.orchestra.run.smithi175.stdout:x86_64 2024-04-15T15:03:39.492 DEBUG:teuthology.orchestra.run.smithi175:> cat /etc/os-release 2024-04-15T15:03:39.550 INFO:teuthology.orchestra.run.smithi175.stdout:NAME="CentOS Stream" 2024-04-15T15:03:39.550 INFO:teuthology.orchestra.run.smithi175.stdout:VERSION="9" 2024-04-15T15:03:39.550 INFO:teuthology.orchestra.run.smithi175.stdout:ID="centos" 2024-04-15T15:03:39.550 INFO:teuthology.orchestra.run.smithi175.stdout:ID_LIKE="rhel fedora" 2024-04-15T15:03:39.550 INFO:teuthology.orchestra.run.smithi175.stdout:VERSION_ID="9" 2024-04-15T15:03:39.550 INFO:teuthology.orchestra.run.smithi175.stdout:PLATFORM_ID="platform:el9" 2024-04-15T15:03:39.550 INFO:teuthology.orchestra.run.smithi175.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-15T15:03:39.550 INFO:teuthology.orchestra.run.smithi175.stdout:ANSI_COLOR="0;31" 2024-04-15T15:03:39.550 INFO:teuthology.orchestra.run.smithi175.stdout:LOGO="fedora-logo-icon" 2024-04-15T15:03:39.551 INFO:teuthology.orchestra.run.smithi175.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-15T15:03:39.551 INFO:teuthology.orchestra.run.smithi175.stdout:HOME_URL="https://centos.org/" 2024-04-15T15:03:39.551 INFO:teuthology.orchestra.run.smithi175.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-15T15:03:39.551 INFO:teuthology.orchestra.run.smithi175.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-15T15:03:39.551 INFO:teuthology.orchestra.run.smithi175.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-15T15:03:39.551 INFO:teuthology.lock.ops:Updating smithi175.front.sepia.ceph.com on lock server 2024-04-15T15:03:39.574 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-15T15:03:39.585 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-15T15:03:39.590 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-15T15:03:39.590 DEBUG:teuthology.orchestra.run.smithi072:> test '!' -e /home/ubuntu/cephtest 2024-04-15T15:03:39.593 DEBUG:teuthology.orchestra.run.smithi175:> test '!' -e /home/ubuntu/cephtest 2024-04-15T15:03:39.608 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-15T15:03:39.614 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-15T15:03:39.615 DEBUG:teuthology.orchestra.run.smithi072:> test -z $(ls -A /var/lib/ceph) 2024-04-15T15:03:39.649 DEBUG:teuthology.orchestra.run.smithi175:> test -z $(ls -A /var/lib/ceph) 2024-04-15T15:03:39.762 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-15T15:03:39.813 INFO:teuthology.run_tasks:Running task kernel... 2024-04-15T15:03:39.827 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-15T15:03:39.827 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-15T15:03:39.828 DEBUG:teuthology.orchestra.run.smithi072:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-15T15:03:39.828 DEBUG:teuthology.orchestra.run.smithi175:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-15T15:03:39.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:03:39.843 DEBUG:teuthology.orchestra.run.smithi072:> uname -r 2024-04-15T15:03:39.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:03:39.844 DEBUG:teuthology.orchestra.run.smithi175:> uname -r 2024-04-15T15:03:39.899 INFO:teuthology.orchestra.run.smithi072.stdout:5.14.0-361.el9.x86_64 2024-04-15T15:03:39.899 INFO:teuthology.task.kernel:Running kernel on smithi072: 5.14.0-361.el9.x86_64 2024-04-15T15:03:39.899 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum install -y kernel 2024-04-15T15:03:39.901 INFO:teuthology.orchestra.run.smithi175.stdout:5.14.0-361.el9.x86_64 2024-04-15T15:03:39.901 INFO:teuthology.task.kernel:Running kernel on smithi175: 5.14.0-361.el9.x86_64 2024-04-15T15:03:39.901 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum install -y kernel 2024-04-15T15:03:42.032 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-04-15T15:03:43.099 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-04-15T15:03:43.466 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.0 MB 00:00 2024-04-15T15:03:44.492 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2024-04-15T15:03:45.435 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-04-15T15:03:47.689 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Stream 9 - BaseOS 2.1 MB/s | 8.0 MB 00:03 2024-04-15T15:03:50.037 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - AppStream 5.5 MB/s | 19 MB 00:03 2024-04-15T15:03:55.131 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Stream 9 - AppStream 3.0 MB/s | 19 MB 00:06 2024-04-15T15:03:55.172 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - CRB 4.5 MB/s | 6.2 MB 00:01 2024-04-15T15:03:56.924 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 16 kB 00:00 2024-04-15T15:03:59.062 INFO:teuthology.orchestra.run.smithi072.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:01 2024-04-15T15:03:59.189 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-15T15:04:01.065 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 16 kB 00:00 2024-04-15T15:04:04.271 INFO:teuthology.orchestra.run.smithi072.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-04-15T15:04:06.008 INFO:teuthology.orchestra.run.smithi175.stdout:Extra Packages for Enterprise Linux 4.3 MB/s | 21 MB 00:04 2024-04-15T15:04:06.692 INFO:teuthology.orchestra.run.smithi072.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-15T15:04:06.692 INFO:teuthology.orchestra.run.smithi072.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-15T15:04:06.793 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-04-15T15:04:06.809 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-04-15T15:04:06.809 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-04-15T15:04:06.809 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-04-15T15:04:06.809 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-04-15T15:04:06.809 INFO:teuthology.orchestra.run.smithi072.stdout: kernel x86_64 5.14.0-438.el9 CentOS-BaseOS 277 k 2024-04-15T15:04:06.809 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-04-15T15:04:06.810 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-core x86_64 5.14.0-438.el9 CentOS-BaseOS 15 M 2024-04-15T15:04:06.810 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules x86_64 5.14.0-438.el9 CentOS-BaseOS 34 M 2024-04-15T15:04:06.810 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules-core x86_64 5.14.0-438.el9 CentOS-BaseOS 28 M 2024-04-15T15:04:06.810 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:04:06.810 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-04-15T15:04:06.810 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-04-15T15:04:06.810 INFO:teuthology.orchestra.run.smithi072.stdout:Install 4 Packages 2024-04-15T15:04:06.810 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:04:06.811 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 77 M 2024-04-15T15:04:06.811 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 123 M 2024-04-15T15:04:06.811 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-04-15T15:04:06.994 INFO:teuthology.orchestra.run.smithi072.stdout:(1/4): kernel-5.14.0-438.el9.x86_64.rpm 1.5 MB/s | 277 kB 00:00 2024-04-15T15:04:07.974 INFO:teuthology.orchestra.run.smithi072.stdout:(2/4): kernel-core-5.14.0-438.el9.x86_64.rpm 13 MB/s | 15 MB 00:01 2024-04-15T15:04:08.749 INFO:teuthology.orchestra.run.smithi072.stdout:(3/4): kernel-modules-5.14.0-438.el9.x86_64.rpm 17 MB/s | 34 MB 00:01 2024-04-15T15:04:09.301 INFO:teuthology.orchestra.run.smithi072.stdout:(4/4): kernel-modules-core-5.14.0-438.el9.x86_6 12 MB/s | 28 MB 00:02 2024-04-15T15:04:09.305 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-04-15T15:04:09.305 INFO:teuthology.orchestra.run.smithi072.stdout:Total 31 MB/s | 77 MB 00:02 2024-04-15T15:04:09.305 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-04-15T15:04:09.714 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-04-15T15:04:09.714 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-04-15T15:04:10.539 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-04-15T15:04:10.540 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-04-15T15:04:11.015 INFO:teuthology.orchestra.run.smithi175.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-15T15:04:12.812 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-04-15T15:04:13.323 INFO:teuthology.orchestra.run.smithi175.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-15T15:04:13.323 INFO:teuthology.orchestra.run.smithi175.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-15T15:04:13.415 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-04-15T15:04:13.422 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-04-15T15:04:13.423 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2024-04-15T15:04:13.423 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-04-15T15:04:13.423 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-04-15T15:04:13.423 INFO:teuthology.orchestra.run.smithi175.stdout: kernel x86_64 5.14.0-438.el9 CentOS-BaseOS 277 k 2024-04-15T15:04:13.423 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2024-04-15T15:04:13.423 INFO:teuthology.orchestra.run.smithi175.stdout: kernel-core x86_64 5.14.0-438.el9 CentOS-BaseOS 15 M 2024-04-15T15:04:13.423 INFO:teuthology.orchestra.run.smithi175.stdout: kernel-modules x86_64 5.14.0-438.el9 CentOS-BaseOS 34 M 2024-04-15T15:04:13.423 INFO:teuthology.orchestra.run.smithi175.stdout: kernel-modules-core x86_64 5.14.0-438.el9 CentOS-BaseOS 28 M 2024-04-15T15:04:13.423 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:04:13.423 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-04-15T15:04:13.423 INFO:teuthology.orchestra.run.smithi175.stdout:================================================================================ 2024-04-15T15:04:13.423 INFO:teuthology.orchestra.run.smithi175.stdout:Install 4 Packages 2024-04-15T15:04:13.423 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:04:13.423 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 77 M 2024-04-15T15:04:13.424 INFO:teuthology.orchestra.run.smithi175.stdout:Installed size: 123 M 2024-04-15T15:04:13.424 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-04-15T15:04:13.473 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-modules-core-5.14.0-438.el9.x86_64 1/4 2024-04-15T15:04:13.580 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T15:04:13.583 INFO:teuthology.orchestra.run.smithi175.stdout:(1/4): kernel-5.14.0-438.el9.x86_64.rpm 1.7 MB/s | 277 kB 00:00 2024-04-15T15:04:14.170 INFO:teuthology.orchestra.run.smithi175.stdout:(2/4): kernel-core-5.14.0-438.el9.x86_64.rpm 20 MB/s | 15 MB 00:00 2024-04-15T15:04:15.667 INFO:teuthology.orchestra.run.smithi175.stdout:(3/4): kernel-modules-core-5.14.0-438.el9.x86_6 13 MB/s | 28 MB 00:02 2024-04-15T15:04:16.067 INFO:teuthology.orchestra.run.smithi175.stdout:(4/4): kernel-modules-5.14.0-438.el9.x86_64.rpm 13 MB/s | 34 MB 00:02 2024-04-15T15:04:16.070 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-04-15T15:04:16.071 INFO:teuthology.orchestra.run.smithi175.stdout:Total 29 MB/s | 77 MB 00:02 2024-04-15T15:04:16.071 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-04-15T15:04:16.314 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T15:04:16.442 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-04-15T15:04:16.442 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-04-15T15:04:16.501 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T15:04:17.294 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-04-15T15:04:17.294 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-04-15T15:04:19.581 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-04-15T15:04:20.194 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : kernel-modules-core-5.14.0-438.el9.x86_64 1/4 2024-04-15T15:04:20.292 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T15:04:21.216 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T15:04:21.263 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:04:23.018 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T15:04:23.142 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T15:04:25.992 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:04:27.851 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T15:04:28.140 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : kernel-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:04:32.854 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:04:57.148 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:04:57.148 INFO:teuthology.orchestra.run.smithi072.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-15T15:04:57.148 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:04:57.193 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:04:57.622 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:04:57.622 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-5.14.0-438.el9.x86_64 1/4 2024-04-15T15:04:57.622 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T15:04:57.622 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T15:04:58.019 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:04:58.019 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:04:58.019 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-04-15T15:04:58.019 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-5.14.0-438.el9.x86_64 2024-04-15T15:04:58.019 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-core-5.14.0-438.el9.x86_64 2024-04-15T15:04:58.019 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules-5.14.0-438.el9.x86_64 2024-04-15T15:04:58.019 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules-core-5.14.0-438.el9.x86_64 2024-04-15T15:04:58.020 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:04:58.020 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-04-15T15:04:58.476 DEBUG:teuthology.orchestra.run.smithi072:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T15:04:58.612 INFO:teuthology.orchestra.run.smithi072.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-15T15:04:58.612 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-15T15:04:58.612 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-438.el9.x86_64 2024-04-15T15:04:58.612 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-15T15:04:58.612 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-15T15:04:58.613 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-15T15:04:58.613 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi072.front.sepia.ceph.com, path=None, version=distro) 2024-04-15T15:04:58.613 DEBUG:teuthology.orchestra.run.smithi072:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T15:04:58.693 INFO:teuthology.orchestra.run.smithi072.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-15T15:04:58.694 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-15T15:04:58.694 DEBUG:teuthology.orchestra.run.smithi072:> sudo rpm -qi grub2-tools 2024-04-15T15:04:58.738 INFO:teuthology.orchestra.run.smithi072.stdout:Name : grub2-tools 2024-04-15T15:04:58.738 INFO:teuthology.orchestra.run.smithi072.stdout:Epoch : 1 2024-04-15T15:04:58.738 INFO:teuthology.orchestra.run.smithi072.stdout:Version : 2.06 2024-04-15T15:04:58.738 INFO:teuthology.orchestra.run.smithi072.stdout:Release : 68.el9 2024-04-15T15:04:58.738 INFO:teuthology.orchestra.run.smithi072.stdout:Architecture: x86_64 2024-04-15T15:04:58.738 INFO:teuthology.orchestra.run.smithi072.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-15T15:04:58.738 INFO:teuthology.orchestra.run.smithi072.stdout:Group : Unspecified 2024-04-15T15:04:58.738 INFO:teuthology.orchestra.run.smithi072.stdout:Size : 8264254 2024-04-15T15:04:58.738 INFO:teuthology.orchestra.run.smithi072.stdout:License : GPLv3+ 2024-04-15T15:04:58.738 INFO:teuthology.orchestra.run.smithi072.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-15T15:04:58.739 INFO:teuthology.orchestra.run.smithi072.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-15T15:04:58.739 INFO:teuthology.orchestra.run.smithi072.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-15T15:04:58.739 INFO:teuthology.orchestra.run.smithi072.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-15T15:04:58.739 INFO:teuthology.orchestra.run.smithi072.stdout:Packager : builder@centos.org 2024-04-15T15:04:58.739 INFO:teuthology.orchestra.run.smithi072.stdout:Vendor : CentOS 2024-04-15T15:04:58.739 INFO:teuthology.orchestra.run.smithi072.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-15T15:04:58.739 INFO:teuthology.orchestra.run.smithi072.stdout:Summary : Support tools for GRUB. 2024-04-15T15:04:58.739 INFO:teuthology.orchestra.run.smithi072.stdout:Description : 2024-04-15T15:04:58.739 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:04:58.739 INFO:teuthology.orchestra.run.smithi072.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-15T15:04:58.739 INFO:teuthology.orchestra.run.smithi072.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-15T15:04:58.739 INFO:teuthology.orchestra.run.smithi072.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-15T15:04:58.739 INFO:teuthology.orchestra.run.smithi072.stdout:hardware devices. 2024-04-15T15:04:58.739 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:04:58.739 INFO:teuthology.orchestra.run.smithi072.stdout:This subpackage provides tools for support of all platforms. 2024-04-15T15:04:58.740 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-15T15:04:58.740 INFO:teuthology.task.kernel:Updating grub on smithi072 to boot 5.14.0-438.el9.x86_64 2024-04-15T15:04:58.740 DEBUG:teuthology.orchestra.run.smithi072:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-15T15:04:59.400 INFO:teuthology.orchestra.run.smithi072.stderr:Generating grub configuration file ... 2024-04-15T15:05:01.002 INFO:teuthology.orchestra.run.smithi072.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-15T15:05:01.027 INFO:teuthology.orchestra.run.smithi072.stderr:done 2024-04-15T15:05:01.029 DEBUG:teuthology.orchestra.run.smithi072:> mktemp 2024-04-15T15:05:01.055 INFO:teuthology.orchestra.run.smithi072.stdout:/tmp/tmp.SY7pFJv5aX 2024-04-15T15:05:01.056 DEBUG:teuthology.orchestra.run.smithi072:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.SY7pFJv5aX 2024-04-15T15:05:01.124 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0666 /tmp/tmp.SY7pFJv5aX 2024-04-15T15:05:01.266 DEBUG:teuthology.orchestra.remote:smithi072:/tmp/tmp.SY7pFJv5aX is 6KB 2024-04-15T15:05:01.315 DEBUG:teuthology.orchestra.run.smithi072:> rm -fr /tmp/tmp.SY7pFJv5aX 2024-04-15T15:05:01.330 DEBUG:teuthology.orchestra.run.smithi072:> sudo /bin/ls /boot/loader/entries || true 2024-04-15T15:05:01.395 INFO:teuthology.orchestra.run.smithi072.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-15T15:05:01.395 INFO:teuthology.orchestra.run.smithi072.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-15T15:05:01.395 INFO:teuthology.orchestra.run.smithi072.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-15T15:05:01.395 INFO:teuthology.orchestra.run.smithi072.stdout:15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64.conf 2024-04-15T15:05:01.396 DEBUG:teuthology.orchestra.run.smithi072:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64 2024-04-15T15:05:01.544 DEBUG:teuthology.orchestra.run.smithi072:> sudo shutdown -r now 2024-04-15T15:05:02.465 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:05:02.465 INFO:teuthology.orchestra.run.smithi175.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-15T15:05:02.465 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:05:02.507 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:05:02.925 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: kernel-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:05:02.925 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : kernel-5.14.0-438.el9.x86_64 1/4 2024-04-15T15:05:02.925 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T15:05:02.925 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T15:05:03.341 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:05:03.341 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:05:03.341 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-04-15T15:05:03.341 INFO:teuthology.orchestra.run.smithi175.stdout: kernel-5.14.0-438.el9.x86_64 2024-04-15T15:05:03.341 INFO:teuthology.orchestra.run.smithi175.stdout: kernel-core-5.14.0-438.el9.x86_64 2024-04-15T15:05:03.341 INFO:teuthology.orchestra.run.smithi175.stdout: kernel-modules-5.14.0-438.el9.x86_64 2024-04-15T15:05:03.341 INFO:teuthology.orchestra.run.smithi175.stdout: kernel-modules-core-5.14.0-438.el9.x86_64 2024-04-15T15:05:03.341 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:05:03.341 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-04-15T15:05:03.690 DEBUG:teuthology.orchestra.run.smithi175:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T15:05:03.779 INFO:teuthology.orchestra.run.smithi175.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-15T15:05:03.780 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-15T15:05:03.780 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-438.el9.x86_64 2024-04-15T15:05:03.780 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-15T15:05:03.780 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-15T15:05:03.780 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-15T15:05:03.780 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi175.front.sepia.ceph.com, path=None, version=distro) 2024-04-15T15:05:03.780 DEBUG:teuthology.orchestra.run.smithi175:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T15:05:03.860 INFO:teuthology.orchestra.run.smithi175.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-15T15:05:03.861 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-15T15:05:03.861 DEBUG:teuthology.orchestra.run.smithi175:> sudo rpm -qi grub2-tools 2024-04-15T15:05:03.900 INFO:teuthology.orchestra.run.smithi175.stdout:Name : grub2-tools 2024-04-15T15:05:03.900 INFO:teuthology.orchestra.run.smithi175.stdout:Epoch : 1 2024-04-15T15:05:03.900 INFO:teuthology.orchestra.run.smithi175.stdout:Version : 2.06 2024-04-15T15:05:03.900 INFO:teuthology.orchestra.run.smithi175.stdout:Release : 68.el9 2024-04-15T15:05:03.900 INFO:teuthology.orchestra.run.smithi175.stdout:Architecture: x86_64 2024-04-15T15:05:03.900 INFO:teuthology.orchestra.run.smithi175.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-15T15:05:03.900 INFO:teuthology.orchestra.run.smithi175.stdout:Group : Unspecified 2024-04-15T15:05:03.900 INFO:teuthology.orchestra.run.smithi175.stdout:Size : 8264254 2024-04-15T15:05:03.900 INFO:teuthology.orchestra.run.smithi175.stdout:License : GPLv3+ 2024-04-15T15:05:03.900 INFO:teuthology.orchestra.run.smithi175.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-15T15:05:03.901 INFO:teuthology.orchestra.run.smithi175.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-15T15:05:03.901 INFO:teuthology.orchestra.run.smithi175.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-15T15:05:03.901 INFO:teuthology.orchestra.run.smithi175.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-15T15:05:03.901 INFO:teuthology.orchestra.run.smithi175.stdout:Packager : builder@centos.org 2024-04-15T15:05:03.901 INFO:teuthology.orchestra.run.smithi175.stdout:Vendor : CentOS 2024-04-15T15:05:03.901 INFO:teuthology.orchestra.run.smithi175.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-15T15:05:03.901 INFO:teuthology.orchestra.run.smithi175.stdout:Summary : Support tools for GRUB. 2024-04-15T15:05:03.901 INFO:teuthology.orchestra.run.smithi175.stdout:Description : 2024-04-15T15:05:03.901 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:05:03.901 INFO:teuthology.orchestra.run.smithi175.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-15T15:05:03.901 INFO:teuthology.orchestra.run.smithi175.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-15T15:05:03.902 INFO:teuthology.orchestra.run.smithi175.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-15T15:05:03.902 INFO:teuthology.orchestra.run.smithi175.stdout:hardware devices. 2024-04-15T15:05:03.902 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:05:03.902 INFO:teuthology.orchestra.run.smithi175.stdout:This subpackage provides tools for support of all platforms. 2024-04-15T15:05:03.903 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-15T15:05:03.903 INFO:teuthology.task.kernel:Updating grub on smithi175 to boot 5.14.0-438.el9.x86_64 2024-04-15T15:05:03.903 DEBUG:teuthology.orchestra.run.smithi175:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-15T15:05:04.460 INFO:teuthology.orchestra.run.smithi175.stderr:Generating grub configuration file ... 2024-04-15T15:05:05.945 INFO:teuthology.orchestra.run.smithi175.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-15T15:05:05.976 INFO:teuthology.orchestra.run.smithi175.stderr:done 2024-04-15T15:05:05.978 DEBUG:teuthology.orchestra.run.smithi175:> mktemp 2024-04-15T15:05:05.995 INFO:teuthology.orchestra.run.smithi175.stdout:/tmp/tmp.GUxsEB8L5p 2024-04-15T15:05:05.995 DEBUG:teuthology.orchestra.run.smithi175:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.GUxsEB8L5p 2024-04-15T15:05:06.064 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 0666 /tmp/tmp.GUxsEB8L5p 2024-04-15T15:05:06.206 DEBUG:teuthology.orchestra.remote:smithi175:/tmp/tmp.GUxsEB8L5p is 6KB 2024-04-15T15:05:06.254 DEBUG:teuthology.orchestra.run.smithi175:> rm -fr /tmp/tmp.GUxsEB8L5p 2024-04-15T15:05:06.268 DEBUG:teuthology.orchestra.run.smithi175:> sudo /bin/ls /boot/loader/entries || true 2024-04-15T15:05:06.332 INFO:teuthology.orchestra.run.smithi175.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-15T15:05:06.333 INFO:teuthology.orchestra.run.smithi175.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-15T15:05:06.333 INFO:teuthology.orchestra.run.smithi175.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-15T15:05:06.333 INFO:teuthology.orchestra.run.smithi175.stdout:15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64.conf 2024-04-15T15:05:06.334 DEBUG:teuthology.orchestra.run.smithi175:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64 2024-04-15T15:05:06.472 DEBUG:teuthology.orchestra.run.smithi175:> sudo shutdown -r now 2024-04-15T15:05:31.570 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-15T15:05:31.571 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-04-15T15:05:31.572 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:05:36.478 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-15T15:05:36.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi175.front.sepia.ceph.com' 2024-04-15T15:05:36.479 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:05:50.110 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.72 2024-04-15T15:05:54.978 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.175 2024-04-15T15:05:59.114 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-04-15T15:05:59.115 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:06:03.984 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi175.front.sepia.ceph.com' 2024-04-15T15:06:04.001 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:06:59.158 DEBUG:teuthology.orchestra.remote:timed out 2024-04-15T15:07:04.008 DEBUG:teuthology.orchestra.remote:timed out 2024-04-15T15:07:11.166 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-04-15T15:07:11.167 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:07:11.574 DEBUG:teuthology.orchestra.run.smithi072:> true 2024-04-15T15:07:12.044 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-04-15T15:07:12.045 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-438.el9.x86_64"... 2024-04-15T15:07:12.045 DEBUG:teuthology.orchestra.run.smithi072:> uname -r 2024-04-15T15:07:12.062 INFO:teuthology.orchestra.run.smithi072.stdout:5.14.0-438.el9.x86_64 2024-04-15T15:07:12.063 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-438.el9.x86_64 vs 5.14.0-438.el9.x86_64 2024-04-15T15:07:12.063 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-15T15:07:12.063 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-15T15:07:13.063 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-15T15:07:13.064 DEBUG:teuthology.orchestra.run.smithi072:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-15T15:07:13.138 INFO:teuthology.orchestra.run.smithi072.stdout:ttyS1 2024-04-15T15:07:13.176 DEBUG:teuthology.parallel:result is None 2024-04-15T15:07:16.009 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi175.front.sepia.ceph.com' 2024-04-15T15:07:16.009 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:07:16.433 DEBUG:teuthology.orchestra.run.smithi175:> true 2024-04-15T15:07:16.809 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi175.front.sepia.ceph.com' 2024-04-15T15:07:16.809 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-438.el9.x86_64"... 2024-04-15T15:07:16.809 DEBUG:teuthology.orchestra.run.smithi175:> uname -r 2024-04-15T15:07:16.824 INFO:teuthology.orchestra.run.smithi175.stdout:5.14.0-438.el9.x86_64 2024-04-15T15:07:16.824 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-438.el9.x86_64 vs 5.14.0-438.el9.x86_64 2024-04-15T15:07:16.825 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-15T15:07:16.825 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-15T15:07:17.825 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-15T15:07:17.826 DEBUG:teuthology.orchestra.run.smithi175:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-15T15:07:17.902 INFO:teuthology.orchestra.run.smithi175.stdout:ttyS1 2024-04-15T15:07:17.933 DEBUG:teuthology.parallel:result is None 2024-04-15T15:07:17.933 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-15T15:07:17.941 INFO:teuthology.task.internal:Creating test directory... 2024-04-15T15:07:17.941 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-15T15:07:17.943 DEBUG:teuthology.orchestra.run.smithi175:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-15T15:07:17.962 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-15T15:07:17.968 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-15T15:07:17.977 INFO:teuthology.task.internal:Creating archive directory... 2024-04-15T15:07:17.977 DEBUG:teuthology.orchestra.run.smithi072:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-15T15:07:18.002 DEBUG:teuthology.orchestra.run.smithi175:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-15T15:07:18.037 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-15T15:07:18.043 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-15T15:07:18.044 DEBUG:teuthology.orchestra.run.smithi072:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-15T15:07:18.073 DEBUG:teuthology.orchestra.run.smithi175:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-15T15:07:18.107 INFO:teuthology.orchestra.run.smithi072.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T15:07:18.110 INFO:teuthology.orchestra.run.smithi175.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T15:07:18.122 INFO:teuthology.orchestra.run.smithi072.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T15:07:18.125 INFO:teuthology.orchestra.run.smithi175.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T15:07:18.126 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-15T15:07:18.138 INFO:teuthology.task.internal:Configuring sudo... 2024-04-15T15:07:18.138 DEBUG:teuthology.orchestra.run.smithi072:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-15T15:07:18.166 DEBUG:teuthology.orchestra.run.smithi175:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-15T15:07:18.196 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-15T15:07:18.205 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-15T15:07:18.205 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-15T15:07:18.237 DEBUG:teuthology.orchestra.run.smithi175:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-15T15:07:18.254 DEBUG:teuthology.orchestra.run.smithi072:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T15:07:18.338 DEBUG:teuthology.orchestra.run.smithi072:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T15:07:18.393 DEBUG:teuthology.orchestra.run.smithi072:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T15:07:18.476 DEBUG:teuthology.orchestra.run.smithi072:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T15:07:18.522 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:07:18.522 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-15T15:07:18.592 DEBUG:teuthology.orchestra.run.smithi175:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T15:07:18.637 DEBUG:teuthology.orchestra.run.smithi175:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T15:07:18.705 DEBUG:teuthology.orchestra.run.smithi175:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T15:07:18.748 DEBUG:teuthology.orchestra.run.smithi175:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T15:07:18.796 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:07:18.796 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-15T15:07:18.865 DEBUG:teuthology.orchestra.run.smithi072:> sudo service rsyslog restart 2024-04-15T15:07:18.867 DEBUG:teuthology.orchestra.run.smithi175:> sudo service rsyslog restart 2024-04-15T15:07:18.913 INFO:teuthology.orchestra.run.smithi072.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T15:07:18.966 INFO:teuthology.orchestra.run.smithi175.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T15:07:19.351 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-15T15:07:19.359 INFO:teuthology.task.internal:Starting timer... 2024-04-15T15:07:19.359 INFO:teuthology.run_tasks:Running task pcp... 2024-04-15T15:07:19.368 INFO:teuthology.run_tasks:Running task selinux... 2024-04-15T15:07:19.382 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-15T15:07:19.382 DEBUG:teuthology.orchestra.run.smithi072:> sudo service auditd rotate 2024-04-15T15:07:19.448 INFO:teuthology.orchestra.run.smithi072.stdout:Rotating logs: 2024-04-15T15:07:19.450 DEBUG:teuthology.orchestra.run.smithi175:> sudo service auditd rotate 2024-04-15T15:07:19.494 INFO:teuthology.orchestra.run.smithi175.stdout:Rotating logs: 2024-04-15T15:07:19.496 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-15T15:07:19.496 DEBUG:teuthology.orchestra.run.smithi072:> /usr/sbin/getenforce 2024-04-15T15:07:19.517 INFO:teuthology.orchestra.run.smithi072.stdout:Permissive 2024-04-15T15:07:19.518 DEBUG:teuthology.orchestra.run.smithi175:> /usr/sbin/getenforce 2024-04-15T15:07:19.555 INFO:teuthology.orchestra.run.smithi175.stdout:Permissive 2024-04-15T15:07:19.555 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi072.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi175.front.sepia.ceph.com': 'permissive'} 2024-04-15T15:07:19.555 DEBUG:teuthology.orchestra.run.smithi072:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-15T15:07:19.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:07:19.589 DEBUG:teuthology.orchestra.run.smithi175:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-15T15:07:19.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:07:19.630 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-15T15:07:19.631 DEBUG:teuthology.orchestra.run.smithi072:> sudo /usr/sbin/setenforce permissive 2024-04-15T15:07:19.670 DEBUG:teuthology.orchestra.run.smithi175:> sudo /usr/sbin/setenforce permissive 2024-04-15T15:07:19.707 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-15T15:07:19.716 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-15T15:07:19.722 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-15T15:07:19.848 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-15T15:07:19.862 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-15T15:07:19.863 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi072.front.sepia.ceph.com,smithi175.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-15T15:11:39.880 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi072.front.sepia.ceph.com'), Remote(name='ubuntu@smithi175.front.sepia.ceph.com')] 2024-04-15T15:11:39.881 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-04-15T15:11:39.882 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:11:39.973 DEBUG:teuthology.orchestra.run.smithi072:> true 2024-04-15T15:11:40.049 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-04-15T15:11:40.050 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi175.front.sepia.ceph.com' 2024-04-15T15:11:40.050 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi175.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:11:40.139 DEBUG:teuthology.orchestra.run.smithi175:> true 2024-04-15T15:11:40.218 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi175.front.sepia.ceph.com' 2024-04-15T15:11:40.218 INFO:teuthology.run_tasks:Running task clock... 2024-04-15T15:11:40.229 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-15T15:11:40.229 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-15T15:11:40.229 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T15:11:40.232 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-15T15:11:40.232 DEBUG:teuthology.orchestra.run.smithi175:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T15:11:40.261 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-15T15:11:40.279 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-15T15:11:40.290 INFO:teuthology.orchestra.run.smithi175.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-15T15:11:40.308 INFO:teuthology.orchestra.run.smithi175.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-15T15:11:40.309 INFO:teuthology.orchestra.run.smithi072.stderr:sudo: ntpd: command not found 2024-04-15T15:11:40.324 INFO:teuthology.orchestra.run.smithi072.stdout:506 Cannot talk to daemon 2024-04-15T15:11:40.338 INFO:teuthology.orchestra.run.smithi175.stderr:sudo: ntpd: command not found 2024-04-15T15:11:40.341 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-15T15:11:40.353 INFO:teuthology.orchestra.run.smithi175.stdout:506 Cannot talk to daemon 2024-04-15T15:11:40.359 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-15T15:11:40.370 INFO:teuthology.orchestra.run.smithi175.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-15T15:11:40.388 INFO:teuthology.orchestra.run.smithi175.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-15T15:11:40.408 INFO:teuthology.orchestra.run.smithi072.stderr:bash: line 1: ntpq: command not found 2024-04-15T15:11:40.412 INFO:teuthology.orchestra.run.smithi072.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T15:11:40.412 INFO:teuthology.orchestra.run.smithi072.stdout:=============================================================================== 2024-04-15T15:11:40.412 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:11:40.412 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:11:40.412 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:11:40.413 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:11:40.439 INFO:teuthology.orchestra.run.smithi175.stderr:bash: line 1: ntpq: command not found 2024-04-15T15:11:40.442 INFO:teuthology.orchestra.run.smithi175.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T15:11:40.443 INFO:teuthology.orchestra.run.smithi175.stdout:=============================================================================== 2024-04-15T15:11:40.443 INFO:teuthology.orchestra.run.smithi175.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:11:40.443 INFO:teuthology.orchestra.run.smithi175.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:11:40.443 INFO:teuthology.orchestra.run.smithi175.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:11:40.443 INFO:teuthology.orchestra.run.smithi175.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:11:40.444 INFO:teuthology.run_tasks:Running task install... 2024-04-15T15:11:40.452 DEBUG:teuthology.task.install:project ceph 2024-04-15T15:11:40.452 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14'}} 2024-04-15T15:11:40.452 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14'} 2024-04-15T15:11:40.452 INFO:teuthology.task.install:Using flavor: default 2024-04-15T15:11:40.458 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-15T15:11:40.458 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-15T15:11:40.458 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14', 'tag': None, 'wait_for_package': False} 2024-04-15T15:11:40.459 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 2024-04-15T15:11:40.460 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14', 'tag': None, 'wait_for_package': False} 2024-04-15T15:11:40.461 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 2024-04-15T15:11:40.605 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-quiesce-xlock/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/centos/9/flavors/default/ 2024-04-15T15:11:40.605 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2861.g9bf1c4ba 2024-04-15T15:11:40.723 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-quiesce-xlock/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/centos/9/flavors/default/ 2024-04-15T15:11:40.724 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2861.g9bf1c4ba 2024-04-15T15:11:40.907 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiesce-xlock/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiesce-xlock/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiesce-xlock/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-15T15:11:40.907 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:11:40.908 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-15T15:11:40.937 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-15T15:11:40.938 DEBUG:teuthology.orchestra.run.smithi072:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-15T15:11:41.007 DEBUG:teuthology.orchestra.run.smithi072:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-15T15:11:41.051 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiesce-xlock/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiesce-xlock/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiesce-xlock/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-15T15:11:41.051 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:11:41.051 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-15T15:11:41.082 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-15T15:11:41.082 DEBUG:teuthology.orchestra.run.smithi175:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-15T15:11:41.088 DEBUG:teuthology.orchestra.run.smithi072:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-15T15:11:41.152 DEBUG:teuthology.orchestra.run.smithi175:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-15T15:11:41.156 INFO:teuthology.orchestra.run.smithi072.stdout:check_obsoletes = 1 2024-04-15T15:11:41.158 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum clean all 2024-04-15T15:11:41.237 DEBUG:teuthology.orchestra.run.smithi175:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-15T15:11:41.306 INFO:teuthology.orchestra.run.smithi175.stdout:check_obsoletes = 1 2024-04-15T15:11:41.307 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum clean all 2024-04-15T15:11:41.397 INFO:teuthology.orchestra.run.smithi072.stdout:74 files removed 2024-04-15T15:11:41.444 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-15T15:11:41.566 INFO:teuthology.orchestra.run.smithi175.stdout:74 files removed 2024-04-15T15:11:41.593 DEBUG:teuthology.orchestra.run.smithi175:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-15T15:11:42.473 INFO:teuthology.orchestra.run.smithi072.stdout:ceph packages for x86_64 115 kB/s | 83 kB 00:00 2024-04-15T15:11:42.627 INFO:teuthology.orchestra.run.smithi175.stdout:ceph packages for x86_64 114 kB/s | 83 kB 00:00 2024-04-15T15:11:43.084 INFO:teuthology.orchestra.run.smithi072.stdout:ceph noarch packages 21 kB/s | 11 kB 00:00 2024-04-15T15:11:43.394 INFO:teuthology.orchestra.run.smithi175.stdout:ceph noarch packages 16 kB/s | 11 kB 00:00 2024-04-15T15:11:43.835 INFO:teuthology.orchestra.run.smithi072.stdout:ceph source packages 2.7 kB/s | 1.9 kB 00:00 2024-04-15T15:11:44.145 INFO:teuthology.orchestra.run.smithi175.stdout:ceph source packages 2.7 kB/s | 1.9 kB 00:00 2024-04-15T15:11:44.480 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-04-15T15:11:44.730 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-04-15T15:11:46.148 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-04-15T15:11:46.822 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS-9 - BaseOS 2.1 MB/s | 2.0 MB 00:00 2024-04-15T15:11:47.006 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-04-15T15:11:48.836 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-04-15T15:11:49.054 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.0 MB 00:01 2024-04-15T15:11:53.123 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - CRB 8.3 MB/s | 6.2 MB 00:00 2024-04-15T15:11:54.723 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 16 kB 00:00 2024-04-15T15:11:56.794 INFO:teuthology.orchestra.run.smithi072.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-04-15T15:12:01.862 INFO:teuthology.orchestra.run.smithi072.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-04-15T15:12:04.743 INFO:teuthology.orchestra.run.smithi072.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-15T15:12:04.744 INFO:teuthology.orchestra.run.smithi072.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-15T15:12:04.759 INFO:teuthology.orchestra.run.smithi072.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-04-15T15:12:04.760 INFO:teuthology.orchestra.run.smithi072.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-15T15:12:04.761 INFO:teuthology.orchestra.run.smithi072.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-04-15T15:12:04.762 INFO:teuthology.orchestra.run.smithi072.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-04-15T15:12:04.764 INFO:teuthology.orchestra.run.smithi072.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-15T15:12:04.767 INFO:teuthology.orchestra.run.smithi072.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-04-15T15:12:04.768 INFO:teuthology.orchestra.run.smithi072.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-04-15T15:12:04.852 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-04-15T15:12:04.879 INFO:teuthology.orchestra.run.smithi072.stdout:============================================================================================ 2024-04-15T15:12:04.879 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-04-15T15:12:04.879 INFO:teuthology.orchestra.run.smithi072.stdout:============================================================================================ 2024-04-15T15:12:04.879 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-04-15T15:12:04.879 INFO:teuthology.orchestra.run.smithi072.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-04-15T15:12:04.879 INFO:teuthology.orchestra.run.smithi072.stdout: ceph x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 6.6 k 2024-04-15T15:12:04.879 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-base x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 5.4 M 2024-04-15T15:12:04.879 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-fuse x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 890 k 2024-04-15T15:12:04.879 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 145 k 2024-04-15T15:12:04.880 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 1.5 M 2024-04-15T15:12:04.880 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 142 k 2024-04-15T15:12:04.880 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 3.5 M 2024-04-15T15:12:04.880 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 7.4 M 2024-04-15T15:12:04.880 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-rook noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 49 k 2024-04-15T15:12:04.880 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-radosgw x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 10 M 2024-04-15T15:12:04.880 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-test x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 49 M 2024-04-15T15:12:04.880 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-volume noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 264 k 2024-04-15T15:12:04.880 INFO:teuthology.orchestra.run.smithi072.stdout: cephadm noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 762 k 2024-04-15T15:12:04.880 INFO:teuthology.orchestra.run.smithi072.stdout: cephfs-mirror x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 224 k 2024-04-15T15:12:04.880 INFO:teuthology.orchestra.run.smithi072.stdout: cephfs-shell x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 27 k 2024-04-15T15:12:04.880 INFO:teuthology.orchestra.run.smithi072.stdout: cephfs-top noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 18 k 2024-04-15T15:12:04.880 INFO:teuthology.orchestra.run.smithi072.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-15T15:12:04.881 INFO:teuthology.orchestra.run.smithi072.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-04-15T15:12:04.881 INFO:teuthology.orchestra.run.smithi072.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-04-15T15:12:04.881 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs-devel x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 32 k 2024-04-15T15:12:04.881 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs2 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 747 k 2024-04-15T15:12:04.881 INFO:teuthology.orchestra.run.smithi072.stdout: librados-devel x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 127 k 2024-04-15T15:12:04.881 INFO:teuthology.orchestra.run.smithi072.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-04-15T15:12:04.881 INFO:teuthology.orchestra.run.smithi072.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-15T15:12:04.881 INFO:teuthology.orchestra.run.smithi072.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-15T15:12:04.881 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql x86_64 13.14-1.el9 CentOS-AppStream 1.5 M 2024-04-15T15:12:04.881 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql-contrib x86_64 13.14-1.el9 CentOS-AppStream 817 k 2024-04-15T15:12:04.881 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql-server x86_64 13.14-1.el9 CentOS-AppStream 5.7 M 2024-04-15T15:12:04.881 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cephfs x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 162 k 2024-04-15T15:12:04.881 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rados x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 320 k 2024-04-15T15:12:04.881 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rbd x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 297 k 2024-04-15T15:12:04.882 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rgw x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 99 k 2024-04-15T15:12:04.882 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-fuse x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 85 k 2024-04-15T15:12:04.882 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-mirror x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 3.1 M 2024-04-15T15:12:04.882 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-nbd x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 171 k 2024-04-15T15:12:04.882 INFO:teuthology.orchestra.run.smithi072.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-15T15:12:04.882 INFO:teuthology.orchestra.run.smithi072.stdout:Upgrading: 2024-04-15T15:12:04.882 INFO:teuthology.orchestra.run.smithi072.stdout: NetworkManager x86_64 1:1.47.4-1.el9 CentOS-BaseOS 2.3 M 2024-04-15T15:12:04.882 INFO:teuthology.orchestra.run.smithi072.stdout: NetworkManager-libnm x86_64 1:1.47.4-1.el9 CentOS-BaseOS 1.8 M 2024-04-15T15:12:04.882 INFO:teuthology.orchestra.run.smithi072.stdout: NetworkManager-team x86_64 1:1.47.4-1.el9 CentOS-BaseOS 41 k 2024-04-15T15:12:04.882 INFO:teuthology.orchestra.run.smithi072.stdout: NetworkManager-tui x86_64 1:1.47.4-1.el9 CentOS-BaseOS 248 k 2024-04-15T15:12:04.882 INFO:teuthology.orchestra.run.smithi072.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-04-15T15:12:04.882 INFO:teuthology.orchestra.run.smithi072.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-04-15T15:12:04.883 INFO:teuthology.orchestra.run.smithi072.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-04-15T15:12:04.883 INFO:teuthology.orchestra.run.smithi072.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-04-15T15:12:04.883 INFO:teuthology.orchestra.run.smithi072.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-04-15T15:12:04.883 INFO:teuthology.orchestra.run.smithi072.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-04-15T15:12:04.883 INFO:teuthology.orchestra.run.smithi072.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-04-15T15:12:04.883 INFO:teuthology.orchestra.run.smithi072.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-04-15T15:12:04.883 INFO:teuthology.orchestra.run.smithi072.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-04-15T15:12:04.883 INFO:teuthology.orchestra.run.smithi072.stdout: librados2 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 3.4 M 2024-04-15T15:12:04.883 INFO:teuthology.orchestra.run.smithi072.stdout: librbd1 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 3.1 M 2024-04-15T15:12:04.883 INFO:teuthology.orchestra.run.smithi072.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-04-15T15:12:04.883 INFO:teuthology.orchestra.run.smithi072.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-04-15T15:12:04.883 INFO:teuthology.orchestra.run.smithi072.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-04-15T15:12:04.883 INFO:teuthology.orchestra.run.smithi072.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-04-15T15:12:04.884 INFO:teuthology.orchestra.run.smithi072.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-04-15T15:12:04.884 INFO:teuthology.orchestra.run.smithi072.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-04-15T15:12:04.884 INFO:teuthology.orchestra.run.smithi072.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-04-15T15:12:04.884 INFO:teuthology.orchestra.run.smithi072.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-04-15T15:12:04.884 INFO:teuthology.orchestra.run.smithi072.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-04-15T15:12:04.884 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-04-15T15:12:04.884 INFO:teuthology.orchestra.run.smithi072.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-15T15:12:04.884 INFO:teuthology.orchestra.run.smithi072.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-04-15T15:12:04.884 INFO:teuthology.orchestra.run.smithi072.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-15T15:12:04.885 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-common x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 21 M 2024-04-15T15:12:04.885 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 34 k 2024-04-15T15:12:04.885 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mds x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 2.4 M 2024-04-15T15:12:04.885 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 249 k 2024-04-15T15:12:04.885 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mon x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 4.7 M 2024-04-15T15:12:04.885 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-osd x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 17 M 2024-04-15T15:12:04.885 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 16 k 2024-04-15T15:12:04.885 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-selinux x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 25 k 2024-04-15T15:12:04.885 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-15T15:12:04.886 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-15T15:12:04.886 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-15T15:12:04.886 INFO:teuthology.orchestra.run.smithi072.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-15T15:12:04.886 INFO:teuthology.orchestra.run.smithi072.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-15T15:12:04.886 INFO:teuthology.orchestra.run.smithi072.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-15T15:12:04.886 INFO:teuthology.orchestra.run.smithi072.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-15T15:12:04.886 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-15T15:12:04.886 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-15T15:12:04.886 INFO:teuthology.orchestra.run.smithi072.stdout: libcephsqlite x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 164 k 2024-04-15T15:12:04.886 INFO:teuthology.orchestra.run.smithi072.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-15T15:12:04.886 INFO:teuthology.orchestra.run.smithi072.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-15T15:12:04.887 INFO:teuthology.orchestra.run.smithi072.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-15T15:12:04.887 INFO:teuthology.orchestra.run.smithi072.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-15T15:12:04.887 INFO:teuthology.orchestra.run.smithi072.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-15T15:12:04.887 INFO:teuthology.orchestra.run.smithi072.stdout: libradosstriper1 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 501 k 2024-04-15T15:12:04.887 INFO:teuthology.orchestra.run.smithi072.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-15T15:12:04.887 INFO:teuthology.orchestra.run.smithi072.stdout: librgw2 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 5.2 M 2024-04-15T15:12:04.887 INFO:teuthology.orchestra.run.smithi072.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-15T15:12:04.887 INFO:teuthology.orchestra.run.smithi072.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-04-15T15:12:04.887 INFO:teuthology.orchestra.run.smithi072.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-15T15:12:04.887 INFO:teuthology.orchestra.run.smithi072.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-15T15:12:04.887 INFO:teuthology.orchestra.run.smithi072.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-15T15:12:04.887 INFO:teuthology.orchestra.run.smithi072.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-15T15:12:04.887 INFO:teuthology.orchestra.run.smithi072.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-15T15:12:04.888 INFO:teuthology.orchestra.run.smithi072.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-15T15:12:04.888 INFO:teuthology.orchestra.run.smithi072.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-15T15:12:04.888 INFO:teuthology.orchestra.run.smithi072.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-15T15:12:04.888 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql-private-libs x86_64 13.14-1.el9 CentOS-AppStream 132 k 2024-04-15T15:12:04.888 INFO:teuthology.orchestra.run.smithi072.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-15T15:12:04.888 INFO:teuthology.orchestra.run.smithi072.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-15T15:12:04.888 INFO:teuthology.orchestra.run.smithi072.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-15T15:12:04.888 INFO:teuthology.orchestra.run.smithi072.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-04-15T15:12:04.888 INFO:teuthology.orchestra.run.smithi072.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-15T15:12:04.888 INFO:teuthology.orchestra.run.smithi072.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-15T15:12:04.888 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-15T15:12:04.888 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-argparse x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 45 k 2024-04-15T15:12:04.888 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-common x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 135 k 2024-04-15T15:12:04.888 INFO:teuthology.orchestra.run.smithi072.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-15T15:12:04.888 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-15T15:12:04.889 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-15T15:12:04.889 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-15T15:12:04.889 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-15T15:12:04.889 INFO:teuthology.orchestra.run.smithi072.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-15T15:12:04.889 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-15T15:12:04.889 INFO:teuthology.orchestra.run.smithi072.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-15T15:12:04.889 INFO:teuthology.orchestra.run.smithi072.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-15T15:12:04.889 INFO:teuthology.orchestra.run.smithi072.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-15T15:12:04.889 INFO:teuthology.orchestra.run.smithi072.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-15T15:12:04.889 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-15T15:12:04.889 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-15T15:12:04.889 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-15T15:12:04.889 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-15T15:12:04.889 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-15T15:12:04.889 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-15T15:12:04.890 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-15T15:12:04.890 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-15T15:12:04.890 INFO:teuthology.orchestra.run.smithi072.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-15T15:12:04.890 INFO:teuthology.orchestra.run.smithi072.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-15T15:12:04.890 INFO:teuthology.orchestra.run.smithi072.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-15T15:12:04.890 INFO:teuthology.orchestra.run.smithi072.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-15T15:12:04.890 INFO:teuthology.orchestra.run.smithi072.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-15T15:12:04.890 INFO:teuthology.orchestra.run.smithi072.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-15T15:12:04.890 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-15T15:12:04.890 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-15T15:12:04.890 INFO:teuthology.orchestra.run.smithi072.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-15T15:12:04.890 INFO:teuthology.orchestra.run.smithi072.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-15T15:12:04.890 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-15T15:12:04.891 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-15T15:12:04.891 INFO:teuthology.orchestra.run.smithi072.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-15T15:12:04.891 INFO:teuthology.orchestra.run.smithi072.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-15T15:12:04.891 INFO:teuthology.orchestra.run.smithi072.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-15T15:12:04.891 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-15T15:12:04.891 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-15T15:12:04.891 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-15T15:12:04.891 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-15T15:12:04.891 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-15T15:12:04.891 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-15T15:12:04.891 INFO:teuthology.orchestra.run.smithi072.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-15T15:12:04.891 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-15T15:12:04.891 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-15T15:12:04.891 INFO:teuthology.orchestra.run.smithi072.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-15T15:12:04.891 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-15T15:12:04.892 INFO:teuthology.orchestra.run.smithi072.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-15T15:12:04.892 INFO:teuthology.orchestra.run.smithi072.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-15T15:12:04.892 INFO:teuthology.orchestra.run.smithi072.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-15T15:12:04.892 INFO:teuthology.orchestra.run.smithi072.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-15T15:12:04.892 INFO:teuthology.orchestra.run.smithi072.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-15T15:12:04.892 INFO:teuthology.orchestra.run.smithi072.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-04-15T15:12:04.892 INFO:teuthology.orchestra.run.smithi072.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-15T15:12:04.892 INFO:teuthology.orchestra.run.smithi072.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-15T15:12:04.892 INFO:teuthology.orchestra.run.smithi072.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-15T15:12:04.892 INFO:teuthology.orchestra.run.smithi072.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-15T15:12:04.892 INFO:teuthology.orchestra.run.smithi072.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-15T15:12:04.892 INFO:teuthology.orchestra.run.smithi072.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-15T15:12:04.892 INFO:teuthology.orchestra.run.smithi072.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-15T15:12:04.892 INFO:teuthology.orchestra.run.smithi072.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-15T15:12:04.893 INFO:teuthology.orchestra.run.smithi072.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-15T15:12:04.893 INFO:teuthology.orchestra.run.smithi072.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-04-15T15:12:04.893 INFO:teuthology.orchestra.run.smithi072.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-15T15:12:04.893 INFO:teuthology.orchestra.run.smithi072.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-04-15T15:12:04.893 INFO:teuthology.orchestra.run.smithi072.stdout:Installing weak dependencies: 2024-04-15T15:12:04.893 INFO:teuthology.orchestra.run.smithi072.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-15T15:12:04.893 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:12:04.893 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-04-15T15:12:04.893 INFO:teuthology.orchestra.run.smithi072.stdout:============================================================================================ 2024-04-15T15:12:04.893 INFO:teuthology.orchestra.run.smithi072.stdout:Install 150 Packages 2024-04-15T15:12:04.893 INFO:teuthology.orchestra.run.smithi072.stdout:Upgrade 24 Packages 2024-04-15T15:12:04.893 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:12:04.894 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 228 M 2024-04-15T15:12:04.894 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-04-15T15:12:05.923 INFO:teuthology.orchestra.run.smithi072.stdout:(1/174): ceph-19.0.0-2861.g9bf1c4ba.el8.x86_64. 31 kB/s | 6.6 kB 00:00 2024-04-15T15:12:06.940 INFO:teuthology.orchestra.run.smithi072.stdout:(2/174): ceph-fuse-19.0.0-2861.g9bf1c4ba.el8.x8 875 kB/s | 890 kB 00:01 2024-04-15T15:12:07.141 INFO:teuthology.orchestra.run.smithi072.stdout:(3/174): ceph-immutable-object-cache-19.0.0-286 723 kB/s | 145 kB 00:00 2024-04-15T15:12:07.621 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Stream 9 - AppStream 1.1 MB/s | 19 MB 00:17 2024-04-15T15:12:08.041 INFO:teuthology.orchestra.run.smithi072.stdout:(4/174): ceph-common-19.0.0-2861.g9bf1c4ba.el8. 8.9 MB/s | 21 MB 00:02 2024-04-15T15:12:08.260 INFO:teuthology.orchestra.run.smithi072.stdout:(5/174): ceph-mgr-19.0.0-2861.g9bf1c4ba.el8.x86 6.9 MB/s | 1.5 MB 00:00 2024-04-15T15:12:08.419 INFO:teuthology.orchestra.run.smithi072.stdout:(6/174): ceph-mds-19.0.0-2861.g9bf1c4ba.el8.x86 1.8 MB/s | 2.4 MB 00:01 2024-04-15T15:12:08.586 INFO:teuthology.orchestra.run.smithi072.stdout:(7/174): ceph-base-19.0.0-2861.g9bf1c4ba.el8.x8 1.9 MB/s | 5.4 MB 00:02 2024-04-15T15:12:08.837 INFO:teuthology.orchestra.run.smithi072.stdout:(8/174): ceph-mon-19.0.0-2861.g9bf1c4ba.el8.x86 8.2 MB/s | 4.7 MB 00:00 2024-04-15T15:12:08.895 INFO:teuthology.orchestra.run.smithi072.stdout:(9/174): ceph-selinux-19.0.0-2861.g9bf1c4ba.el8 432 kB/s | 25 kB 00:00 2024-04-15T15:12:10.038 INFO:teuthology.orchestra.run.smithi072.stdout:(10/174): ceph-radosgw-19.0.0-2861.g9bf1c4ba.el 6.9 MB/s | 10 MB 00:01 2024-04-15T15:12:10.146 INFO:teuthology.orchestra.run.smithi072.stdout:(11/174): cephfs-mirror-19.0.0-2861.g9bf1c4ba.e 2.0 MB/s | 224 kB 00:00 2024-04-15T15:12:10.238 INFO:teuthology.orchestra.run.smithi072.stdout:(12/174): cephfs-shell-19.0.0-2861.g9bf1c4ba.el 295 kB/s | 27 kB 00:00 2024-04-15T15:12:10.322 INFO:teuthology.orchestra.run.smithi072.stdout:(13/174): libcephfs-devel-19.0.0-2861.g9bf1c4ba 380 kB/s | 32 kB 00:00 2024-04-15T15:12:10.439 INFO:teuthology.orchestra.run.smithi072.stdout:(14/174): libcephfs2-19.0.0-2861.g9bf1c4ba.el8. 6.3 MB/s | 747 kB 00:00 2024-04-15T15:12:10.532 INFO:teuthology.orchestra.run.smithi072.stdout:(15/174): libcephsqlite-19.0.0-2861.g9bf1c4ba.e 1.7 MB/s | 164 kB 00:00 2024-04-15T15:12:10.816 INFO:teuthology.orchestra.run.smithi072.stdout:(16/174): ceph-osd-19.0.0-2861.g9bf1c4ba.el8.x8 7.0 MB/s | 17 MB 00:02 2024-04-15T15:12:10.841 INFO:teuthology.orchestra.run.smithi072.stdout:(17/174): librados-devel-19.0.0-2861.g9bf1c4ba. 411 kB/s | 127 kB 00:00 2024-04-15T15:12:10.900 INFO:teuthology.orchestra.run.smithi072.stdout:(18/174): libradosstriper1-19.0.0-2861.g9bf1c4b 5.9 MB/s | 501 kB 00:00 2024-04-15T15:12:11.000 INFO:teuthology.orchestra.run.smithi072.stdout:(19/174): python3-ceph-argparse-19.0.0-2861.g9b 451 kB/s | 45 kB 00:00 2024-04-15T15:12:11.077 INFO:teuthology.orchestra.run.smithi072.stdout:(20/174): python3-ceph-common-19.0.0-2861.g9bf1 1.7 MB/s | 135 kB 00:00 2024-04-15T15:12:11.178 INFO:teuthology.orchestra.run.smithi072.stdout:(21/174): python3-cephfs-19.0.0-2861.g9bf1c4ba. 1.6 MB/s | 162 kB 00:00 2024-04-15T15:12:11.270 INFO:teuthology.orchestra.run.smithi072.stdout:(22/174): python3-rados-19.0.0-2861.g9bf1c4ba.e 3.4 MB/s | 320 kB 00:00 2024-04-15T15:12:11.437 INFO:teuthology.orchestra.run.smithi072.stdout:(23/174): librgw2-19.0.0-2861.g9bf1c4ba.el8.x86 8.8 MB/s | 5.2 MB 00:00 2024-04-15T15:12:11.463 INFO:teuthology.orchestra.run.smithi072.stdout:(24/174): python3-rbd-19.0.0-2861.g9bf1c4ba.el8 1.5 MB/s | 297 kB 00:00 2024-04-15T15:12:11.505 INFO:teuthology.orchestra.run.smithi072.stdout:(25/174): python3-rgw-19.0.0-2861.g9bf1c4ba.el8 1.4 MB/s | 99 kB 00:00 2024-04-15T15:12:11.539 INFO:teuthology.orchestra.run.smithi072.stdout:(26/174): rbd-fuse-19.0.0-2861.g9bf1c4ba.el8.x8 1.1 MB/s | 85 kB 00:00 2024-04-15T15:12:11.631 INFO:teuthology.orchestra.run.smithi072.stdout:(27/174): rbd-nbd-19.0.0-2861.g9bf1c4ba.el8.x86 1.8 MB/s | 171 kB 00:00 2024-04-15T15:12:11.724 INFO:teuthology.orchestra.run.smithi072.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2861.g 371 kB/s | 34 kB 00:00 2024-04-15T15:12:11.824 INFO:teuthology.orchestra.run.smithi072.stdout:(29/174): rbd-mirror-19.0.0-2861.g9bf1c4ba.el8. 9.7 MB/s | 3.1 MB 00:00 2024-04-15T15:12:11.849 INFO:teuthology.orchestra.run.smithi072.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2861.g9bf1c4b 1.1 MB/s | 142 kB 00:00 2024-04-15T15:12:12.258 INFO:teuthology.orchestra.run.smithi072.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2861.g9bf1c 8.2 MB/s | 3.5 MB 00:00 2024-04-15T15:12:12.342 INFO:teuthology.orchestra.run.smithi072.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2861.g9b 2.9 MB/s | 249 kB 00:00 2024-04-15T15:12:12.418 INFO:teuthology.orchestra.run.smithi072.stdout:(33/174): ceph-mgr-rook-19.0.0-2861.g9bf1c4ba.e 658 kB/s | 49 kB 00:00 2024-04-15T15:12:12.494 INFO:teuthology.orchestra.run.smithi072.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2861.g9 219 kB/s | 16 kB 00:00 2024-04-15T15:12:12.587 INFO:teuthology.orchestra.run.smithi072.stdout:(35/174): ceph-volume-19.0.0-2861.g9bf1c4ba.el8 2.8 MB/s | 264 kB 00:00 2024-04-15T15:12:12.755 INFO:teuthology.orchestra.run.smithi072.stdout:(36/174): ceph-mgr-diskprediction-local-19.0.0- 8.2 MB/s | 7.4 MB 00:00 2024-04-15T15:12:12.797 INFO:teuthology.orchestra.run.smithi072.stdout:(37/174): cephadm-19.0.0-2861.g9bf1c4ba.el8.noa 3.6 MB/s | 762 kB 00:00 2024-04-15T15:12:12.831 INFO:teuthology.orchestra.run.smithi072.stdout:(38/174): cephfs-top-19.0.0-2861.g9bf1c4ba.el8. 241 kB/s | 18 kB 00:00 2024-04-15T15:12:13.615 INFO:teuthology.orchestra.run.smithi072.stdout:(39/174): ceph-test-19.0.0-2861.g9bf1c4ba.el8.x 10 MB/s | 49 MB 00:04 2024-04-15T15:12:13.757 INFO:teuthology.orchestra.run.smithi072.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 1.0 MB/s | 920 kB 00:00 2024-04-15T15:12:13.791 INFO:teuthology.orchestra.run.smithi072.stdout:(41/174): automake-1.16.2-8.el9.noarch.rpm 666 kB/s | 662 kB 00:00 2024-04-15T15:12:13.816 INFO:teuthology.orchestra.run.smithi072.stdout:(42/174): boost-program-options-1.75.0-8.el9.x8 612 kB/s | 106 kB 00:00 2024-04-15T15:12:13.841 INFO:teuthology.orchestra.run.smithi072.stdout:(43/174): flex-2.6.4-9.el9.x86_64.rpm 3.6 MB/s | 307 kB 00:00 2024-04-15T15:12:13.867 INFO:teuthology.orchestra.run.smithi072.stdout:(44/174): flexiblas-3.0.4-8.el9.x86_64.rpm 631 kB/s | 32 kB 00:00 2024-04-15T15:12:13.951 INFO:teuthology.orchestra.run.smithi072.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 158 kB/s | 17 kB 00:00 2024-04-15T15:12:14.026 INFO:teuthology.orchestra.run.smithi072.stdout:(46/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 14 MB/s | 3.0 MB 00:00 2024-04-15T15:12:14.051 INFO:teuthology.orchestra.run.smithi072.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 55 kB/s | 10 kB 00:00 2024-04-15T15:12:14.077 INFO:teuthology.orchestra.run.smithi072.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 159 kB 00:00 2024-04-15T15:12:14.102 INFO:teuthology.orchestra.run.smithi072.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 876 kB/s | 44 kB 00:00 2024-04-15T15:12:14.145 INFO:teuthology.orchestra.run.smithi072.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 661 kB 00:00 2024-04-15T15:12:14.178 INFO:teuthology.orchestra.run.smithi072.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 5.6 MB/s | 578 kB 00:00 2024-04-15T15:12:14.205 INFO:teuthology.orchestra.run.smithi072.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 456 kB/s | 46 kB 00:00 2024-04-15T15:12:14.239 INFO:teuthology.orchestra.run.smithi072.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.0 MB/s | 187 kB 00:00 2024-04-15T15:12:14.265 INFO:teuthology.orchestra.run.smithi072.stdout:(54/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.2 MB/s | 282 kB 00:00 2024-04-15T15:12:14.299 INFO:teuthology.orchestra.run.smithi072.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.3 MB/s | 217 kB 00:00 2024-04-15T15:12:14.324 INFO:teuthology.orchestra.run.smithi072.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 406 kB/s | 34 kB 00:00 2024-04-15T15:12:14.525 INFO:teuthology.orchestra.run.smithi072.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 18 MB/s | 4.7 MB 00:00 2024-04-15T15:12:14.575 INFO:teuthology.orchestra.run.smithi072.stdout:(58/174): postgresql-13.14-1.el9.x86_64.rpm 5.5 MB/s | 1.5 MB 00:00 2024-04-15T15:12:14.609 INFO:teuthology.orchestra.run.smithi072.stdout:(59/174): postgresql-contrib-13.14-1.el9.x86_64 2.8 MB/s | 817 kB 00:00 2024-04-15T15:12:14.635 INFO:teuthology.orchestra.run.smithi072.stdout:(60/174): postgresql-private-libs-13.14-1.el9.x 1.2 MB/s | 132 kB 00:00 2024-04-15T15:12:14.693 INFO:teuthology.orchestra.run.smithi072.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 12 MB/s | 1.0 MB 00:00 2024-04-15T15:12:14.777 INFO:teuthology.orchestra.run.smithi072.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 546 kB/s | 78 kB 00:00 2024-04-15T15:12:14.894 INFO:teuthology.orchestra.run.smithi072.stdout:(63/174): postgresql-server-13.14-1.el9.x86_64. 20 MB/s | 5.7 MB 00:00 2024-04-15T15:12:15.020 INFO:teuthology.orchestra.run.smithi072.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-04-15T15:12:15.045 INFO:teuthology.orchestra.run.smithi072.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 92 kB/s | 24 kB 00:00 2024-04-15T15:12:15.070 INFO:teuthology.orchestra.run.smithi072.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 104 kB/s | 18 kB 00:00 2024-04-15T15:12:15.096 INFO:teuthology.orchestra.run.smithi072.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-04-15T15:12:15.123 INFO:teuthology.orchestra.run.smithi072.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 609 kB/s | 32 kB 00:00 2024-04-15T15:12:15.249 INFO:teuthology.orchestra.run.smithi072.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 27 MB/s | 4.8 MB 00:00 2024-04-15T15:12:15.275 INFO:teuthology.orchestra.run.smithi072.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.2 MB/s | 217 kB 00:00 2024-04-15T15:12:15.300 INFO:teuthology.orchestra.run.smithi072.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.0 MB/s | 174 kB 00:00 2024-04-15T15:12:15.334 INFO:teuthology.orchestra.run.smithi072.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 823 kB/s | 69 kB 00:00 2024-04-15T15:12:15.359 INFO:teuthology.orchestra.run.smithi072.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.0 MB/s | 238 kB 00:00 2024-04-15T15:12:15.385 INFO:teuthology.orchestra.run.smithi072.stdout:(74/174): python3-prettytable-0.7.2-27.el9.noar 488 kB/s | 41 kB 00:00 2024-04-15T15:12:15.410 INFO:teuthology.orchestra.run.smithi072.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-04-15T15:12:15.444 INFO:teuthology.orchestra.run.smithi072.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.5 MB/s | 211 kB 00:00 2024-04-15T15:12:15.469 INFO:teuthology.orchestra.run.smithi072.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 733 kB/s | 43 kB 00:00 2024-04-15T15:12:15.495 INFO:teuthology.orchestra.run.smithi072.stdout:(78/174): python3-pytz-2021.1-5.el9.noarch.rpm 563 kB/s | 47 kB 00:00 2024-04-15T15:12:15.604 INFO:teuthology.orchestra.run.smithi072.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 294 kB/s | 39 kB 00:00 2024-04-15T15:12:15.662 INFO:teuthology.orchestra.run.smithi072.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 249 kB/s | 41 kB 00:00 2024-04-15T15:12:15.922 INFO:teuthology.orchestra.run.smithi072.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2024-04-15T15:12:15.947 INFO:teuthology.orchestra.run.smithi072.stdout:(82/174): qatlib-24.02.0-1.el9.x86_64.rpm 645 kB/s | 220 kB 00:00 2024-04-15T15:12:15.972 INFO:teuthology.orchestra.run.smithi072.stdout:(83/174): qatlib-service-24.02.0-1.el9.x86_64.r 113 kB/s | 35 kB 00:00 2024-04-15T15:12:15.998 INFO:teuthology.orchestra.run.smithi072.stdout:(84/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 916 kB/s | 46 kB 00:00 2024-04-15T15:12:16.031 INFO:teuthology.orchestra.run.smithi072.stdout:(85/174): socat-1.7.4.1-5.el9.x86_64.rpm 3.5 MB/s | 300 kB 00:00 2024-04-15T15:12:16.058 INFO:teuthology.orchestra.run.smithi072.stdout:(86/174): uuid-1.6.2-55.el9.x86_64.rpm 660 kB/s | 56 kB 00:00 2024-04-15T15:12:16.084 INFO:teuthology.orchestra.run.smithi072.stdout:(87/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-04-15T15:12:16.109 INFO:teuthology.orchestra.run.smithi072.stdout:(88/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 569 kB/s | 44 kB 00:00 2024-04-15T15:12:16.135 INFO:teuthology.orchestra.run.smithi072.stdout:(89/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-04-15T15:12:16.169 INFO:teuthology.orchestra.run.smithi072.stdout:(90/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-04-15T15:12:16.211 INFO:teuthology.orchestra.run.smithi072.stdout:(91/174): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.1 MB 00:00 2024-04-15T15:12:16.236 INFO:teuthology.orchestra.run.smithi072.stdout:(92/174): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 103 kB 00:00 2024-04-15T15:12:16.262 INFO:teuthology.orchestra.run.smithi072.stdout:(93/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-04-15T15:12:16.287 INFO:teuthology.orchestra.run.smithi072.stdout:(94/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.4 MB/s | 187 kB 00:00 2024-04-15T15:12:16.404 INFO:teuthology.orchestra.run.smithi072.stdout:(95/174): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-04-15T15:12:16.446 INFO:teuthology.orchestra.run.smithi072.stdout:(96/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-04-15T15:12:16.472 INFO:teuthology.orchestra.run.smithi072.stdout:(97/174): python3-requests-2.25.1-8.el9.noarch. 1.8 MB/s | 125 kB 00:00 2024-04-15T15:12:16.531 INFO:teuthology.orchestra.run.smithi072.stdout:(98/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.2 MB/s | 803 kB 00:00 2024-04-15T15:12:16.556 INFO:teuthology.orchestra.run.smithi072.stdout:(99/174): elfutils-libelf-devel-0.190-2.el9.x86 196 kB/s | 21 kB 00:00 2024-04-15T15:12:16.582 INFO:teuthology.orchestra.run.smithi072.stdout:(100/174): libuuid-devel-2.37.4-18.el9.x86_64.r 232 kB/s | 25 kB 00:00 2024-04-15T15:12:16.699 INFO:teuthology.orchestra.run.smithi072.stdout:(101/174): python3-jinja2-2.11.3-5.el9.noarch.r 2.1 MB/s | 248 kB 00:00 2024-04-15T15:12:16.741 INFO:teuthology.orchestra.run.smithi072.stdout:(102/174): python3-devel-3.9.17-2.el9.x86_64.rp 1.3 MB/s | 245 kB 00:00 2024-04-15T15:12:16.777 INFO:teuthology.orchestra.run.smithi072.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.2 MB/s | 94 kB 00:00 2024-04-15T15:12:16.961 INFO:teuthology.orchestra.run.smithi072.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 102 kB/s | 22 kB 00:00 2024-04-15T15:12:17.045 INFO:teuthology.orchestra.run.smithi072.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 3.1 MB/s | 863 kB 00:00 2024-04-15T15:12:17.170 INFO:teuthology.orchestra.run.smithi072.stdout:(106/174): openssl-devel-3.0.7-27.el9.x86_64.rp 6.5 MB/s | 4.1 MB 00:00 2024-04-15T15:12:17.404 INFO:teuthology.orchestra.run.smithi072.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 83 kB/s | 19 kB 00:00 2024-04-15T15:12:17.513 INFO:teuthology.orchestra.run.smithi072.stdout:(108/174): abseil-cpp-20211102.0-3.el9.x86_64.r 993 kB/s | 548 kB 00:00 2024-04-15T15:12:17.547 INFO:teuthology.orchestra.run.smithi072.stdout:(109/174): gperftools-libs-2.9.1-2.el9.x86_64.r 615 kB/s | 309 kB 00:00 2024-04-15T15:12:17.606 INFO:teuthology.orchestra.run.smithi072.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 281 kB/s | 26 kB 00:00 2024-04-15T15:12:17.648 INFO:teuthology.orchestra.run.smithi072.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 482 kB/s | 49 kB 00:00 2024-04-15T15:12:17.707 INFO:teuthology.orchestra.run.smithi072.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 669 kB/s | 67 kB 00:00 2024-04-15T15:12:17.749 INFO:teuthology.orchestra.run.smithi072.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-04-15T15:12:17.883 INFO:teuthology.orchestra.run.smithi072.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.7 MB/s | 838 kB 00:00 2024-04-15T15:12:17.925 INFO:teuthology.orchestra.run.smithi072.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 3.0 MB/s | 548 kB 00:00 2024-04-15T15:12:18.042 INFO:teuthology.orchestra.run.smithi072.stdout:(116/174): libarrow-9.0.0-10.el9.x86_64.rpm 6.9 MB/s | 4.4 MB 00:00 2024-04-15T15:12:18.069 INFO:teuthology.orchestra.run.smithi072.stdout:(117/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 233 kB/s | 43 kB 00:00 2024-04-15T15:12:18.095 INFO:teuthology.orchestra.run.smithi072.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 190 kB/s | 32 kB 00:00 2024-04-15T15:12:18.137 INFO:teuthology.orchestra.run.smithi072.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 159 kB/s | 15 kB 00:00 2024-04-15T15:12:18.196 INFO:teuthology.orchestra.run.smithi072.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 3.5 MB/s | 358 kB 00:00 2024-04-15T15:12:18.221 INFO:teuthology.orchestra.run.smithi072.stdout:(121/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.3 MB/s | 172 kB 00:00 2024-04-15T15:12:18.247 INFO:teuthology.orchestra.run.smithi072.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.7 MB/s | 303 kB 00:00 2024-04-15T15:12:18.289 INFO:teuthology.orchestra.run.smithi072.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 380 kB/s | 35 kB 00:00 2024-04-15T15:12:18.323 INFO:teuthology.orchestra.run.smithi072.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 2.0 MB/s | 203 kB 00:00 2024-04-15T15:12:18.390 INFO:teuthology.orchestra.run.smithi072.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-04-15T15:12:18.424 INFO:teuthology.orchestra.run.smithi072.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 106 kB/s | 11 kB 00:00 2024-04-15T15:12:18.483 INFO:teuthology.orchestra.run.smithi072.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 191 kB/s | 18 kB 00:00 2024-04-15T15:12:18.525 INFO:teuthology.orchestra.run.smithi072.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 230 kB/s | 23 kB 00:00 2024-04-15T15:12:18.584 INFO:teuthology.orchestra.run.smithi072.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 193 kB/s | 19 kB 00:00 2024-04-15T15:12:18.626 INFO:teuthology.orchestra.run.smithi072.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 195 kB/s | 20 kB 00:00 2024-04-15T15:12:18.726 INFO:teuthology.orchestra.run.smithi072.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 462 kB/s | 46 kB 00:00 2024-04-15T15:12:18.769 INFO:teuthology.orchestra.run.smithi072.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 5.5 MB/s | 1.0 MB 00:00 2024-04-15T15:12:18.827 INFO:teuthology.orchestra.run.smithi072.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 782 kB/s | 79 kB 00:00 2024-04-15T15:12:18.864 INFO:teuthology.orchestra.run.smithi072.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 614 kB/s | 58 kB 00:00 2024-04-15T15:12:18.930 INFO:teuthology.orchestra.run.smithi072.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-04-15T15:12:18.955 INFO:teuthology.orchestra.run.smithi072.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 177 kB/s | 16 kB 00:00 2024-04-15T15:12:18.988 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Stream 9 - CRB 812 kB/s | 6.2 MB 00:07 2024-04-15T15:12:19.031 INFO:teuthology.orchestra.run.smithi072.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 893 kB/s | 90 kB 00:00 2024-04-15T15:12:19.057 INFO:teuthology.orchestra.run.smithi072.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 280 kB/s | 28 kB 00:00 2024-04-15T15:12:19.124 INFO:teuthology.orchestra.run.smithi072.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 333 kB/s | 31 kB 00:00 2024-04-15T15:12:19.158 INFO:teuthology.orchestra.run.smithi072.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-04-15T15:12:19.242 INFO:teuthology.orchestra.run.smithi072.stdout:(141/174): python3-grpcio-1.46.7-10.el9.x86_64. 2.0 MB/s | 2.0 MB 00:00 2024-04-15T15:12:19.270 INFO:teuthology.orchestra.run.smithi072.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 328 kB/s | 36 kB 00:00 2024-04-15T15:12:19.293 INFO:teuthology.orchestra.run.smithi072.stdout:(143/174): python3-rsa-4.9-2.el9.noarch.rpm 350 kB/s | 59 kB 00:00 2024-04-15T15:12:19.335 INFO:teuthology.orchestra.run.smithi072.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 548 kB/s | 51 kB 00:00 2024-04-15T15:12:19.469 INFO:teuthology.orchestra.run.smithi072.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.1 MB/s | 230 kB 00:00 2024-04-15T15:12:19.494 INFO:teuthology.orchestra.run.smithi072.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 445 kB/s | 90 kB 00:00 2024-04-15T15:12:19.528 INFO:teuthology.orchestra.run.smithi072.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 2.2 MB/s | 427 kB 00:00 2024-04-15T15:12:19.571 INFO:teuthology.orchestra.run.smithi072.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 197 kB/s | 20 kB 00:00 2024-04-15T15:12:19.598 INFO:teuthology.orchestra.run.smithi072.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.8 MB/s | 192 kB 00:00 2024-04-15T15:12:19.665 INFO:teuthology.orchestra.run.smithi072.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-04-15T15:12:19.765 INFO:teuthology.orchestra.run.smithi072.stdout:(151/174): NetworkManager-1.47.4-1.el9.x86_64.r 23 MB/s | 2.3 MB 00:00 2024-04-15T15:12:19.849 INFO:teuthology.orchestra.run.smithi072.stdout:(152/174): NetworkManager-libnm-1.47.4-1.el9.x8 22 MB/s | 1.8 MB 00:00 2024-04-15T15:12:19.891 INFO:teuthology.orchestra.run.smithi072.stdout:(153/174): NetworkManager-team-1.47.4-1.el9.x86 976 kB/s | 41 kB 00:00 2024-04-15T15:12:19.942 INFO:teuthology.orchestra.run.smithi072.stdout:(154/174): NetworkManager-tui-1.47.4-1.el9.x86_ 4.8 MB/s | 248 kB 00:00 2024-04-15T15:12:19.993 INFO:teuthology.orchestra.run.smithi072.stdout:(155/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-04-15T15:12:20.085 INFO:teuthology.orchestra.run.smithi072.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 779 kB/s | 71 kB 00:00 2024-04-15T15:12:20.160 INFO:teuthology.orchestra.run.smithi072.stdout:(157/174): librbd1-19.0.0-2861.g9bf1c4ba.el8.x8 5.6 MB/s | 3.1 MB 00:00 2024-04-15T15:12:20.186 INFO:teuthology.orchestra.run.smithi072.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 361 kB/s | 36 kB 00:00 2024-04-15T15:12:20.270 INFO:teuthology.orchestra.run.smithi072.stdout:(159/174): librados2-19.0.0-2861.g9bf1c4ba.el8. 4.9 MB/s | 3.4 MB 00:00 2024-04-15T15:12:20.295 INFO:teuthology.orchestra.run.smithi072.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.4 MB/s | 192 kB 00:00 2024-04-15T15:12:20.320 INFO:teuthology.orchestra.run.smithi072.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 1.9 MB/s | 256 kB 00:00 2024-04-15T15:12:20.346 INFO:teuthology.orchestra.run.smithi072.stdout:(162/174): libacl-2.3.1-4.el9.x86_64.rpm 302 kB/s | 23 kB 00:00 2024-04-15T15:12:20.371 INFO:teuthology.orchestra.run.smithi072.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 108 kB 00:00 2024-04-15T15:12:20.397 INFO:teuthology.orchestra.run.smithi072.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.0 MB/s | 155 kB 00:00 2024-04-15T15:12:20.424 INFO:teuthology.orchestra.run.smithi072.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.2 MB/s | 64 kB 00:00 2024-04-15T15:12:20.450 INFO:teuthology.orchestra.run.smithi072.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 548 kB/s | 29 kB 00:00 2024-04-15T15:12:20.475 INFO:teuthology.orchestra.run.smithi072.stdout:(167/174): libmount-2.37.4-18.el9.x86_64.rpm 1.0 MB/s | 136 kB 00:00 2024-04-15T15:12:20.542 INFO:teuthology.orchestra.run.smithi072.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 10 MB/s | 1.2 MB 00:00 2024-04-15T15:12:20.555 INFO:teuthology.orchestra.run.smithi175.stdout:CentOS Stream 9 - Extras packages 69 kB/s | 16 kB 00:00 2024-04-15T15:12:20.610 INFO:teuthology.orchestra.run.smithi072.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 6.8 MB/s | 465 kB 00:00 2024-04-15T15:12:20.668 INFO:teuthology.orchestra.run.smithi072.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 9.9 MB/s | 2.2 MB 00:00 2024-04-15T15:12:20.694 INFO:teuthology.orchestra.run.smithi072.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 371 kB/s | 31 kB 00:00 2024-04-15T15:12:20.761 INFO:teuthology.orchestra.run.smithi072.stdout:(172/174): util-linux-2.37.4-18.el9.x86_64.rpm 8.0 MB/s | 2.3 MB 00:00 2024-04-15T15:12:20.803 INFO:teuthology.orchestra.run.smithi072.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 8.0 MB/s | 1.1 MB 00:00 2024-04-15T15:12:20.828 INFO:teuthology.orchestra.run.smithi072.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 554 kB/s | 74 kB 00:00 2024-04-15T15:12:20.833 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-04-15T15:12:20.833 INFO:teuthology.orchestra.run.smithi072.stdout:Total 14 MB/s | 228 MB 00:15 2024-04-15T15:12:21.230 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-04-15T15:12:21.503 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-04-15T15:12:21.503 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-04-15T15:12:22.117 INFO:teuthology.orchestra.run.smithi175.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-04-15T15:12:23.574 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-04-15T15:12:23.574 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-04-15T15:12:27.181 INFO:teuthology.orchestra.run.smithi175.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-04-15T15:12:29.862 INFO:teuthology.orchestra.run.smithi175.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-15T15:12:29.863 INFO:teuthology.orchestra.run.smithi175.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-15T15:12:29.876 INFO:teuthology.orchestra.run.smithi175.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-04-15T15:12:29.877 INFO:teuthology.orchestra.run.smithi175.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-15T15:12:29.878 INFO:teuthology.orchestra.run.smithi175.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-04-15T15:12:29.879 INFO:teuthology.orchestra.run.smithi175.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-04-15T15:12:29.881 INFO:teuthology.orchestra.run.smithi175.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-15T15:12:29.884 INFO:teuthology.orchestra.run.smithi175.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-04-15T15:12:29.885 INFO:teuthology.orchestra.run.smithi175.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-04-15T15:12:29.967 INFO:teuthology.orchestra.run.smithi175.stdout:Dependencies resolved. 2024-04-15T15:12:29.990 INFO:teuthology.orchestra.run.smithi175.stdout:============================================================================================ 2024-04-15T15:12:29.990 INFO:teuthology.orchestra.run.smithi175.stdout: Package Arch Version Repository Size 2024-04-15T15:12:29.990 INFO:teuthology.orchestra.run.smithi175.stdout:============================================================================================ 2024-04-15T15:12:29.990 INFO:teuthology.orchestra.run.smithi175.stdout:Installing: 2024-04-15T15:12:29.990 INFO:teuthology.orchestra.run.smithi175.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-04-15T15:12:29.990 INFO:teuthology.orchestra.run.smithi175.stdout: ceph x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 6.6 k 2024-04-15T15:12:29.990 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-base x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 5.4 M 2024-04-15T15:12:29.991 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-fuse x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 890 k 2024-04-15T15:12:29.991 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 145 k 2024-04-15T15:12:29.991 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 1.5 M 2024-04-15T15:12:29.991 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 142 k 2024-04-15T15:12:29.991 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 3.5 M 2024-04-15T15:12:29.991 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 7.4 M 2024-04-15T15:12:29.991 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-rook noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 49 k 2024-04-15T15:12:29.991 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-radosgw x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 10 M 2024-04-15T15:12:29.991 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-test x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 49 M 2024-04-15T15:12:29.991 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-volume noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 264 k 2024-04-15T15:12:29.991 INFO:teuthology.orchestra.run.smithi175.stdout: cephadm noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 762 k 2024-04-15T15:12:29.991 INFO:teuthology.orchestra.run.smithi175.stdout: cephfs-mirror x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 224 k 2024-04-15T15:12:29.992 INFO:teuthology.orchestra.run.smithi175.stdout: cephfs-shell x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 27 k 2024-04-15T15:12:29.992 INFO:teuthology.orchestra.run.smithi175.stdout: cephfs-top noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 18 k 2024-04-15T15:12:29.992 INFO:teuthology.orchestra.run.smithi175.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-15T15:12:29.992 INFO:teuthology.orchestra.run.smithi175.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-04-15T15:12:29.992 INFO:teuthology.orchestra.run.smithi175.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-04-15T15:12:29.992 INFO:teuthology.orchestra.run.smithi175.stdout: libcephfs-devel x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 32 k 2024-04-15T15:12:29.992 INFO:teuthology.orchestra.run.smithi175.stdout: libcephfs2 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 747 k 2024-04-15T15:12:29.992 INFO:teuthology.orchestra.run.smithi175.stdout: librados-devel x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 127 k 2024-04-15T15:12:29.992 INFO:teuthology.orchestra.run.smithi175.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-04-15T15:12:29.992 INFO:teuthology.orchestra.run.smithi175.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-15T15:12:29.992 INFO:teuthology.orchestra.run.smithi175.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-15T15:12:29.992 INFO:teuthology.orchestra.run.smithi175.stdout: postgresql x86_64 13.14-1.el9 CentOS-AppStream 1.5 M 2024-04-15T15:12:29.992 INFO:teuthology.orchestra.run.smithi175.stdout: postgresql-contrib x86_64 13.14-1.el9 CentOS-AppStream 817 k 2024-04-15T15:12:29.992 INFO:teuthology.orchestra.run.smithi175.stdout: postgresql-server x86_64 13.14-1.el9 CentOS-AppStream 5.7 M 2024-04-15T15:12:29.993 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cephfs x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 162 k 2024-04-15T15:12:29.993 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rados x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 320 k 2024-04-15T15:12:29.993 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rbd x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 297 k 2024-04-15T15:12:29.993 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rgw x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 99 k 2024-04-15T15:12:29.993 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-fuse x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 85 k 2024-04-15T15:12:29.993 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-mirror x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 3.1 M 2024-04-15T15:12:29.993 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-nbd x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 171 k 2024-04-15T15:12:29.993 INFO:teuthology.orchestra.run.smithi175.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-15T15:12:29.993 INFO:teuthology.orchestra.run.smithi175.stdout:Upgrading: 2024-04-15T15:12:29.993 INFO:teuthology.orchestra.run.smithi175.stdout: NetworkManager x86_64 1:1.47.4-1.el9 CentOS-BaseOS 2.3 M 2024-04-15T15:12:29.993 INFO:teuthology.orchestra.run.smithi175.stdout: NetworkManager-libnm x86_64 1:1.47.4-1.el9 CentOS-BaseOS 1.8 M 2024-04-15T15:12:29.993 INFO:teuthology.orchestra.run.smithi175.stdout: NetworkManager-team x86_64 1:1.47.4-1.el9 CentOS-BaseOS 41 k 2024-04-15T15:12:29.993 INFO:teuthology.orchestra.run.smithi175.stdout: NetworkManager-tui x86_64 1:1.47.4-1.el9 CentOS-BaseOS 248 k 2024-04-15T15:12:29.994 INFO:teuthology.orchestra.run.smithi175.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-04-15T15:12:29.994 INFO:teuthology.orchestra.run.smithi175.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-04-15T15:12:29.994 INFO:teuthology.orchestra.run.smithi175.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-04-15T15:12:29.994 INFO:teuthology.orchestra.run.smithi175.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-04-15T15:12:29.994 INFO:teuthology.orchestra.run.smithi175.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-04-15T15:12:29.994 INFO:teuthology.orchestra.run.smithi175.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-04-15T15:12:29.995 INFO:teuthology.orchestra.run.smithi175.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-04-15T15:12:29.995 INFO:teuthology.orchestra.run.smithi175.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-04-15T15:12:29.995 INFO:teuthology.orchestra.run.smithi175.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-04-15T15:12:29.995 INFO:teuthology.orchestra.run.smithi175.stdout: librados2 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 3.4 M 2024-04-15T15:12:29.995 INFO:teuthology.orchestra.run.smithi175.stdout: librbd1 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 3.1 M 2024-04-15T15:12:29.995 INFO:teuthology.orchestra.run.smithi175.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-04-15T15:12:29.995 INFO:teuthology.orchestra.run.smithi175.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-04-15T15:12:29.995 INFO:teuthology.orchestra.run.smithi175.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-04-15T15:12:29.995 INFO:teuthology.orchestra.run.smithi175.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-04-15T15:12:29.995 INFO:teuthology.orchestra.run.smithi175.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-04-15T15:12:29.995 INFO:teuthology.orchestra.run.smithi175.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-04-15T15:12:29.995 INFO:teuthology.orchestra.run.smithi175.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-04-15T15:12:29.995 INFO:teuthology.orchestra.run.smithi175.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-04-15T15:12:29.996 INFO:teuthology.orchestra.run.smithi175.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-04-15T15:12:29.996 INFO:teuthology.orchestra.run.smithi175.stdout:Installing dependencies: 2024-04-15T15:12:29.996 INFO:teuthology.orchestra.run.smithi175.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-15T15:12:29.996 INFO:teuthology.orchestra.run.smithi175.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-04-15T15:12:29.996 INFO:teuthology.orchestra.run.smithi175.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-15T15:12:29.996 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-common x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 21 M 2024-04-15T15:12:29.996 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 34 k 2024-04-15T15:12:29.996 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mds x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 2.4 M 2024-04-15T15:12:29.996 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 249 k 2024-04-15T15:12:29.996 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mon x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 4.7 M 2024-04-15T15:12:29.996 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-osd x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 17 M 2024-04-15T15:12:29.996 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 16 k 2024-04-15T15:12:29.996 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-selinux x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 25 k 2024-04-15T15:12:29.996 INFO:teuthology.orchestra.run.smithi175.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-15T15:12:29.997 INFO:teuthology.orchestra.run.smithi175.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-15T15:12:29.997 INFO:teuthology.orchestra.run.smithi175.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-15T15:12:29.997 INFO:teuthology.orchestra.run.smithi175.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-15T15:12:29.997 INFO:teuthology.orchestra.run.smithi175.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-15T15:12:29.997 INFO:teuthology.orchestra.run.smithi175.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-15T15:12:29.997 INFO:teuthology.orchestra.run.smithi175.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-15T15:12:29.997 INFO:teuthology.orchestra.run.smithi175.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-15T15:12:29.997 INFO:teuthology.orchestra.run.smithi175.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-15T15:12:29.997 INFO:teuthology.orchestra.run.smithi175.stdout: libcephsqlite x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 164 k 2024-04-15T15:12:29.997 INFO:teuthology.orchestra.run.smithi175.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-15T15:12:29.998 INFO:teuthology.orchestra.run.smithi175.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-15T15:12:29.998 INFO:teuthology.orchestra.run.smithi175.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-15T15:12:29.998 INFO:teuthology.orchestra.run.smithi175.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-15T15:12:29.998 INFO:teuthology.orchestra.run.smithi175.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-15T15:12:29.998 INFO:teuthology.orchestra.run.smithi175.stdout: libradosstriper1 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 501 k 2024-04-15T15:12:29.998 INFO:teuthology.orchestra.run.smithi175.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-15T15:12:29.998 INFO:teuthology.orchestra.run.smithi175.stdout: librgw2 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 5.2 M 2024-04-15T15:12:29.998 INFO:teuthology.orchestra.run.smithi175.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-15T15:12:29.998 INFO:teuthology.orchestra.run.smithi175.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-04-15T15:12:29.998 INFO:teuthology.orchestra.run.smithi175.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-15T15:12:29.998 INFO:teuthology.orchestra.run.smithi175.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-15T15:12:29.998 INFO:teuthology.orchestra.run.smithi175.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-15T15:12:29.998 INFO:teuthology.orchestra.run.smithi175.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-15T15:12:29.999 INFO:teuthology.orchestra.run.smithi175.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-15T15:12:29.999 INFO:teuthology.orchestra.run.smithi175.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-15T15:12:29.999 INFO:teuthology.orchestra.run.smithi175.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-15T15:12:29.999 INFO:teuthology.orchestra.run.smithi175.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-15T15:12:29.999 INFO:teuthology.orchestra.run.smithi175.stdout: postgresql-private-libs x86_64 13.14-1.el9 CentOS-AppStream 132 k 2024-04-15T15:12:29.999 INFO:teuthology.orchestra.run.smithi175.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-15T15:12:29.999 INFO:teuthology.orchestra.run.smithi175.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-15T15:12:29.999 INFO:teuthology.orchestra.run.smithi175.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-15T15:12:29.999 INFO:teuthology.orchestra.run.smithi175.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-04-15T15:12:29.999 INFO:teuthology.orchestra.run.smithi175.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-15T15:12:29.999 INFO:teuthology.orchestra.run.smithi175.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-15T15:12:29.999 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-15T15:12:29.999 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ceph-argparse x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 45 k 2024-04-15T15:12:29.999 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ceph-common x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 135 k 2024-04-15T15:12:30.000 INFO:teuthology.orchestra.run.smithi175.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-15T15:12:30.000 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-15T15:12:30.000 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-15T15:12:30.000 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-15T15:12:30.000 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-15T15:12:30.000 INFO:teuthology.orchestra.run.smithi175.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-15T15:12:30.000 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-15T15:12:30.000 INFO:teuthology.orchestra.run.smithi175.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-15T15:12:30.000 INFO:teuthology.orchestra.run.smithi175.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-15T15:12:30.000 INFO:teuthology.orchestra.run.smithi175.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-15T15:12:30.001 INFO:teuthology.orchestra.run.smithi175.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-15T15:12:30.001 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-15T15:12:30.001 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-15T15:12:30.001 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-15T15:12:30.001 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-15T15:12:30.001 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-15T15:12:30.001 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-15T15:12:30.001 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-15T15:12:30.001 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-15T15:12:30.001 INFO:teuthology.orchestra.run.smithi175.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-15T15:12:30.001 INFO:teuthology.orchestra.run.smithi175.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-15T15:12:30.001 INFO:teuthology.orchestra.run.smithi175.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-15T15:12:30.001 INFO:teuthology.orchestra.run.smithi175.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-15T15:12:30.002 INFO:teuthology.orchestra.run.smithi175.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-15T15:12:30.002 INFO:teuthology.orchestra.run.smithi175.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-15T15:12:30.002 INFO:teuthology.orchestra.run.smithi175.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-15T15:12:30.002 INFO:teuthology.orchestra.run.smithi175.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-15T15:12:30.002 INFO:teuthology.orchestra.run.smithi175.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-15T15:12:30.003 INFO:teuthology.orchestra.run.smithi175.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-15T15:12:30.003 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-15T15:12:30.003 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-15T15:12:30.003 INFO:teuthology.orchestra.run.smithi175.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-15T15:12:30.003 INFO:teuthology.orchestra.run.smithi175.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-15T15:12:30.003 INFO:teuthology.orchestra.run.smithi175.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-15T15:12:30.003 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-15T15:12:30.003 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-15T15:12:30.004 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-15T15:12:30.004 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-15T15:12:30.004 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-15T15:12:30.004 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-15T15:12:30.004 INFO:teuthology.orchestra.run.smithi175.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-15T15:12:30.004 INFO:teuthology.orchestra.run.smithi175.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-15T15:12:30.004 INFO:teuthology.orchestra.run.smithi175.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-15T15:12:30.004 INFO:teuthology.orchestra.run.smithi175.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-15T15:12:30.004 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-15T15:12:30.004 INFO:teuthology.orchestra.run.smithi175.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-15T15:12:30.004 INFO:teuthology.orchestra.run.smithi175.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-15T15:12:30.004 INFO:teuthology.orchestra.run.smithi175.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-15T15:12:30.004 INFO:teuthology.orchestra.run.smithi175.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-15T15:12:30.005 INFO:teuthology.orchestra.run.smithi175.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-15T15:12:30.005 INFO:teuthology.orchestra.run.smithi175.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-04-15T15:12:30.005 INFO:teuthology.orchestra.run.smithi175.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-15T15:12:30.005 INFO:teuthology.orchestra.run.smithi175.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-15T15:12:30.005 INFO:teuthology.orchestra.run.smithi175.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-15T15:12:30.005 INFO:teuthology.orchestra.run.smithi175.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-15T15:12:30.005 INFO:teuthology.orchestra.run.smithi175.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-15T15:12:30.005 INFO:teuthology.orchestra.run.smithi175.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-15T15:12:30.005 INFO:teuthology.orchestra.run.smithi175.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-15T15:12:30.005 INFO:teuthology.orchestra.run.smithi175.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-15T15:12:30.005 INFO:teuthology.orchestra.run.smithi175.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-15T15:12:30.005 INFO:teuthology.orchestra.run.smithi175.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-04-15T15:12:30.005 INFO:teuthology.orchestra.run.smithi175.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-15T15:12:30.006 INFO:teuthology.orchestra.run.smithi175.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-04-15T15:12:30.006 INFO:teuthology.orchestra.run.smithi175.stdout:Installing weak dependencies: 2024-04-15T15:12:30.006 INFO:teuthology.orchestra.run.smithi175.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-15T15:12:30.006 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:12:30.006 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction Summary 2024-04-15T15:12:30.006 INFO:teuthology.orchestra.run.smithi175.stdout:============================================================================================ 2024-04-15T15:12:30.006 INFO:teuthology.orchestra.run.smithi175.stdout:Install 150 Packages 2024-04-15T15:12:30.006 INFO:teuthology.orchestra.run.smithi175.stdout:Upgrade 24 Packages 2024-04-15T15:12:30.006 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:12:30.007 INFO:teuthology.orchestra.run.smithi175.stdout:Total download size: 228 M 2024-04-15T15:12:30.007 INFO:teuthology.orchestra.run.smithi175.stdout:Downloading Packages: 2024-04-15T15:12:31.070 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-04-15T15:12:31.187 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-04-15T15:12:31.222 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-04-15T15:12:31.341 INFO:teuthology.orchestra.run.smithi175.stdout:(1/174): ceph-19.0.0-2861.g9bf1c4ba.el8.x86_64. 15 kB/s | 6.6 kB 00:00 2024-04-15T15:12:31.344 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-15T15:12:31.387 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-15T15:12:31.674 INFO:teuthology.orchestra.run.smithi175.stdout:(2/174): ceph-fuse-19.0.0-2861.g9bf1c4ba.el8.x8 2.6 MB/s | 890 kB 00:00 2024-04-15T15:12:31.705 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-04-15T15:12:31.768 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-04-15T15:12:31.774 INFO:teuthology.orchestra.run.smithi175.stdout:(3/174): ceph-immutable-object-cache-19.0.0-286 1.4 MB/s | 145 kB 00:00 2024-04-15T15:12:31.808 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-04-15T15:12:31.842 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-04-15T15:12:31.886 INFO:teuthology.orchestra.run.smithi175.stdout:(4/174): ceph-base-19.0.0-2861.g9bf1c4ba.el8.x8 5.5 MB/s | 5.4 MB 00:00 2024-04-15T15:12:31.905 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-04-15T15:12:32.025 INFO:teuthology.orchestra.run.smithi175.stdout:(5/174): ceph-mgr-19.0.0-2861.g9bf1c4ba.el8.x86 11 MB/s | 1.5 MB 00:00 2024-04-15T15:12:32.225 INFO:teuthology.orchestra.run.smithi175.stdout:(6/174): ceph-mds-19.0.0-2861.g9bf1c4ba.el8.x86 5.2 MB/s | 2.4 MB 00:00 2024-04-15T15:12:32.242 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-04-15T15:12:32.372 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : librados2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 10/198 2024-04-15T15:12:32.409 INFO:teuthology.orchestra.run.smithi175.stdout:(7/174): ceph-mon-19.0.0-2861.g9bf1c4ba.el8.x86 12 MB/s | 4.7 MB 00:00 2024-04-15T15:12:32.494 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librados2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 10/198 2024-04-15T15:12:32.523 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libcephfs2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 11/198 2024-04-15T15:12:32.620 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libcephfs2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 11/198 2024-04-15T15:12:32.650 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rados-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 12/198 2024-04-15T15:12:32.725 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-04-15T15:12:32.898 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-04-15T15:12:32.953 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-04-15T15:12:32.982 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libcephsqlite-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 16/198 2024-04-15T15:12:33.086 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 16/198 2024-04-15T15:12:33.117 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libradosstriper1-2:19.0.0-2861.g9bf1c4ba.el8.x86 17/198 2024-04-15T15:12:33.185 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2861.g9bf1c4ba.el8.x86 17/198 2024-04-15T15:12:33.224 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-04-15T15:12:33.226 INFO:teuthology.orchestra.run.smithi175.stdout:(8/174): ceph-radosgw-19.0.0-2861.g9bf1c4ba.el8 12 MB/s | 10 MB 00:00 2024-04-15T15:12:33.247 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-15T15:12:33.263 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-15T15:12:33.309 INFO:teuthology.orchestra.run.smithi175.stdout:(9/174): ceph-selinux-19.0.0-2861.g9bf1c4ba.el8 303 kB/s | 25 kB 00:00 2024-04-15T15:12:33.458 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-15T15:12:33.489 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-15T15:12:33.522 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-04-15T15:12:33.973 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-04-15T15:12:34.053 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-15T15:12:34.107 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-15T15:12:34.143 INFO:teuthology.orchestra.run.smithi175.stdout:(10/174): ceph-common-19.0.0-2861.g9bf1c4ba.el8 6.4 MB/s | 21 MB 00:03 2024-04-15T15:12:34.226 INFO:teuthology.orchestra.run.smithi175.stdout:(11/174): cephfs-mirror-19.0.0-2861.g9bf1c4ba.e 2.6 MB/s | 224 kB 00:00 2024-04-15T15:12:34.227 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-15T15:12:34.293 INFO:teuthology.orchestra.run.smithi175.stdout:(12/174): cephfs-shell-19.0.0-2861.g9bf1c4ba.el 405 kB/s | 27 kB 00:00 2024-04-15T15:12:34.332 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-15T15:12:34.360 INFO:teuthology.orchestra.run.smithi175.stdout:(13/174): libcephfs-devel-19.0.0-2861.g9bf1c4ba 478 kB/s | 32 kB 00:00 2024-04-15T15:12:34.519 INFO:teuthology.orchestra.run.smithi175.stdout:(14/174): libcephfs2-19.0.0-2861.g9bf1c4ba.el8. 4.6 MB/s | 747 kB 00:00 2024-04-15T15:12:34.574 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-15T15:12:34.682 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-04-15T15:12:34.686 INFO:teuthology.orchestra.run.smithi175.stdout:(15/174): ceph-osd-19.0.0-2861.g9bf1c4ba.el8.x8 6.8 MB/s | 17 MB 00:02 2024-04-15T15:12:34.711 INFO:teuthology.orchestra.run.smithi175.stdout:(16/174): libcephsqlite-19.0.0-2861.g9bf1c4ba.e 853 kB/s | 164 kB 00:00 2024-04-15T15:12:34.753 INFO:teuthology.orchestra.run.smithi175.stdout:(17/174): librados-devel-19.0.0-2861.g9bf1c4ba. 1.9 MB/s | 127 kB 00:00 2024-04-15T15:12:34.777 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-04-15T15:12:34.786 INFO:teuthology.orchestra.run.smithi175.stdout:(18/174): libradosstriper1-19.0.0-2861.g9bf1c4b 6.5 MB/s | 501 kB 00:00 2024-04-15T15:12:34.815 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-04-15T15:12:34.852 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-04-15T15:12:34.870 INFO:teuthology.orchestra.run.smithi175.stdout:(19/174): python3-ceph-argparse-19.0.0-2861.g9b 542 kB/s | 45 kB 00:00 2024-04-15T15:12:34.896 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-04-15T15:12:34.962 INFO:teuthology.orchestra.run.smithi175.stdout:(20/174): python3-ceph-common-19.0.0-2861.g9bf1 1.4 MB/s | 135 kB 00:00 2024-04-15T15:12:34.971 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-04-15T15:12:35.034 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-04-15T15:12:35.071 INFO:teuthology.orchestra.run.smithi175.stdout:(21/174): python3-cephfs-19.0.0-2861.g9bf1c4ba. 1.5 MB/s | 162 kB 00:00 2024-04-15T15:12:35.072 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-04-15T15:12:35.138 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-04-15T15:12:35.180 INFO:teuthology.orchestra.run.smithi175.stdout:(22/174): python3-rados-19.0.0-2861.g9bf1c4ba.e 2.9 MB/s | 320 kB 00:00 2024-04-15T15:12:35.181 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-04-15T15:12:35.228 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-04-15T15:12:35.286 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-04-15T15:12:35.296 INFO:teuthology.orchestra.run.smithi175.stdout:(23/174): python3-rbd-19.0.0-2861.g9bf1c4ba.el8 2.5 MB/s | 297 kB 00:00 2024-04-15T15:12:35.314 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-04-15T15:12:35.383 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-04-15T15:12:35.405 INFO:teuthology.orchestra.run.smithi175.stdout:(24/174): python3-rgw-19.0.0-2861.g9bf1c4ba.el8 915 kB/s | 99 kB 00:00 2024-04-15T15:12:35.420 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-ceph-common-2:19.0.0-2861.g9bf1c4ba.el8. 39/198 2024-04-15T15:12:35.464 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-ceph-argparse-2:19.0.0-2861.g9bf1c4ba.el 40/198 2024-04-15T15:12:35.493 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cephfs-2:19.0.0-2861.g9bf1c4ba.el8.x86_6 41/198 2024-04-15T15:12:35.547 INFO:teuthology.orchestra.run.smithi175.stdout:(25/174): librgw2-19.0.0-2861.g9bf1c4ba.el8.x86 6.6 MB/s | 5.2 MB 00:00 2024-04-15T15:12:35.562 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-04-15T15:12:35.572 INFO:teuthology.orchestra.run.smithi175.stdout:(26/174): rbd-fuse-19.0.0-2861.g9bf1c4ba.el8.x8 511 kB/s | 85 kB 00:00 2024-04-15T15:12:35.619 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-04-15T15:12:35.681 INFO:teuthology.orchestra.run.smithi175.stdout:(27/174): rbd-nbd-19.0.0-2861.g9bf1c4ba.el8.x86 1.5 MB/s | 171 kB 00:00 2024-04-15T15:12:35.701 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-04-15T15:12:35.781 INFO:teuthology.orchestra.run.smithi175.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2861.g 342 kB/s | 34 kB 00:00 2024-04-15T15:12:35.836 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-04-15T15:12:35.890 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-04-15T15:12:35.894 INFO:teuthology.orchestra.run.smithi175.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2861.g9bf1c4b 1.2 MB/s | 142 kB 00:00 2024-04-15T15:12:36.007 INFO:teuthology.orchestra.run.smithi175.stdout:(30/174): rbd-mirror-19.0.0-2861.g9bf1c4ba.el8. 6.7 MB/s | 3.1 MB 00:00 2024-04-15T15:12:36.424 INFO:teuthology.orchestra.run.smithi175.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2861.g9bf1c 6.7 MB/s | 3.5 MB 00:00 2024-04-15T15:12:36.427 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-04-15T15:12:36.514 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-04-15T15:12:36.644 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-04-15T15:12:36.689 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-15T15:12:36.699 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-15T15:12:36.736 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-04-15T15:12:36.849 INFO:teuthology.orchestra.run.smithi175.stdout:(32/174): ceph-test-19.0.0-2861.g9bf1c4ba.el8.x 14 MB/s | 49 MB 00:03 2024-04-15T15:12:36.874 INFO:teuthology.orchestra.run.smithi175.stdout:(33/174): ceph-mgr-modules-core-19.0.0-2861.g9b 552 kB/s | 249 kB 00:00 2024-04-15T15:12:36.891 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-04-15T15:12:36.916 INFO:teuthology.orchestra.run.smithi175.stdout:(34/174): ceph-mgr-rook-19.0.0-2861.g9bf1c4ba.e 743 kB/s | 49 kB 00:00 2024-04-15T15:12:36.933 INFO:teuthology.orchestra.run.smithi175.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2861.g9 281 kB/s | 16 kB 00:00 2024-04-15T15:12:36.993 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librados-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_6 53/198 2024-04-15T15:12:37.042 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-04-15T15:12:37.050 INFO:teuthology.orchestra.run.smithi175.stdout:(36/174): ceph-mgr-diskprediction-local-19.0.0- 7.1 MB/s | 7.4 MB 00:01 2024-04-15T15:12:37.075 INFO:teuthology.orchestra.run.smithi175.stdout:(37/174): ceph-volume-19.0.0-2861.g9bf1c4ba.el8 1.6 MB/s | 264 kB 00:00 2024-04-15T15:12:37.117 INFO:teuthology.orchestra.run.smithi175.stdout:(38/174): cephadm-19.0.0-2861.g9bf1c4ba.el8.noa 4.0 MB/s | 762 kB 00:00 2024-04-15T15:12:37.168 INFO:teuthology.orchestra.run.smithi175.stdout:(39/174): cephfs-top-19.0.0-2861.g9bf1c4ba.el8. 156 kB/s | 18 kB 00:00 2024-04-15T15:12:37.288 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-04-15T15:12:37.309 INFO:teuthology.orchestra.run.smithi175.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 5.4 MB/s | 920 kB 00:00 2024-04-15T15:12:37.335 INFO:teuthology.orchestra.run.smithi175.stdout:(41/174): automake-1.16.2-8.el9.noarch.rpm 2.5 MB/s | 662 kB 00:00 2024-04-15T15:12:37.360 INFO:teuthology.orchestra.run.smithi175.stdout:(42/174): boost-program-options-1.75.0-8.el9.x8 643 kB/s | 106 kB 00:00 2024-04-15T15:12:37.385 INFO:teuthology.orchestra.run.smithi175.stdout:(43/174): flex-2.6.4-9.el9.x86_64.rpm 4.0 MB/s | 307 kB 00:00 2024-04-15T15:12:37.411 INFO:teuthology.orchestra.run.smithi175.stdout:(44/174): flexiblas-3.0.4-8.el9.x86_64.rpm 638 kB/s | 32 kB 00:00 2024-04-15T15:12:37.510 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-04-15T15:12:37.519 INFO:teuthology.orchestra.run.smithi175.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 128 kB/s | 17 kB 00:00 2024-04-15T15:12:37.562 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-04-15T15:12:37.569 INFO:teuthology.orchestra.run.smithi175.stdout:(46/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 14 MB/s | 3.0 MB 00:00 2024-04-15T15:12:37.595 INFO:teuthology.orchestra.run.smithi175.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 55 kB/s | 10 kB 00:00 2024-04-15T15:12:37.608 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-04-15T15:12:37.620 INFO:teuthology.orchestra.run.smithi175.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.6 MB/s | 159 kB 00:00 2024-04-15T15:12:37.645 INFO:teuthology.orchestra.run.smithi175.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 891 kB/s | 44 kB 00:00 2024-04-15T15:12:37.687 INFO:teuthology.orchestra.run.smithi175.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 7.1 MB/s | 661 kB 00:00 2024-04-15T15:12:37.720 INFO:teuthology.orchestra.run.smithi175.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 5.6 MB/s | 578 kB 00:00 2024-04-15T15:12:37.746 INFO:teuthology.orchestra.run.smithi175.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 464 kB/s | 46 kB 00:00 2024-04-15T15:12:37.771 INFO:teuthology.orchestra.run.smithi175.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-04-15T15:12:37.780 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-04-15T15:12:37.796 INFO:teuthology.orchestra.run.smithi175.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 187 kB 00:00 2024-04-15T15:12:37.823 INFO:teuthology.orchestra.run.smithi175.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2024-04-15T15:12:37.847 INFO:teuthology.orchestra.run.smithi175.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 455 kB/s | 34 kB 00:00 2024-04-15T15:12:37.856 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-04-15T15:12:37.912 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-04-15T15:12:37.986 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-04-15T15:12:38.019 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-04-15T15:12:38.047 INFO:teuthology.orchestra.run.smithi175.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 19 MB/s | 4.7 MB 00:00 2024-04-15T15:12:38.047 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-04-15T15:12:38.081 INFO:teuthology.orchestra.run.smithi175.stdout:(58/174): postgresql-contrib-13.14-1.el9.x86_64 3.5 MB/s | 817 kB 00:00 2024-04-15T15:12:38.103 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-04-15T15:12:38.114 INFO:teuthology.orchestra.run.smithi175.stdout:(59/174): postgresql-13.14-1.el9.x86_64.rpm 5.8 MB/s | 1.5 MB 00:00 2024-04-15T15:12:38.139 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-04-15T15:12:38.140 INFO:teuthology.orchestra.run.smithi175.stdout:(60/174): postgresql-private-libs-13.14-1.el9.x 1.4 MB/s | 132 kB 00:00 2024-04-15T15:12:38.183 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-04-15T15:12:38.236 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-04-15T15:12:38.299 INFO:teuthology.orchestra.run.smithi175.stdout:(61/174): postgresql-server-13.14-1.el9.x86_64. 26 MB/s | 5.7 MB 00:00 2024-04-15T15:12:38.332 INFO:teuthology.orchestra.run.smithi175.stdout:(62/174): protobuf-3.14.0-13.el9.x86_64.rpm 4.7 MB/s | 1.0 MB 00:00 2024-04-15T15:12:38.357 INFO:teuthology.orchestra.run.smithi175.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 359 kB/s | 78 kB 00:00 2024-04-15T15:12:38.379 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-04-15T15:12:38.436 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-04-15T15:12:38.474 INFO:teuthology.orchestra.run.smithi175.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 33 MB/s | 5.8 MB 00:00 2024-04-15T15:12:38.475 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-04-15T15:12:38.499 INFO:teuthology.orchestra.run.smithi175.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 177 kB/s | 24 kB 00:00 2024-04-15T15:12:38.517 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-04-15T15:12:38.524 INFO:teuthology.orchestra.run.smithi175.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 111 kB/s | 18 kB 00:00 2024-04-15T15:12:38.550 INFO:teuthology.orchestra.run.smithi175.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 633 kB/s | 32 kB 00:00 2024-04-15T15:12:38.575 INFO:teuthology.orchestra.run.smithi175.stdout:(68/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-04-15T15:12:38.613 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-04-15T15:12:38.653 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-04-15T15:12:38.667 INFO:teuthology.orchestra.run.smithi175.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-04-15T15:12:38.692 INFO:teuthology.orchestra.run.smithi175.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.5 MB/s | 217 kB 00:00 2024-04-15T15:12:38.694 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-04-15T15:12:38.717 INFO:teuthology.orchestra.run.smithi175.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.2 MB/s | 174 kB 00:00 2024-04-15T15:12:38.729 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-04-15T15:12:38.743 INFO:teuthology.orchestra.run.smithi175.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 919 kB/s | 69 kB 00:00 2024-04-15T15:12:38.753 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-04-15T15:12:38.768 INFO:teuthology.orchestra.run.smithi175.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 825 kB/s | 41 kB 00:00 2024-04-15T15:12:38.793 INFO:teuthology.orchestra.run.smithi175.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-04-15T15:12:38.818 INFO:teuthology.orchestra.run.smithi175.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-04-15T15:12:38.841 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-04-15T15:12:38.844 INFO:teuthology.orchestra.run.smithi175.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 950 kB/s | 47 kB 00:00 2024-04-15T15:12:38.869 INFO:teuthology.orchestra.run.smithi175.stdout:(77/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 211 kB 00:00 2024-04-15T15:12:38.894 INFO:teuthology.orchestra.run.smithi175.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 569 kB/s | 43 kB 00:00 2024-04-15T15:12:38.977 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-04-15T15:12:39.088 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-04-15T15:12:39.111 INFO:teuthology.orchestra.run.smithi175.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 181 kB/s | 39 kB 00:00 2024-04-15T15:12:39.137 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-04-15T15:12:39.195 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-04-15T15:12:39.260 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-04-15T15:12:39.269 INFO:teuthology.orchestra.run.smithi175.stdout:(80/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-04-15T15:12:39.294 INFO:teuthology.orchestra.run.smithi175.stdout:(81/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 103 kB/s | 41 kB 00:00 2024-04-15T15:12:39.320 INFO:teuthology.orchestra.run.smithi175.stdout:(82/174): qatlib-24.02.0-1.el9.x86_64.rpm 1.0 MB/s | 220 kB 00:00 2024-04-15T15:12:39.345 INFO:teuthology.orchestra.run.smithi175.stdout:(83/174): qatlib-service-24.02.0-1.el9.x86_64.r 698 kB/s | 35 kB 00:00 2024-04-15T15:12:39.370 INFO:teuthology.orchestra.run.smithi175.stdout:(84/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 618 kB/s | 46 kB 00:00 2024-04-15T15:12:39.426 INFO:teuthology.orchestra.run.smithi175.stdout:(85/174): socat-1.7.4.1-5.el9.x86_64.rpm 2.8 MB/s | 300 kB 00:00 2024-04-15T15:12:39.454 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-04-15T15:12:39.454 INFO:teuthology.orchestra.run.smithi175.stdout:(86/174): uuid-1.6.2-55.el9.x86_64.rpm 677 kB/s | 56 kB 00:00 2024-04-15T15:12:39.479 INFO:teuthology.orchestra.run.smithi175.stdout:(87/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 578 kB/s | 63 kB 00:00 2024-04-15T15:12:39.505 INFO:teuthology.orchestra.run.smithi175.stdout:(88/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 559 kB/s | 44 kB 00:00 2024-04-15T15:12:39.514 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-04-15T15:12:39.530 INFO:teuthology.orchestra.run.smithi175.stdout:(89/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-04-15T15:12:39.555 INFO:teuthology.orchestra.run.smithi175.stdout:(90/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-04-15T15:12:39.589 INFO:teuthology.orchestra.run.smithi175.stdout:(91/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-04-15T15:12:39.614 INFO:teuthology.orchestra.run.smithi175.stdout:(92/174): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-04-15T15:12:39.639 INFO:teuthology.orchestra.run.smithi175.stdout:(93/174): python3-pycparser-2.20-6.el9.noarch.r 1.5 MB/s | 124 kB 00:00 2024-04-15T15:12:39.651 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-04-15T15:12:39.665 INFO:teuthology.orchestra.run.smithi175.stdout:(94/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.4 MB/s | 187 kB 00:00 2024-04-15T15:12:39.696 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-04-15T15:12:39.729 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-04-15T15:12:39.732 INFO:teuthology.orchestra.run.smithi175.stdout:(95/174): jq-1.6-16.el9.x86_64.rpm 2.0 MB/s | 187 kB 00:00 2024-04-15T15:12:39.757 INFO:teuthology.orchestra.run.smithi175.stdout:(96/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 195 kB 00:00 2024-04-15T15:12:39.769 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-15T15:12:39.806 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-04-15T15:12:39.815 INFO:teuthology.orchestra.run.smithi175.stdout:(97/174): libgfortran-11.4.1-3.el9.x86_64.rpm 4.5 MB/s | 803 kB 00:00 2024-04-15T15:12:39.858 INFO:teuthology.orchestra.run.smithi175.stdout:(98/174): python3-requests-2.25.1-8.el9.noarch. 995 kB/s | 125 kB 00:00 2024-04-15T15:12:39.858 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-04-15T15:12:39.898 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-04-15T15:12:39.939 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-04-15T15:12:39.978 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-04-15T15:12:40.054 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-04-15T15:12:40.089 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-04-15T15:12:40.123 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-04-15T15:12:40.166 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-04-15T15:12:40.194 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-04-15T15:12:40.283 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-04-15T15:12:40.374 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-04-15T15:12:40.426 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-04-15T15:12:40.459 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-04-15T15:12:40.491 INFO:teuthology.orchestra.run.smithi175.stdout:(99/174): elfutils-libelf-devel-0.190-2.el9.x86 29 kB/s | 21 kB 00:00 2024-04-15T15:12:40.508 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-04-15T15:12:40.575 INFO:teuthology.orchestra.run.smithi175.stdout:(100/174): libuuid-devel-2.37.4-18.el9.x86_64.r 33 kB/s | 25 kB 00:00 2024-04-15T15:12:40.606 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-04-15T15:12:40.664 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-04-15T15:12:41.396 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-04-15T15:12:41.458 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-04-15T15:12:41.492 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-04-15T15:12:41.542 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-04-15T15:12:41.583 INFO:teuthology.orchestra.run.smithi175.stdout:(101/174): python3-devel-3.9.17-2.el9.x86_64.rp 225 kB/s | 245 kB 00:01 2024-04-15T15:12:41.623 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-04-15T15:12:41.642 INFO:teuthology.orchestra.run.smithi175.stdout:(102/174): python3-jinja2-2.11.3-5.el9.noarch.r 232 kB/s | 248 kB 00:01 2024-04-15T15:12:41.664 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-04-15T15:12:41.684 INFO:teuthology.orchestra.run.smithi175.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 536 kB/s | 22 kB 00:00 2024-04-15T15:12:41.784 INFO:teuthology.orchestra.run.smithi175.stdout:(104/174): protobuf-compiler-3.14.0-13.el9.x86_ 8.4 MB/s | 863 kB 00:00 2024-04-15T15:12:41.841 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-04-15T15:12:41.879 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-04-15T15:12:41.935 INFO:teuthology.orchestra.run.smithi175.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 3.6 MB/s | 548 kB 00:00 2024-04-15T15:12:41.965 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-04-15T15:12:41.985 INFO:teuthology.orchestra.run.smithi175.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 6.1 MB/s | 309 kB 00:00 2024-04-15T15:12:42.010 INFO:teuthology.orchestra.run.smithi175.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 774 kB/s | 19 kB 00:00 2024-04-15T15:12:42.186 INFO:teuthology.orchestra.run.smithi175.stdout:(108/174): libarrow-9.0.0-10.el9.x86_64.rpm 25 MB/s | 4.4 MB 00:00 2024-04-15T15:12:42.219 INFO:teuthology.orchestra.run.smithi175.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 782 kB/s | 26 kB 00:00 2024-04-15T15:12:42.253 INFO:teuthology.orchestra.run.smithi175.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-04-15T15:12:42.295 INFO:teuthology.orchestra.run.smithi175.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-04-15T15:12:42.345 INFO:teuthology.orchestra.run.smithi175.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-04-15T15:12:42.378 INFO:teuthology.orchestra.run.smithi175.stdout:(113/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 118 kB/s | 94 kB 00:00 2024-04-15T15:12:42.421 INFO:teuthology.orchestra.run.smithi175.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 11 MB/s | 838 kB 00:00 2024-04-15T15:12:42.454 INFO:teuthology.orchestra.run.smithi175.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.3 MB/s | 43 kB 00:00 2024-04-15T15:12:42.488 INFO:teuthology.orchestra.run.smithi175.stdout:(116/174): python3-cachetools-4.2.4-1.el9.noarc 967 kB/s | 32 kB 00:00 2024-04-15T15:12:42.521 INFO:teuthology.orchestra.run.smithi175.stdout:(117/174): python3-certifi-2021.10.8-2.el9.noar 448 kB/s | 15 kB 00:00 2024-04-15T15:12:42.547 INFO:teuthology.orchestra.run.smithi175.stdout:(118/174): python3-asyncssh-2.13.2-5.el9.noarch 3.2 MB/s | 548 kB 00:00 2024-04-15T15:12:42.572 INFO:teuthology.orchestra.run.smithi175.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-04-15T15:12:42.597 INFO:teuthology.orchestra.run.smithi175.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-04-15T15:12:42.607 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-04-15T15:12:42.727 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-04-15T15:12:42.731 INFO:teuthology.orchestra.run.smithi175.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.9 MB/s | 303 kB 00:00 2024-04-15T15:12:42.823 INFO:teuthology.orchestra.run.smithi175.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 156 kB/s | 35 kB 00:00 2024-04-15T15:12:42.848 INFO:teuthology.orchestra.run.smithi175.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 1.7 MB/s | 203 kB 00:00 2024-04-15T15:12:42.882 INFO:teuthology.orchestra.run.smithi175.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 4.2 MB/s | 144 kB 00:00 2024-04-15T15:12:42.949 INFO:teuthology.orchestra.run.smithi175.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 16 MB/s | 2.0 MB 00:00 2024-04-15T15:12:42.974 INFO:teuthology.orchestra.run.smithi175.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 116 kB/s | 11 kB 00:00 2024-04-15T15:12:42.991 INFO:teuthology.orchestra.run.smithi175.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 422 kB/s | 18 kB 00:00 2024-04-15T15:12:43.008 INFO:teuthology.orchestra.run.smithi175.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 691 kB/s | 23 kB 00:00 2024-04-15T15:12:43.025 INFO:teuthology.orchestra.run.smithi175.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 581 kB/s | 19 kB 00:00 2024-04-15T15:12:43.042 INFO:teuthology.orchestra.run.smithi175.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 588 kB/s | 20 kB 00:00 2024-04-15T15:12:43.076 INFO:teuthology.orchestra.run.smithi175.stdout:(131/174): python3-kubernetes-26.1.0-2.el9.noar 20 MB/s | 1.0 MB 00:00 2024-04-15T15:12:43.093 INFO:teuthology.orchestra.run.smithi175.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 921 kB/s | 46 kB 00:00 2024-04-15T15:12:43.118 INFO:teuthology.orchestra.run.smithi175.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.8 MB/s | 79 kB 00:00 2024-04-15T15:12:43.136 INFO:teuthology.orchestra.run.smithi175.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 1.3 MB/s | 58 kB 00:00 2024-04-15T15:12:43.160 INFO:teuthology.orchestra.run.smithi175.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 6.4 MB/s | 272 kB 00:00 2024-04-15T15:12:43.177 INFO:teuthology.orchestra.run.smithi175.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 393 kB/s | 16 kB 00:00 2024-04-15T15:12:43.211 INFO:teuthology.orchestra.run.smithi175.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-04-15T15:12:43.236 INFO:teuthology.orchestra.run.smithi175.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 481 kB/s | 28 kB 00:00 2024-04-15T15:12:43.254 INFO:teuthology.orchestra.run.smithi175.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 732 kB/s | 31 kB 00:00 2024-04-15T15:12:43.278 INFO:teuthology.orchestra.run.smithi175.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 4.4 MB/s | 188 kB 00:00 2024-04-15T15:12:43.295 INFO:teuthology.orchestra.run.smithi175.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 1.4 MB/s | 59 kB 00:00 2024-04-15T15:12:43.312 INFO:teuthology.orchestra.run.smithi175.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 1.0 MB/s | 36 kB 00:00 2024-04-15T15:12:43.329 INFO:teuthology.orchestra.run.smithi175.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 1.5 MB/s | 51 kB 00:00 2024-04-15T15:12:43.355 INFO:teuthology.orchestra.run.smithi175.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 5.4 MB/s | 230 kB 00:00 2024-04-15T15:12:43.380 INFO:teuthology.orchestra.run.smithi175.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.8 MB/s | 90 kB 00:00 2024-04-15T15:12:43.405 INFO:teuthology.orchestra.run.smithi175.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 8.3 MB/s | 427 kB 00:00 2024-04-15T15:12:43.422 INFO:teuthology.orchestra.run.smithi175.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 477 kB/s | 20 kB 00:00 2024-04-15T15:12:43.447 INFO:teuthology.orchestra.run.smithi175.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 4.5 MB/s | 192 kB 00:00 2024-04-15T15:12:43.498 INFO:teuthology.orchestra.run.smithi175.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 21 MB/s | 1.6 MB 00:00 2024-04-15T15:12:43.940 INFO:teuthology.orchestra.run.smithi175.stdout:(150/174): librbd1-19.0.0-2861.g9bf1c4ba.el8.x8 7.1 MB/s | 3.1 MB 00:00 2024-04-15T15:12:43.960 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-04-15T15:12:43.999 INFO:teuthology.orchestra.run.smithi175.stdout:(151/174): librados2-19.0.0-2861.g9bf1c4ba.el8. 6.2 MB/s | 3.4 MB 00:00 2024-04-15T15:12:44.057 INFO:teuthology.orchestra.run.smithi175.stdout:(152/174): NetworkManager-1.47.4-1.el9.x86_64.r 20 MB/s | 2.3 MB 00:00 2024-04-15T15:12:44.099 INFO:teuthology.orchestra.run.smithi175.stdout:(153/174): NetworkManager-libnm-1.47.4-1.el9.x8 19 MB/s | 1.8 MB 00:00 2024-04-15T15:12:44.124 INFO:teuthology.orchestra.run.smithi175.stdout:(154/174): NetworkManager-team-1.47.4-1.el9.x86 633 kB/s | 41 kB 00:00 2024-04-15T15:12:44.150 INFO:teuthology.orchestra.run.smithi175.stdout:(155/174): NetworkManager-tui-1.47.4-1.el9.x86_ 4.8 MB/s | 248 kB 00:00 2024-04-15T15:12:44.183 INFO:teuthology.orchestra.run.smithi175.stdout:(156/174): xfsdump-3.1.12-4.el9.x86_64.rpm 5.4 MB/s | 321 kB 00:00 2024-04-15T15:12:44.233 INFO:teuthology.orchestra.run.smithi175.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 859 kB/s | 71 kB 00:00 2024-04-15T15:12:44.258 INFO:teuthology.orchestra.run.smithi175.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 485 kB/s | 36 kB 00:00 2024-04-15T15:12:44.317 INFO:teuthology.orchestra.run.smithi175.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.3 MB/s | 192 kB 00:00 2024-04-15T15:12:44.325 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-04-15T15:12:44.351 INFO:teuthology.orchestra.run.smithi175.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.7 MB/s | 256 kB 00:00 2024-04-15T15:12:44.393 INFO:teuthology.orchestra.run.smithi175.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 306 kB/s | 23 kB 00:00 2024-04-15T15:12:44.435 INFO:teuthology.orchestra.run.smithi175.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 108 kB 00:00 2024-04-15T15:12:44.477 INFO:teuthology.orchestra.run.smithi175.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 155 kB 00:00 2024-04-15T15:12:44.510 INFO:teuthology.orchestra.run.smithi175.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 136 kB 00:00 2024-04-15T15:12:44.527 INFO:teuthology.orchestra.run.smithi175.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.2 MB/s | 64 kB 00:00 2024-04-15T15:12:44.539 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-04-15T15:12:44.561 INFO:teuthology.orchestra.run.smithi175.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 574 kB/s | 29 kB 00:00 2024-04-15T15:12:44.567 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-04-15T15:12:44.596 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-04-15T15:12:44.645 INFO:teuthology.orchestra.run.smithi175.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 10 MB/s | 1.2 MB 00:00 2024-04-15T15:12:44.753 INFO:teuthology.orchestra.run.smithi175.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 11 MB/s | 2.2 MB 00:00 2024-04-15T15:12:44.853 INFO:teuthology.orchestra.run.smithi175.stdout:(169/174): util-linux-2.37.4-18.el9.x86_64.rpm 11 MB/s | 2.3 MB 00:00 2024-04-15T15:12:44.896 INFO:teuthology.orchestra.run.smithi175.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 3.2 MB/s | 465 kB 00:00 2024-04-15T15:12:44.912 INFO:teuthology.orchestra.run.smithi175.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 532 kB/s | 31 kB 00:00 2024-04-15T15:12:44.916 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-04-15T15:12:44.943 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : librbd1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 124/198 2024-04-15T15:12:45.013 INFO:teuthology.orchestra.run.smithi175.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 9.2 MB/s | 1.1 MB 00:00 2024-04-15T15:12:45.074 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librbd1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 124/198 2024-04-15T15:12:45.101 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 125/198 2024-04-15T15:12:45.152 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-04-15T15:12:45.619 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-04-15T15:12:45.651 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librgw2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 128/198 2024-04-15T15:12:45.696 INFO:teuthology.orchestra.run.smithi175.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 95 kB/s | 74 kB 00:00 2024-04-15T15:12:45.723 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librgw2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 128/198 2024-04-15T15:12:45.754 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 129/198 2024-04-15T15:12:46.445 INFO:teuthology.orchestra.run.smithi175.stdout:(174/174): openssl-devel-3.0.7-27.el9.x86_64.rp 642 kB/s | 4.1 MB 00:06 2024-04-15T15:12:46.452 INFO:teuthology.orchestra.run.smithi175.stdout:-------------------------------------------------------------------------------- 2024-04-15T15:12:46.452 INFO:teuthology.orchestra.run.smithi175.stdout:Total 14 MB/s | 228 MB 00:16 2024-04-15T15:12:46.846 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction check 2024-04-15T15:12:47.119 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction check succeeded. 2024-04-15T15:12:47.119 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction test 2024-04-15T15:12:48.022 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 130/198 2024-04-15T15:12:48.083 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 130/198 2024-04-15T15:12:48.642 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 130/198 2024-04-15T15:12:48.681 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-base-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 131/198 2024-04-15T15:12:48.765 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-base-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 131/198 2024-04-15T15:12:48.766 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-15T15:12:48.766 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-15T15:12:48.766 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:12:48.810 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-selinux-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 132/198 2024-04-15T15:12:49.152 INFO:teuthology.orchestra.run.smithi175.stdout:Transaction test succeeded. 2024-04-15T15:12:49.152 INFO:teuthology.orchestra.run.smithi175.stdout:Running transaction 2024-04-15T15:12:56.070 INFO:teuthology.orchestra.run.smithi175.stdout: Preparing : 1/1 2024-04-15T15:12:56.201 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-04-15T15:12:56.235 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-04-15T15:12:56.273 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-15T15:12:56.284 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-15T15:12:56.327 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-04-15T15:12:56.385 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-04-15T15:12:56.418 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-04-15T15:12:56.444 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-04-15T15:12:56.503 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-04-15T15:12:56.833 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-04-15T15:12:56.887 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : librados2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 10/198 2024-04-15T15:12:56.993 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librados2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 10/198 2024-04-15T15:12:57.038 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libcephfs2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 11/198 2024-04-15T15:12:57.130 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: libcephfs2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 11/198 2024-04-15T15:12:57.159 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-rados-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 12/198 2024-04-15T15:12:57.203 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-04-15T15:12:57.384 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-04-15T15:12:57.434 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-04-15T15:12:57.461 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libcephsqlite-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 16/198 2024-04-15T15:12:57.573 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 16/198 2024-04-15T15:12:57.595 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libradosstriper1-2:19.0.0-2861.g9bf1c4ba.el8.x86 17/198 2024-04-15T15:12:57.661 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2861.g9bf1c4ba.el8.x86 17/198 2024-04-15T15:12:57.699 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-04-15T15:12:57.722 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-15T15:12:57.738 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-15T15:12:57.915 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-15T15:12:57.955 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-15T15:12:57.993 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-04-15T15:12:58.204 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-04-15T15:12:58.337 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-15T15:12:58.397 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-15T15:12:58.486 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-15T15:12:58.552 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-15T15:12:58.759 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-15T15:12:58.866 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-04-15T15:12:58.960 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-04-15T15:12:58.990 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-04-15T15:12:59.017 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-04-15T15:12:59.052 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-04-15T15:12:59.127 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-04-15T15:12:59.180 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-04-15T15:12:59.218 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-04-15T15:12:59.282 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-04-15T15:12:59.318 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-04-15T15:12:59.361 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-04-15T15:12:59.403 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-04-15T15:12:59.430 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-04-15T15:12:59.500 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-04-15T15:12:59.535 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-ceph-common-2:19.0.0-2861.g9bf1c4ba.el8. 39/198 2024-04-15T15:12:59.735 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-ceph-argparse-2:19.0.0-2861.g9bf1c4ba.el 40/198 2024-04-15T15:12:59.916 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cephfs-2:19.0.0-2861.g9bf1c4ba.el8.x86_6 41/198 2024-04-15T15:13:00.154 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-04-15T15:13:00.239 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-04-15T15:13:00.313 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-04-15T15:13:00.446 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-04-15T15:13:00.499 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-04-15T15:13:01.032 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-04-15T15:13:01.110 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-04-15T15:13:01.223 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-04-15T15:13:01.281 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-15T15:13:01.291 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-15T15:13:01.329 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-04-15T15:13:01.380 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-04-15T15:13:01.416 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : librados-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_6 53/198 2024-04-15T15:13:01.447 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-04-15T15:13:01.729 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-04-15T15:13:01.923 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-04-15T15:13:02.002 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-04-15T15:13:02.046 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-04-15T15:13:02.202 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-04-15T15:13:02.248 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-04-15T15:13:02.303 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-04-15T15:13:02.375 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-04-15T15:13:02.415 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-04-15T15:13:02.451 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-04-15T15:13:02.497 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-04-15T15:13:02.534 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-04-15T15:13:02.577 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-04-15T15:13:02.619 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-04-15T15:13:02.751 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-04-15T15:13:02.791 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-04-15T15:13:02.821 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-04-15T15:13:02.863 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-04-15T15:13:02.956 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-04-15T15:13:02.988 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-04-15T15:13:03.028 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-04-15T15:13:03.054 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-04-15T15:13:03.078 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-04-15T15:13:03.155 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-04-15T15:13:03.278 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-04-15T15:13:03.385 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-04-15T15:13:03.436 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-04-15T15:13:03.484 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-04-15T15:13:03.540 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-04-15T15:13:03.722 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-04-15T15:13:03.773 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-04-15T15:13:03.900 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-04-15T15:13:03.943 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-04-15T15:13:03.979 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-04-15T15:13:04.009 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-15T15:13:04.067 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-04-15T15:13:04.097 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-04-15T15:13:04.144 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-04-15T15:13:04.176 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-04-15T15:13:04.206 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-04-15T15:13:04.282 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-04-15T15:13:04.316 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-04-15T15:13:04.318 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 132/198 2024-04-15T15:13:04.318 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /sys 2024-04-15T15:13:04.318 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /proc 2024-04-15T15:13:04.318 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /mnt 2024-04-15T15:13:04.318 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /var/tmp 2024-04-15T15:13:04.319 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /home 2024-04-15T15:13:04.319 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /root 2024-04-15T15:13:04.319 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /tmp 2024-04-15T15:13:04.319 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:04.349 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-04-15T15:13:04.391 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-04-15T15:13:04.425 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-04-15T15:13:04.489 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-volume-2:19.0.0-2861.g9bf1c4ba.el8.noarch 133/198 2024-04-15T15:13:04.505 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-volume-2:19.0.0-2861.g9bf1c4ba.el8.noarch 133/198 2024-04-15T15:13:04.506 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:04.506 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-15T15:13:04.506 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:04.513 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-04-15T15:13:04.603 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-04-15T15:13:04.647 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-04-15T15:13:04.679 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-04-15T15:13:04.726 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-04-15T15:13:04.822 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-04-15T15:13:04.871 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-04-15T15:13:05.594 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-04-15T15:13:05.636 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-04-15T15:13:05.670 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-04-15T15:13:05.717 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-04-15T15:13:05.796 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-04-15T15:13:05.837 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-04-15T15:13:05.999 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-osd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 134/198 2024-04-15T15:13:06.017 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-04-15T15:13:06.027 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-osd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 134/198 2024-04-15T15:13:06.027 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:06.027 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-15T15:13:06.027 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-15T15:13:06.027 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-15T15:13:06.027 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:06.043 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-04-15T15:13:06.129 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-04-15T15:13:06.259 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mds-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 135/198 2024-04-15T15:13:06.282 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mds-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 135/198 2024-04-15T15:13:06.282 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:06.282 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-15T15:13:06.282 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-15T15:13:06.283 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-15T15:13:06.283 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:06.727 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mon-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 136/198 2024-04-15T15:13:06.750 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mon-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 136/198 2024-04-15T15:13:06.751 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:06.751 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-15T15:13:06.751 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-15T15:13:06.751 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-15T15:13:06.751 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:06.769 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-04-15T15:13:06.846 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-04-15T15:13:06.881 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-04-15T15:13:07.271 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 138/198 2024-04-15T15:13:07.300 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 138/198 2024-04-15T15:13:07.333 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2861.g9bf1c4ba.e 139/198 2024-04-15T15:13:07.403 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2861.g9bf1c4ba. 140/198 2024-04-15T15:13:07.435 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noa 141/198 2024-04-15T15:13:08.093 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-04-15T15:13:08.166 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-04-15T15:13:08.197 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-04-15T15:13:08.229 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-04-15T15:13:08.256 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-04-15T15:13:08.477 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noa 141/198 2024-04-15T15:13:08.546 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2861.g9bf1c4ba.el8.n 142/198 2024-04-15T15:13:08.571 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-04-15T15:13:08.602 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : librbd1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 124/198 2024-04-15T15:13:08.693 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librbd1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 124/198 2024-04-15T15:13:08.724 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-rbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 125/198 2024-04-15T15:13:08.761 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-04-15T15:13:09.228 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-04-15T15:13:09.272 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : librgw2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 128/198 2024-04-15T15:13:09.352 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librgw2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 128/198 2024-04-15T15:13:09.382 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : python3-rgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 129/198 2024-04-15T15:13:09.516 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2861.g9bf1c4ba.el8.n 142/198 2024-04-15T15:13:09.564 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2861.g9bf 143/198 2024-04-15T15:13:09.666 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2861.g9bf 143/198 2024-04-15T15:13:09.829 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2861.g9bf1c4ba.el 144/198 2024-04-15T15:13:09.861 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 145/198 2024-04-15T15:13:09.885 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 145/198 2024-04-15T15:13:09.885 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:09.885 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-15T15:13:09.885 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-15T15:13:09.885 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-15T15:13:09.885 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:09.929 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-rook-2:19.0.0-2861.g9bf1c4ba.el8.noarch 146/198 2024-04-15T15:13:09.941 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2861.g9bf1c4ba.el8.noarch 146/198 2024-04-15T15:13:09.996 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 147/198 2024-04-15T15:13:10.882 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-04-15T15:13:10.916 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-radosgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 149/198 2024-04-15T15:13:10.940 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 149/198 2024-04-15T15:13:10.940 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:10.940 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-15T15:13:10.940 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-15T15:13:10.940 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-15T15:13:10.940 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:10.984 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2861.g9bf1c 150/198 2024-04-15T15:13:11.007 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2861.g9bf1c 150/198 2024-04-15T15:13:11.007 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:11.007 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-15T15:13:11.007 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:11.081 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : cephfs-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 151/198 2024-04-15T15:13:11.104 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 151/198 2024-04-15T15:13:11.104 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:11.104 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-15T15:13:11.104 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-15T15:13:11.105 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-15T15:13:11.105 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:11.405 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : rbd-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 152/198 2024-04-15T15:13:11.428 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 152/198 2024-04-15T15:13:11.428 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:11.428 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-15T15:13:11.428 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-15T15:13:11.429 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-15T15:13:11.429 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:11.666 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 130/198 2024-04-15T15:13:11.705 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 130/198 2024-04-15T15:13:12.251 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 130/198 2024-04-15T15:13:12.283 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-base-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 131/198 2024-04-15T15:13:12.357 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-base-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 131/198 2024-04-15T15:13:12.357 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-15T15:13:12.357 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-15T15:13:12.357 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:12.392 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-selinux-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 132/198 2024-04-15T15:13:16.008 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-test-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 153/198 2024-04-15T15:13:16.058 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : rbd-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 154/198 2024-04-15T15:13:16.089 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : rbd-nbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 155/198 2024-04-15T15:13:16.125 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-04-15T15:13:16.307 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : cephfs-shell-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 157/198 2024-04-15T15:13:16.433 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-04-15T15:13:16.480 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 159/198 2024-04-15T15:13:16.545 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-04-15T15:13:16.591 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-04-15T15:13:16.627 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-04-15T15:13:16.660 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libcephfs-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_ 163/198 2024-04-15T15:13:16.694 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-04-15T15:13:16.733 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : NetworkManager-team-1:1.47.4-1.el9.x86_64 165/198 2024-04-15T15:13:16.818 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : NetworkManager-tui-1:1.47.4-1.el9.x86_64 166/198 2024-04-15T15:13:16.870 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-04-15T15:13:16.903 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-04-15T15:13:17.481 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-15T15:13:17.525 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-15T15:13:17.546 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-15T15:13:18.370 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : cephfs-top-2:19.0.0-2861.g9bf1c4ba.el8.noarch 170/198 2024-04-15T15:13:18.476 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-04-15T15:13:18.517 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-04-15T15:13:18.608 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-04-15T15:13:18.641 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-04-15T15:13:18.660 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-04-15T15:13:18.686 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-04-15T15:13:18.713 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-04-15T15:13:18.713 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-15T15:13:18.827 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-15T15:13:18.857 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-04-15T15:13:18.857 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-15T15:13:18.942 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-15T15:13:18.980 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-04-15T15:13:19.013 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-04-15T15:13:19.038 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-04-15T15:13:19.064 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-04-15T15:13:19.115 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-04-15T15:13:19.173 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-04-15T15:13:19.208 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-04-15T15:13:19.233 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-04-15T15:13:19.259 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-04-15T15:13:19.293 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-04-15T15:13:19.318 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-04-15T15:13:19.325 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-15T15:13:19.325 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-15T15:13:19.514 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-15T15:13:19.514 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-15T15:13:19.616 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-15T15:13:19.655 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-04-15T15:13:19.683 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-04-15T15:13:19.709 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-04-15T15:13:19.745 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-04-15T15:13:19.818 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-15T15:13:24.509 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 132/198 2024-04-15T15:13:24.510 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /sys 2024-04-15T15:13:24.510 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /proc 2024-04-15T15:13:24.510 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /mnt 2024-04-15T15:13:24.510 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /var/tmp 2024-04-15T15:13:24.510 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /home 2024-04-15T15:13:24.510 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /root 2024-04-15T15:13:24.510 INFO:teuthology.orchestra.run.smithi175.stdout:skipping the directory /tmp 2024-04-15T15:13:24.510 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:24.669 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-volume-2:19.0.0-2861.g9bf1c4ba.el8.noarch 133/198 2024-04-15T15:13:24.685 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-volume-2:19.0.0-2861.g9bf1c4ba.el8.noarch 133/198 2024-04-15T15:13:24.685 INFO:teuthology.orchestra.run.smithi175.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:24.685 INFO:teuthology.orchestra.run.smithi175.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-15T15:13:24.685 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:26.145 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-osd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 134/198 2024-04-15T15:13:26.176 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-osd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 134/198 2024-04-15T15:13:26.176 INFO:teuthology.orchestra.run.smithi175.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:26.176 INFO:teuthology.orchestra.run.smithi175.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-15T15:13:26.176 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-15T15:13:26.176 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-15T15:13:26.176 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:26.429 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mds-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 135/198 2024-04-15T15:13:26.459 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mds-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 135/198 2024-04-15T15:13:26.459 INFO:teuthology.orchestra.run.smithi175.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:26.459 INFO:teuthology.orchestra.run.smithi175.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-15T15:13:26.459 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-15T15:13:26.459 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-15T15:13:26.459 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:26.915 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mon-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 136/198 2024-04-15T15:13:26.939 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mon-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 136/198 2024-04-15T15:13:26.939 INFO:teuthology.orchestra.run.smithi175.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:26.939 INFO:teuthology.orchestra.run.smithi175.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-15T15:13:26.939 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-15T15:13:26.939 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-15T15:13:26.939 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:27.033 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-04-15T15:13:27.504 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 138/198 2024-04-15T15:13:27.531 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 138/198 2024-04-15T15:13:27.555 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2861.g9bf1c4ba.e 139/198 2024-04-15T15:13:27.624 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2861.g9bf1c4ba. 140/198 2024-04-15T15:13:27.654 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noa 141/198 2024-04-15T15:13:28.690 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noa 141/198 2024-04-15T15:13:28.747 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2861.g9bf1c4ba.el8.n 142/198 2024-04-15T15:13:29.603 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-15T15:13:29.604 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 1/198 2024-04-15T15:13:29.604 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-base-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2/198 2024-04-15T15:13:29.604 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 3/198 2024-04-15T15:13:29.604 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 4/198 2024-04-15T15:13:29.604 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2861.g9bf1c 5/198 2024-04-15T15:13:29.604 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mds-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 6/198 2024-04-15T15:13:29.604 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 7/198 2024-04-15T15:13:29.604 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mon-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 8/198 2024-04-15T15:13:29.604 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-osd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 9/198 2024-04-15T15:13:29.604 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-radosgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 10/198 2024-04-15T15:13:29.604 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-selinux-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 11/198 2024-04-15T15:13:29.604 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-test-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 12/198 2024-04-15T15:13:29.605 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : cephfs-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 13/198 2024-04-15T15:13:29.605 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : cephfs-shell-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 14/198 2024-04-15T15:13:29.605 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libcephfs-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_ 15/198 2024-04-15T15:13:29.605 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libcephfs2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 16/198 2024-04-15T15:13:29.605 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libcephsqlite-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 17/198 2024-04-15T15:13:29.605 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librados-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_6 18/198 2024-04-15T15:13:29.605 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libradosstriper1-2:19.0.0-2861.g9bf1c4ba.el8.x86 19/198 2024-04-15T15:13:29.605 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librgw2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 20/198 2024-04-15T15:13:29.605 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2861.g9bf1c4ba.el 21/198 2024-04-15T15:13:29.605 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-ceph-common-2:19.0.0-2861.g9bf1c4ba.el8. 22/198 2024-04-15T15:13:29.605 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cephfs-2:19.0.0-2861.g9bf1c4ba.el8.x86_6 23/198 2024-04-15T15:13:29.605 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rados-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 24/198 2024-04-15T15:13:29.605 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 25/198 2024-04-15T15:13:29.606 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 26/198 2024-04-15T15:13:29.606 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : rbd-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 27/198 2024-04-15T15:13:29.606 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : rbd-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 28/198 2024-04-15T15:13:29.606 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : rbd-nbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 29/198 2024-04-15T15:13:29.606 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2861.g9bf1c4ba. 30/198 2024-04-15T15:13:29.606 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noa 31/198 2024-04-15T15:13:29.606 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2861.g9bf1c4ba.el8.n 32/198 2024-04-15T15:13:29.606 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2861.g9bf 33/198 2024-04-15T15:13:29.606 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2861.g9bf1c4ba.el 34/198 2024-04-15T15:13:29.606 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2861.g9bf1c4ba.el8.noarch 35/198 2024-04-15T15:13:29.606 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2861.g9bf1c4ba.e 36/198 2024-04-15T15:13:29.607 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-volume-2:19.0.0-2861.g9bf1c4ba.el8.noarch 37/198 2024-04-15T15:13:29.607 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 38/198 2024-04-15T15:13:29.607 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : cephfs-top-2:19.0.0-2861.g9bf1c4ba.el8.noarch 39/198 2024-04-15T15:13:29.607 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-04-15T15:13:29.607 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-04-15T15:13:29.607 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-04-15T15:13:29.607 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-04-15T15:13:29.607 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-04-15T15:13:29.607 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-04-15T15:13:29.607 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-04-15T15:13:29.607 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-04-15T15:13:29.607 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-04-15T15:13:29.607 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-04-15T15:13:29.607 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-04-15T15:13:29.608 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-04-15T15:13:29.608 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-04-15T15:13:29.608 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-04-15T15:13:29.608 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-04-15T15:13:29.608 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-04-15T15:13:29.608 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-04-15T15:13:29.608 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-04-15T15:13:29.608 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : postgresql-13.14-1.el9.x86_64 58/198 2024-04-15T15:13:29.608 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 59/198 2024-04-15T15:13:29.608 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 60/198 2024-04-15T15:13:29.608 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 61/198 2024-04-15T15:13:29.609 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-04-15T15:13:29.609 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-04-15T15:13:29.609 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-04-15T15:13:29.609 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-04-15T15:13:29.609 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-04-15T15:13:29.609 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-04-15T15:13:29.609 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-04-15T15:13:29.609 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-04-15T15:13:29.609 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-04-15T15:13:29.609 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-04-15T15:13:29.609 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-04-15T15:13:29.609 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-04-15T15:13:29.610 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-04-15T15:13:29.610 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-04-15T15:13:29.610 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-04-15T15:13:29.610 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-04-15T15:13:29.610 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-04-15T15:13:29.610 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-04-15T15:13:29.610 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-04-15T15:13:29.610 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-04-15T15:13:29.610 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 82/198 2024-04-15T15:13:29.610 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 83/198 2024-04-15T15:13:29.610 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 84/198 2024-04-15T15:13:29.610 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 85/198 2024-04-15T15:13:29.610 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 86/198 2024-04-15T15:13:29.611 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 87/198 2024-04-15T15:13:29.611 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 88/198 2024-04-15T15:13:29.611 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-15T15:13:29.611 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 90/198 2024-04-15T15:13:29.611 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 91/198 2024-04-15T15:13:29.613 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-ply-3.11-14.el9.noarch 92/198 2024-04-15T15:13:29.613 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 93/198 2024-04-15T15:13:29.613 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/198 2024-04-15T15:13:29.613 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : jq-1.6-16.el9.x86_64 95/198 2024-04-15T15:13:29.613 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 96/198 2024-04-15T15:13:29.613 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 97/198 2024-04-15T15:13:29.613 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 98/198 2024-04-15T15:13:29.613 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 99/198 2024-04-15T15:13:29.614 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 100/198 2024-04-15T15:13:29.614 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 101/198 2024-04-15T15:13:29.614 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 102/198 2024-04-15T15:13:29.614 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 103/198 2024-04-15T15:13:29.614 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-04-15T15:13:29.614 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-04-15T15:13:29.614 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-04-15T15:13:29.614 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-04-15T15:13:29.614 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-04-15T15:13:29.614 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-04-15T15:13:29.614 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-04-15T15:13:29.614 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-04-15T15:13:29.615 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-04-15T15:13:29.615 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-04-15T15:13:29.615 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-04-15T15:13:29.615 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-15T15:13:29.615 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-04-15T15:13:29.615 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-04-15T15:13:29.615 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-04-15T15:13:29.615 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-04-15T15:13:29.615 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-04-15T15:13:29.615 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-04-15T15:13:29.615 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-04-15T15:13:29.615 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-04-15T15:13:29.615 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-04-15T15:13:29.616 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-04-15T15:13:29.616 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-04-15T15:13:29.616 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-04-15T15:13:29.616 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-04-15T15:13:29.616 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-04-15T15:13:29.616 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-04-15T15:13:29.616 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-04-15T15:13:29.616 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-04-15T15:13:29.616 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-04-15T15:13:29.616 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-04-15T15:13:29.616 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-04-15T15:13:29.616 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-04-15T15:13:29.616 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-04-15T15:13:29.616 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-04-15T15:13:29.616 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-04-15T15:13:29.617 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-04-15T15:13:29.617 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-04-15T15:13:29.617 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-04-15T15:13:29.617 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-04-15T15:13:29.617 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-04-15T15:13:29.617 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-04-15T15:13:29.617 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-04-15T15:13:29.617 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-04-15T15:13:29.617 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-04-15T15:13:29.617 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-04-15T15:13:29.617 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-04-15T15:13:29.617 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librados2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 151/198 2024-04-15T15:13:29.617 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-04-15T15:13:29.618 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librbd1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 153/198 2024-04-15T15:13:29.618 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-04-15T15:13:29.618 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 155/198 2024-04-15T15:13:29.618 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-04-15T15:13:29.618 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 157/198 2024-04-15T15:13:29.618 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-04-15T15:13:29.618 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 159/198 2024-04-15T15:13:29.618 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-04-15T15:13:29.618 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/198 2024-04-15T15:13:29.618 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-04-15T15:13:29.618 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 163/198 2024-04-15T15:13:29.618 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 164/198 2024-04-15T15:13:29.618 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-04-15T15:13:29.618 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-04-15T15:13:29.619 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-04-15T15:13:29.619 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-04-15T15:13:29.619 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-04-15T15:13:29.619 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-04-15T15:13:29.619 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-04-15T15:13:29.619 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-04-15T15:13:29.619 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-04-15T15:13:29.619 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-04-15T15:13:29.619 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-04-15T15:13:29.619 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-04-15T15:13:29.619 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-04-15T15:13:29.619 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-04-15T15:13:29.619 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-04-15T15:13:29.619 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-04-15T15:13:29.619 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-04-15T15:13:29.620 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-04-15T15:13:29.620 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-04-15T15:13:29.620 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-04-15T15:13:29.620 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-04-15T15:13:29.620 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-04-15T15:13:29.620 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-04-15T15:13:29.620 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-04-15T15:13:29.620 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-04-15T15:13:29.620 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-04-15T15:13:29.620 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-04-15T15:13:29.620 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-04-15T15:13:29.620 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-04-15T15:13:29.620 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-04-15T15:13:29.620 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-04-15T15:13:29.621 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-04-15T15:13:29.621 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-04-15T15:13:29.713 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2861.g9bf1c4ba.el8.n 142/198 2024-04-15T15:13:29.739 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2861.g9bf 143/198 2024-04-15T15:13:29.840 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2861.g9bf 143/198 2024-04-15T15:13:30.002 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2861.g9bf1c4ba.el 144/198 2024-04-15T15:13:30.033 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 145/198 2024-04-15T15:13:30.057 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 145/198 2024-04-15T15:13:30.057 INFO:teuthology.orchestra.run.smithi175.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:30.057 INFO:teuthology.orchestra.run.smithi175.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-15T15:13:30.057 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-15T15:13:30.057 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-15T15:13:30.057 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:30.100 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-mgr-rook-2:19.0.0-2861.g9bf1c4ba.el8.noarch 146/198 2024-04-15T15:13:30.112 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2861.g9bf1c4ba.el8.noarch 146/198 2024-04-15T15:13:30.159 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 147/198 2024-04-15T15:13:31.038 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-04-15T15:13:31.069 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-radosgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 149/198 2024-04-15T15:13:31.093 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 149/198 2024-04-15T15:13:31.093 INFO:teuthology.orchestra.run.smithi175.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:31.093 INFO:teuthology.orchestra.run.smithi175.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-15T15:13:31.093 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-15T15:13:31.093 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-15T15:13:31.093 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:31.136 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2861.g9bf1c 150/198 2024-04-15T15:13:31.159 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2861.g9bf1c 150/198 2024-04-15T15:13:31.159 INFO:teuthology.orchestra.run.smithi175.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:31.159 INFO:teuthology.orchestra.run.smithi175.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-15T15:13:31.159 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:31.212 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : cephfs-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 151/198 2024-04-15T15:13:31.235 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 151/198 2024-04-15T15:13:31.235 INFO:teuthology.orchestra.run.smithi175.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:31.235 INFO:teuthology.orchestra.run.smithi175.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-15T15:13:31.235 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-15T15:13:31.235 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-15T15:13:31.235 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:31.546 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : rbd-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 152/198 2024-04-15T15:13:31.569 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 152/198 2024-04-15T15:13:31.570 INFO:teuthology.orchestra.run.smithi175.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:31.570 INFO:teuthology.orchestra.run.smithi175.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-15T15:13:31.570 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-15T15:13:31.570 INFO:teuthology.orchestra.run.smithi175.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-15T15:13:31.570 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:32.433 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-04-15T15:13:32.433 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:32.433 INFO:teuthology.orchestra.run.smithi072.stdout:Upgraded: 2024-04-15T15:13:32.433 INFO:teuthology.orchestra.run.smithi072.stdout: NetworkManager-1:1.47.4-1.el9.x86_64 2024-04-15T15:13:32.434 INFO:teuthology.orchestra.run.smithi072.stdout: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 2024-04-15T15:13:32.434 INFO:teuthology.orchestra.run.smithi072.stdout: NetworkManager-team-1:1.47.4-1.el9.x86_64 2024-04-15T15:13:32.434 INFO:teuthology.orchestra.run.smithi072.stdout: NetworkManager-tui-1:1.47.4-1.el9.x86_64 2024-04-15T15:13:32.434 INFO:teuthology.orchestra.run.smithi072.stdout: acl-2.3.1-4.el9.x86_64 2024-04-15T15:13:32.434 INFO:teuthology.orchestra.run.smithi072.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-04-15T15:13:32.434 INFO:teuthology.orchestra.run.smithi072.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-04-15T15:13:32.434 INFO:teuthology.orchestra.run.smithi072.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-04-15T15:13:32.434 INFO:teuthology.orchestra.run.smithi072.stdout: libacl-2.3.1-4.el9.x86_64 2024-04-15T15:13:32.434 INFO:teuthology.orchestra.run.smithi072.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-04-15T15:13:32.434 INFO:teuthology.orchestra.run.smithi072.stdout: libblkid-2.37.4-18.el9.x86_64 2024-04-15T15:13:32.434 INFO:teuthology.orchestra.run.smithi072.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-04-15T15:13:32.434 INFO:teuthology.orchestra.run.smithi072.stdout: libmount-2.37.4-18.el9.x86_64 2024-04-15T15:13:32.435 INFO:teuthology.orchestra.run.smithi072.stdout: librados2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:32.435 INFO:teuthology.orchestra.run.smithi072.stdout: librbd1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:32.435 INFO:teuthology.orchestra.run.smithi072.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-04-15T15:13:32.435 INFO:teuthology.orchestra.run.smithi072.stdout: libuuid-2.37.4-18.el9.x86_64 2024-04-15T15:13:32.435 INFO:teuthology.orchestra.run.smithi072.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-04-15T15:13:32.435 INFO:teuthology.orchestra.run.smithi072.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-04-15T15:13:32.435 INFO:teuthology.orchestra.run.smithi072.stdout: util-linux-2.37.4-18.el9.x86_64 2024-04-15T15:13:32.436 INFO:teuthology.orchestra.run.smithi072.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-04-15T15:13:32.436 INFO:teuthology.orchestra.run.smithi072.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-04-15T15:13:32.436 INFO:teuthology.orchestra.run.smithi072.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-04-15T15:13:32.436 INFO:teuthology.orchestra.run.smithi072.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-04-15T15:13:32.436 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-04-15T15:13:32.436 INFO:teuthology.orchestra.run.smithi072.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-15T15:13:32.437 INFO:teuthology.orchestra.run.smithi072.stdout: automake-1.16.2-8.el9.noarch 2024-04-15T15:13:32.437 INFO:teuthology.orchestra.run.smithi072.stdout: bison-3.7.4-5.el9.x86_64 2024-04-15T15:13:32.437 INFO:teuthology.orchestra.run.smithi072.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-15T15:13:32.437 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:32.437 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-base-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:32.437 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:32.437 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:32.437 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-grafana-dashboards-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:13:32.438 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-immutable-object-cache-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:32.438 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mds-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:32.438 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:32.438 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:13:32.438 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-dashboard-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:13:32.438 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:13:32.438 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-modules-core-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:13:32.439 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-rook-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:13:32.439 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mon-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:32.439 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-osd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:32.439 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-prometheus-alerts-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:13:32.439 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-radosgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:32.439 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-selinux-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:32.440 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-test-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:32.440 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-volume-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:13:32.440 INFO:teuthology.orchestra.run.smithi072.stdout: cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:13:32.440 INFO:teuthology.orchestra.run.smithi072.stdout: cephfs-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:32.440 INFO:teuthology.orchestra.run.smithi072.stdout: cephfs-shell-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:32.440 INFO:teuthology.orchestra.run.smithi072.stdout: cephfs-top-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:13:32.441 INFO:teuthology.orchestra.run.smithi072.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-15T15:13:32.441 INFO:teuthology.orchestra.run.smithi072.stdout: flex-2.6.4-9.el9.x86_64 2024-04-15T15:13:32.441 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-15T15:13:32.441 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-15T15:13:32.441 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-15T15:13:32.441 INFO:teuthology.orchestra.run.smithi072.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-15T15:13:32.441 INFO:teuthology.orchestra.run.smithi072.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-15T15:13:32.441 INFO:teuthology.orchestra.run.smithi072.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-15T15:13:32.441 INFO:teuthology.orchestra.run.smithi072.stdout: jq-1.6-16.el9.x86_64 2024-04-15T15:13:32.441 INFO:teuthology.orchestra.run.smithi072.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-15T15:13:32.441 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-15T15:13:32.442 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-15T15:13:32.442 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:32.442 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:32.442 INFO:teuthology.orchestra.run.smithi072.stdout: libcephsqlite-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:32.442 INFO:teuthology.orchestra.run.smithi072.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-15T15:13:32.442 INFO:teuthology.orchestra.run.smithi072.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-15T15:13:32.442 INFO:teuthology.orchestra.run.smithi072.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-15T15:13:32.443 INFO:teuthology.orchestra.run.smithi072.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-15T15:13:32.443 INFO:teuthology.orchestra.run.smithi072.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-15T15:13:32.443 INFO:teuthology.orchestra.run.smithi072.stdout: librados-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:32.443 INFO:teuthology.orchestra.run.smithi072.stdout: libradosstriper1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:32.443 INFO:teuthology.orchestra.run.smithi072.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-15T15:13:32.444 INFO:teuthology.orchestra.run.smithi072.stdout: librgw2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:32.444 INFO:teuthology.orchestra.run.smithi072.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-15T15:13:32.444 INFO:teuthology.orchestra.run.smithi072.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-15T15:13:32.444 INFO:teuthology.orchestra.run.smithi072.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-15T15:13:32.444 INFO:teuthology.orchestra.run.smithi072.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-15T15:13:32.444 INFO:teuthology.orchestra.run.smithi072.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-15T15:13:32.444 INFO:teuthology.orchestra.run.smithi072.stdout: lua-5.4.4-4.el9.x86_64 2024-04-15T15:13:32.444 INFO:teuthology.orchestra.run.smithi072.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-15T15:13:32.444 INFO:teuthology.orchestra.run.smithi072.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-15T15:13:32.444 INFO:teuthology.orchestra.run.smithi072.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-15T15:13:32.445 INFO:teuthology.orchestra.run.smithi072.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-15T15:13:32.445 INFO:teuthology.orchestra.run.smithi072.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-15T15:13:32.445 INFO:teuthology.orchestra.run.smithi072.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-15T15:13:32.445 INFO:teuthology.orchestra.run.smithi072.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-15T15:13:32.445 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql-13.14-1.el9.x86_64 2024-04-15T15:13:32.445 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-15T15:13:32.445 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-15T15:13:32.445 INFO:teuthology.orchestra.run.smithi072.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-15T15:13:32.445 INFO:teuthology.orchestra.run.smithi072.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-15T15:13:32.445 INFO:teuthology.orchestra.run.smithi072.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-15T15:13:32.445 INFO:teuthology.orchestra.run.smithi072.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-15T15:13:32.446 INFO:teuthology.orchestra.run.smithi072.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-15T15:13:32.446 INFO:teuthology.orchestra.run.smithi072.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-15T15:13:32.446 INFO:teuthology.orchestra.run.smithi072.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-15T15:13:32.446 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-15T15:13:32.446 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-argparse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:32.446 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:32.446 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cephfs-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:32.446 INFO:teuthology.orchestra.run.smithi072.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-15T15:13:32.446 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-15T15:13:32.446 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-15T15:13:32.447 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-15T15:13:32.447 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-15T15:13:32.447 INFO:teuthology.orchestra.run.smithi072.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-15T15:13:32.447 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-15T15:13:32.447 INFO:teuthology.orchestra.run.smithi072.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-15T15:13:32.447 INFO:teuthology.orchestra.run.smithi072.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-15T15:13:32.447 INFO:teuthology.orchestra.run.smithi072.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-15T15:13:32.447 INFO:teuthology.orchestra.run.smithi072.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-15T15:13:32.447 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-15T15:13:32.447 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-15T15:13:32.447 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-15T15:13:32.447 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-15T15:13:32.447 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-15T15:13:32.448 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-15T15:13:32.448 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-15T15:13:32.448 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-15T15:13:32.448 INFO:teuthology.orchestra.run.smithi072.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-15T15:13:32.448 INFO:teuthology.orchestra.run.smithi072.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-15T15:13:32.448 INFO:teuthology.orchestra.run.smithi072.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-15T15:13:32.448 INFO:teuthology.orchestra.run.smithi072.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-15T15:13:32.448 INFO:teuthology.orchestra.run.smithi072.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-15T15:13:32.448 INFO:teuthology.orchestra.run.smithi072.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-15T15:13:32.448 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-15T15:13:32.448 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-15T15:13:32.448 INFO:teuthology.orchestra.run.smithi072.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-15T15:13:32.448 INFO:teuthology.orchestra.run.smithi072.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-15T15:13:32.448 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-15T15:13:32.448 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ply-3.11-14.el9.noarch 2024-04-15T15:13:32.449 INFO:teuthology.orchestra.run.smithi072.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-15T15:13:32.449 INFO:teuthology.orchestra.run.smithi072.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-15T15:13:32.449 INFO:teuthology.orchestra.run.smithi072.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-15T15:13:32.449 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-15T15:13:32.449 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-15T15:13:32.449 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-15T15:13:32.449 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-15T15:13:32.449 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-15T15:13:32.449 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-15T15:13:32.449 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rados-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:32.449 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:32.449 INFO:teuthology.orchestra.run.smithi072.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-15T15:13:32.449 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-15T15:13:32.449 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-15T15:13:32.449 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:32.450 INFO:teuthology.orchestra.run.smithi072.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-15T15:13:32.450 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-15T15:13:32.450 INFO:teuthology.orchestra.run.smithi072.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-15T15:13:32.450 INFO:teuthology.orchestra.run.smithi072.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-15T15:13:32.450 INFO:teuthology.orchestra.run.smithi072.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-15T15:13:32.450 INFO:teuthology.orchestra.run.smithi072.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-15T15:13:32.450 INFO:teuthology.orchestra.run.smithi072.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-15T15:13:32.450 INFO:teuthology.orchestra.run.smithi072.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-15T15:13:32.450 INFO:teuthology.orchestra.run.smithi072.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-15T15:13:32.450 INFO:teuthology.orchestra.run.smithi072.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-15T15:13:32.450 INFO:teuthology.orchestra.run.smithi072.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-15T15:13:32.450 INFO:teuthology.orchestra.run.smithi072.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-15T15:13:32.450 INFO:teuthology.orchestra.run.smithi072.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-15T15:13:32.451 INFO:teuthology.orchestra.run.smithi072.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-15T15:13:32.451 INFO:teuthology.orchestra.run.smithi072.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-15T15:13:32.451 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:32.451 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:32.451 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-nbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:32.451 INFO:teuthology.orchestra.run.smithi072.stdout: re2-1:20211101-3.el9.x86_64 2024-04-15T15:13:32.451 INFO:teuthology.orchestra.run.smithi072.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-15T15:13:32.451 INFO:teuthology.orchestra.run.smithi072.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-15T15:13:32.451 INFO:teuthology.orchestra.run.smithi072.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-15T15:13:32.451 INFO:teuthology.orchestra.run.smithi072.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-15T15:13:32.452 INFO:teuthology.orchestra.run.smithi072.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-15T15:13:32.452 INFO:teuthology.orchestra.run.smithi072.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-04-15T15:13:32.452 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:32.452 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-04-15T15:13:33.067 DEBUG:teuthology.parallel:result is None 2024-04-15T15:13:36.150 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-test-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 153/198 2024-04-15T15:13:36.191 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : rbd-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 154/198 2024-04-15T15:13:36.221 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : rbd-nbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 155/198 2024-04-15T15:13:36.249 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-04-15T15:13:36.433 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : cephfs-shell-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 157/198 2024-04-15T15:13:36.548 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-04-15T15:13:36.596 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : ceph-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 159/198 2024-04-15T15:13:36.654 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-04-15T15:13:36.713 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-04-15T15:13:36.756 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-04-15T15:13:36.797 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libcephfs-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_ 163/198 2024-04-15T15:13:36.847 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-04-15T15:13:36.876 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : NetworkManager-team-1:1.47.4-1.el9.x86_64 165/198 2024-04-15T15:13:36.952 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : NetworkManager-tui-1:1.47.4-1.el9.x86_64 166/198 2024-04-15T15:13:37.012 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-04-15T15:13:37.049 INFO:teuthology.orchestra.run.smithi175.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-04-15T15:13:37.663 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-15T15:13:37.705 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-15T15:13:37.725 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-15T15:13:38.542 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : cephfs-top-2:19.0.0-2861.g9bf1c4ba.el8.noarch 170/198 2024-04-15T15:13:38.633 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-04-15T15:13:38.674 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-04-15T15:13:38.759 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-04-15T15:13:38.786 INFO:teuthology.orchestra.run.smithi175.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-04-15T15:13:38.803 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-04-15T15:13:38.828 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-04-15T15:13:38.855 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-04-15T15:13:38.855 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-15T15:13:38.951 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-15T15:13:38.980 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-04-15T15:13:38.981 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-15T15:13:39.065 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-15T15:13:39.096 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-04-15T15:13:39.124 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-04-15T15:13:39.150 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-04-15T15:13:39.175 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-04-15T15:13:39.210 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-04-15T15:13:39.248 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-04-15T15:13:39.281 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-04-15T15:13:39.329 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-04-15T15:13:39.355 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-04-15T15:13:39.380 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-04-15T15:13:39.405 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-04-15T15:13:39.412 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-15T15:13:39.412 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-15T15:13:39.600 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-15T15:13:39.600 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-15T15:13:39.694 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-15T15:13:39.723 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-04-15T15:13:39.750 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-04-15T15:13:39.775 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-04-15T15:13:39.800 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-04-15T15:13:39.868 INFO:teuthology.orchestra.run.smithi175.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-15T15:13:45.190 INFO:teuthology.orchestra.run.smithi175.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-15T15:13:45.190 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 1/198 2024-04-15T15:13:45.190 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-base-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2/198 2024-04-15T15:13:45.190 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 3/198 2024-04-15T15:13:45.190 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 4/198 2024-04-15T15:13:45.190 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2861.g9bf1c 5/198 2024-04-15T15:13:45.190 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mds-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 6/198 2024-04-15T15:13:45.190 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 7/198 2024-04-15T15:13:45.191 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mon-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 8/198 2024-04-15T15:13:45.191 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-osd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 9/198 2024-04-15T15:13:45.191 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-radosgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 10/198 2024-04-15T15:13:45.191 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-selinux-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 11/198 2024-04-15T15:13:45.191 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-test-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 12/198 2024-04-15T15:13:45.191 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : cephfs-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 13/198 2024-04-15T15:13:45.191 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : cephfs-shell-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 14/198 2024-04-15T15:13:45.191 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libcephfs-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_ 15/198 2024-04-15T15:13:45.191 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libcephfs2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 16/198 2024-04-15T15:13:45.191 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libcephsqlite-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 17/198 2024-04-15T15:13:45.191 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librados-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_6 18/198 2024-04-15T15:13:45.191 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libradosstriper1-2:19.0.0-2861.g9bf1c4ba.el8.x86 19/198 2024-04-15T15:13:45.191 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librgw2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 20/198 2024-04-15T15:13:45.191 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2861.g9bf1c4ba.el 21/198 2024-04-15T15:13:45.191 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-ceph-common-2:19.0.0-2861.g9bf1c4ba.el8. 22/198 2024-04-15T15:13:45.192 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cephfs-2:19.0.0-2861.g9bf1c4ba.el8.x86_6 23/198 2024-04-15T15:13:45.192 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-rados-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 24/198 2024-04-15T15:13:45.192 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-rbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 25/198 2024-04-15T15:13:45.192 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-rgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 26/198 2024-04-15T15:13:45.192 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : rbd-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 27/198 2024-04-15T15:13:45.192 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : rbd-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 28/198 2024-04-15T15:13:45.192 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : rbd-nbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 29/198 2024-04-15T15:13:45.192 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2861.g9bf1c4ba. 30/198 2024-04-15T15:13:45.192 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noa 31/198 2024-04-15T15:13:45.192 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2861.g9bf1c4ba.el8.n 32/198 2024-04-15T15:13:45.192 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2861.g9bf 33/198 2024-04-15T15:13:45.192 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2861.g9bf1c4ba.el 34/198 2024-04-15T15:13:45.192 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2861.g9bf1c4ba.el8.noarch 35/198 2024-04-15T15:13:45.192 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2861.g9bf1c4ba.e 36/198 2024-04-15T15:13:45.192 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : ceph-volume-2:19.0.0-2861.g9bf1c4ba.el8.noarch 37/198 2024-04-15T15:13:45.193 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 38/198 2024-04-15T15:13:45.193 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : cephfs-top-2:19.0.0-2861.g9bf1c4ba.el8.noarch 39/198 2024-04-15T15:13:45.193 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-04-15T15:13:45.193 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-04-15T15:13:45.193 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-04-15T15:13:45.193 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-04-15T15:13:45.193 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-04-15T15:13:45.193 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-04-15T15:13:45.193 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-04-15T15:13:45.193 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-04-15T15:13:45.193 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-04-15T15:13:45.193 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-04-15T15:13:45.193 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-04-15T15:13:45.193 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-04-15T15:13:45.193 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-04-15T15:13:45.194 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-04-15T15:13:45.194 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-04-15T15:13:45.194 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-04-15T15:13:45.194 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-04-15T15:13:45.194 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-04-15T15:13:45.194 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : postgresql-13.14-1.el9.x86_64 58/198 2024-04-15T15:13:45.194 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 59/198 2024-04-15T15:13:45.194 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 60/198 2024-04-15T15:13:45.194 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 61/198 2024-04-15T15:13:45.194 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-04-15T15:13:45.194 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-04-15T15:13:45.194 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-04-15T15:13:45.194 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-04-15T15:13:45.194 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-04-15T15:13:45.195 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-04-15T15:13:45.195 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-04-15T15:13:45.195 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-04-15T15:13:45.195 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-04-15T15:13:45.195 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-04-15T15:13:45.195 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-04-15T15:13:45.195 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-04-15T15:13:45.195 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-04-15T15:13:45.195 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-04-15T15:13:45.195 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-04-15T15:13:45.195 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-04-15T15:13:45.195 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-04-15T15:13:45.195 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-04-15T15:13:45.196 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-04-15T15:13:45.196 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-04-15T15:13:45.196 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 82/198 2024-04-15T15:13:45.196 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 83/198 2024-04-15T15:13:45.196 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 84/198 2024-04-15T15:13:45.196 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 85/198 2024-04-15T15:13:45.196 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 86/198 2024-04-15T15:13:45.196 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 87/198 2024-04-15T15:13:45.196 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 88/198 2024-04-15T15:13:45.196 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-15T15:13:45.196 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 90/198 2024-04-15T15:13:45.196 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 91/198 2024-04-15T15:13:45.196 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-ply-3.11-14.el9.noarch 92/198 2024-04-15T15:13:45.196 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 93/198 2024-04-15T15:13:45.196 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/198 2024-04-15T15:13:45.196 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : jq-1.6-16.el9.x86_64 95/198 2024-04-15T15:13:45.197 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 96/198 2024-04-15T15:13:45.197 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 97/198 2024-04-15T15:13:45.197 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 98/198 2024-04-15T15:13:45.197 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 99/198 2024-04-15T15:13:45.197 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 100/198 2024-04-15T15:13:45.197 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 101/198 2024-04-15T15:13:45.197 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 102/198 2024-04-15T15:13:45.197 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 103/198 2024-04-15T15:13:45.197 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-04-15T15:13:45.197 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-04-15T15:13:45.197 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-04-15T15:13:45.197 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-04-15T15:13:45.197 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-04-15T15:13:45.197 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-04-15T15:13:45.197 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-04-15T15:13:45.198 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-04-15T15:13:45.198 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-04-15T15:13:45.198 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-04-15T15:13:45.198 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-04-15T15:13:45.198 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-15T15:13:45.198 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-04-15T15:13:45.198 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-04-15T15:13:45.198 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-04-15T15:13:45.198 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-04-15T15:13:45.198 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-04-15T15:13:45.198 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-04-15T15:13:45.198 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-04-15T15:13:45.198 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-04-15T15:13:45.199 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-04-15T15:13:45.199 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-04-15T15:13:45.199 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-04-15T15:13:45.199 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-04-15T15:13:45.199 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-04-15T15:13:45.199 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-04-15T15:13:45.199 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-04-15T15:13:45.199 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-04-15T15:13:45.199 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-04-15T15:13:45.199 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-04-15T15:13:45.199 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-04-15T15:13:45.199 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-04-15T15:13:45.199 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-04-15T15:13:45.199 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-04-15T15:13:45.200 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-04-15T15:13:45.200 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-04-15T15:13:45.200 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-04-15T15:13:45.200 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-04-15T15:13:45.200 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-04-15T15:13:45.200 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-04-15T15:13:45.200 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-04-15T15:13:45.200 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-04-15T15:13:45.200 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-04-15T15:13:45.200 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-04-15T15:13:45.200 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-04-15T15:13:45.200 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-04-15T15:13:45.200 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-04-15T15:13:45.200 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librados2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 151/198 2024-04-15T15:13:45.201 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-04-15T15:13:45.201 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librbd1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 153/198 2024-04-15T15:13:45.201 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-04-15T15:13:45.201 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 155/198 2024-04-15T15:13:45.201 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-04-15T15:13:45.201 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 157/198 2024-04-15T15:13:45.201 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-04-15T15:13:45.201 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 159/198 2024-04-15T15:13:45.201 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-04-15T15:13:45.201 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/198 2024-04-15T15:13:45.201 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-04-15T15:13:45.201 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 163/198 2024-04-15T15:13:45.201 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 164/198 2024-04-15T15:13:45.201 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-04-15T15:13:45.201 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-04-15T15:13:45.202 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-04-15T15:13:45.202 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-04-15T15:13:45.202 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-04-15T15:13:45.202 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-04-15T15:13:45.202 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-04-15T15:13:45.202 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-04-15T15:13:45.202 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-04-15T15:13:45.202 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-04-15T15:13:45.202 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-04-15T15:13:45.202 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-04-15T15:13:45.202 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-04-15T15:13:45.202 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-04-15T15:13:45.202 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-04-15T15:13:45.202 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-04-15T15:13:45.203 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-04-15T15:13:45.203 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-04-15T15:13:45.203 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-04-15T15:13:45.203 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-04-15T15:13:45.203 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-04-15T15:13:45.203 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-04-15T15:13:45.203 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-04-15T15:13:45.203 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-04-15T15:13:45.203 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-04-15T15:13:45.203 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-04-15T15:13:45.203 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-04-15T15:13:45.203 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-04-15T15:13:45.203 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-04-15T15:13:45.203 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-04-15T15:13:45.203 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-04-15T15:13:45.203 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-04-15T15:13:45.204 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-04-15T15:13:48.455 INFO:teuthology.orchestra.run.smithi175.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-04-15T15:13:48.456 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:48.456 INFO:teuthology.orchestra.run.smithi175.stdout:Upgraded: 2024-04-15T15:13:48.456 INFO:teuthology.orchestra.run.smithi175.stdout: NetworkManager-1:1.47.4-1.el9.x86_64 2024-04-15T15:13:48.456 INFO:teuthology.orchestra.run.smithi175.stdout: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 2024-04-15T15:13:48.456 INFO:teuthology.orchestra.run.smithi175.stdout: NetworkManager-team-1:1.47.4-1.el9.x86_64 2024-04-15T15:13:48.456 INFO:teuthology.orchestra.run.smithi175.stdout: NetworkManager-tui-1:1.47.4-1.el9.x86_64 2024-04-15T15:13:48.456 INFO:teuthology.orchestra.run.smithi175.stdout: acl-2.3.1-4.el9.x86_64 2024-04-15T15:13:48.456 INFO:teuthology.orchestra.run.smithi175.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-04-15T15:13:48.456 INFO:teuthology.orchestra.run.smithi175.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-04-15T15:13:48.456 INFO:teuthology.orchestra.run.smithi175.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-04-15T15:13:48.456 INFO:teuthology.orchestra.run.smithi175.stdout: libacl-2.3.1-4.el9.x86_64 2024-04-15T15:13:48.457 INFO:teuthology.orchestra.run.smithi175.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-04-15T15:13:48.457 INFO:teuthology.orchestra.run.smithi175.stdout: libblkid-2.37.4-18.el9.x86_64 2024-04-15T15:13:48.457 INFO:teuthology.orchestra.run.smithi175.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-04-15T15:13:48.457 INFO:teuthology.orchestra.run.smithi175.stdout: libmount-2.37.4-18.el9.x86_64 2024-04-15T15:13:48.457 INFO:teuthology.orchestra.run.smithi175.stdout: librados2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:48.457 INFO:teuthology.orchestra.run.smithi175.stdout: librbd1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:48.457 INFO:teuthology.orchestra.run.smithi175.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-04-15T15:13:48.457 INFO:teuthology.orchestra.run.smithi175.stdout: libuuid-2.37.4-18.el9.x86_64 2024-04-15T15:13:48.457 INFO:teuthology.orchestra.run.smithi175.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-04-15T15:13:48.457 INFO:teuthology.orchestra.run.smithi175.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-04-15T15:13:48.457 INFO:teuthology.orchestra.run.smithi175.stdout: util-linux-2.37.4-18.el9.x86_64 2024-04-15T15:13:48.457 INFO:teuthology.orchestra.run.smithi175.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-04-15T15:13:48.457 INFO:teuthology.orchestra.run.smithi175.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-04-15T15:13:48.458 INFO:teuthology.orchestra.run.smithi175.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-04-15T15:13:48.458 INFO:teuthology.orchestra.run.smithi175.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-04-15T15:13:48.458 INFO:teuthology.orchestra.run.smithi175.stdout:Installed: 2024-04-15T15:13:48.458 INFO:teuthology.orchestra.run.smithi175.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-15T15:13:48.458 INFO:teuthology.orchestra.run.smithi175.stdout: automake-1.16.2-8.el9.noarch 2024-04-15T15:13:48.458 INFO:teuthology.orchestra.run.smithi175.stdout: bison-3.7.4-5.el9.x86_64 2024-04-15T15:13:48.458 INFO:teuthology.orchestra.run.smithi175.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-15T15:13:48.458 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:48.458 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-base-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:48.458 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:48.458 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:48.458 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-grafana-dashboards-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:13:48.458 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-immutable-object-cache-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:48.459 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mds-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:48.459 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:48.459 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:13:48.459 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-dashboard-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:13:48.459 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:13:48.459 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-modules-core-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:13:48.459 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mgr-rook-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:13:48.459 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-mon-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:48.459 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-osd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:48.459 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-prometheus-alerts-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:13:48.459 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-radosgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:48.459 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-selinux-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:48.459 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-test-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:48.459 INFO:teuthology.orchestra.run.smithi175.stdout: ceph-volume-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:13:48.460 INFO:teuthology.orchestra.run.smithi175.stdout: cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:13:48.460 INFO:teuthology.orchestra.run.smithi175.stdout: cephfs-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:48.460 INFO:teuthology.orchestra.run.smithi175.stdout: cephfs-shell-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:48.460 INFO:teuthology.orchestra.run.smithi175.stdout: cephfs-top-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:13:48.460 INFO:teuthology.orchestra.run.smithi175.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-15T15:13:48.460 INFO:teuthology.orchestra.run.smithi175.stdout: flex-2.6.4-9.el9.x86_64 2024-04-15T15:13:48.460 INFO:teuthology.orchestra.run.smithi175.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-15T15:13:48.460 INFO:teuthology.orchestra.run.smithi175.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-15T15:13:48.460 INFO:teuthology.orchestra.run.smithi175.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-15T15:13:48.460 INFO:teuthology.orchestra.run.smithi175.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-15T15:13:48.460 INFO:teuthology.orchestra.run.smithi175.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-15T15:13:48.460 INFO:teuthology.orchestra.run.smithi175.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-15T15:13:48.461 INFO:teuthology.orchestra.run.smithi175.stdout: jq-1.6-16.el9.x86_64 2024-04-15T15:13:48.461 INFO:teuthology.orchestra.run.smithi175.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-15T15:13:48.461 INFO:teuthology.orchestra.run.smithi175.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-15T15:13:48.461 INFO:teuthology.orchestra.run.smithi175.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-15T15:13:48.461 INFO:teuthology.orchestra.run.smithi175.stdout: libcephfs-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:48.461 INFO:teuthology.orchestra.run.smithi175.stdout: libcephfs2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:48.461 INFO:teuthology.orchestra.run.smithi175.stdout: libcephsqlite-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:48.461 INFO:teuthology.orchestra.run.smithi175.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-15T15:13:48.461 INFO:teuthology.orchestra.run.smithi175.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-15T15:13:48.461 INFO:teuthology.orchestra.run.smithi175.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-15T15:13:48.461 INFO:teuthology.orchestra.run.smithi175.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-15T15:13:48.461 INFO:teuthology.orchestra.run.smithi175.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-15T15:13:48.461 INFO:teuthology.orchestra.run.smithi175.stdout: librados-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:48.461 INFO:teuthology.orchestra.run.smithi175.stdout: libradosstriper1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:48.462 INFO:teuthology.orchestra.run.smithi175.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-15T15:13:48.462 INFO:teuthology.orchestra.run.smithi175.stdout: librgw2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:48.462 INFO:teuthology.orchestra.run.smithi175.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-15T15:13:48.462 INFO:teuthology.orchestra.run.smithi175.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-15T15:13:48.462 INFO:teuthology.orchestra.run.smithi175.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-15T15:13:48.462 INFO:teuthology.orchestra.run.smithi175.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-15T15:13:48.462 INFO:teuthology.orchestra.run.smithi175.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-15T15:13:48.462 INFO:teuthology.orchestra.run.smithi175.stdout: lua-5.4.4-4.el9.x86_64 2024-04-15T15:13:48.462 INFO:teuthology.orchestra.run.smithi175.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-15T15:13:48.462 INFO:teuthology.orchestra.run.smithi175.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-15T15:13:48.462 INFO:teuthology.orchestra.run.smithi175.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-15T15:13:48.462 INFO:teuthology.orchestra.run.smithi175.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-15T15:13:48.463 INFO:teuthology.orchestra.run.smithi175.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-15T15:13:48.463 INFO:teuthology.orchestra.run.smithi175.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-15T15:13:48.463 INFO:teuthology.orchestra.run.smithi175.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-15T15:13:48.463 INFO:teuthology.orchestra.run.smithi175.stdout: postgresql-13.14-1.el9.x86_64 2024-04-15T15:13:48.463 INFO:teuthology.orchestra.run.smithi175.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-15T15:13:48.463 INFO:teuthology.orchestra.run.smithi175.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-15T15:13:48.463 INFO:teuthology.orchestra.run.smithi175.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-15T15:13:48.463 INFO:teuthology.orchestra.run.smithi175.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-15T15:13:48.463 INFO:teuthology.orchestra.run.smithi175.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-15T15:13:48.463 INFO:teuthology.orchestra.run.smithi175.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-15T15:13:48.463 INFO:teuthology.orchestra.run.smithi175.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-15T15:13:48.463 INFO:teuthology.orchestra.run.smithi175.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-15T15:13:48.463 INFO:teuthology.orchestra.run.smithi175.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-15T15:13:48.463 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-15T15:13:48.463 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ceph-argparse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:48.464 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:48.464 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cephfs-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:48.464 INFO:teuthology.orchestra.run.smithi175.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-15T15:13:48.464 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-15T15:13:48.464 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-15T15:13:48.464 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-15T15:13:48.464 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-15T15:13:48.464 INFO:teuthology.orchestra.run.smithi175.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-15T15:13:48.464 INFO:teuthology.orchestra.run.smithi175.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-15T15:13:48.464 INFO:teuthology.orchestra.run.smithi175.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-15T15:13:48.464 INFO:teuthology.orchestra.run.smithi175.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-15T15:13:48.464 INFO:teuthology.orchestra.run.smithi175.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-15T15:13:48.464 INFO:teuthology.orchestra.run.smithi175.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-15T15:13:48.465 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-15T15:13:48.465 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-15T15:13:48.465 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-15T15:13:48.465 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-15T15:13:48.465 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-15T15:13:48.465 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-15T15:13:48.465 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-15T15:13:48.465 INFO:teuthology.orchestra.run.smithi175.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-15T15:13:48.465 INFO:teuthology.orchestra.run.smithi175.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-15T15:13:48.465 INFO:teuthology.orchestra.run.smithi175.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-15T15:13:48.465 INFO:teuthology.orchestra.run.smithi175.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-15T15:13:48.465 INFO:teuthology.orchestra.run.smithi175.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-15T15:13:48.465 INFO:teuthology.orchestra.run.smithi175.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-15T15:13:48.466 INFO:teuthology.orchestra.run.smithi175.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-15T15:13:48.466 INFO:teuthology.orchestra.run.smithi175.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-15T15:13:48.466 INFO:teuthology.orchestra.run.smithi175.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-15T15:13:48.466 INFO:teuthology.orchestra.run.smithi175.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-15T15:13:48.466 INFO:teuthology.orchestra.run.smithi175.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-15T15:13:48.466 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-15T15:13:48.466 INFO:teuthology.orchestra.run.smithi175.stdout: python3-ply-3.11-14.el9.noarch 2024-04-15T15:13:48.466 INFO:teuthology.orchestra.run.smithi175.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-15T15:13:48.466 INFO:teuthology.orchestra.run.smithi175.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-15T15:13:48.466 INFO:teuthology.orchestra.run.smithi175.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-15T15:13:48.466 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-15T15:13:48.466 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-15T15:13:48.466 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-15T15:13:48.467 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-15T15:13:48.467 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-15T15:13:48.467 INFO:teuthology.orchestra.run.smithi175.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-15T15:13:48.467 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rados-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:48.467 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:48.467 INFO:teuthology.orchestra.run.smithi175.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-15T15:13:48.467 INFO:teuthology.orchestra.run.smithi175.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-15T15:13:48.467 INFO:teuthology.orchestra.run.smithi175.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-15T15:13:48.467 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:48.467 INFO:teuthology.orchestra.run.smithi175.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-15T15:13:48.467 INFO:teuthology.orchestra.run.smithi175.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-15T15:13:48.467 INFO:teuthology.orchestra.run.smithi175.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-15T15:13:48.467 INFO:teuthology.orchestra.run.smithi175.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-15T15:13:48.468 INFO:teuthology.orchestra.run.smithi175.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-15T15:13:48.468 INFO:teuthology.orchestra.run.smithi175.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-15T15:13:48.468 INFO:teuthology.orchestra.run.smithi175.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-15T15:13:48.468 INFO:teuthology.orchestra.run.smithi175.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-15T15:13:48.468 INFO:teuthology.orchestra.run.smithi175.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-15T15:13:48.468 INFO:teuthology.orchestra.run.smithi175.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-15T15:13:48.468 INFO:teuthology.orchestra.run.smithi175.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-15T15:13:48.468 INFO:teuthology.orchestra.run.smithi175.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-15T15:13:48.468 INFO:teuthology.orchestra.run.smithi175.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-15T15:13:48.468 INFO:teuthology.orchestra.run.smithi175.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-15T15:13:48.468 INFO:teuthology.orchestra.run.smithi175.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-15T15:13:48.468 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:48.468 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:48.468 INFO:teuthology.orchestra.run.smithi175.stdout: rbd-nbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:13:48.468 INFO:teuthology.orchestra.run.smithi175.stdout: re2-1:20211101-3.el9.x86_64 2024-04-15T15:13:48.469 INFO:teuthology.orchestra.run.smithi175.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-15T15:13:48.469 INFO:teuthology.orchestra.run.smithi175.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-15T15:13:48.469 INFO:teuthology.orchestra.run.smithi175.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-15T15:13:48.469 INFO:teuthology.orchestra.run.smithi175.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-15T15:13:48.469 INFO:teuthology.orchestra.run.smithi175.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-15T15:13:48.469 INFO:teuthology.orchestra.run.smithi175.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-04-15T15:13:48.469 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:48.470 INFO:teuthology.orchestra.run.smithi175.stdout:Complete! 2024-04-15T15:13:48.921 DEBUG:teuthology.parallel:result is None 2024-04-15T15:13:48.921 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 2024-04-15T15:13:49.065 DEBUG:teuthology.orchestra.run.smithi072:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-15T15:13:49.092 INFO:teuthology.orchestra.run.smithi072.stdout:19.0.0-2861.g9bf1c4ba.el8 2024-04-15T15:13:49.093 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2861.g9bf1c4ba.el8 2024-04-15T15:13:49.093 INFO:teuthology.task.install:The correct ceph version 19.0.0-2861.g9bf1c4ba is installed. 2024-04-15T15:13:49.095 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 2024-04-15T15:13:49.316 DEBUG:teuthology.orchestra.run.smithi175:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-15T15:13:49.339 INFO:teuthology.orchestra.run.smithi175.stdout:19.0.0-2861.g9bf1c4ba.el8 2024-04-15T15:13:49.340 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2861.g9bf1c4ba.el8 2024-04-15T15:13:49.340 INFO:teuthology.task.install:The correct ceph version 19.0.0-2861.g9bf1c4ba is installed. 2024-04-15T15:13:49.341 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-15T15:13:49.342 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:13:49.342 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-15T15:13:49.379 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:13:49.443 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-15T15:13:49.475 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-15T15:13:49.475 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:13:49.475 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/usr/bin/daemon-helper 2024-04-15T15:13:49.507 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-15T15:13:49.580 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:13:49.581 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/usr/bin/daemon-helper 2024-04-15T15:13:49.610 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-15T15:13:49.679 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-15T15:13:49.680 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:13:49.680 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-15T15:13:49.708 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-15T15:13:49.777 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:13:49.777 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-15T15:13:49.806 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-15T15:13:49.874 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-15T15:13:49.874 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:13:49.875 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/usr/bin/stdin-killer 2024-04-15T15:13:49.903 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-15T15:13:49.972 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:13:49.972 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/usr/bin/stdin-killer 2024-04-15T15:13:50.001 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-15T15:13:50.070 INFO:teuthology.run_tasks:Running task ceph... 2024-04-15T15:13:50.158 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-15T15:13:50.158 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 777 /var/log/ceph 2024-04-15T15:13:50.160 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 777 /var/log/ceph 2024-04-15T15:13:50.188 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-15T15:13:50.188 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-15T15:13:50.229 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-15T15:13:50.257 INFO:tasks.ceph:Creating extra log directories... 2024-04-15T15:13:50.257 DEBUG:teuthology.orchestra.run.smithi072:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-15T15:13:50.297 DEBUG:teuthology.orchestra.run.smithi175:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-15T15:13:50.332 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-04-15T15:13:50.332 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-04-15T15:13:50.332 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-04-15T15:13:50.332 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:13:50.332 DEBUG:teuthology.orchestra.run.smithi072:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-15T15:13:50.379 DEBUG:teuthology.orchestra.run.smithi072:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-15T15:13:50.499 DEBUG:teuthology.orchestra.run.smithi072:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-15T15:13:50.528 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:13:50.528 DEBUG:teuthology.orchestra.run.smithi175:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-15T15:13:50.544 DEBUG:teuthology.orchestra.run.smithi175:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-15T15:13:50.667 DEBUG:teuthology.orchestra.run.smithi175:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-15T15:13:50.702 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-15T15:13:50.702 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-15T15:13:50.702 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657120', 'branch': 'wip-lusov-quiesce-xlock', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/quiesce}', 'email': None, 'first_in_suite': False, 'job_id': '7657120', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-lusov-quiesce-xlock'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow request'], 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'subvolume_version': 1, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-lusov-quiesce-xlock', 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14'}}, 'owner': 'scheduled_leonidus@teuthology', 'priority': 70, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 2288, 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'wip-lusov-quiesce-xlock', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi072.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG+jvD3lC11LrXoo70wa+dUFmGv6veuLETc7MspcK85RTHe7zimRkbrxwOjw8TurdnsOI7szwqxlzsmbcgrpAUA=', 'smithi175.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGpS+fS1n7SHFRiy96aTxyNYq1ubnETNBoHqo+uqNcC3llhjvSVSFpQM/r0JijXdeTsCN9v8KOD5Q2Fkkvx/hlQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow request'], 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'ceph-fuse': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_quiesce']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': '6c637841c215537a4502385240412f1966e0faab', 'timestamp': '2024-04-15_14:46:31', 'tube': 'smithi', 'user': 'leonidus', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-15T15:13:50.702 DEBUG:teuthology.orchestra.run.smithi072:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-15T15:13:50.705 DEBUG:teuthology.orchestra.run.smithi175:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-15T15:13:50.753 DEBUG:teuthology.orchestra.run.smithi072:> sudo install -d -m0777 -- /var/run/ceph 2024-04-15T15:13:50.760 DEBUG:teuthology.orchestra.run.smithi175:> sudo install -d -m0777 -- /var/run/ceph 2024-04-15T15:13:50.820 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:13:50.821 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/scratch_devs of=/dev/stdout 2024-04-15T15:13:50.848 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-15T15:13:50.849 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_1 2024-04-15T15:13:50.904 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-15T15:13:50.904 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T15:13:50.904 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-04-15T15:13:50.904 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T15:13:50.904 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T15:13:50.904 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2024-04-15 15:13:20.162602724 +0000 2024-04-15T15:13:50.905 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2024-04-15 15:11:16.975130942 +0000 2024-04-15T15:13:50.905 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2024-04-15 15:11:16.975130942 +0000 2024-04-15T15:13:50.905 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2024-04-15 15:11:16.975130942 +0000 2024-04-15T15:13:50.905 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-15T15:13:50.971 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2024-04-15T15:13:50.971 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2024-04-15T15:13:50.971 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000258224 s, 2.0 MB/s 2024-04-15T15:13:50.972 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-15T15:13:51.030 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_2 2024-04-15T15:13:51.085 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-15T15:13:51.086 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T15:13:51.086 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-15T15:13:51.086 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T15:13:51.086 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T15:13:51.086 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2024-04-15 15:13:20.162602724 +0000 2024-04-15T15:13:51.086 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2024-04-15 15:11:17.309123617 +0000 2024-04-15T15:13:51.086 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2024-04-15 15:11:17.309123617 +0000 2024-04-15T15:13:51.086 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2024-04-15 15:11:17.309123617 +0000 2024-04-15T15:13:51.086 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-15T15:13:51.153 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2024-04-15T15:13:51.153 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2024-04-15T15:13:51.153 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000274241 s, 1.9 MB/s 2024-04-15T15:13:51.154 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-15T15:13:51.210 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_3 2024-04-15T15:13:51.266 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-15T15:13:51.266 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T15:13:51.266 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-04-15T15:13:51.266 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T15:13:51.266 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T15:13:51.266 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2024-04-15 15:13:20.162602724 +0000 2024-04-15T15:13:51.266 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2024-04-15 15:11:17.638116403 +0000 2024-04-15T15:13:51.266 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2024-04-15 15:11:17.638116403 +0000 2024-04-15T15:13:51.266 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2024-04-15 15:11:17.638116403 +0000 2024-04-15T15:13:51.267 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-15T15:13:51.332 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2024-04-15T15:13:51.333 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2024-04-15T15:13:51.333 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000184868 s, 2.8 MB/s 2024-04-15T15:13:51.334 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-15T15:13:51.390 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_4 2024-04-15T15:13:51.445 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-15T15:13:51.446 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T15:13:51.446 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-15T15:13:51.446 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T15:13:51.446 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T15:13:51.446 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2024-04-15 15:13:20.163602702 +0000 2024-04-15T15:13:51.446 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2024-04-15 15:11:17.971109101 +0000 2024-04-15T15:13:51.446 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2024-04-15 15:11:17.971109101 +0000 2024-04-15T15:13:51.446 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2024-04-15 15:11:17.971109101 +0000 2024-04-15T15:13:51.447 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-15T15:13:51.512 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2024-04-15T15:13:51.513 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2024-04-15T15:13:51.513 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000190826 s, 2.7 MB/s 2024-04-15T15:13:51.514 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-15T15:13:51.570 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-15T15:13:51.571 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:13:51.571 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/scratch_devs of=/dev/stdout 2024-04-15T15:13:51.588 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-15T15:13:51.588 DEBUG:teuthology.orchestra.run.smithi175:> stat /dev/vg_nvme/lv_1 2024-04-15T15:13:51.645 INFO:teuthology.orchestra.run.smithi175.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-15T15:13:51.645 INFO:teuthology.orchestra.run.smithi175.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T15:13:51.645 INFO:teuthology.orchestra.run.smithi175.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-15T15:13:51.645 INFO:teuthology.orchestra.run.smithi175.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T15:13:51.645 INFO:teuthology.orchestra.run.smithi175.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T15:13:51.646 INFO:teuthology.orchestra.run.smithi175.stdout:Access: 2024-04-15 15:13:40.217758683 +0000 2024-04-15T15:13:51.646 INFO:teuthology.orchestra.run.smithi175.stdout:Modify: 2024-04-15 15:11:12.762390252 +0000 2024-04-15T15:13:51.646 INFO:teuthology.orchestra.run.smithi175.stdout:Change: 2024-04-15 15:11:12.762390252 +0000 2024-04-15T15:13:51.646 INFO:teuthology.orchestra.run.smithi175.stdout: Birth: 2024-04-15 15:11:12.762390252 +0000 2024-04-15T15:13:51.646 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-15T15:13:51.714 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records in 2024-04-15T15:13:51.714 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records out 2024-04-15T15:13:51.714 INFO:teuthology.orchestra.run.smithi175.stderr:512 bytes copied, 0.000228106 s, 2.2 MB/s 2024-04-15T15:13:51.715 DEBUG:teuthology.orchestra.run.smithi175:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-15T15:13:51.772 DEBUG:teuthology.orchestra.run.smithi175:> stat /dev/vg_nvme/lv_2 2024-04-15T15:13:51.828 INFO:teuthology.orchestra.run.smithi175.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-15T15:13:51.829 INFO:teuthology.orchestra.run.smithi175.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T15:13:51.829 INFO:teuthology.orchestra.run.smithi175.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-04-15T15:13:51.829 INFO:teuthology.orchestra.run.smithi175.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T15:13:51.829 INFO:teuthology.orchestra.run.smithi175.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T15:13:51.829 INFO:teuthology.orchestra.run.smithi175.stdout:Access: 2024-04-15 15:13:40.218758668 +0000 2024-04-15T15:13:51.829 INFO:teuthology.orchestra.run.smithi175.stdout:Modify: 2024-04-15 15:11:13.100385085 +0000 2024-04-15T15:13:51.829 INFO:teuthology.orchestra.run.smithi175.stdout:Change: 2024-04-15 15:11:13.100385085 +0000 2024-04-15T15:13:51.829 INFO:teuthology.orchestra.run.smithi175.stdout: Birth: 2024-04-15 15:11:13.100385085 +0000 2024-04-15T15:13:51.829 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-15T15:13:51.896 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records in 2024-04-15T15:13:51.896 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records out 2024-04-15T15:13:51.896 INFO:teuthology.orchestra.run.smithi175.stderr:512 bytes copied, 0.000250556 s, 2.0 MB/s 2024-04-15T15:13:51.897 DEBUG:teuthology.orchestra.run.smithi175:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-15T15:13:51.954 DEBUG:teuthology.orchestra.run.smithi175:> stat /dev/vg_nvme/lv_3 2024-04-15T15:13:52.009 INFO:teuthology.orchestra.run.smithi175.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-15T15:13:52.010 INFO:teuthology.orchestra.run.smithi175.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T15:13:52.010 INFO:teuthology.orchestra.run.smithi175.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-04-15T15:13:52.010 INFO:teuthology.orchestra.run.smithi175.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T15:13:52.010 INFO:teuthology.orchestra.run.smithi175.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T15:13:52.010 INFO:teuthology.orchestra.run.smithi175.stdout:Access: 2024-04-15 15:13:40.218758668 +0000 2024-04-15T15:13:52.010 INFO:teuthology.orchestra.run.smithi175.stdout:Modify: 2024-04-15 15:11:13.435379964 +0000 2024-04-15T15:13:52.010 INFO:teuthology.orchestra.run.smithi175.stdout:Change: 2024-04-15 15:11:13.435379964 +0000 2024-04-15T15:13:52.010 INFO:teuthology.orchestra.run.smithi175.stdout: Birth: 2024-04-15 15:11:13.435379964 +0000 2024-04-15T15:13:52.010 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-15T15:13:52.076 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records in 2024-04-15T15:13:52.076 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records out 2024-04-15T15:13:52.076 INFO:teuthology.orchestra.run.smithi175.stderr:512 bytes copied, 0.000227598 s, 2.2 MB/s 2024-04-15T15:13:52.077 DEBUG:teuthology.orchestra.run.smithi175:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-15T15:13:52.134 DEBUG:teuthology.orchestra.run.smithi175:> stat /dev/vg_nvme/lv_4 2024-04-15T15:13:52.190 INFO:teuthology.orchestra.run.smithi175.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-15T15:13:52.191 INFO:teuthology.orchestra.run.smithi175.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T15:13:52.191 INFO:teuthology.orchestra.run.smithi175.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-15T15:13:52.191 INFO:teuthology.orchestra.run.smithi175.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T15:13:52.191 INFO:teuthology.orchestra.run.smithi175.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T15:13:52.191 INFO:teuthology.orchestra.run.smithi175.stdout:Access: 2024-04-15 15:13:40.219758653 +0000 2024-04-15T15:13:52.191 INFO:teuthology.orchestra.run.smithi175.stdout:Modify: 2024-04-15 15:11:13.772374812 +0000 2024-04-15T15:13:52.191 INFO:teuthology.orchestra.run.smithi175.stdout:Change: 2024-04-15 15:11:13.772374812 +0000 2024-04-15T15:13:52.191 INFO:teuthology.orchestra.run.smithi175.stdout: Birth: 2024-04-15 15:11:13.772374812 +0000 2024-04-15T15:13:52.191 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-15T15:13:52.257 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records in 2024-04-15T15:13:52.258 INFO:teuthology.orchestra.run.smithi175.stderr:1+0 records out 2024-04-15T15:13:52.258 INFO:teuthology.orchestra.run.smithi175.stderr:512 bytes copied, 0.000208525 s, 2.5 MB/s 2024-04-15T15:13:52.259 DEBUG:teuthology.orchestra.run.smithi175:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-15T15:13:52.315 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-15T15:13:52.316 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi072.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi175.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-15T15:13:52.316 INFO:tasks.ceph:Generating config... 2024-04-15T15:13:52.317 INFO:tasks.ceph:[client] client mount timeout = 600 2024-04-15T15:13:52.317 INFO:tasks.ceph:[client] debug client = 20 2024-04-15T15:13:52.317 INFO:tasks.ceph:[client] debug ms = 1 2024-04-15T15:13:52.317 INFO:tasks.ceph:[client] pid file = 2024-04-15T15:13:52.317 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-04-15T15:13:52.317 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-04-15T15:13:52.317 INFO:tasks.ceph:[mds] debug mds = 20 2024-04-15T15:13:52.317 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-04-15T15:13:52.317 INFO:tasks.ceph:[mds] debug ms = 1 2024-04-15T15:13:52.317 INFO:tasks.ceph:[mds] mds debug frag = True 2024-04-15T15:13:52.318 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-04-15T15:13:52.318 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-04-15T15:13:52.318 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-04-15T15:13:52.318 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-04-15T15:13:52.318 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-04-15T15:13:52.318 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-04-15T15:13:52.318 INFO:tasks.ceph:[mgr] debug client = 20 2024-04-15T15:13:52.318 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-15T15:13:52.318 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-15T15:13:52.318 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-15T15:13:52.318 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-15T15:13:52.318 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-15T15:13:52.318 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-04-15T15:13:52.318 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-04-15T15:13:52.319 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-15T15:13:52.319 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-15T15:13:52.319 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-15T15:13:52.319 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-15T15:13:52.319 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-15T15:13:52.319 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-15T15:13:52.319 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-15T15:13:52.319 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-15T15:13:52.319 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-15T15:13:52.319 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-15T15:13:52.319 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-15T15:13:52.319 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-15T15:13:52.319 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-04-15T15:13:52.320 INFO:tasks.ceph:Setting up mon.a... 2024-04-15T15:13:52.320 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-15T15:13:52.376 INFO:teuthology.orchestra.run.smithi072.stdout:creating /etc/ceph/ceph.keyring 2024-04-15T15:13:52.379 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-15T15:13:52.425 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-15T15:13:52.453 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.72'), ('mon.b', '172.21.15.175'), ('mon.c', '[v2:172.21.15.175:3301,v1:172.21.15.175:6790]')] 2024-04-15T15:13:52.454 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.72,172.21.15.175,[v2:172.21.15.175:3301,v1:172.21.15.175:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-04-15T15:13:52.455 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:13:52.455 DEBUG:teuthology.orchestra.run.smithi072:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-15T15:13:52.509 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.72 --add b 172.21.15.175 --addv c '[v2:172.21.15.175:3301,v1:172.21.15.175:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:13:52.595 INFO:teuthology.orchestra.run.smithi072.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:13:52.595 INFO:teuthology.orchestra.run.smithi072.stdout:monmaptool: generated fsid 05091e39-b18a-4545-8b68-c0ae4699e116 2024-04-15T15:13:52.595 INFO:teuthology.orchestra.run.smithi072.stdout:setting min_mon_release = quincy 2024-04-15T15:13:52.595 INFO:teuthology.orchestra.run.smithi072.stdout:epoch 0 2024-04-15T15:13:52.595 INFO:teuthology.orchestra.run.smithi072.stdout:fsid 05091e39-b18a-4545-8b68-c0ae4699e116 2024-04-15T15:13:52.595 INFO:teuthology.orchestra.run.smithi072.stdout:last_changed 2024-04-15T15:13:52.594387+0000 2024-04-15T15:13:52.595 INFO:teuthology.orchestra.run.smithi072.stdout:created 2024-04-15T15:13:52.594387+0000 2024-04-15T15:13:52.595 INFO:teuthology.orchestra.run.smithi072.stdout:min_mon_release 17 (quincy) 2024-04-15T15:13:52.596 INFO:teuthology.orchestra.run.smithi072.stdout:election_strategy: 1 2024-04-15T15:13:52.596 INFO:teuthology.orchestra.run.smithi072.stdout:0: [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] mon.a 2024-04-15T15:13:52.596 INFO:teuthology.orchestra.run.smithi072.stdout:1: [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] mon.b 2024-04-15T15:13:52.596 INFO:teuthology.orchestra.run.smithi072.stdout:2: [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] mon.c 2024-04-15T15:13:52.596 INFO:teuthology.orchestra.run.smithi072.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-15T15:13:52.597 DEBUG:teuthology.orchestra.run.smithi072:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-15T15:13:52.612 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 05091e39-b18a-4545-8b68-c0ae4699e116... 2024-04-15T15:13:52.613 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-15T15:13:52.655 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-15T15:13:52.706 INFO:teuthology.orchestra.run.smithi072.stdout:[global] 2024-04-15T15:13:52.706 INFO:teuthology.orchestra.run.smithi072.stdout: chdir = "" 2024-04-15T15:13:52.706 INFO:teuthology.orchestra.run.smithi072.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-15T15:13:52.706 INFO:teuthology.orchestra.run.smithi072.stdout: auth supported = cephx 2024-04-15T15:13:52.706 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:52.706 INFO:teuthology.orchestra.run.smithi072.stdout: filestore xattr use omap = true 2024-04-15T15:13:52.706 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:52.706 INFO:teuthology.orchestra.run.smithi072.stdout: mon clock drift allowed = 1.000 2024-04-15T15:13:52.706 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:52.706 INFO:teuthology.orchestra.run.smithi072.stdout: osd crush chooseleaf type = 0 2024-04-15T15:13:52.706 INFO:teuthology.orchestra.run.smithi072.stdout: auth debug = true 2024-04-15T15:13:52.707 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:52.707 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on old message = true 2024-04-15T15:13:52.707 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on bug = true 2024-04-15T15:13:52.707 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:52.707 INFO:teuthology.orchestra.run.smithi072.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-15T15:13:52.707 INFO:teuthology.orchestra.run.smithi072.stdout: mon pg warn max object skew = 0 2024-04-15T15:13:52.707 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:52.707 INFO:teuthology.orchestra.run.smithi072.stdout: # disable pg_autoscaler by default for new pools 2024-04-15T15:13:52.707 INFO:teuthology.orchestra.run.smithi072.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-15T15:13:52.707 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:52.707 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default size = 2 2024-04-15T15:13:52.707 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:52.707 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd allow primary affinity = true 2024-04-15T15:13:52.707 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd allow pg remap = true 2024-04-15T15:13:52.707 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on legacy crush tunables = false 2024-04-15T15:13:52.708 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on crush straw calc version zero = false 2024-04-15T15:13:52.708 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on no sortbitwise = false 2024-04-15T15:13:52.708 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on osd down out interval zero = false 2024-04-15T15:13:52.708 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on too few osds = false 2024-04-15T15:13:52.708 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-15T15:13:52.708 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-15T15:13:52.708 INFO:teuthology.orchestra.run.smithi072.stdout: mon_allow_pool_size_one = true 2024-04-15T15:13:52.708 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:52.708 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-15T15:13:52.708 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:52.708 INFO:teuthology.orchestra.run.smithi072.stdout: osd default data pool replay window = 5 2024-04-15T15:13:52.708 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:52.708 INFO:teuthology.orchestra.run.smithi072.stdout: mon allow pool delete = true 2024-04-15T15:13:52.708 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:52.708 INFO:teuthology.orchestra.run.smithi072.stdout: mon cluster log file level = debug 2024-04-15T15:13:52.709 INFO:teuthology.orchestra.run.smithi072.stdout: debug asserts on shutdown = true 2024-04-15T15:13:52.709 INFO:teuthology.orchestra.run.smithi072.stdout: mon health detail to clog = false 2024-04-15T15:13:52.709 INFO:teuthology.orchestra.run.smithi072.stdout: mon host = "172.21.15.72,172.21.15.175,[v2:172.21.15.175:3301,v1:172.21.15.175:6790]" 2024-04-15T15:13:52.709 INFO:teuthology.orchestra.run.smithi072.stdout: fsid = 05091e39-b18a-4545-8b68-c0ae4699e116 2024-04-15T15:13:52.709 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:52.709 INFO:teuthology.orchestra.run.smithi072.stdout:[osd] 2024-04-15T15:13:52.709 INFO:teuthology.orchestra.run.smithi072.stdout: osd journal size = 100 2024-04-15T15:13:52.709 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:52.709 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub load threshold = 5.0 2024-04-15T15:13:52.709 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub max interval = 600 2024-04-15T15:13:52.709 INFO:teuthology.orchestra.run.smithi072.stdout: osd mclock profile = high_recovery_ops 2024-04-15T15:13:52.709 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:52.709 INFO:teuthology.orchestra.run.smithi072.stdout: osd recover clone overlap = true 2024-04-15T15:13:52.709 INFO:teuthology.orchestra.run.smithi072.stdout: osd recovery max chunk = 1048576 2024-04-15T15:13:52.709 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:52.710 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug shutdown = true 2024-04-15T15:13:52.710 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug op order = true 2024-04-15T15:13:52.710 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug verify stray on activate = true 2024-04-15T15:13:52.710 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug trim objects = true 2024-04-15T15:13:52.710 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:52.710 INFO:teuthology.orchestra.run.smithi072.stdout: osd open classes on start = true 2024-04-15T15:13:52.710 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug pg log writeout = true 2024-04-15T15:13:52.710 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:52.710 INFO:teuthology.orchestra.run.smithi072.stdout: osd deep scrub update digest min age = 30 2024-04-15T15:13:52.710 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:52.710 INFO:teuthology.orchestra.run.smithi072.stdout: osd map max advance = 10 2024-04-15T15:13:52.710 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:52.710 INFO:teuthology.orchestra.run.smithi072.stdout: journal zero on create = true 2024-04-15T15:13:52.710 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:52.710 INFO:teuthology.orchestra.run.smithi072.stdout: filestore ondisk finisher threads = 3 2024-04-15T15:13:52.711 INFO:teuthology.orchestra.run.smithi072.stdout: filestore apply finisher threads = 3 2024-04-15T15:13:52.711 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:52.711 INFO:teuthology.orchestra.run.smithi072.stdout: bdev debug aio = true 2024-04-15T15:13:52.711 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug misdirected ops = true 2024-04-15T15:13:52.711 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore block size = 96636764160 2024-04-15T15:13:52.711 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore fsck on mount = True 2024-04-15T15:13:52.711 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluefs = 20 2024-04-15T15:13:52.711 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluestore = 20 2024-04-15T15:13:52.711 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-04-15T15:13:52.711 INFO:teuthology.orchestra.run.smithi072.stdout: debug osd = 20 2024-04-15T15:13:52.711 INFO:teuthology.orchestra.run.smithi072.stdout: debug rocksdb = 10 2024-04-15T15:13:52.711 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd backfillfull_ratio = 0.85 2024-04-15T15:13:52.711 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd full ratio = 0.9 2024-04-15T15:13:52.711 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd nearfull ratio = 0.8 2024-04-15T15:13:52.712 INFO:teuthology.orchestra.run.smithi072.stdout: osd failsafe full ratio = 0.95 2024-04-15T15:13:52.712 INFO:teuthology.orchestra.run.smithi072.stdout: osd objectstore = bluestore 2024-04-15T15:13:52.712 INFO:teuthology.orchestra.run.smithi072.stdout: osd op complaint time = 180 2024-04-15T15:13:52.712 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:52.712 INFO:teuthology.orchestra.run.smithi072.stdout:[mgr] 2024-04-15T15:13:52.712 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-04-15T15:13:52.712 INFO:teuthology.orchestra.run.smithi072.stdout: debug mgr = 20 2024-04-15T15:13:52.712 INFO:teuthology.orchestra.run.smithi072.stdout: debug mon = 20 2024-04-15T15:13:52.712 INFO:teuthology.orchestra.run.smithi072.stdout: debug auth = 20 2024-04-15T15:13:52.712 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min pgs per osd = 4 2024-04-15T15:13:52.713 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min bytes per osd = 10 2024-04-15T15:13:52.713 INFO:teuthology.orchestra.run.smithi072.stdout: mgr/telemetry/nag = false 2024-04-15T15:13:52.713 INFO:teuthology.orchestra.run.smithi072.stdout: debug client = 20 2024-04-15T15:13:52.713 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:52.713 INFO:teuthology.orchestra.run.smithi072.stdout:[mon] 2024-04-15T15:13:52.713 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-04-15T15:13:52.713 INFO:teuthology.orchestra.run.smithi072.stdout: debug mon = 20 2024-04-15T15:13:52.713 INFO:teuthology.orchestra.run.smithi072.stdout: debug paxos = 20 2024-04-15T15:13:52.713 INFO:teuthology.orchestra.run.smithi072.stdout: debug auth = 20 2024-04-15T15:13:52.713 INFO:teuthology.orchestra.run.smithi072.stdout: mon data avail warn = 5 2024-04-15T15:13:52.714 INFO:teuthology.orchestra.run.smithi072.stdout: mon mgr mkfs grace = 240 2024-04-15T15:13:52.714 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min pgs per osd = 4 2024-04-15T15:13:52.714 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd reporter subtree level = osd 2024-04-15T15:13:52.714 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd prime pg temp = true 2024-04-15T15:13:52.714 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min bytes per osd = 10 2024-04-15T15:13:52.714 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:52.714 INFO:teuthology.orchestra.run.smithi072.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-15T15:13:52.715 INFO:teuthology.orchestra.run.smithi072.stdout: auth mon ticket ttl = 660 # 11m 2024-04-15T15:13:52.715 INFO:teuthology.orchestra.run.smithi072.stdout: auth service ticket ttl = 240 # 4m 2024-04-15T15:13:52.715 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:52.715 INFO:teuthology.orchestra.run.smithi072.stdout: # don't complain about insecure global_id in the test suite 2024-04-15T15:13:52.715 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-15T15:13:52.715 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-15T15:13:52.715 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:52.715 INFO:teuthology.orchestra.run.smithi072.stdout: # 1m isn't quite enough 2024-04-15T15:13:52.715 INFO:teuthology.orchestra.run.smithi072.stdout: mon_down_mkfs_grace = 2m 2024-04-15T15:13:52.715 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:52.715 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_filestore_osds = false 2024-04-15T15:13:52.715 INFO:teuthology.orchestra.run.smithi072.stdout: mon down mkfs grace = 300 2024-04-15T15:13:52.715 INFO:teuthology.orchestra.run.smithi072.stdout: mon op complaint time = 120 2024-04-15T15:13:52.715 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:13:52.715 INFO:teuthology.orchestra.run.smithi072.stdout:[client] 2024-04-15T15:13:52.715 INFO:teuthology.orchestra.run.smithi072.stdout: rgw cache enabled = true 2024-04-15T15:13:52.716 INFO:teuthology.orchestra.run.smithi072.stdout: rgw enable ops log = true 2024-04-15T15:13:52.716 INFO:teuthology.orchestra.run.smithi072.stdout: rgw enable usage log = true 2024-04-15T15:13:52.716 INFO:teuthology.orchestra.run.smithi072.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-15T15:13:52.716 INFO:teuthology.orchestra.run.smithi072.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-15T15:13:52.716 INFO:teuthology.orchestra.run.smithi072.stdout: client mount timeout = 600 2024-04-15T15:13:52.716 INFO:teuthology.orchestra.run.smithi072.stdout: debug client = 20 2024-04-15T15:13:52.716 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-04-15T15:13:52.716 INFO:teuthology.orchestra.run.smithi072.stdout: pid file = "" 2024-04-15T15:13:52.716 INFO:teuthology.orchestra.run.smithi072.stdout: rados mon op timeout = 900 2024-04-15T15:13:52.716 INFO:teuthology.orchestra.run.smithi072.stdout: rados osd op timeout = 900 2024-04-15T15:13:52.716 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.a] 2024-04-15T15:13:52.716 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.b] 2024-04-15T15:13:52.716 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.c] 2024-04-15T15:13:52.716 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.a] 2024-04-15T15:13:52.717 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.c] 2024-04-15T15:13:52.717 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.b] 2024-04-15T15:13:52.717 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.d] 2024-04-15T15:13:52.717 INFO:teuthology.orchestra.run.smithi072.stdout:[mds] 2024-04-15T15:13:52.717 INFO:teuthology.orchestra.run.smithi072.stdout: debug mds = 20 2024-04-15T15:13:52.717 INFO:teuthology.orchestra.run.smithi072.stdout: debug mds balancer = 20 2024-04-15T15:13:52.717 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2024-04-15T15:13:52.717 INFO:teuthology.orchestra.run.smithi072.stdout: mds debug frag = True 2024-04-15T15:13:52.717 INFO:teuthology.orchestra.run.smithi072.stdout: mds debug scatterstat = True 2024-04-15T15:13:52.717 INFO:teuthology.orchestra.run.smithi072.stdout: mds op complaint time = 180 2024-04-15T15:13:52.717 INFO:teuthology.orchestra.run.smithi072.stdout: mds verify scatter = True 2024-04-15T15:13:52.717 INFO:teuthology.orchestra.run.smithi072.stdout: osd op complaint time = 180 2024-04-15T15:13:52.718 INFO:teuthology.orchestra.run.smithi072.stdout: rados mon op timeout = 900 2024-04-15T15:13:52.718 INFO:teuthology.orchestra.run.smithi072.stdout: rados osd op timeout = 900 2024-04-15T15:13:52.718 INFO:teuthology.orchestra.run.smithi175.stdout:[global] 2024-04-15T15:13:52.718 INFO:teuthology.orchestra.run.smithi175.stdout: chdir = "" 2024-04-15T15:13:52.719 INFO:teuthology.orchestra.run.smithi175.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-15T15:13:52.719 INFO:teuthology.orchestra.run.smithi175.stdout: auth supported = cephx 2024-04-15T15:13:52.719 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:52.719 INFO:teuthology.orchestra.run.smithi175.stdout: filestore xattr use omap = true 2024-04-15T15:13:52.719 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:52.719 INFO:teuthology.orchestra.run.smithi175.stdout: mon clock drift allowed = 1.000 2024-04-15T15:13:52.719 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:52.719 INFO:teuthology.orchestra.run.smithi175.stdout: osd crush chooseleaf type = 0 2024-04-15T15:13:52.719 INFO:teuthology.orchestra.run.smithi175.stdout: auth debug = true 2024-04-15T15:13:52.719 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:52.719 INFO:teuthology.orchestra.run.smithi175.stdout: ms die on old message = true 2024-04-15T15:13:52.719 INFO:teuthology.orchestra.run.smithi175.stdout: ms die on bug = true 2024-04-15T15:13:52.719 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:52.719 INFO:teuthology.orchestra.run.smithi175.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-15T15:13:52.720 INFO:teuthology.orchestra.run.smithi175.stdout: mon pg warn max object skew = 0 2024-04-15T15:13:52.720 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:52.720 INFO:teuthology.orchestra.run.smithi175.stdout: # disable pg_autoscaler by default for new pools 2024-04-15T15:13:52.720 INFO:teuthology.orchestra.run.smithi175.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-15T15:13:52.720 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:52.720 INFO:teuthology.orchestra.run.smithi175.stdout: osd pool default size = 2 2024-04-15T15:13:52.720 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:52.720 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd allow primary affinity = true 2024-04-15T15:13:52.720 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd allow pg remap = true 2024-04-15T15:13:52.720 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on legacy crush tunables = false 2024-04-15T15:13:52.720 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on crush straw calc version zero = false 2024-04-15T15:13:52.720 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on no sortbitwise = false 2024-04-15T15:13:52.722 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on osd down out interval zero = false 2024-04-15T15:13:52.722 INFO:teuthology.orchestra.run.smithi175.stdout: mon warn on too few osds = false 2024-04-15T15:13:52.722 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-15T15:13:52.722 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-15T15:13:52.723 INFO:teuthology.orchestra.run.smithi175.stdout: mon_allow_pool_size_one = true 2024-04-15T15:13:52.723 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:52.723 INFO:teuthology.orchestra.run.smithi175.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-15T15:13:52.723 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:52.723 INFO:teuthology.orchestra.run.smithi175.stdout: osd default data pool replay window = 5 2024-04-15T15:13:52.723 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:52.723 INFO:teuthology.orchestra.run.smithi175.stdout: mon allow pool delete = true 2024-04-15T15:13:52.723 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:52.723 INFO:teuthology.orchestra.run.smithi175.stdout: mon cluster log file level = debug 2024-04-15T15:13:52.723 INFO:teuthology.orchestra.run.smithi175.stdout: debug asserts on shutdown = true 2024-04-15T15:13:52.723 INFO:teuthology.orchestra.run.smithi175.stdout: mon health detail to clog = false 2024-04-15T15:13:52.723 INFO:teuthology.orchestra.run.smithi175.stdout: mon host = "172.21.15.72,172.21.15.175,[v2:172.21.15.175:3301,v1:172.21.15.175:6790]" 2024-04-15T15:13:52.723 INFO:teuthology.orchestra.run.smithi175.stdout: fsid = 05091e39-b18a-4545-8b68-c0ae4699e116 2024-04-15T15:13:52.723 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:52.724 INFO:teuthology.orchestra.run.smithi175.stdout:[osd] 2024-04-15T15:13:52.724 INFO:teuthology.orchestra.run.smithi175.stdout: osd journal size = 100 2024-04-15T15:13:52.724 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:52.724 INFO:teuthology.orchestra.run.smithi175.stdout: osd scrub load threshold = 5.0 2024-04-15T15:13:52.724 INFO:teuthology.orchestra.run.smithi175.stdout: osd scrub max interval = 600 2024-04-15T15:13:52.724 INFO:teuthology.orchestra.run.smithi175.stdout: osd mclock profile = high_recovery_ops 2024-04-15T15:13:52.724 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:52.724 INFO:teuthology.orchestra.run.smithi175.stdout: osd recover clone overlap = true 2024-04-15T15:13:52.725 INFO:teuthology.orchestra.run.smithi175.stdout: osd recovery max chunk = 1048576 2024-04-15T15:13:52.725 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:52.725 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug shutdown = true 2024-04-15T15:13:52.725 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug op order = true 2024-04-15T15:13:52.725 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug verify stray on activate = true 2024-04-15T15:13:52.725 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug trim objects = true 2024-04-15T15:13:52.725 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:52.725 INFO:teuthology.orchestra.run.smithi175.stdout: osd open classes on start = true 2024-04-15T15:13:52.725 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug pg log writeout = true 2024-04-15T15:13:52.725 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:52.725 INFO:teuthology.orchestra.run.smithi175.stdout: osd deep scrub update digest min age = 30 2024-04-15T15:13:52.725 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:52.725 INFO:teuthology.orchestra.run.smithi175.stdout: osd map max advance = 10 2024-04-15T15:13:52.725 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:52.725 INFO:teuthology.orchestra.run.smithi175.stdout: journal zero on create = true 2024-04-15T15:13:52.725 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:52.726 INFO:teuthology.orchestra.run.smithi175.stdout: filestore ondisk finisher threads = 3 2024-04-15T15:13:52.726 INFO:teuthology.orchestra.run.smithi175.stdout: filestore apply finisher threads = 3 2024-04-15T15:13:52.726 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:52.726 INFO:teuthology.orchestra.run.smithi175.stdout: bdev debug aio = true 2024-04-15T15:13:52.726 INFO:teuthology.orchestra.run.smithi175.stdout: osd debug misdirected ops = true 2024-04-15T15:13:52.726 INFO:teuthology.orchestra.run.smithi175.stdout: bluestore block size = 96636764160 2024-04-15T15:13:52.726 INFO:teuthology.orchestra.run.smithi175.stdout: bluestore fsck on mount = True 2024-04-15T15:13:52.726 INFO:teuthology.orchestra.run.smithi175.stdout: debug bluefs = 20 2024-04-15T15:13:52.726 INFO:teuthology.orchestra.run.smithi175.stdout: debug bluestore = 20 2024-04-15T15:13:52.726 INFO:teuthology.orchestra.run.smithi175.stdout: debug ms = 1 2024-04-15T15:13:52.726 INFO:teuthology.orchestra.run.smithi175.stdout: debug osd = 20 2024-04-15T15:13:52.726 INFO:teuthology.orchestra.run.smithi175.stdout: debug rocksdb = 10 2024-04-15T15:13:52.726 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd backfillfull_ratio = 0.85 2024-04-15T15:13:52.726 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd full ratio = 0.9 2024-04-15T15:13:52.726 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd nearfull ratio = 0.8 2024-04-15T15:13:52.727 INFO:teuthology.orchestra.run.smithi175.stdout: osd failsafe full ratio = 0.95 2024-04-15T15:13:52.727 INFO:teuthology.orchestra.run.smithi175.stdout: osd objectstore = bluestore 2024-04-15T15:13:52.727 INFO:teuthology.orchestra.run.smithi175.stdout: osd op complaint time = 180 2024-04-15T15:13:52.727 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:52.727 INFO:teuthology.orchestra.run.smithi175.stdout:[mgr] 2024-04-15T15:13:52.727 INFO:teuthology.orchestra.run.smithi175.stdout: debug ms = 1 2024-04-15T15:13:52.727 INFO:teuthology.orchestra.run.smithi175.stdout: debug mgr = 20 2024-04-15T15:13:52.727 INFO:teuthology.orchestra.run.smithi175.stdout: debug mon = 20 2024-04-15T15:13:52.727 INFO:teuthology.orchestra.run.smithi175.stdout: debug auth = 20 2024-04-15T15:13:52.727 INFO:teuthology.orchestra.run.smithi175.stdout: mon reweight min pgs per osd = 4 2024-04-15T15:13:52.727 INFO:teuthology.orchestra.run.smithi175.stdout: mon reweight min bytes per osd = 10 2024-04-15T15:13:52.727 INFO:teuthology.orchestra.run.smithi175.stdout: mgr/telemetry/nag = false 2024-04-15T15:13:52.727 INFO:teuthology.orchestra.run.smithi175.stdout: debug client = 20 2024-04-15T15:13:52.727 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:52.727 INFO:teuthology.orchestra.run.smithi175.stdout:[mon] 2024-04-15T15:13:52.728 INFO:teuthology.orchestra.run.smithi175.stdout: debug ms = 1 2024-04-15T15:13:52.728 INFO:teuthology.orchestra.run.smithi175.stdout: debug mon = 20 2024-04-15T15:13:52.728 INFO:teuthology.orchestra.run.smithi175.stdout: debug paxos = 20 2024-04-15T15:13:52.728 INFO:teuthology.orchestra.run.smithi175.stdout: debug auth = 20 2024-04-15T15:13:52.728 INFO:teuthology.orchestra.run.smithi175.stdout: mon data avail warn = 5 2024-04-15T15:13:52.728 INFO:teuthology.orchestra.run.smithi175.stdout: mon mgr mkfs grace = 240 2024-04-15T15:13:52.728 INFO:teuthology.orchestra.run.smithi175.stdout: mon reweight min pgs per osd = 4 2024-04-15T15:13:52.728 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd reporter subtree level = osd 2024-04-15T15:13:52.728 INFO:teuthology.orchestra.run.smithi175.stdout: mon osd prime pg temp = true 2024-04-15T15:13:52.728 INFO:teuthology.orchestra.run.smithi175.stdout: mon reweight min bytes per osd = 10 2024-04-15T15:13:52.728 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:52.728 INFO:teuthology.orchestra.run.smithi175.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-15T15:13:52.728 INFO:teuthology.orchestra.run.smithi175.stdout: auth mon ticket ttl = 660 # 11m 2024-04-15T15:13:52.728 INFO:teuthology.orchestra.run.smithi175.stdout: auth service ticket ttl = 240 # 4m 2024-04-15T15:13:52.728 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:52.729 INFO:teuthology.orchestra.run.smithi175.stdout: # don't complain about insecure global_id in the test suite 2024-04-15T15:13:52.729 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-15T15:13:52.729 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-15T15:13:52.729 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:52.729 INFO:teuthology.orchestra.run.smithi175.stdout: # 1m isn't quite enough 2024-04-15T15:13:52.729 INFO:teuthology.orchestra.run.smithi175.stdout: mon_down_mkfs_grace = 2m 2024-04-15T15:13:52.729 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:52.729 INFO:teuthology.orchestra.run.smithi175.stdout: mon_warn_on_filestore_osds = false 2024-04-15T15:13:52.729 INFO:teuthology.orchestra.run.smithi175.stdout: mon down mkfs grace = 300 2024-04-15T15:13:52.729 INFO:teuthology.orchestra.run.smithi175.stdout: mon op complaint time = 120 2024-04-15T15:13:52.729 INFO:teuthology.orchestra.run.smithi175.stdout: 2024-04-15T15:13:52.730 INFO:teuthology.orchestra.run.smithi175.stdout:[client] 2024-04-15T15:13:52.730 INFO:teuthology.orchestra.run.smithi175.stdout: rgw cache enabled = true 2024-04-15T15:13:52.730 INFO:teuthology.orchestra.run.smithi175.stdout: rgw enable ops log = true 2024-04-15T15:13:52.730 INFO:teuthology.orchestra.run.smithi175.stdout: rgw enable usage log = true 2024-04-15T15:13:52.730 INFO:teuthology.orchestra.run.smithi175.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-15T15:13:52.730 INFO:teuthology.orchestra.run.smithi175.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-15T15:13:52.730 INFO:teuthology.orchestra.run.smithi175.stdout: client mount timeout = 600 2024-04-15T15:13:52.730 INFO:teuthology.orchestra.run.smithi175.stdout: debug client = 20 2024-04-15T15:13:52.730 INFO:teuthology.orchestra.run.smithi175.stdout: debug ms = 1 2024-04-15T15:13:52.730 INFO:teuthology.orchestra.run.smithi175.stdout: pid file = "" 2024-04-15T15:13:52.730 INFO:teuthology.orchestra.run.smithi175.stdout: rados mon op timeout = 900 2024-04-15T15:13:52.730 INFO:teuthology.orchestra.run.smithi175.stdout: rados osd op timeout = 900 2024-04-15T15:13:52.730 INFO:teuthology.orchestra.run.smithi175.stdout:[mon.a] 2024-04-15T15:13:52.731 INFO:teuthology.orchestra.run.smithi175.stdout:[mon.b] 2024-04-15T15:13:52.731 INFO:teuthology.orchestra.run.smithi175.stdout:[mon.c] 2024-04-15T15:13:52.731 INFO:teuthology.orchestra.run.smithi175.stdout:[mds.a] 2024-04-15T15:13:52.731 INFO:teuthology.orchestra.run.smithi175.stdout:[mds.c] 2024-04-15T15:13:52.731 INFO:teuthology.orchestra.run.smithi175.stdout:[mds.b] 2024-04-15T15:13:52.731 INFO:teuthology.orchestra.run.smithi175.stdout:[mds.d] 2024-04-15T15:13:52.731 INFO:teuthology.orchestra.run.smithi175.stdout:[mds] 2024-04-15T15:13:52.731 INFO:teuthology.orchestra.run.smithi175.stdout: debug mds = 20 2024-04-15T15:13:52.731 INFO:teuthology.orchestra.run.smithi175.stdout: debug mds balancer = 20 2024-04-15T15:13:52.731 INFO:teuthology.orchestra.run.smithi175.stdout: debug ms = 1 2024-04-15T15:13:52.731 INFO:teuthology.orchestra.run.smithi175.stdout: mds debug frag = True 2024-04-15T15:13:52.731 INFO:teuthology.orchestra.run.smithi175.stdout: mds debug scatterstat = True 2024-04-15T15:13:52.732 INFO:teuthology.orchestra.run.smithi175.stdout: mds op complaint time = 180 2024-04-15T15:13:52.733 INFO:teuthology.orchestra.run.smithi175.stdout: mds verify scatter = True 2024-04-15T15:13:52.733 INFO:teuthology.orchestra.run.smithi175.stdout: osd op complaint time = 180 2024-04-15T15:13:52.733 INFO:teuthology.orchestra.run.smithi175.stdout: rados mon op timeout = 900 2024-04-15T15:13:52.733 INFO:teuthology.orchestra.run.smithi175.stdout: rados osd op timeout = 900 2024-04-15T15:13:52.734 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-15T15:13:52.735 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-15T15:13:52.832 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-15T15:13:52.832 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:13:52.832 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-15T15:13:52.848 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:13:52.848 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-15T15:13:52.904 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi072.front.sepia.ceph.com 2024-04-15T15:13:52.904 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:13:52.904 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-15T15:13:52.904 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-15T15:13:52.985 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:13:52.985 DEBUG:teuthology.orchestra.run.smithi072:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:13:53.041 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi175.front.sepia.ceph.com 2024-04-15T15:13:53.041 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:13:53.041 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-15T15:13:53.041 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-15T15:13:53.084 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:13:53.084 DEBUG:teuthology.orchestra.run.smithi175:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:13:53.139 INFO:tasks.ceph:Setting up mon nodes... 2024-04-15T15:13:53.139 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-15T15:13:53.140 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-15T15:13:53.196 INFO:teuthology.orchestra.run.smithi072.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-15T15:13:53.199 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-15T15:13:53.262 INFO:teuthology.orchestra.run.smithi175.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-15T15:13:53.265 INFO:tasks.ceph:Setting up mds nodes... 2024-04-15T15:13:53.265 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-04-15T15:13:53.323 INFO:teuthology.orchestra.run.smithi072.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-04-15T15:13:53.326 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-04-15T15:13:53.355 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-04-15T15:13:53.452 INFO:teuthology.orchestra.run.smithi072.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-04-15T15:13:53.455 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-04-15T15:13:53.484 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-04-15T15:13:53.542 INFO:teuthology.orchestra.run.smithi175.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-04-15T15:13:53.545 DEBUG:teuthology.orchestra.run.smithi175:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-04-15T15:13:53.574 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-04-15T15:13:53.672 INFO:teuthology.orchestra.run.smithi175.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-04-15T15:13:53.674 DEBUG:teuthology.orchestra.run.smithi175:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-04-15T15:13:53.702 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-15T15:13:53.703 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-15T15:13:53.746 INFO:teuthology.orchestra.run.smithi072.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-15T15:13:53.763 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-04-15T15:13:53.806 INFO:teuthology.orchestra.run.smithi072.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-04-15T15:13:53.822 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-15T15:13:53.871 INFO:teuthology.orchestra.run.smithi175.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-15T15:13:53.889 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-04-15T15:13:53.935 INFO:teuthology.orchestra.run.smithi175.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-04-15T15:13:53.951 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-15T15:13:53.951 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi072.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi175.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-15T15:13:53.951 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-15T15:13:53.979 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-15T15:13:53.979 INFO:tasks.ceph:role: osd.0 2024-04-15T15:13:53.979 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi072.front.sepia.ceph.com 2024-04-15T15:13:53.979 DEBUG:teuthology.orchestra.run.smithi072:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-15T15:13:54.054 INFO:teuthology.orchestra.run.smithi072.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-15T15:13:54.054 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-15T15:13:54.054 INFO:teuthology.orchestra.run.smithi072.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-15T15:13:54.054 INFO:teuthology.orchestra.run.smithi072.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-15T15:13:54.054 INFO:teuthology.orchestra.run.smithi072.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-15T15:13:54.054 INFO:teuthology.orchestra.run.smithi072.stdout: = sunit=0 swidth=0 blks 2024-04-15T15:13:54.055 INFO:teuthology.orchestra.run.smithi072.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-15T15:13:54.055 INFO:teuthology.orchestra.run.smithi072.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-15T15:13:54.055 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-15T15:13:54.055 INFO:teuthology.orchestra.run.smithi072.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-15T15:13:54.538 INFO:teuthology.orchestra.run.smithi072.stdout:Discarding blocks...Done. 2024-04-15T15:13:54.540 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi072.front.sepia.ceph.com -o noatime 2024-04-15T15:13:54.540 DEBUG:teuthology.orchestra.run.smithi072:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-15T15:13:54.644 DEBUG:teuthology.orchestra.run.smithi072:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-15T15:13:54.678 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-15T15:13:54.747 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-15T15:13:54.747 INFO:tasks.ceph:role: osd.1 2024-04-15T15:13:54.747 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi072.front.sepia.ceph.com 2024-04-15T15:13:54.747 DEBUG:teuthology.orchestra.run.smithi072:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-15T15:13:54.823 INFO:teuthology.orchestra.run.smithi072.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-15T15:13:54.823 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-15T15:13:54.823 INFO:teuthology.orchestra.run.smithi072.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-15T15:13:54.823 INFO:teuthology.orchestra.run.smithi072.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-15T15:13:54.823 INFO:teuthology.orchestra.run.smithi072.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-15T15:13:54.823 INFO:teuthology.orchestra.run.smithi072.stdout: = sunit=0 swidth=0 blks 2024-04-15T15:13:54.824 INFO:teuthology.orchestra.run.smithi072.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-15T15:13:54.824 INFO:teuthology.orchestra.run.smithi072.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-15T15:13:54.824 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-15T15:13:54.824 INFO:teuthology.orchestra.run.smithi072.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-15T15:13:55.286 INFO:teuthology.orchestra.run.smithi072.stdout:Discarding blocks...Done. 2024-04-15T15:13:55.288 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi072.front.sepia.ceph.com -o noatime 2024-04-15T15:13:55.288 DEBUG:teuthology.orchestra.run.smithi072:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-15T15:13:55.366 DEBUG:teuthology.orchestra.run.smithi072:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-15T15:13:55.440 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-15T15:13:55.509 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-15T15:13:55.509 INFO:tasks.ceph:role: osd.2 2024-04-15T15:13:55.509 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi072.front.sepia.ceph.com 2024-04-15T15:13:55.509 DEBUG:teuthology.orchestra.run.smithi072:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-15T15:13:55.585 INFO:teuthology.orchestra.run.smithi072.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-15T15:13:55.585 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-15T15:13:55.586 INFO:teuthology.orchestra.run.smithi072.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-15T15:13:55.586 INFO:teuthology.orchestra.run.smithi072.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-15T15:13:55.586 INFO:teuthology.orchestra.run.smithi072.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-15T15:13:55.586 INFO:teuthology.orchestra.run.smithi072.stdout: = sunit=0 swidth=0 blks 2024-04-15T15:13:55.586 INFO:teuthology.orchestra.run.smithi072.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-15T15:13:55.586 INFO:teuthology.orchestra.run.smithi072.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-15T15:13:55.586 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-15T15:13:55.586 INFO:teuthology.orchestra.run.smithi072.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-15T15:13:56.070 INFO:teuthology.orchestra.run.smithi072.stdout:Discarding blocks...Done. 2024-04-15T15:13:56.072 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi072.front.sepia.ceph.com -o noatime 2024-04-15T15:13:56.072 DEBUG:teuthology.orchestra.run.smithi072:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-15T15:13:56.167 DEBUG:teuthology.orchestra.run.smithi072:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-15T15:13:56.201 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-15T15:13:56.269 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-15T15:13:56.269 INFO:tasks.ceph:role: osd.3 2024-04-15T15:13:56.270 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi072.front.sepia.ceph.com 2024-04-15T15:13:56.270 DEBUG:teuthology.orchestra.run.smithi072:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-15T15:13:56.346 INFO:teuthology.orchestra.run.smithi072.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-15T15:13:56.346 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-15T15:13:56.346 INFO:teuthology.orchestra.run.smithi072.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-15T15:13:56.346 INFO:teuthology.orchestra.run.smithi072.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-15T15:13:56.346 INFO:teuthology.orchestra.run.smithi072.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-15T15:13:56.346 INFO:teuthology.orchestra.run.smithi072.stdout: = sunit=0 swidth=0 blks 2024-04-15T15:13:56.346 INFO:teuthology.orchestra.run.smithi072.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-15T15:13:56.346 INFO:teuthology.orchestra.run.smithi072.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-15T15:13:56.346 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-15T15:13:56.346 INFO:teuthology.orchestra.run.smithi072.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-15T15:13:56.867 INFO:teuthology.orchestra.run.smithi072.stdout:Discarding blocks...Done. 2024-04-15T15:13:56.869 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi072.front.sepia.ceph.com -o noatime 2024-04-15T15:13:56.869 DEBUG:teuthology.orchestra.run.smithi072:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-15T15:13:57.008 DEBUG:teuthology.orchestra.run.smithi072:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-15T15:13:57.043 DEBUG:teuthology.orchestra.run.smithi072:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:13:57.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:13:57.141+0000 7f32dc689640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-15T15:13:57.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:13:57.143+0000 7f32dc689640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-15T15:13:57.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:13:57.143+0000 7f32dc689640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-15T15:13:57.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:13:57.143+0000 7f32dc689640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-15T15:13:57.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:13:57.143+0000 7f32dc689640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-15T15:13:57.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:13:57.151+0000 7f32dc689640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-15T15:14:00.001 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-15T15:14:00.030 DEBUG:teuthology.orchestra.run.smithi072:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:14:00.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:00.117+0000 7fef8eb38640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-15T15:14:00.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:00.118+0000 7fef8eb38640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-15T15:14:00.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:00.118+0000 7fef8eb38640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-15T15:14:00.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:00.118+0000 7fef8eb38640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-15T15:14:00.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:00.118+0000 7fef8eb38640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-15T15:14:00.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:00.118+0000 7fef8eb38640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-15T15:14:02.970 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-15T15:14:03.003 DEBUG:teuthology.orchestra.run.smithi072:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:14:03.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:03.096+0000 7f68df6d7640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-15T15:14:03.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:03.096+0000 7f68df6d7640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-15T15:14:03.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:03.096+0000 7f68df6d7640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-15T15:14:03.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:03.096+0000 7f68df6d7640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-15T15:14:03.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:03.096+0000 7f68df6d7640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-15T15:14:03.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:03.097+0000 7f68df6d7640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-15T15:14:05.951 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-15T15:14:05.984 DEBUG:teuthology.orchestra.run.smithi072:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:14:06.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:06.075+0000 7f5d1280d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-15T15:14:06.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:06.075+0000 7f5d1280d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-15T15:14:06.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:06.075+0000 7f5d1280d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-15T15:14:06.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:06.075+0000 7f5d1280d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-15T15:14:06.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:06.075+0000 7f5d1280d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-15T15:14:06.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:06.076+0000 7f5d1280d640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-15T15:14:08.929 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-15T15:14:08.962 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-15T15:14:08.997 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-15T15:14:08.997 INFO:tasks.ceph:role: osd.4 2024-04-15T15:14:08.997 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi175.front.sepia.ceph.com 2024-04-15T15:14:08.997 DEBUG:teuthology.orchestra.run.smithi175:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-15T15:14:09.075 INFO:teuthology.orchestra.run.smithi175.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-15T15:14:09.076 INFO:teuthology.orchestra.run.smithi175.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-15T15:14:09.076 INFO:teuthology.orchestra.run.smithi175.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-15T15:14:09.076 INFO:teuthology.orchestra.run.smithi175.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-15T15:14:09.076 INFO:teuthology.orchestra.run.smithi175.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-15T15:14:09.076 INFO:teuthology.orchestra.run.smithi175.stdout: = sunit=0 swidth=0 blks 2024-04-15T15:14:09.076 INFO:teuthology.orchestra.run.smithi175.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-15T15:14:09.076 INFO:teuthology.orchestra.run.smithi175.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-15T15:14:09.076 INFO:teuthology.orchestra.run.smithi175.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-15T15:14:09.076 INFO:teuthology.orchestra.run.smithi175.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-15T15:14:09.578 INFO:teuthology.orchestra.run.smithi175.stdout:Discarding blocks...Done. 2024-04-15T15:14:09.580 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi175.front.sepia.ceph.com -o noatime 2024-04-15T15:14:09.581 DEBUG:teuthology.orchestra.run.smithi175:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-15T15:14:09.712 DEBUG:teuthology.orchestra.run.smithi175:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-15T15:14:09.750 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-15T15:14:09.822 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-15T15:14:09.822 INFO:tasks.ceph:role: osd.5 2024-04-15T15:14:09.822 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi175.front.sepia.ceph.com 2024-04-15T15:14:09.823 DEBUG:teuthology.orchestra.run.smithi175:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-15T15:14:09.899 INFO:teuthology.orchestra.run.smithi175.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-15T15:14:09.899 INFO:teuthology.orchestra.run.smithi175.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-15T15:14:09.899 INFO:teuthology.orchestra.run.smithi175.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-15T15:14:09.899 INFO:teuthology.orchestra.run.smithi175.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-15T15:14:09.900 INFO:teuthology.orchestra.run.smithi175.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-15T15:14:09.900 INFO:teuthology.orchestra.run.smithi175.stdout: = sunit=0 swidth=0 blks 2024-04-15T15:14:09.900 INFO:teuthology.orchestra.run.smithi175.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-15T15:14:09.900 INFO:teuthology.orchestra.run.smithi175.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-15T15:14:09.900 INFO:teuthology.orchestra.run.smithi175.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-15T15:14:09.900 INFO:teuthology.orchestra.run.smithi175.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-15T15:14:10.354 INFO:teuthology.orchestra.run.smithi175.stdout:Discarding blocks...Done. 2024-04-15T15:14:10.356 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi175.front.sepia.ceph.com -o noatime 2024-04-15T15:14:10.357 DEBUG:teuthology.orchestra.run.smithi175:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-15T15:14:10.455 DEBUG:teuthology.orchestra.run.smithi175:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-15T15:14:10.489 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-15T15:14:10.558 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-15T15:14:10.558 INFO:tasks.ceph:role: osd.6 2024-04-15T15:14:10.559 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi175.front.sepia.ceph.com 2024-04-15T15:14:10.559 DEBUG:teuthology.orchestra.run.smithi175:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-15T15:14:10.634 INFO:teuthology.orchestra.run.smithi175.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-15T15:14:10.635 INFO:teuthology.orchestra.run.smithi175.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-15T15:14:10.635 INFO:teuthology.orchestra.run.smithi175.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-15T15:14:10.635 INFO:teuthology.orchestra.run.smithi175.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-15T15:14:10.635 INFO:teuthology.orchestra.run.smithi175.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-15T15:14:10.635 INFO:teuthology.orchestra.run.smithi175.stdout: = sunit=0 swidth=0 blks 2024-04-15T15:14:10.635 INFO:teuthology.orchestra.run.smithi175.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-15T15:14:10.635 INFO:teuthology.orchestra.run.smithi175.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-15T15:14:10.635 INFO:teuthology.orchestra.run.smithi175.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-15T15:14:10.635 INFO:teuthology.orchestra.run.smithi175.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-15T15:14:11.113 INFO:teuthology.orchestra.run.smithi175.stdout:Discarding blocks...Done. 2024-04-15T15:14:11.116 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi175.front.sepia.ceph.com -o noatime 2024-04-15T15:14:11.116 DEBUG:teuthology.orchestra.run.smithi175:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-15T15:14:11.272 DEBUG:teuthology.orchestra.run.smithi175:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-15T15:14:11.310 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-15T15:14:11.379 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-15T15:14:11.379 INFO:tasks.ceph:role: osd.7 2024-04-15T15:14:11.379 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi175.front.sepia.ceph.com 2024-04-15T15:14:11.380 DEBUG:teuthology.orchestra.run.smithi175:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-15T15:14:11.456 INFO:teuthology.orchestra.run.smithi175.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-15T15:14:11.456 INFO:teuthology.orchestra.run.smithi175.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-15T15:14:11.456 INFO:teuthology.orchestra.run.smithi175.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-15T15:14:11.456 INFO:teuthology.orchestra.run.smithi175.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-15T15:14:11.456 INFO:teuthology.orchestra.run.smithi175.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-15T15:14:11.456 INFO:teuthology.orchestra.run.smithi175.stdout: = sunit=0 swidth=0 blks 2024-04-15T15:14:11.456 INFO:teuthology.orchestra.run.smithi175.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-15T15:14:11.456 INFO:teuthology.orchestra.run.smithi175.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-15T15:14:11.457 INFO:teuthology.orchestra.run.smithi175.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-15T15:14:11.457 INFO:teuthology.orchestra.run.smithi175.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-15T15:14:11.958 INFO:teuthology.orchestra.run.smithi175.stdout:Discarding blocks...Done. 2024-04-15T15:14:11.960 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi175.front.sepia.ceph.com -o noatime 2024-04-15T15:14:11.960 DEBUG:teuthology.orchestra.run.smithi175:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-15T15:14:12.092 DEBUG:teuthology.orchestra.run.smithi175:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-15T15:14:12.128 DEBUG:teuthology.orchestra.run.smithi175:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:14:12.231 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:12.228+0000 7fa6f9d47640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-15T15:14:12.231 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:12.229+0000 7fa6f9d47640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-15T15:14:12.231 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:12.229+0000 7fa6f9d47640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-15T15:14:12.231 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:12.229+0000 7fa6f9d47640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-15T15:14:12.231 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:12.229+0000 7fa6f9d47640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-15T15:14:12.231 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:12.229+0000 7fa6f9d47640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-15T15:14:15.079 DEBUG:teuthology.orchestra.run.smithi175:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-15T15:14:15.113 DEBUG:teuthology.orchestra.run.smithi175:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:14:15.206 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:15.204+0000 7f4d3bd38640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-15T15:14:15.207 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:15.204+0000 7f4d3bd38640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-15T15:14:15.207 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:15.204+0000 7f4d3bd38640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-15T15:14:15.207 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:15.205+0000 7f4d3bd38640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-15T15:14:15.207 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:15.205+0000 7f4d3bd38640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-15T15:14:15.207 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:15.205+0000 7f4d3bd38640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-15T15:14:18.050 DEBUG:teuthology.orchestra.run.smithi175:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-15T15:14:18.081 DEBUG:teuthology.orchestra.run.smithi175:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:14:18.173 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:18.171+0000 7f5272ca2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-15T15:14:18.173 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:18.171+0000 7f5272ca2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-15T15:14:18.173 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:18.171+0000 7f5272ca2640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-15T15:14:18.173 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:18.171+0000 7f5272ca2640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-15T15:14:18.173 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:18.171+0000 7f5272ca2640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-15T15:14:18.174 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:18.172+0000 7f5272ca2640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-15T15:14:20.704 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:14:20.707 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:14:21.026 DEBUG:teuthology.orchestra.run.smithi175:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-15T15:14:21.056 DEBUG:teuthology.orchestra.run.smithi175:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:14:21.147 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:21.145+0000 7f274501c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-15T15:14:21.147 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:21.145+0000 7f274501c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-15T15:14:21.147 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:21.145+0000 7f274501c640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-15T15:14:21.147 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:21.145+0000 7f274501c640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-15T15:14:21.148 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:21.145+0000 7f274501c640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-15T15:14:21.148 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:21.146+0000 7f274501c640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-15T15:14:24.006 DEBUG:teuthology.orchestra.run.smithi175:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-15T15:14:24.036 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-15T15:14:24.036 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:14:24.036 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-15T15:14:24.070 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:14:24.070 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-04-15T15:14:24.142 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:14:24.142 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-04-15T15:14:24.213 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:14:24.213 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-15T15:14:24.283 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:14:24.283 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-15T15:14:24.352 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:14:24.352 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-15T15:14:24.420 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:14:24.421 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-15T15:14:24.488 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:14:24.488 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-15T15:14:24.519 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:14:24.520 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-04-15T15:14:24.591 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:14:24.591 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-04-15T15:14:24.660 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:14:24.661 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-15T15:14:24.729 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:14:24.730 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-15T15:14:24.798 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:14:24.798 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-15T15:14:24.867 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:14:24.867 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-15T15:14:24.935 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:14:24.936 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-15T15:14:24.952 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:14:24.952 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-04-15T15:14:25.008 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:14:25.008 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-15T15:14:25.024 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:14:25.024 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-04-15T15:14:25.079 INFO:tasks.ceph:Adding keys to all mons... 2024-04-15T15:14:25.079 DEBUG:teuthology.orchestra.run.smithi072:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-15T15:14:25.082 DEBUG:teuthology.orchestra.run.smithi175:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-15T15:14:25.123 INFO:teuthology.orchestra.run.smithi072.stdout:[mgr.y] 2024-04-15T15:14:25.123 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQAxRB1mprGxCxAARc8uGtfv/vNbLoNQsdp9eA== 2024-04-15T15:14:25.123 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.a] 2024-04-15T15:14:25.123 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQAxRB1mb8hFExAA+z3Hr0Di3fl2HSUuF4loaQ== 2024-04-15T15:14:25.123 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.c] 2024-04-15T15:14:25.124 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQAxRB1mnrrtGhAAXFFmF2XPvAGcCa4ubdIbsA== 2024-04-15T15:14:25.124 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.0] 2024-04-15T15:14:25.124 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQA1RB1mwxiNCBAAsgQtq6EfdWgfqR2lHVYO/A== 2024-04-15T15:14:25.124 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.1] 2024-04-15T15:14:25.124 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQA4RB1mTmAgBxAAvuGNnF95Ma3ts7xme4REzg== 2024-04-15T15:14:25.124 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.2] 2024-04-15T15:14:25.124 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQA7RB1mKV/EBRAATVdq1Hth//pt0NhGibGDyg== 2024-04-15T15:14:25.124 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.3] 2024-04-15T15:14:25.124 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQA+RB1mpLCJBBAAmJmD6JCL4sA8dbrBkcWaoQ== 2024-04-15T15:14:25.124 INFO:teuthology.orchestra.run.smithi072.stdout:[mgr.x] 2024-04-15T15:14:25.125 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQAxRB1mjTijDxAAFiyrGLVXMufbRf+Jjf3edg== 2024-04-15T15:14:25.125 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.b] 2024-04-15T15:14:25.125 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQAxRB1mmEhOIBAA3r0+6X19RkbYSMGKLjo4wQ== 2024-04-15T15:14:25.125 INFO:teuthology.orchestra.run.smithi072.stdout:[mds.d] 2024-04-15T15:14:25.125 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQAxRB1mczkKKBAAmqVjJjk52ob0NzjfIVsMsg== 2024-04-15T15:14:25.125 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.4] 2024-04-15T15:14:25.125 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBERB1mz26yDRAAiMp1uUjnORA7M3jIe7cL+A== 2024-04-15T15:14:25.125 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.5] 2024-04-15T15:14:25.125 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBHRB1mdaVCDBAAG7XtCTHarMsw41QSGKWT7w== 2024-04-15T15:14:25.125 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.6] 2024-04-15T15:14:25.125 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBKRB1mBadEChAAPPP/cq0rfWeb0vI0oEhy9Q== 2024-04-15T15:14:25.125 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.7] 2024-04-15T15:14:25.125 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBNRB1m1FK8CBAABbyg/11z7CwNO2k948Z4/Q== 2024-04-15T15:14:25.126 INFO:teuthology.orchestra.run.smithi072.stdout:[client.0] 2024-04-15T15:14:25.126 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQAxRB1mycl8LBAA87k203Ws8lFHsc/7l5rqbg== 2024-04-15T15:14:25.126 INFO:teuthology.orchestra.run.smithi072.stdout:[client.2] 2024-04-15T15:14:25.126 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQAxRB1m2XANMBAAc5PeZVI7uPVoptzUL2mlEQ== 2024-04-15T15:14:25.126 INFO:teuthology.orchestra.run.smithi072.stdout:[client.1] 2024-04-15T15:14:25.126 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQAxRB1mx0/iMxAAbLexbWtlBoCaESA8AH7sFg== 2024-04-15T15:14:25.126 INFO:teuthology.orchestra.run.smithi072.stdout:[client.3] 2024-04-15T15:14:25.126 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQAxRB1mF8myNxAADFNnmx+5pm9grpwJK3K3lw== 2024-04-15T15:14:25.148 INFO:teuthology.orchestra.run.smithi175.stdout:[mgr.y] 2024-04-15T15:14:25.148 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQAxRB1mprGxCxAARc8uGtfv/vNbLoNQsdp9eA== 2024-04-15T15:14:25.148 INFO:teuthology.orchestra.run.smithi175.stdout:[mds.a] 2024-04-15T15:14:25.148 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQAxRB1mb8hFExAA+z3Hr0Di3fl2HSUuF4loaQ== 2024-04-15T15:14:25.148 INFO:teuthology.orchestra.run.smithi175.stdout:[mds.c] 2024-04-15T15:14:25.148 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQAxRB1mnrrtGhAAXFFmF2XPvAGcCa4ubdIbsA== 2024-04-15T15:14:25.148 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.0] 2024-04-15T15:14:25.148 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQA1RB1mwxiNCBAAsgQtq6EfdWgfqR2lHVYO/A== 2024-04-15T15:14:25.148 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.1] 2024-04-15T15:14:25.149 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQA4RB1mTmAgBxAAvuGNnF95Ma3ts7xme4REzg== 2024-04-15T15:14:25.149 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.2] 2024-04-15T15:14:25.149 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQA7RB1mKV/EBRAATVdq1Hth//pt0NhGibGDyg== 2024-04-15T15:14:25.149 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.3] 2024-04-15T15:14:25.149 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQA+RB1mpLCJBBAAmJmD6JCL4sA8dbrBkcWaoQ== 2024-04-15T15:14:25.149 INFO:teuthology.orchestra.run.smithi175.stdout:[mgr.x] 2024-04-15T15:14:25.149 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQAxRB1mjTijDxAAFiyrGLVXMufbRf+Jjf3edg== 2024-04-15T15:14:25.149 INFO:teuthology.orchestra.run.smithi175.stdout:[mds.b] 2024-04-15T15:14:25.149 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQAxRB1mmEhOIBAA3r0+6X19RkbYSMGKLjo4wQ== 2024-04-15T15:14:25.149 INFO:teuthology.orchestra.run.smithi175.stdout:[mds.d] 2024-04-15T15:14:25.149 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQAxRB1mczkKKBAAmqVjJjk52ob0NzjfIVsMsg== 2024-04-15T15:14:25.149 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.4] 2024-04-15T15:14:25.149 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQBERB1mz26yDRAAiMp1uUjnORA7M3jIe7cL+A== 2024-04-15T15:14:25.149 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.5] 2024-04-15T15:14:25.149 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQBHRB1mdaVCDBAAG7XtCTHarMsw41QSGKWT7w== 2024-04-15T15:14:25.150 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.6] 2024-04-15T15:14:25.150 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQBKRB1mBadEChAAPPP/cq0rfWeb0vI0oEhy9Q== 2024-04-15T15:14:25.150 INFO:teuthology.orchestra.run.smithi175.stdout:[osd.7] 2024-04-15T15:14:25.150 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQBNRB1m1FK8CBAABbyg/11z7CwNO2k948Z4/Q== 2024-04-15T15:14:25.150 INFO:teuthology.orchestra.run.smithi175.stdout:[client.0] 2024-04-15T15:14:25.150 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQAxRB1mycl8LBAA87k203Ws8lFHsc/7l5rqbg== 2024-04-15T15:14:25.150 INFO:teuthology.orchestra.run.smithi175.stdout:[client.2] 2024-04-15T15:14:25.150 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQAxRB1m2XANMBAAc5PeZVI7uPVoptzUL2mlEQ== 2024-04-15T15:14:25.150 INFO:teuthology.orchestra.run.smithi175.stdout:[client.1] 2024-04-15T15:14:25.150 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQAxRB1mx0/iMxAAbLexbWtlBoCaESA8AH7sFg== 2024-04-15T15:14:25.150 INFO:teuthology.orchestra.run.smithi175.stdout:[client.3] 2024-04-15T15:14:25.150 INFO:teuthology.orchestra.run.smithi175.stdout: key = AQAxRB1mF8myNxAADFNnmx+5pm9grpwJK3K3lw== 2024-04-15T15:14:25.151 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-15T15:14:25.170 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-15T15:14:25.242 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-15T15:14:25.245 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-15T15:14:25.302 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-15T15:14:25.304 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-15T15:14:25.363 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:14:25.365 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:14:25.419 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:14:25.421 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:14:25.475 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:14:25.477 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:14:25.531 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:14:25.533 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:14:25.587 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-15T15:14:25.589 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-15T15:14:25.643 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-15T15:14:25.645 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-15T15:14:25.702 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-15T15:14:25.704 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-15T15:14:25.763 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:14:25.767 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:14:25.821 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:14:25.823 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:14:25.876 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:14:25.879 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:14:25.933 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:14:25.935 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:14:25.989 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-15T15:14:25.992 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-15T15:14:26.049 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-15T15:14:26.052 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-15T15:14:26.108 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-15T15:14:26.111 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-15T15:14:26.168 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-15T15:14:26.171 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-15T15:14:26.231 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-15T15:14:26.231 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-15T15:14:26.261 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-15T15:14:26.361 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-15T15:14:26.390 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-15T15:14:26.418 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-15T15:14:26.517 DEBUG:teuthology.orchestra.run.smithi175:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-15T15:14:26.545 DEBUG:teuthology.orchestra.run.smithi175:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-15T15:14:26.614 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-15T15:14:26.714 DEBUG:teuthology.orchestra.run.smithi175:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-15T15:14:26.742 DEBUG:teuthology.orchestra.run.smithi072:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:14:26.744 DEBUG:teuthology.orchestra.run.smithi175:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:14:26.801 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-15T15:14:26.801 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-15T15:14:26.802 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-15T15:14:26.804 INFO:tasks.ceph.mon.a:Started 2024-04-15T15:14:26.804 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-15T15:14:26.804 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-15T15:14:26.844 INFO:tasks.ceph.mon.b:Started 2024-04-15T15:14:26.844 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-15T15:14:26.844 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-15T15:14:26.847 INFO:tasks.ceph.mon.c:Started 2024-04-15T15:14:26.847 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-15T15:14:26.847 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-15T15:14:26.847 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-15T15:14:26.849 INFO:tasks.ceph.mgr.y:Started 2024-04-15T15:14:26.849 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-15T15:14:26.849 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-15T15:14:26.851 INFO:tasks.ceph.mgr.x:Started 2024-04-15T15:14:26.851 INFO:tasks.ceph:Setting crush tunables to default 2024-04-15T15:14:26.852 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --cluster ceph osd crush tunables default 2024-04-15T15:14:26.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:26.954+0000 7f8e4850d640 1 Processor -- start 2024-04-15T15:14:26.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:26.954+0000 7f8e4850d640 1 -- start start 2024-04-15T15:14:26.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:26.955+0000 7f8e4850d640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8e4015f9a0 0x7f8e4006f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:26.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:26.955+0000 7f8e4850d640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8e400b5fc0 0x7f8e400b6450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:26.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:26.955+0000 7f8e4850d640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e4015fe50 con 0x7f8e40158fd0 2024-04-15T15:14:26.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:26.955+0000 7f8e4850d640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f8e401593b0 con 0x7f8e4015f9a0 2024-04-15T15:14:26.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:26.955+0000 7f8e4850d640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f8e400b58d0 con 0x7f8e400b5fc0 2024-04-15T15:14:26.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:26.955+0000 7f8e45aa8640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8e4015f9a0 0x7f8e4006f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:26.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:26.955+0000 7f8e46aaa640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8e400b5fc0 0x7f8e400b6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:26.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:26.955+0000 7f8e46aaa640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8e400b5fc0 0x7f8e400b6450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3301/0 says I am v2:172.21.15.72:44244/0 (socket says 172.21.15.72:44244) 2024-04-15T15:14:26.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:26.955+0000 7f8e45aa8640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8e4015f9a0 0x7f8e4006f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.72:49986/0 (socket says 172.21.15.72:49986) 2024-04-15T15:14:26.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:26.955+0000 7f8e462a9640 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f8e40158fd0 0x7f8e400b51c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.72:43172/0 (socket says 172.21.15.72:43172) 2024-04-15T15:14:26.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:26.955+0000 7f8e46aaa640 1 -- 172.21.15.72:0/123959349 learned_addr learned my addr 172.21.15.72:0/123959349 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:26.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:26.955+0000 7f8e46aaa640 1 -- 172.21.15.72:0/123959349 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8e400b5fc0 msgr2=0x7f8e400b6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-15T15:14:26.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:26.955+0000 7f8e46aaa640 1 -- 172.21.15.72:0/123959349 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8e400b5fc0 msgr2=0x7f8e400b6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-15T15:14:26.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:26.955+0000 7f8e46aaa640 1 --2- 172.21.15.72:0/123959349 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8e400b5fc0 0x7f8e400b6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-15T15:14:26.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:26.955+0000 7f8e45aa8640 1 -- 172.21.15.72:0/123959349 >> v2:172.21.15.175:3300/0 conn(0x7f8e4015f9a0 msgr2=0x7f8e4006f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-04-15T15:14:26.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:26.955+0000 7f8e46aaa640 1 --2- 172.21.15.72:0/123959349 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8e400b5fc0 0x7f8e400b6450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-15T15:14:26.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:26.955+0000 7f8e45aa8640 1 -- 172.21.15.72:0/123959349 >> v2:172.21.15.175:3300/0 conn(0x7f8e4015f9a0 msgr2=0x7f8e4006f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-15T15:14:26.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:26.955+0000 7f8e45aa8640 1 --2- 172.21.15.72:0/123959349 >> v2:172.21.15.175:3300/0 conn(0x7f8e4015f9a0 0x7f8e4006f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-15T15:14:26.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:26.955+0000 7f8e45aa8640 1 --2- 172.21.15.72:0/123959349 >> v2:172.21.15.175:3300/0 conn(0x7f8e4015f9a0 0x7f8e4006f6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-15T15:14:27.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.120+0000 7f8e452a7640 1 -- 172.21.15.72:0/123959349 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 646477279 0 0) 0x7f8e3c001110 con 0x7f8e40158fd0 2024-04-15T15:14:27.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.120+0000 7f8e452a7640 1 -- 172.21.15.72:0/123959349 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e2c003520 con 0x7f8e40158fd0 2024-04-15T15:14:27.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.123+0000 7f8e452a7640 1 -- 172.21.15.72:0/123959349 <== mon.1 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2548716371 0 0) 0x7f8e2c003520 con 0x7f8e40158fd0 2024-04-15T15:14:27.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.123+0000 7f8e452a7640 1 -- 172.21.15.72:0/123959349 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8e400b5fc0 msgr2=0x7f8e400b6450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-15T15:14:27.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.123+0000 7f8e452a7640 1 --2- 172.21.15.72:0/123959349 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8e400b5fc0 0x7f8e400b6450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:27.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.123+0000 7f8e452a7640 1 -- 172.21.15.72:0/123959349 >> v2:172.21.15.175:3300/0 conn(0x7f8e4015f9a0 msgr2=0x7f8e4006f6e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-15T15:14:27.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.123+0000 7f8e452a7640 1 --2- 172.21.15.72:0/123959349 >> v2:172.21.15.175:3300/0 conn(0x7f8e4015f9a0 0x7f8e4006f6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:27.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.123+0000 7f8e452a7640 1 -- 172.21.15.72:0/123959349 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e400b4400 con 0x7f8e40158fd0 2024-04-15T15:14:27.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.123+0000 7f8e452a7640 1 -- 172.21.15.72:0/123959349 <== mon.1 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f8e3c001840 con 0x7f8e40158fd0 2024-04-15T15:14:27.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.123+0000 7f8e452a7640 1 -- 172.21.15.72:0/123959349 >> v1:172.21.15.175:6789/0 conn(0x7f8e40158fd0 legacy=0x7f8e400b51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:27.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.123+0000 7f8e452a7640 1 --2- 172.21.15.72:0/123959349 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e2c0039b0 0x7f8e2c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:27.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.124+0000 7f8e452a7640 1 --2- 172.21.15.72:0/123959349 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8e2c004800 0x7f8e2c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:27.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.124+0000 7f8e452a7640 1 --2- 172.21.15.72:0/123959349 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8e400b5fc0 0x7f8e2c0051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:27.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.124+0000 7f8e452a7640 1 -- 172.21.15.72:0/123959349 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8e401593b0 con 0x7f8e2c0039b0 2024-04-15T15:14:27.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.124+0000 7f8e452a7640 1 -- 172.21.15.72:0/123959349 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f8e400b58d0 con 0x7f8e2c004800 2024-04-15T15:14:27.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.124+0000 7f8e452a7640 1 -- 172.21.15.72:0/123959349 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f8e2c00bb50 con 0x7f8e400b5fc0 2024-04-15T15:14:27.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.124+0000 7f8e462a9640 1 --2- 172.21.15.72:0/123959349 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e2c0039b0 0x7f8e2c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:27.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.124+0000 7f8e45aa8640 1 --2- 172.21.15.72:0/123959349 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8e2c004800 0x7f8e2c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:27.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.124+0000 7f8e46aaa640 1 --2- 172.21.15.72:0/123959349 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8e400b5fc0 0x7f8e2c0051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:27.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.124+0000 7f8e45aa8640 1 -- 172.21.15.72:0/123959349 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8e400b5fc0 msgr2=0x7f8e2c0051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:27.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.124+0000 7f8e45aa8640 1 --2- 172.21.15.72:0/123959349 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8e400b5fc0 0x7f8e2c0051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:27.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.124+0000 7f8e45aa8640 1 -- 172.21.15.72:0/123959349 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e2c0039b0 msgr2=0x7f8e2c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:27.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.124+0000 7f8e45aa8640 1 --2- 172.21.15.72:0/123959349 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e2c0039b0 0x7f8e2c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:27.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.124+0000 7f8e45aa8640 1 -- 172.21.15.72:0/123959349 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e2c003520 con 0x7f8e2c004800 2024-04-15T15:14:27.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.124+0000 7f8e46aaa640 1 --2- 172.21.15.72:0/123959349 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8e400b5fc0 0x7f8e2c0051e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:14:27.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.124+0000 7f8e462a9640 1 --2- 172.21.15.72:0/123959349 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e2c0039b0 0x7f8e2c003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:14:27.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.125+0000 7f8e45aa8640 1 --2- 172.21.15.72:0/123959349 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8e2c004800 0x7f8e2c004c80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8e3000b9e0 tx=0x7f8e2c005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3706a8cf504fb639 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:27.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.125+0000 7f8e452a7640 1 -- 172.21.15.72:0/123959349 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e30036020 con 0x7f8e2c004800 2024-04-15T15:14:27.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.125+0000 7f8e452a7640 1 -- 172.21.15.72:0/123959349 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e30032630 con 0x7f8e2c004800 2024-04-15T15:14:27.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.125+0000 7f8e452a7640 1 -- 172.21.15.72:0/123959349 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e3000b5e0 con 0x7f8e2c004800 2024-04-15T15:14:27.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.126+0000 7f8e4850d640 1 -- 172.21.15.72:0/123959349 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8e2c004800 msgr2=0x7f8e2c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:27.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.126+0000 7f8e4850d640 1 --2- 172.21.15.72:0/123959349 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8e2c004800 0x7f8e2c004c80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8e3000b9e0 tx=0x7f8e2c005830 comp rx=0 tx=0).stop 2024-04-15T15:14:27.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.126+0000 7f8e46aaa640 1 -- 172.21.15.72:0/123959349 reap_dead start 2024-04-15T15:14:27.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.126+0000 7f8e4850d640 1 -- 172.21.15.72:0/123959349 shutdown_connections 2024-04-15T15:14:27.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.126+0000 7f8e4850d640 1 -- 172.21.15.72:0/123959349 >> 172.21.15.72:0/123959349 conn(0x7f8e40072c80 msgr2=0x7f8e40073090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:27.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.126+0000 7f8e4850d640 1 -- 172.21.15.72:0/123959349 shutdown_connections 2024-04-15T15:14:27.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.126+0000 7f8e4850d640 1 -- 172.21.15.72:0/123959349 wait complete. 2024-04-15T15:14:27.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.127+0000 7f8e4850d640 1 Processor -- start 2024-04-15T15:14:27.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.127+0000 7f8e4850d640 1 -- start start 2024-04-15T15:14:27.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.128+0000 7f8e4850d640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8e2c004800 0x7f8e4015e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:27.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.128+0000 7f8e4850d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e40155e10 0x7f8e400b0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:27.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.128+0000 7f8e4850d640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8e401ea0e0 0x7f8e401ea560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:27.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.128+0000 7f8e4850d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8e400b49c0 con 0x7f8e40155e10 2024-04-15T15:14:27.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.128+0000 7f8e4850d640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f8e4006fe30 con 0x7f8e2c004800 2024-04-15T15:14:27.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.128+0000 7f8e4850d640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f8e400b4ca0 con 0x7f8e401ea0e0 2024-04-15T15:14:27.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.128+0000 7f8e45aa8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e40155e10 0x7f8e400b0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:27.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.128+0000 7f8e45aa8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e40155e10 0x7f8e400b0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54446/0 (socket says 172.21.15.72:54446) 2024-04-15T15:14:27.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.128+0000 7f8e45aa8640 1 -- 172.21.15.72:0/2232918438 learned_addr learned my addr 172.21.15.72:0/2232918438 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:27.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.128+0000 7f8e462a9640 1 --2- 172.21.15.72:0/2232918438 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8e2c004800 0x7f8e4015e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:27.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.128+0000 7f8e46aaa640 1 --2- 172.21.15.72:0/2232918438 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8e401ea0e0 0x7f8e401ea560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:27.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.128+0000 7f8e45aa8640 1 -- 172.21.15.72:0/2232918438 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8e401ea0e0 msgr2=0x7f8e401ea560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:27.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.128+0000 7f8e45aa8640 1 --2- 172.21.15.72:0/2232918438 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8e401ea0e0 0x7f8e401ea560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:27.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.128+0000 7f8e45aa8640 1 -- 172.21.15.72:0/2232918438 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8e2c004800 msgr2=0x7f8e4015e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:27.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.128+0000 7f8e45aa8640 1 --2- 172.21.15.72:0/2232918438 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8e2c004800 0x7f8e4015e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:27.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.128+0000 7f8e45aa8640 1 -- 172.21.15.72:0/2232918438 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e4015cb30 con 0x7f8e40155e10 2024-04-15T15:14:27.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.129+0000 7f8e45aa8640 1 --2- 172.21.15.72:0/2232918438 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e40155e10 0x7f8e400b0610 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8e3000bb00 tx=0x7f8e30004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:27.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.129+0000 7f8e36ffd640 1 -- 172.21.15.72:0/2232918438 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e3000b5e0 con 0x7f8e40155e10 2024-04-15T15:14:27.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.129+0000 7f8e36ffd640 1 -- 172.21.15.72:0/2232918438 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e3003ba60 con 0x7f8e40155e10 2024-04-15T15:14:27.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.129+0000 7f8e4850d640 1 -- 172.21.15.72:0/2232918438 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e400b0b50 con 0x7f8e40155e10 2024-04-15T15:14:27.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.129+0000 7f8e36ffd640 1 -- 172.21.15.72:0/2232918438 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e3003be30 con 0x7f8e40155e10 2024-04-15T15:14:27.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.129+0000 7f8e4850d640 1 -- 172.21.15.72:0/2232918438 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e4015d080 con 0x7f8e40155e10 2024-04-15T15:14:27.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.129+0000 7f8e36ffd640 1 -- 172.21.15.72:0/2232918438 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8e30039020 con 0x7f8e40155e10 2024-04-15T15:14:27.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.129+0000 7f8e36ffd640 1 -- 172.21.15.72:0/2232918438 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f8e300453f0 con 0x7f8e40155e10 2024-04-15T15:14:27.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.130+0000 7f8e4850d640 1 -- 172.21.15.72:0/2232918438 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e401eaaa0 con 0x7f8e40155e10 2024-04-15T15:14:27.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.132+0000 7f8e36ffd640 1 -- 172.21.15.72:0/2232918438 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f8e30040070 con 0x7f8e40155e10 2024-04-15T15:14:27.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.198+0000 7f8e4850d640 1 -- 172.21.15.72:0/2232918438 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f8e400cfc90 con 0x7f8e40155e10 2024-04-15T15:14:27.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.199+0000 7f8e36ffd640 1 -- 172.21.15.72:0/2232918438 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f8e3004e3f0 con 0x7f8e40155e10 2024-04-15T15:14:27.201 INFO:teuthology.orchestra.run.smithi072.stderr:adjusted tunables profile to default 2024-04-15T15:14:27.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.200+0000 7f8e4850d640 1 -- 172.21.15.72:0/2232918438 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e40155e10 msgr2=0x7f8e400b0610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:27.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.200+0000 7f8e4850d640 1 --2- 172.21.15.72:0/2232918438 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e40155e10 0x7f8e400b0610 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8e3000bb00 tx=0x7f8e30004730 comp rx=0 tx=0).stop 2024-04-15T15:14:27.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.200+0000 7f8e4850d640 1 -- 172.21.15.72:0/2232918438 shutdown_connections 2024-04-15T15:14:27.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.200+0000 7f8e4850d640 1 --2- 172.21.15.72:0/2232918438 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8e401ea0e0 0x7f8e401ea560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:27.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.200+0000 7f8e4850d640 1 --2- 172.21.15.72:0/2232918438 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e40155e10 0x7f8e400b0610 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:27.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.200+0000 7f8e4850d640 1 --2- 172.21.15.72:0/2232918438 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8e2c004800 0x7f8e4015e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:27.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.200+0000 7f8e4850d640 1 -- 172.21.15.72:0/2232918438 >> 172.21.15.72:0/2232918438 conn(0x7f8e40072c80 msgr2=0x7f8e400d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:27.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.201+0000 7f8e4850d640 1 -- 172.21.15.72:0/2232918438 shutdown_connections 2024-04-15T15:14:27.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:27.201+0000 7f8e4850d640 1 -- 172.21.15.72:0/2232918438 wait complete. 2024-04-15T15:14:27.213 INFO:tasks.ceph:check_enable_crimson: False 2024-04-15T15:14:27.213 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-15T15:14:27.213 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:14:27.213 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-15T15:14:27.242 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:14:27.242 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-15T15:14:27.310 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:14:27.310 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-15T15:14:27.378 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:14:27.379 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-15T15:14:27.447 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:14:27.447 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-15T15:14:27.475 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:14:27.476 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-15T15:14:27.546 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:14:27.546 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-15T15:14:27.589 INFO:tasks.ceph.mgr.x.smithi175.stderr:2024-04-15T15:14:27.588+0000 7f4b2580f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-15T15:14:27.607 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-04-15T15:14:27.606+0000 7f8ede82d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-15T15:14:27.615 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:14:27.615 DEBUG:teuthology.orchestra.run.smithi175:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-15T15:14:27.646 INFO:tasks.ceph.mgr.x.smithi175.stderr:2024-04-15T15:14:27.645+0000 7f4b2580f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-15T15:14:27.660 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-04-15T15:14:27.659+0000 7f8ede82d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-15T15:14:27.673 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --cluster ceph osd new 38ec1d7c-98f9-412c-bf00-97f8d82a075e 0 2024-04-15T15:14:27.816 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.816+0000 7fe17e79d640 1 Processor -- start 2024-04-15T15:14:27.817 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.816+0000 7fe17e79d640 1 -- start start 2024-04-15T15:14:27.818 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.817+0000 7fe17e79d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe17815b210 0x7fe17815eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:27.818 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.817+0000 7fe17e79d640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe17815ba80 0x7fe17806db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:27.818 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.817+0000 7fe17e79d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1780cecd0 con 0x7fe17815b5f0 2024-04-15T15:14:27.818 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.817+0000 7fe17e79d640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fe17815f730 con 0x7fe17815b210 2024-04-15T15:14:27.818 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.817+0000 7fe17e79d640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fe17815fa10 con 0x7fe17815ba80 2024-04-15T15:14:27.819 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.817+0000 7fe17df9c640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe17815ba80 0x7fe17806db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:27.819 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.817+0000 7fe17df9c640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe17815ba80 0x7fe17806db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59504/0 (socket says 172.21.15.175:59504) 2024-04-15T15:14:27.819 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.817+0000 7fe17df9c640 1 -- 172.21.15.175:0/3736330403 learned_addr learned my addr 172.21.15.175:0/3736330403 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-04-15T15:14:27.819 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.817+0000 7fe17d79b640 1 --2- 172.21.15.175:0/3736330403 >> v2:172.21.15.72:3300/0 conn(0x7fe17815b210 0x7fe17815eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:27.819 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.817+0000 7fe17df9c640 1 -- 172.21.15.175:0/3736330403 >> v2:172.21.15.72:3300/0 conn(0x7fe17815b210 msgr2=0x7fe17815eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:27.819 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.817+0000 7fe16ffff640 1 -- 172.21.15.175:0/3736330403 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3604829078 0 0) 0x7fe168001110 con 0x7fe17815b5f0 2024-04-15T15:14:27.819 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.817+0000 7fe17df9c640 1 --2- 172.21.15.175:0/3736330403 >> v2:172.21.15.72:3300/0 conn(0x7fe17815b210 0x7fe17815eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:27.819 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.817+0000 7fe17df9c640 1 -- 172.21.15.175:0/3736330403 >> v1:172.21.15.72:6789/0 conn(0x7fe17815b5f0 legacy=0x7fe1780b5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:27.819 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.817+0000 7fe17df9c640 1 -- 172.21.15.175:0/3736330403 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe17815fcf0 con 0x7fe17815ba80 2024-04-15T15:14:27.820 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.818+0000 7fe17df9c640 1 --2- 172.21.15.175:0/3736330403 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe17815ba80 0x7fe17806db00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe17806f860 tx=0x7fe17002bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5cbf9db6dd4539e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:27.820 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.818+0000 7fe16ffff640 1 -- 172.21.15.175:0/3736330403 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe17002ca80 con 0x7fe17815ba80 2024-04-15T15:14:27.821 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.819+0000 7fe16ffff640 1 -- 172.21.15.175:0/3736330403 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe17002cc20 con 0x7fe17815ba80 2024-04-15T15:14:27.821 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.819+0000 7fe16ffff640 1 -- 172.21.15.175:0/3736330403 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1700343f0 con 0x7fe17815ba80 2024-04-15T15:14:27.821 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.819+0000 7fe17e79d640 1 -- 172.21.15.175:0/3736330403 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe17815ba80 msgr2=0x7fe17806db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:27.821 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.819+0000 7fe17e79d640 1 --2- 172.21.15.175:0/3736330403 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe17815ba80 0x7fe17806db00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe17806f860 tx=0x7fe17002bee0 comp rx=0 tx=0).stop 2024-04-15T15:14:27.821 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.819+0000 7fe17e79d640 1 -- 172.21.15.175:0/3736330403 shutdown_connections 2024-04-15T15:14:27.821 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.819+0000 7fe17e79d640 1 --2- 172.21.15.175:0/3736330403 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe17815ba80 0x7fe17806db00 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:27.821 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.819+0000 7fe17e79d640 1 --2- 172.21.15.175:0/3736330403 >> v2:172.21.15.72:3300/0 conn(0x7fe17815b210 0x7fe17815eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:27.821 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.819+0000 7fe17e79d640 1 -- 172.21.15.175:0/3736330403 >> 172.21.15.175:0/3736330403 conn(0x7fe1780d1190 msgr2=0x7fe1780d15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:27.821 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.819+0000 7fe17e79d640 1 -- 172.21.15.175:0/3736330403 shutdown_connections 2024-04-15T15:14:27.821 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.819+0000 7fe17e79d640 1 -- 172.21.15.175:0/3736330403 wait complete. 2024-04-15T15:14:27.824 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.820+0000 7fe17e79d640 1 Processor -- start 2024-04-15T15:14:27.824 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.820+0000 7fe17e79d640 1 -- start start 2024-04-15T15:14:27.824 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.821+0000 7fe17e79d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe17815b210 0x7fe17815dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:27.824 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.821+0000 7fe17e79d640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe17815b5f0 0x7fe1781e4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:27.824 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.821+0000 7fe17e79d640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe17815ba80 0x7fe17814c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:27.824 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.821+0000 7fe17e79d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe17815f420 con 0x7fe17815b210 2024-04-15T15:14:27.824 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.821+0000 7fe17e79d640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fe17806d5e0 con 0x7fe17815ba80 2024-04-15T15:14:27.824 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.821+0000 7fe17e79d640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fe1780b64d0 con 0x7fe17815b5f0 2024-04-15T15:14:27.824 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.821+0000 7fe17cf9a640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe17815b5f0 0x7fe1781e4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:27.824 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.821+0000 7fe17df9c640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe17815ba80 0x7fe17814c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:27.824 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.821+0000 7fe17cf9a640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe17815b5f0 0x7fe1781e4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3301/0 says I am v2:172.21.15.175:42910/0 (socket says 172.21.15.175:42910) 2024-04-15T15:14:27.824 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.821+0000 7fe17cf9a640 1 -- 172.21.15.175:0/4072119035 learned_addr learned my addr 172.21.15.175:0/4072119035 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-04-15T15:14:27.824 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.821+0000 7fe17d79b640 1 --2- 172.21.15.175:0/4072119035 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe17815b210 0x7fe17815dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:27.825 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.822+0000 7fe17cf9a640 1 -- 172.21.15.175:0/4072119035 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe17815ba80 msgr2=0x7fe17814c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:27.825 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.822+0000 7fe17cf9a640 1 --2- 172.21.15.175:0/4072119035 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe17815ba80 0x7fe17814c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:27.825 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.822+0000 7fe17cf9a640 1 -- 172.21.15.175:0/4072119035 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe17815b210 msgr2=0x7fe17815dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:27.825 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.822+0000 7fe17cf9a640 1 --2- 172.21.15.175:0/4072119035 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe17815b210 0x7fe17815dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:27.825 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.822+0000 7fe17cf9a640 1 -- 172.21.15.175:0/4072119035 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1780ca760 con 0x7fe17815b5f0 2024-04-15T15:14:27.825 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.822+0000 7fe17df9c640 1 --2- 172.21.15.175:0/4072119035 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe17815ba80 0x7fe17814c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:14:27.825 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.822+0000 7fe17d79b640 1 --2- 172.21.15.175:0/4072119035 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe17815b210 0x7fe17815dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:14:27.825 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.822+0000 7fe17cf9a640 1 --2- 172.21.15.175:0/4072119035 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe17815b5f0 0x7fe1781e4d40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe16800c750 tx=0x7fe16800cc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:27.825 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.822+0000 7fe16dffb640 1 -- 172.21.15.175:0/4072119035 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe168017070 con 0x7fe17815b5f0 2024-04-15T15:14:27.825 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.822+0000 7fe16dffb640 1 -- 172.21.15.175:0/4072119035 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1680052b0 con 0x7fe17815b5f0 2024-04-15T15:14:27.825 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.822+0000 7fe17e79d640 1 -- 172.21.15.175:0/4072119035 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1780caa40 con 0x7fe17815b5f0 2024-04-15T15:14:27.825 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.822+0000 7fe16dffb640 1 -- 172.21.15.175:0/4072119035 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe16800a880 con 0x7fe17815b5f0 2024-04-15T15:14:27.825 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.822+0000 7fe17e79d640 1 -- 172.21.15.175:0/4072119035 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1780cb000 con 0x7fe17815b5f0 2024-04-15T15:14:27.825 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.822+0000 7fe16dffb640 1 -- 172.21.15.175:0/4072119035 <== mon.2 v2:172.21.15.175:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe16801a070 con 0x7fe17815b5f0 2024-04-15T15:14:27.825 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.822+0000 7fe16dffb640 1 -- 172.21.15.175:0/4072119035 <== mon.2 v2:172.21.15.175:3301/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7fe168003de0 con 0x7fe17815b5f0 2024-04-15T15:14:27.826 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.823+0000 7fe17e79d640 1 -- 172.21.15.175:0/4072119035 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe178075aa0 con 0x7fe17815b5f0 2024-04-15T15:14:27.826 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.826+0000 7fe16dffb640 1 -- 172.21.15.175:0/4072119035 <== mon.2 v2:172.21.15.175:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fe1780cecd0 con 0x7fe17815b5f0 2024-04-15T15:14:27.961 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.960+0000 7fe17e79d640 1 -- 172.21.15.175:0/4072119035 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_command({"prefix": "osd new", "uuid": "38ec1d7c-98f9-412c-bf00-97f8d82a075e", "id": 0} v 0) -- 0x7fe17815aa30 con 0x7fe17815b5f0 2024-04-15T15:14:27.964 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.964+0000 7fe16dffb640 1 -- 172.21.15.175:0/4072119035 <== mon.2 v2:172.21.15.175:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "38ec1d7c-98f9-412c-bf00-97f8d82a075e", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7fe1680035f0 con 0x7fe17815b5f0 2024-04-15T15:14:27.965 INFO:teuthology.orchestra.run.smithi175.stdout:0 2024-04-15T15:14:27.966 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.965+0000 7fe17e79d640 1 -- 172.21.15.175:0/4072119035 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe17815b5f0 msgr2=0x7fe1781e4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:27.966 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.965+0000 7fe17e79d640 1 --2- 172.21.15.175:0/4072119035 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe17815b5f0 0x7fe1781e4d40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe16800c750 tx=0x7fe16800cc10 comp rx=0 tx=0).stop 2024-04-15T15:14:27.966 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.965+0000 7fe17e79d640 1 -- 172.21.15.175:0/4072119035 shutdown_connections 2024-04-15T15:14:27.966 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.965+0000 7fe17e79d640 1 --2- 172.21.15.175:0/4072119035 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe17815ba80 0x7fe17814c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:27.966 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.965+0000 7fe17e79d640 1 --2- 172.21.15.175:0/4072119035 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe17815b5f0 0x7fe1781e4d40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:27.966 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.965+0000 7fe17e79d640 1 --2- 172.21.15.175:0/4072119035 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe17815b210 0x7fe17815dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:27.966 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.965+0000 7fe17e79d640 1 -- 172.21.15.175:0/4072119035 >> 172.21.15.175:0/4072119035 conn(0x7fe1780d1190 msgr2=0x7fe1780b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:27.966 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.965+0000 7fe17e79d640 1 -- 172.21.15.175:0/4072119035 shutdown_connections 2024-04-15T15:14:27.967 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:27.965+0000 7fe17e79d640 1 -- 172.21.15.175:0/4072119035 wait complete. 2024-04-15T15:14:27.976 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --cluster ceph osd new 0e798110-9f52-4dfb-9f05-43236921d829 1 2024-04-15T15:14:28.080 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.079+0000 7fafcc21b640 1 Processor -- start 2024-04-15T15:14:28.080 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.079+0000 7fafcc21b640 1 -- start start 2024-04-15T15:14:28.081 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.080+0000 7fafcc21b640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fafc40cf160 0x7fafc40d1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:28.081 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.080+0000 7fafcc21b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fafc415e610 0x7fafc415ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:28.081 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.080+0000 7fafcc21b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafc415d570 con 0x7fafc40cf540 2024-04-15T15:14:28.081 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.080+0000 7fafcc21b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fafc415d850 con 0x7fafc415e610 2024-04-15T15:14:28.081 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.080+0000 7fafcc21b640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fafc415db30 con 0x7fafc40cf160 2024-04-15T15:14:28.082 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.080+0000 7fafc9fb7640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fafc40cf160 0x7fafc40d1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:28.082 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.080+0000 7fafc9fb7640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fafc40cf160 0x7fafc40d1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59516/0 (socket says 172.21.15.175:59516) 2024-04-15T15:14:28.082 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.080+0000 7fafc9fb7640 1 -- 172.21.15.175:0/2063861974 learned_addr learned my addr 172.21.15.175:0/2063861974 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-04-15T15:14:28.082 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.080+0000 7fafca7b8640 1 --2- 172.21.15.175:0/2063861974 >> v2:172.21.15.72:3300/0 conn(0x7fafc415e610 0x7fafc415ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:28.082 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.080+0000 7fafc9fb7640 1 -- 172.21.15.175:0/2063861974 >> v2:172.21.15.72:3300/0 conn(0x7fafc415e610 msgr2=0x7fafc415ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:28.082 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.080+0000 7fafc9fb7640 1 --2- 172.21.15.175:0/2063861974 >> v2:172.21.15.72:3300/0 conn(0x7fafc415e610 0x7fafc415ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.082 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.080+0000 7fafc9fb7640 1 -- 172.21.15.175:0/2063861974 >> v1:172.21.15.72:6789/0 conn(0x7fafc40cf540 legacy=0x7fafc415cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:28.082 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.080+0000 7fafc9fb7640 1 -- 172.21.15.175:0/2063861974 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafc415de10 con 0x7fafc40cf160 2024-04-15T15:14:28.084 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.081+0000 7fafc9fb7640 1 --2- 172.21.15.175:0/2063861974 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafc40cf160 0x7fafc40d1ea0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fafc406c950 tx=0x7fafc002b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c96ba7fe5018e884 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:28.084 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.081+0000 7fafc8fb5640 1 -- 172.21.15.175:0/2063861974 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc0034070 con 0x7fafc40cf160 2024-04-15T15:14:28.084 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.081+0000 7fafc8fb5640 1 -- 172.21.15.175:0/2063861974 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafc0030860 con 0x7fafc40cf160 2024-04-15T15:14:28.084 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.081+0000 7fafc8fb5640 1 -- 172.21.15.175:0/2063861974 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc0030c30 con 0x7fafc40cf160 2024-04-15T15:14:28.084 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.082+0000 7fafcc21b640 1 -- 172.21.15.175:0/2063861974 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafc40cf160 msgr2=0x7fafc40d1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:28.084 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.082+0000 7fafcc21b640 1 --2- 172.21.15.175:0/2063861974 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafc40cf160 0x7fafc40d1ea0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fafc406c950 tx=0x7fafc002b9a0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.084 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.082+0000 7fafcc21b640 1 -- 172.21.15.175:0/2063861974 shutdown_connections 2024-04-15T15:14:28.084 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.082+0000 7fafcc21b640 1 --2- 172.21.15.175:0/2063861974 >> v2:172.21.15.72:3300/0 conn(0x7fafc415e610 0x7fafc415ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.084 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.082+0000 7fafcc21b640 1 --2- 172.21.15.175:0/2063861974 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafc40cf160 0x7fafc40d1ea0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.084 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.082+0000 7fafcc21b640 1 -- 172.21.15.175:0/2063861974 >> 172.21.15.175:0/2063861974 conn(0x7fafc40d24d0 msgr2=0x7fafc40d28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:28.084 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.082+0000 7fafcc21b640 1 -- 172.21.15.175:0/2063861974 shutdown_connections 2024-04-15T15:14:28.084 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.082+0000 7fafcc21b640 1 -- 172.21.15.175:0/2063861974 wait complete. 2024-04-15T15:14:28.084 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.083+0000 7fafcc21b640 1 Processor -- start 2024-04-15T15:14:28.085 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.083+0000 7fafcc21b640 1 -- start start 2024-04-15T15:14:28.087 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.084+0000 7fafcc21b640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fafc40cf160 0x7fafc413b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:28.087 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.084+0000 7fafcc21b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fafc40cf540 0x7fafc413b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:28.087 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.084+0000 7fafcc21b640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafc415e610 0x7fafc41f2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:28.087 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.084+0000 7fafcc21b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fafc4146250 con 0x7fafc40cf540 2024-04-15T15:14:28.087 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.084+0000 7fafcc21b640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fafc40cf9c0 con 0x7fafc415e610 2024-04-15T15:14:28.087 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.084+0000 7fafcc21b640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fafc41463d0 con 0x7fafc40cf160 2024-04-15T15:14:28.087 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.084+0000 7fafc9fb7640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fafc40cf160 0x7fafc413b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:28.087 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.084+0000 7fafc9fb7640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fafc40cf160 0x7fafc413b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3301/0 says I am v2:172.21.15.175:42924/0 (socket says 172.21.15.175:42924) 2024-04-15T15:14:28.087 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.084+0000 7fafc9fb7640 1 -- 172.21.15.175:0/3261359256 learned_addr learned my addr 172.21.15.175:0/3261359256 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-04-15T15:14:28.087 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.084+0000 7fafca7b8640 1 --2- 172.21.15.175:0/3261359256 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafc415e610 0x7fafc41f2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:28.087 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.084+0000 7fafc97b6640 1 --2- 172.21.15.175:0/3261359256 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fafc40cf540 0x7fafc413b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:28.087 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.084+0000 7fafc9fb7640 1 -- 172.21.15.175:0/3261359256 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafc415e610 msgr2=0x7fafc41f2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:28.087 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.084+0000 7fafc9fb7640 1 --2- 172.21.15.175:0/3261359256 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafc415e610 0x7fafc41f2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.088 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.084+0000 7fafc9fb7640 1 -- 172.21.15.175:0/3261359256 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fafc40cf540 msgr2=0x7fafc413b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:28.088 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.084+0000 7fafc9fb7640 1 --2- 172.21.15.175:0/3261359256 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fafc40cf540 0x7fafc413b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.088 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.084+0000 7fafc9fb7640 1 -- 172.21.15.175:0/3261359256 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafc413c0b0 con 0x7fafc40cf160 2024-04-15T15:14:28.088 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.084+0000 7fafc9fb7640 1 --2- 172.21.15.175:0/3261359256 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fafc40cf160 0x7fafc413b1e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fafc00097c0 tx=0x7fafc0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:28.088 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.085+0000 7fafba7fc640 1 -- 172.21.15.175:0/3261359256 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc0048070 con 0x7fafc40cf160 2024-04-15T15:14:28.088 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.085+0000 7fafba7fc640 1 -- 172.21.15.175:0/3261359256 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafc0030660 con 0x7fafc40cf160 2024-04-15T15:14:28.088 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.085+0000 7fafba7fc640 1 -- 172.21.15.175:0/3261359256 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc0034040 con 0x7fafc40cf160 2024-04-15T15:14:28.088 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.085+0000 7fafcc21b640 1 -- 172.21.15.175:0/3261359256 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafc413c390 con 0x7fafc40cf160 2024-04-15T15:14:28.088 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.085+0000 7fafcc21b640 1 -- 172.21.15.175:0/3261359256 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fafc41f4230 con 0x7fafc40cf160 2024-04-15T15:14:28.088 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.085+0000 7fafba7fc640 1 -- 172.21.15.175:0/3261359256 <== mon.2 v2:172.21.15.175:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fafc004b070 con 0x7fafc40cf160 2024-04-15T15:14:28.088 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.085+0000 7fafba7fc640 1 -- 172.21.15.175:0/3261359256 <== mon.2 v2:172.21.15.175:3301/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7fafc00437c0 con 0x7fafc40cf160 2024-04-15T15:14:28.088 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.086+0000 7fafcc21b640 1 -- 172.21.15.175:0/3261359256 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafc4141ed0 con 0x7fafc40cf160 2024-04-15T15:14:28.089 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.088+0000 7fafba7fc640 1 -- 172.21.15.175:0/3261359256 <== mon.2 v2:172.21.15.175:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fafc4141ed0 con 0x7fafc40cf160 2024-04-15T15:14:28.162 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.161+0000 7fafcc21b640 1 -- 172.21.15.175:0/3261359256 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_command({"prefix": "osd new", "uuid": "0e798110-9f52-4dfb-9f05-43236921d829", "id": 1} v 0) -- 0x7fafc40b6070 con 0x7fafc40cf160 2024-04-15T15:14:28.168 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.167+0000 7fafba7fc640 1 -- 172.21.15.175:0/3261359256 <== mon.2 v2:172.21.15.175:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0e798110-9f52-4dfb-9f05-43236921d829", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7fafc40b6070 con 0x7fafc40cf160 2024-04-15T15:14:28.168 INFO:teuthology.orchestra.run.smithi175.stdout:1 2024-04-15T15:14:28.169 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.168+0000 7fafcc21b640 1 -- 172.21.15.175:0/3261359256 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fafc40cf160 msgr2=0x7fafc413b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:28.169 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.168+0000 7fafcc21b640 1 --2- 172.21.15.175:0/3261359256 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fafc40cf160 0x7fafc413b1e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fafc00097c0 tx=0x7fafc0004730 comp rx=0 tx=0).stop 2024-04-15T15:14:28.169 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.168+0000 7fafcc21b640 1 -- 172.21.15.175:0/3261359256 shutdown_connections 2024-04-15T15:14:28.169 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.168+0000 7fafcc21b640 1 --2- 172.21.15.175:0/3261359256 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fafc415e610 0x7fafc41f2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.169 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.168+0000 7fafcc21b640 1 --2- 172.21.15.175:0/3261359256 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fafc40cf540 0x7fafc413b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.169 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.168+0000 7fafcc21b640 1 --2- 172.21.15.175:0/3261359256 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fafc40cf160 0x7fafc413b1e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.169 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.168+0000 7fafcc21b640 1 -- 172.21.15.175:0/3261359256 >> 172.21.15.175:0/3261359256 conn(0x7fafc40d24d0 msgr2=0x7fafc40b5960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:28.169 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.168+0000 7fafcc21b640 1 -- 172.21.15.175:0/3261359256 shutdown_connections 2024-04-15T15:14:28.170 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.168+0000 7fafcc21b640 1 -- 172.21.15.175:0/3261359256 wait complete. 2024-04-15T15:14:28.180 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --cluster ceph osd new 28be1d7e-11ab-4c1c-8202-b443bac0bc04 2 2024-04-15T15:14:28.200 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-04-15T15:14:28.199+0000 7f8ede82d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-15T15:14:28.252 INFO:tasks.ceph.mgr.x.smithi175.stderr:2024-04-15T15:14:28.251+0000 7f4b2580f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-15T15:14:28.253 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-04-15T15:14:28.252+0000 7f8ede82d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-15T15:14:28.283 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.282+0000 7fdb7a7d5640 1 Processor -- start 2024-04-15T15:14:28.284 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.283+0000 7fdb7a7d5640 1 -- start start 2024-04-15T15:14:28.287 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.283+0000 7fdb7a7d5640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fdb7406f7f0 0x7fdb7406d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:28.288 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.283+0000 7fdb7a7d5640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb7415f630 con 0x7fdb74158d60 2024-04-15T15:14:28.288 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.283+0000 7fdb7a7d5640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb7415f910 con 0x7fdb7406d740 2024-04-15T15:14:28.288 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.283+0000 7fdb7a7d5640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fdb7415ffe0 con 0x7fdb7406f7f0 2024-04-15T15:14:28.288 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.283+0000 7fdb78d72640 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fdb7406d740 0x7fdb74160750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:60960/0 (socket says 172.21.15.175:60960) 2024-04-15T15:14:28.288 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.284+0000 7fdb78d72640 1 -- 172.21.15.175:0/3893687877 learned_addr learned my addr 172.21.15.175:0/3893687877 (peer_addr_for_me v1:172.21.15.175:0/0) 2024-04-15T15:14:28.288 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.284+0000 7fdb737fe640 1 --2- 172.21.15.175:0/3893687877 >> v2:172.21.15.72:3300/0 conn(0x7fdb7406f7f0 0x7fdb7406d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:28.288 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.284+0000 7fdb72ffd640 1 -- 172.21.15.175:0/3893687877 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1328633618 0 0) 0x7fdb7415f910 con 0x7fdb7406d740 2024-04-15T15:14:28.288 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.284+0000 7fdb72ffd640 1 -- 172.21.15.175:0/3893687877 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb5c003520 con 0x7fdb7406d740 2024-04-15T15:14:28.288 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.284+0000 7fdb72ffd640 1 -- 172.21.15.175:0/3893687877 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 333384651 0 0) 0x7fdb7415f630 con 0x7fdb74158d60 2024-04-15T15:14:28.288 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.284+0000 7fdb72ffd640 1 -- 172.21.15.175:0/3893687877 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb7415f910 con 0x7fdb74158d60 2024-04-15T15:14:28.288 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.284+0000 7fdb72ffd640 1 -- 172.21.15.175:0/3893687877 <== mon.1 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2441987204 0 0) 0x7fdb5c003520 con 0x7fdb7406d740 2024-04-15T15:14:28.288 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.284+0000 7fdb72ffd640 1 -- 172.21.15.175:0/3893687877 >> v2:172.21.15.72:3300/0 conn(0x7fdb7406f7f0 msgr2=0x7fdb7406d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:28.288 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.284+0000 7fdb72ffd640 1 --2- 172.21.15.175:0/3893687877 >> v2:172.21.15.72:3300/0 conn(0x7fdb7406f7f0 0x7fdb7406d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.288 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.284+0000 7fdb72ffd640 1 -- 172.21.15.175:0/3893687877 >> v1:172.21.15.72:6789/0 conn(0x7fdb74158d60 legacy=0x7fdb7415ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:28.289 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.284+0000 7fdb72ffd640 1 -- 172.21.15.175:0/3893687877 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb7415fbf0 con 0x7fdb7406d740 2024-04-15T15:14:28.289 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.284+0000 7fdb72ffd640 1 -- 172.21.15.175:0/3893687877 <== mon.1 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7fdb6c0019f0 con 0x7fdb7406d740 2024-04-15T15:14:28.289 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.284+0000 7fdb72ffd640 1 -- 172.21.15.175:0/3893687877 >> v1:172.21.15.175:6789/0 conn(0x7fdb7406d740 legacy=0x7fdb74160750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:28.289 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.284+0000 7fdb72ffd640 1 --2- 172.21.15.175:0/3893687877 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdb5c003b40 0x7fdb5c003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:28.289 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.284+0000 7fdb72ffd640 1 --2- 172.21.15.175:0/3893687877 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdb5c004800 0x7fdb5c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:28.289 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.284+0000 7fdb72ffd640 1 --2- 172.21.15.175:0/3893687877 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb5c0051c0 0x7fdb5c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:28.289 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.284+0000 7fdb72ffd640 1 -- 172.21.15.175:0/3893687877 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7415ffe0 con 0x7fdb5c0051c0 2024-04-15T15:14:28.289 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.284+0000 7fdb72ffd640 1 -- 172.21.15.175:0/3893687877 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fdb5c00c160 con 0x7fdb5c004800 2024-04-15T15:14:28.289 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.284+0000 7fdb72ffd640 1 -- 172.21.15.175:0/3893687877 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fdb5c00c360 con 0x7fdb5c003b40 2024-04-15T15:14:28.289 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.284+0000 7fdb73fff640 1 --2- 172.21.15.175:0/3893687877 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdb5c003b40 0x7fdb5c003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:28.289 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.284+0000 7fdb78d72640 1 --2- 172.21.15.175:0/3893687877 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb5c0051c0 0x7fdb5c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:28.289 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.284+0000 7fdb737fe640 1 --2- 172.21.15.175:0/3893687877 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdb5c004800 0x7fdb5c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:28.290 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.284+0000 7fdb73fff640 1 -- 172.21.15.175:0/3893687877 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdb5c004800 msgr2=0x7fdb5c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:28.290 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.284+0000 7fdb73fff640 1 --2- 172.21.15.175:0/3893687877 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdb5c004800 0x7fdb5c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.290 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.284+0000 7fdb73fff640 1 -- 172.21.15.175:0/3893687877 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb5c0051c0 msgr2=0x7fdb5c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:28.290 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.284+0000 7fdb73fff640 1 --2- 172.21.15.175:0/3893687877 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb5c0051c0 0x7fdb5c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.290 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.285+0000 7fdb73fff640 1 -- 172.21.15.175:0/3893687877 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb5c003520 con 0x7fdb5c003b40 2024-04-15T15:14:28.290 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.285+0000 7fdb78d72640 1 -- 172.21.15.175:0/3893687877 reap_dead start 2024-04-15T15:14:28.290 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.285+0000 7fdb73fff640 1 --2- 172.21.15.175:0/3893687877 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdb5c003b40 0x7fdb5c003f60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fdb640018d0 tx=0x7fdb5c005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5380a1caf62dc86 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:28.290 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.285+0000 7fdb72ffd640 1 -- 172.21.15.175:0/3893687877 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb64037070 con 0x7fdb5c003b40 2024-04-15T15:14:28.290 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.285+0000 7fdb72ffd640 1 -- 172.21.15.175:0/3893687877 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb640052d0 con 0x7fdb5c003b40 2024-04-15T15:14:28.290 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.285+0000 7fdb72ffd640 1 -- 172.21.15.175:0/3893687877 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb640014f0 con 0x7fdb5c003b40 2024-04-15T15:14:28.290 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.286+0000 7fdb7a7d5640 1 -- 172.21.15.175:0/3893687877 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdb5c003b40 msgr2=0x7fdb5c003f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:28.290 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.286+0000 7fdb7a7d5640 1 --2- 172.21.15.175:0/3893687877 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdb5c003b40 0x7fdb5c003f60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fdb640018d0 tx=0x7fdb5c005830 comp rx=0 tx=0).stop 2024-04-15T15:14:28.290 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.286+0000 7fdb7a7d5640 1 -- 172.21.15.175:0/3893687877 shutdown_connections 2024-04-15T15:14:28.290 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.286+0000 7fdb7a7d5640 1 --2- 172.21.15.175:0/3893687877 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdb5c003b40 0x7fdb5c003f60 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.291 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.286+0000 7fdb7a7d5640 1 -- 172.21.15.175:0/3893687877 >> 172.21.15.175:0/3893687877 conn(0x7fdb74072400 msgr2=0x7fdb74072810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:28.291 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.286+0000 7fdb7a7d5640 1 -- 172.21.15.175:0/3893687877 shutdown_connections 2024-04-15T15:14:28.291 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.286+0000 7fdb7a7d5640 1 -- 172.21.15.175:0/3893687877 wait complete. 2024-04-15T15:14:28.294 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.288+0000 7fdb7a7d5640 1 Processor -- start 2024-04-15T15:14:28.294 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.288+0000 7fdb7a7d5640 1 -- start start 2024-04-15T15:14:28.295 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.289+0000 7fdb7a7d5640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdb5c003b40 0x7fdb741f8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:28.295 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.289+0000 7fdb7a7d5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb74149100 0x7fdb7414ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:28.295 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.289+0000 7fdb7a7d5640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdb74149560 0x7fdb740b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:28.295 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.289+0000 7fdb7a7d5640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fdb74159140 con 0x7fdb74149100 2024-04-15T15:14:28.295 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.289+0000 7fdb7a7d5640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fdb7406db80 con 0x7fdb5c003b40 2024-04-15T15:14:28.295 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.289+0000 7fdb7a7d5640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fdb741592c0 con 0x7fdb74149560 2024-04-15T15:14:28.295 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.290+0000 7fdb73fff640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdb5c003b40 0x7fdb741f8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:28.295 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.290+0000 7fdb73fff640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdb5c003b40 0x7fdb741f8520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59548/0 (socket says 172.21.15.175:59548) 2024-04-15T15:14:28.295 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.290+0000 7fdb78d72640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdb74149560 0x7fdb740b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:28.295 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.290+0000 7fdb73fff640 1 -- 172.21.15.175:0/3780035172 learned_addr learned my addr 172.21.15.175:0/3780035172 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-04-15T15:14:28.295 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.290+0000 7fdb737fe640 1 --2- 172.21.15.175:0/3780035172 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb74149100 0x7fdb7414ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:28.295 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.290+0000 7fdb78d72640 1 -- 172.21.15.175:0/3780035172 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdb5c003b40 msgr2=0x7fdb741f8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:28.296 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.290+0000 7fdb78d72640 1 --2- 172.21.15.175:0/3780035172 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdb5c003b40 0x7fdb741f8520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.296 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.290+0000 7fdb78d72640 1 -- 172.21.15.175:0/3780035172 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb74149100 msgr2=0x7fdb7414ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:28.296 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.290+0000 7fdb78d72640 1 --2- 172.21.15.175:0/3780035172 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb74149100 0x7fdb7414ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.296 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.290+0000 7fdb78d72640 1 -- 172.21.15.175:0/3780035172 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb7414c560 con 0x7fdb74149560 2024-04-15T15:14:28.296 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.290+0000 7fdb78d72640 1 --2- 172.21.15.175:0/3780035172 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdb74149560 0x7fdb740b15e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdb74075170 tx=0x7fdb6c007a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:28.296 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.290+0000 7fdb70ff9640 1 -- 172.21.15.175:0/3780035172 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb6c00a9b0 con 0x7fdb74149560 2024-04-15T15:14:28.296 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.290+0000 7fdb70ff9640 1 -- 172.21.15.175:0/3780035172 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb6c019070 con 0x7fdb74149560 2024-04-15T15:14:28.296 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.290+0000 7fdb70ff9640 1 -- 172.21.15.175:0/3780035172 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb6c00adf0 con 0x7fdb74149560 2024-04-15T15:14:28.296 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.290+0000 7fdb7a7d5640 1 -- 172.21.15.175:0/3780035172 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb7414c840 con 0x7fdb74149560 2024-04-15T15:14:28.296 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.290+0000 7fdb7a7d5640 1 -- 172.21.15.175:0/3780035172 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb741f8a60 con 0x7fdb74149560 2024-04-15T15:14:28.296 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.290+0000 7fdb70ff9640 1 -- 172.21.15.175:0/3780035172 <== mon.2 v2:172.21.15.175:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fdb6c01e460 con 0x7fdb74149560 2024-04-15T15:14:28.296 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.290+0000 7fdb70ff9640 1 -- 172.21.15.175:0/3780035172 <== mon.2 v2:172.21.15.175:3301/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7fdb6c013c70 con 0x7fdb74149560 2024-04-15T15:14:28.297 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.291+0000 7fdb7a7d5640 1 -- 172.21.15.175:0/3780035172 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb74075c50 con 0x7fdb74149560 2024-04-15T15:14:28.297 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.293+0000 7fdb70ff9640 1 -- 172.21.15.175:0/3780035172 <== mon.2 v2:172.21.15.175:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fdb6c0133f0 con 0x7fdb74149560 2024-04-15T15:14:28.299 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-04-15T15:14:28.297+0000 7f8ede82d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-15T15:14:28.312 INFO:tasks.ceph.mgr.x.smithi175.stderr:2024-04-15T15:14:28.311+0000 7f4b2580f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-15T15:14:28.351 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-04-15T15:14:28.350+0000 7f8ede82d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-15T15:14:28.358 INFO:tasks.ceph.mgr.x.smithi175.stderr:2024-04-15T15:14:28.357+0000 7f4b2580f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-15T15:14:28.367 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.367+0000 7fdb7a7d5640 1 -- 172.21.15.175:0/3780035172 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_command({"prefix": "osd new", "uuid": "28be1d7e-11ab-4c1c-8202-b443bac0bc04", "id": 2} v 0) -- 0x7fdb74072ff0 con 0x7fdb74149560 2024-04-15T15:14:28.373 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.372+0000 7fdb70ff9640 1 -- 172.21.15.175:0/3780035172 <== mon.2 v2:172.21.15.175:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "28be1d7e-11ab-4c1c-8202-b443bac0bc04", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7fdb6c013850 con 0x7fdb74149560 2024-04-15T15:14:28.373 INFO:teuthology.orchestra.run.smithi175.stdout:2 2024-04-15T15:14:28.374 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.373+0000 7fdb7a7d5640 1 -- 172.21.15.175:0/3780035172 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdb74149560 msgr2=0x7fdb740b15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:28.375 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.373+0000 7fdb7a7d5640 1 --2- 172.21.15.175:0/3780035172 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdb74149560 0x7fdb740b15e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdb74075170 tx=0x7fdb6c007a40 comp rx=0 tx=0).stop 2024-04-15T15:14:28.375 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.373+0000 7fdb7a7d5640 1 -- 172.21.15.175:0/3780035172 shutdown_connections 2024-04-15T15:14:28.375 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.373+0000 7fdb7a7d5640 1 --2- 172.21.15.175:0/3780035172 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdb74149560 0x7fdb740b15e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.375 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.373+0000 7fdb7a7d5640 1 --2- 172.21.15.175:0/3780035172 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb74149100 0x7fdb7414ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.375 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.373+0000 7fdb7a7d5640 1 --2- 172.21.15.175:0/3780035172 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdb5c003b40 0x7fdb741f8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.375 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.373+0000 7fdb7a7d5640 1 -- 172.21.15.175:0/3780035172 >> 172.21.15.175:0/3780035172 conn(0x7fdb74072400 msgr2=0x7fdb740aeb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:28.375 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.373+0000 7fdb7a7d5640 1 -- 172.21.15.175:0/3780035172 shutdown_connections 2024-04-15T15:14:28.375 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.373+0000 7fdb7a7d5640 1 -- 172.21.15.175:0/3780035172 wait complete. 2024-04-15T15:14:28.385 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --cluster ceph osd new 5de0f86b-c336-4ea1-972e-10a267b9678d 3 2024-04-15T15:14:28.396 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-04-15T15:14:28.392+0000 7f8ede82d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-15T15:14:28.416 INFO:tasks.ceph.mgr.x.smithi175.stderr:2024-04-15T15:14:28.415+0000 7f4b2580f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-15T15:14:28.461 INFO:tasks.ceph.mgr.x.smithi175.stderr:2024-04-15T15:14:28.460+0000 7f4b2580f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-15T15:14:28.489 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.488+0000 7f282c924640 1 Processor -- start 2024-04-15T15:14:28.489 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.488+0000 7f282c924640 1 -- start start 2024-04-15T15:14:28.491 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.488+0000 7f282c924640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f28280cff00 0x7f28280d0360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:28.491 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.488+0000 7f282c924640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f282806ed70 0x7f282806f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:28.491 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.488+0000 7f282c924640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f282806f6e0 con 0x7f282815b620 2024-04-15T15:14:28.491 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.489+0000 7f282c924640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f28280d08a0 con 0x7f282806ed70 2024-04-15T15:14:28.491 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.489+0000 7f282c924640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f282806d300 con 0x7f28280cff00 2024-04-15T15:14:28.491 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.489+0000 7f281ffff640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f28280cff00 0x7f28280d0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:28.491 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.489+0000 7f281ffff640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f28280cff00 0x7f28280d0360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59564/0 (socket says 172.21.15.175:59564) 2024-04-15T15:14:28.491 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.489+0000 7f281ffff640 1 -- 172.21.15.175:0/2549969622 learned_addr learned my addr 172.21.15.175:0/2549969622 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-04-15T15:14:28.491 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.489+0000 7f2826d9d640 1 --2- 172.21.15.175:0/2549969622 >> v2:172.21.15.72:3300/0 conn(0x7f282806ed70 0x7f282806f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:28.491 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.489+0000 7f281ffff640 1 -- 172.21.15.175:0/2549969622 >> v2:172.21.15.72:3300/0 conn(0x7f282806ed70 msgr2=0x7f282806f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:28.491 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.489+0000 7f281ffff640 1 --2- 172.21.15.175:0/2549969622 >> v2:172.21.15.72:3300/0 conn(0x7f282806ed70 0x7f282806f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.491 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.489+0000 7f281ffff640 1 -- 172.21.15.175:0/2549969622 >> v1:172.21.15.72:6789/0 conn(0x7f282815b620 legacy=0x7f282806e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:28.491 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.489+0000 7f281ffff640 1 -- 172.21.15.175:0/2549969622 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f282815ca10 con 0x7f28280cff00 2024-04-15T15:14:28.492 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.490+0000 7f281ffff640 1 --2- 172.21.15.175:0/2549969622 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f28280cff00 0x7f28280d0360 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2828075240 tx=0x7f281002e170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d7fa02855d9651e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:28.492 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.490+0000 7f2825d9b640 1 -- 172.21.15.175:0/2549969622 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f281003b070 con 0x7f28280cff00 2024-04-15T15:14:28.492 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.490+0000 7f2825d9b640 1 -- 172.21.15.175:0/2549969622 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2810036440 con 0x7f28280cff00 2024-04-15T15:14:28.492 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.490+0000 7f2825d9b640 1 -- 172.21.15.175:0/2549969622 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2810036810 con 0x7f28280cff00 2024-04-15T15:14:28.493 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.491+0000 7f282c924640 1 -- 172.21.15.175:0/2549969622 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f28280cff00 msgr2=0x7f28280d0360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:28.493 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.491+0000 7f282c924640 1 --2- 172.21.15.175:0/2549969622 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f28280cff00 0x7f28280d0360 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2828075240 tx=0x7f281002e170 comp rx=0 tx=0).stop 2024-04-15T15:14:28.493 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.491+0000 7f282c924640 1 -- 172.21.15.175:0/2549969622 shutdown_connections 2024-04-15T15:14:28.493 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.491+0000 7f282c924640 1 --2- 172.21.15.175:0/2549969622 >> v2:172.21.15.72:3300/0 conn(0x7f282806ed70 0x7f282806f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.493 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.491+0000 7f282c924640 1 --2- 172.21.15.175:0/2549969622 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f28280cff00 0x7f28280d0360 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.493 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.491+0000 7f282c924640 1 -- 172.21.15.175:0/2549969622 >> 172.21.15.175:0/2549969622 conn(0x7f2828070540 msgr2=0x7f2828070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:28.493 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.491+0000 7f282c924640 1 -- 172.21.15.175:0/2549969622 shutdown_connections 2024-04-15T15:14:28.493 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.491+0000 7f282c924640 1 -- 172.21.15.175:0/2549969622 wait complete. 2024-04-15T15:14:28.493 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.492+0000 7f282c924640 1 Processor -- start 2024-04-15T15:14:28.494 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.492+0000 7f282c924640 1 -- start start 2024-04-15T15:14:28.496 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.493+0000 7f282c924640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f282806ed70 0x7f2828150930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:28.496 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.493+0000 7f282c924640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f28280cff00 0x7f28280c6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:28.496 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.493+0000 7f282c924640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f282815b620 0x7f2828070920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:28.496 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.493+0000 7f282c924640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f282806db10 con 0x7f282806ed70 2024-04-15T15:14:28.496 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.493+0000 7f282c924640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f282806d830 con 0x7f28280cff00 2024-04-15T15:14:28.496 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.493+0000 7f282c924640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f282806ddf0 con 0x7f282815b620 2024-04-15T15:14:28.496 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.493+0000 7f281ffff640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f28280cff00 0x7f28280c6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:28.496 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.493+0000 7f281ffff640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f28280cff00 0x7f28280c6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59568/0 (socket says 172.21.15.175:59568) 2024-04-15T15:14:28.496 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.493+0000 7f281ffff640 1 -- 172.21.15.175:0/1041051243 learned_addr learned my addr 172.21.15.175:0/1041051243 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-04-15T15:14:28.496 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.493+0000 7f2826d9d640 1 --2- 172.21.15.175:0/1041051243 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f282815b620 0x7f2828070920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:28.497 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.493+0000 7f282659c640 1 --2- 172.21.15.175:0/1041051243 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f282806ed70 0x7f2828150930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:28.497 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.493+0000 7f281ffff640 1 -- 172.21.15.175:0/1041051243 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f282815b620 msgr2=0x7f2828070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:28.497 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.493+0000 7f281ffff640 1 --2- 172.21.15.175:0/1041051243 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f282815b620 0x7f2828070920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.497 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.493+0000 7f281ffff640 1 -- 172.21.15.175:0/1041051243 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f282806ed70 msgr2=0x7f2828150930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:28.497 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.493+0000 7f281ffff640 1 --2- 172.21.15.175:0/1041051243 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f282806ed70 0x7f2828150930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.497 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.493+0000 7f281ffff640 1 -- 172.21.15.175:0/1041051243 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2828150e70 con 0x7f28280cff00 2024-04-15T15:14:28.497 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.493+0000 7f281ffff640 1 --2- 172.21.15.175:0/1041051243 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f28280cff00 0x7f28280c6390 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f281002e1b0 tx=0x7f28100046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:28.497 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.493+0000 7f281effd640 1 -- 172.21.15.175:0/1041051243 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f281003b040 con 0x7f28280cff00 2024-04-15T15:14:28.497 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.493+0000 7f281effd640 1 -- 172.21.15.175:0/1041051243 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2810046070 con 0x7f28280cff00 2024-04-15T15:14:28.497 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.493+0000 7f281effd640 1 -- 172.21.15.175:0/1041051243 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2810034880 con 0x7f28280cff00 2024-04-15T15:14:28.497 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.493+0000 7f282c924640 1 -- 172.21.15.175:0/1041051243 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28280c6be0 con 0x7f28280cff00 2024-04-15T15:14:28.497 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.494+0000 7f282c924640 1 -- 172.21.15.175:0/1041051243 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28280c71a0 con 0x7f28280cff00 2024-04-15T15:14:28.497 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.494+0000 7f281effd640 1 -- 172.21.15.175:0/1041051243 <== mon.1 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2810034a20 con 0x7f28280cff00 2024-04-15T15:14:28.497 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.494+0000 7f281effd640 1 -- 172.21.15.175:0/1041051243 <== mon.1 v2:172.21.15.175:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f281003f430 con 0x7f28280cff00 2024-04-15T15:14:28.497 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.494+0000 7f282c924640 1 -- 172.21.15.175:0/1041051243 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2828075c50 con 0x7f28280cff00 2024-04-15T15:14:28.498 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.497+0000 7f281effd640 1 -- 172.21.15.175:0/1041051243 <== mon.1 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f281004b460 con 0x7f28280cff00 2024-04-15T15:14:28.509 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-04-15T15:14:28.508+0000 7f8ede82d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-15T15:14:28.570 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.569+0000 7f282c924640 1 -- 172.21.15.175:0/1041051243 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5de0f86b-c336-4ea1-972e-10a267b9678d", "id": 3} v 0) -- 0x7f282806e0f0 con 0x7f28280cff00 2024-04-15T15:14:28.575 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.574+0000 7f281effd640 1 -- 172.21.15.175:0/1041051243 <== mon.1 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5de0f86b-c336-4ea1-972e-10a267b9678d", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f281004b680 con 0x7f28280cff00 2024-04-15T15:14:28.576 INFO:teuthology.orchestra.run.smithi175.stdout:3 2024-04-15T15:14:28.576 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.575+0000 7f282c924640 1 -- 172.21.15.175:0/1041051243 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f28280cff00 msgr2=0x7f28280c6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:28.576 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.575+0000 7f282c924640 1 --2- 172.21.15.175:0/1041051243 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f28280cff00 0x7f28280c6390 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f281002e1b0 tx=0x7f28100046c0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.576 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.575+0000 7f282c924640 1 -- 172.21.15.175:0/1041051243 shutdown_connections 2024-04-15T15:14:28.577 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.575+0000 7f282c924640 1 --2- 172.21.15.175:0/1041051243 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f282815b620 0x7f2828070920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.577 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.575+0000 7f282c924640 1 --2- 172.21.15.175:0/1041051243 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f28280cff00 0x7f28280c6390 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.577 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.575+0000 7f282c924640 1 --2- 172.21.15.175:0/1041051243 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f282806ed70 0x7f2828150930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.577 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.575+0000 7f282c924640 1 -- 172.21.15.175:0/1041051243 >> 172.21.15.175:0/1041051243 conn(0x7f2828070540 msgr2=0x7f28280b26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:28.577 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.575+0000 7f282c924640 1 -- 172.21.15.175:0/1041051243 shutdown_connections 2024-04-15T15:14:28.577 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.575+0000 7f282c924640 1 -- 172.21.15.175:0/1041051243 wait complete. 2024-04-15T15:14:28.579 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-04-15T15:14:28.577+0000 7f8ede82d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-15T15:14:28.584 INFO:tasks.ceph.mgr.x.smithi175.stderr:2024-04-15T15:14:28.583+0000 7f4b2580f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-15T15:14:28.587 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --cluster ceph osd new c5bece86-901e-49b0-b7f1-05531bc856d1 4 2024-04-15T15:14:28.624 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-04-15T15:14:28.623+0000 7f8ede82d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-15T15:14:28.661 INFO:tasks.ceph.mgr.x.smithi175.stderr:2024-04-15T15:14:28.657+0000 7f4b2580f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-15T15:14:28.675 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-04-15T15:14:28.674+0000 7f8ede82d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-15T15:14:28.706 INFO:tasks.ceph.mgr.x.smithi175.stderr:2024-04-15T15:14:28.705+0000 7f4b2580f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-15T15:14:28.721 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-04-15T15:14:28.720+0000 7f8ede82d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-15T15:14:28.731 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.730+0000 7f8c96063640 1 Processor -- start 2024-04-15T15:14:28.731 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.730+0000 7f8c96063640 1 -- start start 2024-04-15T15:14:28.732 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.731+0000 7f8c96063640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8c900cf4d0 0x7f8c90071420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:28.732 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.731+0000 7f8c96063640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c900d0520 con 0x7f8c9015c6b0 2024-04-15T15:14:28.732 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.731+0000 7f8c96063640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c900d0800 con 0x7f8c900b52a0 2024-04-15T15:14:28.732 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.731+0000 7f8c96063640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f8c900d0ae0 con 0x7f8c900cf4d0 2024-04-15T15:14:28.732 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.731+0000 7f8c8effd640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8c900cf4d0 0x7f8c90071420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:28.733 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.731+0000 7f8c8effd640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8c900cf4d0 0x7f8c90071420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59584/0 (socket says 172.21.15.175:59584) 2024-04-15T15:14:28.733 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.731+0000 7f8c8effd640 1 -- 172.21.15.175:0/788701398 learned_addr learned my addr 172.21.15.175:0/788701398 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-04-15T15:14:28.733 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.731+0000 7f8c8ffff640 1 --1- 172.21.15.175:0/788701398 >> v1:172.21.15.175:6789/0 conn(0x7f8c900b52a0 0x7f8c900ccee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:60968/0 (socket says 172.21.15.175:60968) 2024-04-15T15:14:28.733 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.731+0000 7f8c8effd640 1 -- 172.21.15.175:0/788701398 >> v1:172.21.15.175:6789/0 conn(0x7f8c900b52a0 legacy=0x7f8c900ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:28.733 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.731+0000 7f8c8effd640 1 -- 172.21.15.175:0/788701398 >> v1:172.21.15.72:6789/0 conn(0x7f8c9015c6b0 legacy=0x7f8c900cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:28.733 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.731+0000 7f8c8effd640 1 -- 172.21.15.175:0/788701398 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c900d0dc0 con 0x7f8c900cf4d0 2024-04-15T15:14:28.734 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.732+0000 7f8c8effd640 1 --2- 172.21.15.175:0/788701398 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8c900cf4d0 0x7f8c90071420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8c84009870 tx=0x7f8c8402ba20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ddffebf42209fe server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:28.734 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.732+0000 7f8c8e7fc640 1 -- 172.21.15.175:0/788701398 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c8402f840 con 0x7f8c900cf4d0 2024-04-15T15:14:28.734 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.732+0000 7f8c8e7fc640 1 -- 172.21.15.175:0/788701398 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c8402f9e0 con 0x7f8c900cf4d0 2024-04-15T15:14:28.734 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.732+0000 7f8c8e7fc640 1 -- 172.21.15.175:0/788701398 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c8402fdb0 con 0x7f8c900cf4d0 2024-04-15T15:14:28.734 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.733+0000 7f8c96063640 1 -- 172.21.15.175:0/788701398 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8c900cf4d0 msgr2=0x7f8c90071420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:28.734 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.733+0000 7f8c96063640 1 --2- 172.21.15.175:0/788701398 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8c900cf4d0 0x7f8c90071420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8c84009870 tx=0x7f8c8402ba20 comp rx=0 tx=0).stop 2024-04-15T15:14:28.735 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.733+0000 7f8c96063640 1 -- 172.21.15.175:0/788701398 shutdown_connections 2024-04-15T15:14:28.735 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.733+0000 7f8c96063640 1 --2- 172.21.15.175:0/788701398 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8c900cf4d0 0x7f8c90071420 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.735 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.733+0000 7f8c96063640 1 -- 172.21.15.175:0/788701398 >> 172.21.15.175:0/788701398 conn(0x7f8c900b3fa0 msgr2=0x7f8c900b4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:28.735 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.733+0000 7f8c96063640 1 -- 172.21.15.175:0/788701398 shutdown_connections 2024-04-15T15:14:28.735 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.733+0000 7f8c96063640 1 -- 172.21.15.175:0/788701398 wait complete. 2024-04-15T15:14:28.737 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.734+0000 7f8c96063640 1 Processor -- start 2024-04-15T15:14:28.737 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.734+0000 7f8c96063640 1 -- start start 2024-04-15T15:14:28.737 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.735+0000 7f8c96063640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8c900b52a0 0x7f8c9015ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:28.737 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.735+0000 7f8c96063640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8c900cf4d0 0x7f8c901e7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:28.737 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.735+0000 7f8c96063640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8c9015c6b0 0x7f8c90149a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:28.737 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.735+0000 7f8c96063640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8c900d0100 con 0x7f8c900cf4d0 2024-04-15T15:14:28.737 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.735+0000 7f8c96063640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f8c900cff80 con 0x7f8c9015c6b0 2024-04-15T15:14:28.738 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.735+0000 7f8c96063640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f8c900d0280 con 0x7f8c900b52a0 2024-04-15T15:14:28.738 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.735+0000 7f8c8ffff640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8c9015c6b0 0x7f8c90149a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:28.738 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.735+0000 7f8c8f7fe640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8c900b52a0 0x7f8c9015ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:28.738 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.735+0000 7f8c8ffff640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8c9015c6b0 0x7f8c90149a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59590/0 (socket says 172.21.15.175:59590) 2024-04-15T15:14:28.738 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.735+0000 7f8c8ffff640 1 -- 172.21.15.175:0/849136328 learned_addr learned my addr 172.21.15.175:0/849136328 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-04-15T15:14:28.738 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.735+0000 7f8c8effd640 1 --2- 172.21.15.175:0/849136328 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8c900cf4d0 0x7f8c901e7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:28.738 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.735+0000 7f8c8ffff640 1 -- 172.21.15.175:0/849136328 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8c900b52a0 msgr2=0x7f8c9015ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:28.738 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.735+0000 7f8c8ffff640 1 --2- 172.21.15.175:0/849136328 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8c900b52a0 0x7f8c9015ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.738 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.735+0000 7f8c8ffff640 1 -- 172.21.15.175:0/849136328 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8c900cf4d0 msgr2=0x7f8c901e7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:28.739 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.735+0000 7f8c8ffff640 1 --2- 172.21.15.175:0/849136328 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8c900cf4d0 0x7f8c901e7da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.739 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.735+0000 7f8c8ffff640 1 -- 172.21.15.175:0/849136328 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c90148970 con 0x7f8c9015c6b0 2024-04-15T15:14:28.739 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.735+0000 7f8c8f7fe640 1 --2- 172.21.15.175:0/849136328 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8c900b52a0 0x7f8c9015ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:14:28.739 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.736+0000 7f8c8ffff640 1 --2- 172.21.15.175:0/849136328 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8c9015c6b0 0x7f8c90149a10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8c8800c730 tx=0x7f8c8800cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:28.739 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.736+0000 7f8c77fff640 1 -- 172.21.15.175:0/849136328 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c88018070 con 0x7f8c9015c6b0 2024-04-15T15:14:28.739 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.736+0000 7f8c77fff640 1 -- 172.21.15.175:0/849136328 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c88004d60 con 0x7f8c9015c6b0 2024-04-15T15:14:28.739 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.736+0000 7f8c96063640 1 -- 172.21.15.175:0/849136328 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c90148c50 con 0x7f8c9015c6b0 2024-04-15T15:14:28.739 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.736+0000 7f8c77fff640 1 -- 172.21.15.175:0/849136328 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c8800a840 con 0x7f8c9015c6b0 2024-04-15T15:14:28.739 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.736+0000 7f8c96063640 1 -- 172.21.15.175:0/849136328 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c90149210 con 0x7f8c9015c6b0 2024-04-15T15:14:28.739 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.736+0000 7f8c77fff640 1 -- 172.21.15.175:0/849136328 <== mon.1 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8c8801b070 con 0x7f8c9015c6b0 2024-04-15T15:14:28.739 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.736+0000 7f8c77fff640 1 -- 172.21.15.175:0/849136328 <== mon.1 v2:172.21.15.175:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f8c88007930 con 0x7f8c9015c6b0 2024-04-15T15:14:28.740 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.737+0000 7f8c96063640 1 -- 172.21.15.175:0/849136328 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c90075c50 con 0x7f8c9015c6b0 2024-04-15T15:14:28.740 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.739+0000 7f8c77fff640 1 -- 172.21.15.175:0/849136328 <== mon.1 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f8c88022070 con 0x7f8c9015c6b0 2024-04-15T15:14:28.762 INFO:tasks.ceph.mgr.x.smithi175.stderr:2024-04-15T15:14:28.762+0000 7f4b2580f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-15T15:14:28.811 INFO:tasks.ceph.mgr.x.smithi175.stderr:2024-04-15T15:14:28.810+0000 7f4b2580f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-15T15:14:28.812 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.812+0000 7f8c96063640 1 -- 172.21.15.175:0/849136328 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c5bece86-901e-49b0-b7f1-05531bc856d1", "id": 4} v 0) -- 0x7f8c900c8d00 con 0x7f8c9015c6b0 2024-04-15T15:14:28.818 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.818+0000 7f8c77fff640 1 -- 172.21.15.175:0/849136328 <== mon.1 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c5bece86-901e-49b0-b7f1-05531bc856d1", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f8c8800adb0 con 0x7f8c9015c6b0 2024-04-15T15:14:28.819 INFO:teuthology.orchestra.run.smithi175.stdout:4 2024-04-15T15:14:28.820 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.819+0000 7f8c96063640 1 -- 172.21.15.175:0/849136328 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8c9015c6b0 msgr2=0x7f8c90149a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:28.820 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.819+0000 7f8c96063640 1 --2- 172.21.15.175:0/849136328 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8c9015c6b0 0x7f8c90149a10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8c8800c730 tx=0x7f8c8800cbf0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.820 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.819+0000 7f8c96063640 1 -- 172.21.15.175:0/849136328 shutdown_connections 2024-04-15T15:14:28.820 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.819+0000 7f8c96063640 1 --2- 172.21.15.175:0/849136328 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8c9015c6b0 0x7f8c90149a10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.820 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.819+0000 7f8c96063640 1 --2- 172.21.15.175:0/849136328 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8c900cf4d0 0x7f8c901e7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.820 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.819+0000 7f8c96063640 1 --2- 172.21.15.175:0/849136328 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8c900b52a0 0x7f8c9015ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.820 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.819+0000 7f8c96063640 1 -- 172.21.15.175:0/849136328 >> 172.21.15.175:0/849136328 conn(0x7f8c900b3fa0 msgr2=0x7f8c900b7860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:28.820 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.819+0000 7f8c96063640 1 -- 172.21.15.175:0/849136328 shutdown_connections 2024-04-15T15:14:28.820 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.819+0000 7f8c96063640 1 -- 172.21.15.175:0/849136328 wait complete. 2024-04-15T15:14:28.830 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --cluster ceph osd new e50375c9-553a-46fb-8747-1471841434c0 5 2024-04-15T15:14:28.973 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.973+0000 7fd02ba37640 1 Processor -- start 2024-04-15T15:14:28.974 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.973+0000 7fd02ba37640 1 -- start start 2024-04-15T15:14:28.975 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.974+0000 7fd02ba37640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd0240b0800 0x7fd0240b5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:28.975 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.974+0000 7fd02ba37640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0240b4b70 con 0x7fd0240cfa00 2024-04-15T15:14:28.976 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.974+0000 7fd02ba37640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0240b4e50 con 0x7fd0240cddb0 2024-04-15T15:14:28.976 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.974+0000 7fd02ba37640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fd0240b5130 con 0x7fd0240b0800 2024-04-15T15:14:28.976 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.974+0000 7fd028fd2640 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fd0240cddb0 0x7fd0240b43b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.175:60980/0 (socket says 172.21.15.175:60980) 2024-04-15T15:14:28.976 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.974+0000 7fd028fd2640 1 -- 172.21.15.175:0/2338144553 learned_addr learned my addr 172.21.15.175:0/2338144553 (peer_addr_for_me v1:172.21.15.175:0/0) 2024-04-15T15:14:28.976 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.974+0000 7fd0297d3640 1 --2- 172.21.15.175:0/2338144553 >> v2:172.21.15.72:3300/0 conn(0x7fd0240b0800 0x7fd0240b5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:28.976 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.974+0000 7fd01bfff640 1 -- 172.21.15.175:0/2338144553 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 123622485 0 0) 0x7fd0240b4e50 con 0x7fd0240cddb0 2024-04-15T15:14:28.976 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.974+0000 7fd01bfff640 1 -- 172.21.15.175:0/2338144553 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd010003520 con 0x7fd0240cddb0 2024-04-15T15:14:28.976 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.974+0000 7fd01bfff640 1 -- 172.21.15.175:0/2338144553 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3150169974 0 0) 0x7fd0240b4b70 con 0x7fd0240cfa00 2024-04-15T15:14:28.976 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.974+0000 7fd01bfff640 1 -- 172.21.15.175:0/2338144553 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd0240b4e50 con 0x7fd0240cfa00 2024-04-15T15:14:28.976 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.974+0000 7fd0297d3640 1 -- 172.21.15.175:0/2338144553 >> v1:172.21.15.175:6789/0 conn(0x7fd0240cddb0 legacy=0x7fd0240b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:28.976 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.974+0000 7fd0297d3640 1 -- 172.21.15.175:0/2338144553 >> v1:172.21.15.72:6789/0 conn(0x7fd0240cfa00 legacy=0x7fd0240cfe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:28.976 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.974+0000 7fd0297d3640 1 -- 172.21.15.175:0/2338144553 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0240b5410 con 0x7fd0240b0800 2024-04-15T15:14:28.980 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.975+0000 7fd0297d3640 1 --2- 172.21.15.175:0/2338144553 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0240b0800 0x7fd0240b5c10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd0140097a0 tx=0x7fd01402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=311fca1f779f414b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:28.980 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.975+0000 7fd01bfff640 1 -- 172.21.15.175:0/2338144553 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd014034070 con 0x7fd0240b0800 2024-04-15T15:14:28.980 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.975+0000 7fd01bfff640 1 -- 172.21.15.175:0/2338144553 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd014030610 con 0x7fd0240b0800 2024-04-15T15:14:28.980 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.975+0000 7fd01bfff640 1 -- 172.21.15.175:0/2338144553 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0140309c0 con 0x7fd0240b0800 2024-04-15T15:14:28.980 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.976+0000 7fd02ba37640 1 -- 172.21.15.175:0/2338144553 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0240b0800 msgr2=0x7fd0240b5c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:28.980 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.976+0000 7fd02ba37640 1 --2- 172.21.15.175:0/2338144553 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0240b0800 0x7fd0240b5c10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd0140097a0 tx=0x7fd01402b960 comp rx=0 tx=0).stop 2024-04-15T15:14:28.980 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.976+0000 7fd02ba37640 1 -- 172.21.15.175:0/2338144553 shutdown_connections 2024-04-15T15:14:28.980 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.976+0000 7fd02ba37640 1 --2- 172.21.15.175:0/2338144553 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0240b0800 0x7fd0240b5c10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.981 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.976+0000 7fd02ba37640 1 -- 172.21.15.175:0/2338144553 >> 172.21.15.175:0/2338144553 conn(0x7fd0240703a0 msgr2=0x7fd0240707b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:28.981 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.976+0000 7fd02ba37640 1 -- 172.21.15.175:0/2338144553 shutdown_connections 2024-04-15T15:14:28.981 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.976+0000 7fd02ba37640 1 -- 172.21.15.175:0/2338144553 wait complete. 2024-04-15T15:14:28.981 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.977+0000 7fd02ba37640 1 Processor -- start 2024-04-15T15:14:28.981 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.977+0000 7fd02ba37640 1 -- start start 2024-04-15T15:14:28.981 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.978+0000 7fd02ba37640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0240b0800 0x7fd02406cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:28.981 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.978+0000 7fd02ba37640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd0240cddb0 0x7fd02406d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:28.981 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.978+0000 7fd02ba37640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0240cfa00 0x7fd02406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:28.981 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.978+0000 7fd02ba37640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd0240b0d60 con 0x7fd0240b0800 2024-04-15T15:14:28.981 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.978+0000 7fd02ba37640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fd0240b0be0 con 0x7fd0240cfa00 2024-04-15T15:14:28.981 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.978+0000 7fd02ba37640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fd0240ae160 con 0x7fd0240cddb0 2024-04-15T15:14:28.981 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.978+0000 7fd028fd2640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd0240cddb0 0x7fd02406d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:28.981 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.978+0000 7fd028fd2640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd0240cddb0 0x7fd02406d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3301/0 says I am v2:172.21.15.175:42968/0 (socket says 172.21.15.175:42968) 2024-04-15T15:14:28.982 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.978+0000 7fd029fd4640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0240cfa00 0x7fd02406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:28.982 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.978+0000 7fd028fd2640 1 -- 172.21.15.175:0/1531971798 learned_addr learned my addr 172.21.15.175:0/1531971798 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-04-15T15:14:28.982 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.978+0000 7fd0297d3640 1 --2- 172.21.15.175:0/1531971798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0240b0800 0x7fd02406cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:28.982 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.978+0000 7fd029fd4640 1 -- 172.21.15.175:0/1531971798 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd0240cddb0 msgr2=0x7fd02406d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:28.982 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.978+0000 7fd029fd4640 1 --2- 172.21.15.175:0/1531971798 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd0240cddb0 0x7fd02406d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.982 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.978+0000 7fd029fd4640 1 -- 172.21.15.175:0/1531971798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0240b0800 msgr2=0x7fd02406cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:28.982 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.978+0000 7fd029fd4640 1 --2- 172.21.15.175:0/1531971798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0240b0800 0x7fd02406cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:28.982 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.978+0000 7fd029fd4640 1 -- 172.21.15.175:0/1531971798 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0241603f0 con 0x7fd0240cfa00 2024-04-15T15:14:28.982 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.979+0000 7fd029fd4640 1 --2- 172.21.15.175:0/1531971798 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0240cfa00 0x7fd02406d7b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd01c00c730 tx=0x7fd01c00cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:28.982 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.979+0000 7fd019ffb640 1 -- 172.21.15.175:0/1531971798 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd01c003b70 con 0x7fd0240cfa00 2024-04-15T15:14:28.982 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.979+0000 7fd019ffb640 1 -- 172.21.15.175:0/1531971798 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd01c003d30 con 0x7fd0240cfa00 2024-04-15T15:14:28.982 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.979+0000 7fd02ba37640 1 -- 172.21.15.175:0/1531971798 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0241606d0 con 0x7fd0240cfa00 2024-04-15T15:14:28.982 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.979+0000 7fd019ffb640 1 -- 172.21.15.175:0/1531971798 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd01c005310 con 0x7fd0240cfa00 2024-04-15T15:14:28.982 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.979+0000 7fd02ba37640 1 -- 172.21.15.175:0/1531971798 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd02406c550 con 0x7fd0240cfa00 2024-04-15T15:14:28.982 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.979+0000 7fd019ffb640 1 -- 172.21.15.175:0/1531971798 <== mon.1 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd01c0056c0 con 0x7fd0240cfa00 2024-04-15T15:14:28.983 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.979+0000 7fd019ffb640 1 -- 172.21.15.175:0/1531971798 <== mon.1 v2:172.21.15.175:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7fd01c012cc0 con 0x7fd0240cfa00 2024-04-15T15:14:28.983 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.980+0000 7fd02ba37640 1 -- 172.21.15.175:0/1531971798 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd024075c50 con 0x7fd0240cfa00 2024-04-15T15:14:28.983 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:28.982+0000 7fd019ffb640 1 -- 172.21.15.175:0/1531971798 <== mon.1 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fd01c012740 con 0x7fd0240cfa00 2024-04-15T15:14:29.056 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.055+0000 7fd02ba37640 1 -- 172.21.15.175:0/1531971798 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e50375c9-553a-46fb-8747-1471841434c0", "id": 5} v 0) -- 0x7fd02415cdc0 con 0x7fd0240cfa00 2024-04-15T15:14:29.061 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.060+0000 7fd019ffb640 1 -- 172.21.15.175:0/1531971798 <== mon.1 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e50375c9-553a-46fb-8747-1471841434c0", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7fd01c012920 con 0x7fd0240cfa00 2024-04-15T15:14:29.061 INFO:teuthology.orchestra.run.smithi175.stdout:5 2024-04-15T15:14:29.062 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.061+0000 7fd02ba37640 1 -- 172.21.15.175:0/1531971798 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0240cfa00 msgr2=0x7fd02406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:29.062 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.061+0000 7fd02ba37640 1 --2- 172.21.15.175:0/1531971798 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0240cfa00 0x7fd02406d7b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd01c00c730 tx=0x7fd01c00cbf0 comp rx=0 tx=0).stop 2024-04-15T15:14:29.062 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.061+0000 7fd02ba37640 1 -- 172.21.15.175:0/1531971798 shutdown_connections 2024-04-15T15:14:29.062 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.061+0000 7fd02ba37640 1 --2- 172.21.15.175:0/1531971798 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0240cfa00 0x7fd02406d7b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:29.063 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.061+0000 7fd02ba37640 1 --2- 172.21.15.175:0/1531971798 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd0240cddb0 0x7fd02406d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:29.063 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.061+0000 7fd02ba37640 1 --2- 172.21.15.175:0/1531971798 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0240b0800 0x7fd02406cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:29.063 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.061+0000 7fd02ba37640 1 -- 172.21.15.175:0/1531971798 >> 172.21.15.175:0/1531971798 conn(0x7fd0240703a0 msgr2=0x7fd0240ae520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:29.063 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.061+0000 7fd02ba37640 1 -- 172.21.15.175:0/1531971798 shutdown_connections 2024-04-15T15:14:29.063 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.061+0000 7fd02ba37640 1 -- 172.21.15.175:0/1531971798 wait complete. 2024-04-15T15:14:29.074 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --cluster ceph osd new 7a629a23-4ffe-4b27-a2d0-b42070d22084 6 2024-04-15T15:14:29.099 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-04-15T15:14:29.098+0000 7f8ede82d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-15T15:14:29.178 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.177+0000 7f35097af640 1 Processor -- start 2024-04-15T15:14:29.178 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.177+0000 7f35097af640 1 -- start start 2024-04-15T15:14:29.179 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.178+0000 7f35097af640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f350415f810 0x7f35040ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:29.180 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.178+0000 7f35097af640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35040cd390 con 0x7f350406d300 2024-04-15T15:14:29.180 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.178+0000 7f35097af640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f350406d950 con 0x7f35040cc8a0 2024-04-15T15:14:29.180 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.178+0000 7f35097af640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f350406e460 con 0x7f350415f810 2024-04-15T15:14:29.180 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.178+0000 7f35037fe640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f350415f810 0x7f35040ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:29.180 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.178+0000 7f35037fe640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f350415f810 0x7f35040ce6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59612/0 (socket says 172.21.15.175:59612) 2024-04-15T15:14:29.180 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.178+0000 7f35037fe640 1 -- 172.21.15.175:0/45624554 learned_addr learned my addr 172.21.15.175:0/45624554 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-04-15T15:14:29.180 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.179+0000 7f35037fe640 1 -- 172.21.15.175:0/45624554 >> v1:172.21.15.175:6789/0 conn(0x7f35040cc8a0 legacy=0x7f35040ccc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:29.180 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.179+0000 7f35037fe640 1 -- 172.21.15.175:0/45624554 >> v1:172.21.15.72:6789/0 conn(0x7f350406d300 legacy=0x7f350415f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:29.180 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.179+0000 7f35037fe640 1 -- 172.21.15.175:0/45624554 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f350406e6b0 con 0x7f350415f810 2024-04-15T15:14:29.181 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.179+0000 7f35037fe640 1 --2- 172.21.15.175:0/45624554 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f350415f810 0x7f35040ce6e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f34fc009e50 tx=0x7f34fc02bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b77fb7e76611df4 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:29.182 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.179+0000 7f3501ffb640 1 -- 172.21.15.175:0/45624554 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34fc02cb20 con 0x7f350415f810 2024-04-15T15:14:29.182 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.180+0000 7f3501ffb640 1 -- 172.21.15.175:0/45624554 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34fc02ccc0 con 0x7f350415f810 2024-04-15T15:14:29.182 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.180+0000 7f3501ffb640 1 -- 172.21.15.175:0/45624554 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34fc034610 con 0x7f350415f810 2024-04-15T15:14:29.182 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.180+0000 7f35097af640 1 -- 172.21.15.175:0/45624554 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f350415f810 msgr2=0x7f35040ce6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:29.182 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.180+0000 7f35097af640 1 --2- 172.21.15.175:0/45624554 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f350415f810 0x7f35040ce6e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f34fc009e50 tx=0x7f34fc02bf80 comp rx=0 tx=0).stop 2024-04-15T15:14:29.182 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.180+0000 7f35097af640 1 -- 172.21.15.175:0/45624554 shutdown_connections 2024-04-15T15:14:29.182 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.180+0000 7f35097af640 1 --2- 172.21.15.175:0/45624554 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f350415f810 0x7f35040ce6e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:29.182 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.180+0000 7f35097af640 1 -- 172.21.15.175:0/45624554 >> 172.21.15.175:0/45624554 conn(0x7f35040cd590 msgr2=0x7f35040cd9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:29.182 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.180+0000 7f35097af640 1 -- 172.21.15.175:0/45624554 shutdown_connections 2024-04-15T15:14:29.183 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.180+0000 7f35097af640 1 -- 172.21.15.175:0/45624554 wait complete. 2024-04-15T15:14:29.185 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.182+0000 7f35097af640 1 Processor -- start 2024-04-15T15:14:29.185 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.182+0000 7f35097af640 1 -- start start 2024-04-15T15:14:29.185 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.182+0000 7f35097af640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f350406d300 0x7f35041549e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:29.185 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.182+0000 7f35097af640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f35040cc8a0 0x7f35040b5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:29.186 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.182+0000 7f35097af640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f350415f810 0x7f3504159120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:29.186 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.182+0000 7f35097af640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f350406f190 con 0x7f350406d300 2024-04-15T15:14:29.186 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.182+0000 7f35097af640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f350406ec70 con 0x7f350415f810 2024-04-15T15:14:29.186 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.182+0000 7f35097af640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f350406f310 con 0x7f35040cc8a0 2024-04-15T15:14:29.187 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.182+0000 7f35037fe640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f350415f810 0x7f3504159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:29.187 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.182+0000 7f35027fc640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f35040cc8a0 0x7f35040b5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:29.187 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.182+0000 7f35037fe640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f350415f810 0x7f3504159120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59624/0 (socket says 172.21.15.175:59624) 2024-04-15T15:14:29.187 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.182+0000 7f35037fe640 1 -- 172.21.15.175:0/2773566969 learned_addr learned my addr 172.21.15.175:0/2773566969 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-04-15T15:14:29.187 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.182+0000 7f3502ffd640 1 --2- 172.21.15.175:0/2773566969 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f350406d300 0x7f35041549e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:29.187 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.183+0000 7f35027fc640 1 -- 172.21.15.175:0/2773566969 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f350415f810 msgr2=0x7f3504159120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:29.187 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.183+0000 7f35027fc640 1 --2- 172.21.15.175:0/2773566969 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f350415f810 0x7f3504159120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:29.187 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.183+0000 7f35027fc640 1 -- 172.21.15.175:0/2773566969 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f350406d300 msgr2=0x7f35041549e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:29.188 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.183+0000 7f35027fc640 1 --2- 172.21.15.175:0/2773566969 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f350406d300 0x7f35041549e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:29.188 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.183+0000 7f35027fc640 1 -- 172.21.15.175:0/2773566969 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35040b63a0 con 0x7f35040cc8a0 2024-04-15T15:14:29.188 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.183+0000 7f35037fe640 1 --2- 172.21.15.175:0/2773566969 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f350415f810 0x7f3504159120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:14:29.188 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.183+0000 7f35027fc640 1 --2- 172.21.15.175:0/2773566969 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f35040cc8a0 0x7f35040b5950 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f35040cd560 tx=0x7f34ec00c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:29.188 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.183+0000 7f34fb7fe640 1 -- 172.21.15.175:0/2773566969 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ec017070 con 0x7f35040cc8a0 2024-04-15T15:14:29.188 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.183+0000 7f34fb7fe640 1 -- 172.21.15.175:0/2773566969 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34ec0052f0 con 0x7f35040cc8a0 2024-04-15T15:14:29.188 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.183+0000 7f35097af640 1 -- 172.21.15.175:0/2773566969 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35040b6680 con 0x7f35040cc8a0 2024-04-15T15:14:29.188 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.183+0000 7f34fb7fe640 1 -- 172.21.15.175:0/2773566969 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ec003850 con 0x7f35040cc8a0 2024-04-15T15:14:29.188 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.183+0000 7f35097af640 1 -- 172.21.15.175:0/2773566969 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f350413f910 con 0x7f35040cc8a0 2024-04-15T15:14:29.188 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.183+0000 7f34fb7fe640 1 -- 172.21.15.175:0/2773566969 <== mon.2 v2:172.21.15.175:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f34ec01a070 con 0x7f35040cc8a0 2024-04-15T15:14:29.188 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.183+0000 7f34fb7fe640 1 -- 172.21.15.175:0/2773566969 <== mon.2 v2:172.21.15.175:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f34ec011bd0 con 0x7f35040cc8a0 2024-04-15T15:14:29.188 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.184+0000 7f35097af640 1 -- 172.21.15.175:0/2773566969 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3504075c50 con 0x7f35040cc8a0 2024-04-15T15:14:29.189 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.186+0000 7f34fb7fe640 1 -- 172.21.15.175:0/2773566969 <== mon.2 v2:172.21.15.175:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f34ec021080 con 0x7f35040cc8a0 2024-04-15T15:14:29.214 INFO:tasks.ceph.mgr.x.smithi175.stderr:2024-04-15T15:14:29.213+0000 7f4b2580f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-15T15:14:29.234 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-04-15T15:14:29.233+0000 7f8ede82d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-15T15:14:29.260 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.259+0000 7f35097af640 1 -- 172.21.15.175:0/2773566969 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_command({"prefix": "osd new", "uuid": "7a629a23-4ffe-4b27-a2d0-b42070d22084", "id": 6} v 0) -- 0x7f350406bc10 con 0x7f35040cc8a0 2024-04-15T15:14:29.264 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.264+0000 7f34fb7fe640 1 -- 172.21.15.175:0/2773566969 <== mon.2 v2:172.21.15.175:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7a629a23-4ffe-4b27-a2d0-b42070d22084", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f34ec011e20 con 0x7f35040cc8a0 2024-04-15T15:14:29.265 INFO:teuthology.orchestra.run.smithi175.stdout:6 2024-04-15T15:14:29.266 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.265+0000 7f35097af640 1 -- 172.21.15.175:0/2773566969 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f35040cc8a0 msgr2=0x7f35040b5950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:29.266 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.265+0000 7f35097af640 1 --2- 172.21.15.175:0/2773566969 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f35040cc8a0 0x7f35040b5950 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f35040cd560 tx=0x7f34ec00c820 comp rx=0 tx=0).stop 2024-04-15T15:14:29.266 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.265+0000 7f35097af640 1 -- 172.21.15.175:0/2773566969 shutdown_connections 2024-04-15T15:14:29.266 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.265+0000 7f35097af640 1 --2- 172.21.15.175:0/2773566969 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f350415f810 0x7f3504159120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:29.266 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.265+0000 7f35097af640 1 --2- 172.21.15.175:0/2773566969 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f35040cc8a0 0x7f35040b5950 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:29.266 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.265+0000 7f35097af640 1 --2- 172.21.15.175:0/2773566969 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f350406d300 0x7f35041549e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:29.266 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.265+0000 7f35097af640 1 -- 172.21.15.175:0/2773566969 >> 172.21.15.175:0/2773566969 conn(0x7f35040cd590 msgr2=0x7f3504140ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:29.266 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.265+0000 7f35097af640 1 -- 172.21.15.175:0/2773566969 shutdown_connections 2024-04-15T15:14:29.267 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.265+0000 7f35097af640 1 -- 172.21.15.175:0/2773566969 wait complete. 2024-04-15T15:14:29.277 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --cluster ceph osd new c120037a-a16b-4504-847c-81cb859d31be 7 2024-04-15T15:14:29.340 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-04-15T15:14:29.339+0000 7f8ede82d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-15T15:14:29.356 INFO:tasks.ceph.mgr.x.smithi175.stderr:2024-04-15T15:14:29.356+0000 7f4b2580f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-15T15:14:29.380 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.379+0000 7f2ff21cd640 1 Processor -- start 2024-04-15T15:14:29.380 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.379+0000 7f2ff21cd640 1 -- start start 2024-04-15T15:14:29.382 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.380+0000 7f2ff21cd640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2fec0717a0 0x7f2fec071c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:29.382 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.380+0000 7f2ff21cd640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2fec15cbe0 0x7f2fec15d070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:29.383 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.380+0000 7f2ff21cd640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fec0d1e70 con 0x7f2fec0ccfa0 2024-04-15T15:14:29.383 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.380+0000 7f2ff21cd640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f2fec0b2390 con 0x7f2fec0717a0 2024-04-15T15:14:29.383 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.380+0000 7f2ff21cd640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f2fec15d5b0 con 0x7f2fec15cbe0 2024-04-15T15:14:29.383 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.380+0000 7f2fe3fff640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2fec0717a0 0x7f2fec071c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:29.383 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.380+0000 7f2febfff640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2fec15cbe0 0x7f2fec15d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:29.383 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.380+0000 7f2fe3fff640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2fec0717a0 0x7f2fec071c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.175:59628/0 (socket says 172.21.15.175:59628) 2024-04-15T15:14:29.383 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.380+0000 7f2fe3fff640 1 -- 172.21.15.175:0/3338209381 learned_addr learned my addr 172.21.15.175:0/3338209381 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-04-15T15:14:29.383 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.381+0000 7f2fe3fff640 1 -- 172.21.15.175:0/3338209381 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2fec15cbe0 msgr2=0x7f2fec15d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:29.383 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.381+0000 7f2fe3fff640 1 --2- 172.21.15.175:0/3338209381 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2fec15cbe0 0x7f2fec15d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:29.383 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.381+0000 7f2fe3fff640 1 -- 172.21.15.175:0/3338209381 >> v1:172.21.15.72:6789/0 conn(0x7f2fec0ccfa0 legacy=0x7f2fec0b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:29.383 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.381+0000 7f2fe3fff640 1 -- 172.21.15.175:0/3338209381 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fec15d810 con 0x7f2fec0717a0 2024-04-15T15:14:29.384 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.381+0000 7f2febfff640 1 --2- 172.21.15.175:0/3338209381 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2fec15cbe0 0x7f2fec15d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:14:29.384 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.381+0000 7f2fe3fff640 1 --2- 172.21.15.175:0/3338209381 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2fec0717a0 0x7f2fec071c50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2fd4009870 tx=0x7f2fd402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0c4986c12d70be9 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:29.384 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.382+0000 7f2feaffd640 1 -- 172.21.15.175:0/3338209381 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd4035020 con 0x7f2fec0717a0 2024-04-15T15:14:29.384 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.382+0000 7f2feaffd640 1 -- 172.21.15.175:0/3338209381 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fd4031640 con 0x7f2fec0717a0 2024-04-15T15:14:29.384 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.382+0000 7f2feaffd640 1 -- 172.21.15.175:0/3338209381 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd402d040 con 0x7f2fec0717a0 2024-04-15T15:14:29.389 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.382+0000 7f2ff21cd640 1 -- 172.21.15.175:0/3338209381 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2fec0717a0 msgr2=0x7f2fec071c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:29.389 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.382+0000 7f2ff21cd640 1 --2- 172.21.15.175:0/3338209381 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2fec0717a0 0x7f2fec071c50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2fd4009870 tx=0x7f2fd402b8f0 comp rx=0 tx=0).stop 2024-04-15T15:14:29.389 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.382+0000 7f2ff21cd640 1 -- 172.21.15.175:0/3338209381 shutdown_connections 2024-04-15T15:14:29.389 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.382+0000 7f2ff21cd640 1 --2- 172.21.15.175:0/3338209381 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2fec15cbe0 0x7f2fec15d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:29.389 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.382+0000 7f2ff21cd640 1 --2- 172.21.15.175:0/3338209381 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2fec0717a0 0x7f2fec071c50 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:29.389 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.382+0000 7f2ff21cd640 1 -- 172.21.15.175:0/3338209381 >> 172.21.15.175:0/3338209381 conn(0x7f2fec0d24b0 msgr2=0x7f2fec0d28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:29.390 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.382+0000 7f2ff21cd640 1 -- 172.21.15.175:0/3338209381 shutdown_connections 2024-04-15T15:14:29.390 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.382+0000 7f2ff21cd640 1 -- 172.21.15.175:0/3338209381 wait complete. 2024-04-15T15:14:29.390 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.383+0000 7f2ff21cd640 1 Processor -- start 2024-04-15T15:14:29.390 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.383+0000 7f2ff21cd640 1 -- start start 2024-04-15T15:14:29.390 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.384+0000 7f2ff21cd640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2fec0717a0 0x7f2fec0ca170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:29.390 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.384+0000 7f2ff21cd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2fec0ccfa0 0x7f2fec0cb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:29.390 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.384+0000 7f2ff21cd640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2fec15cbe0 0x7f2fec1f22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:29.390 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.384+0000 7f2ff21cd640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2fec0b2020 con 0x7f2fec0ccfa0 2024-04-15T15:14:29.390 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.384+0000 7f2ff21cd640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f2fec0b1d40 con 0x7f2fec15cbe0 2024-04-15T15:14:29.391 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.384+0000 7f2ff21cd640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f2fec0cd380 con 0x7f2fec0717a0 2024-04-15T15:14:29.391 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.384+0000 7f2feb7fe640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2fec0717a0 0x7f2fec0ca170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:29.391 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.384+0000 7f2feb7fe640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2fec0717a0 0x7f2fec0ca170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3301/0 says I am v2:172.21.15.175:42992/0 (socket says 172.21.15.175:42992) 2024-04-15T15:14:29.391 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.384+0000 7f2feb7fe640 1 -- 172.21.15.175:0/2950163197 learned_addr learned my addr 172.21.15.175:0/2950163197 (peer_addr_for_me v2:172.21.15.175:0/0) 2024-04-15T15:14:29.391 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.384+0000 7f2febfff640 1 --2- 172.21.15.175:0/2950163197 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2fec15cbe0 0x7f2fec1f22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:29.391 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.384+0000 7f2fe3fff640 1 --2- 172.21.15.175:0/2950163197 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2fec0ccfa0 0x7f2fec0cb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:29.391 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.385+0000 7f2feb7fe640 1 -- 172.21.15.175:0/2950163197 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2fec15cbe0 msgr2=0x7f2fec1f22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:29.391 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.385+0000 7f2feb7fe640 1 --2- 172.21.15.175:0/2950163197 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2fec15cbe0 0x7f2fec1f22a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:29.391 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.385+0000 7f2feb7fe640 1 -- 172.21.15.175:0/2950163197 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2fec0ccfa0 msgr2=0x7f2fec0cb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:29.391 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.385+0000 7f2feb7fe640 1 --2- 172.21.15.175:0/2950163197 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2fec0ccfa0 0x7f2fec0cb8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:29.392 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.385+0000 7f2feb7fe640 1 -- 172.21.15.175:0/2950163197 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fec0cc010 con 0x7f2fec0717a0 2024-04-15T15:14:29.392 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.385+0000 7f2feb7fe640 1 --2- 172.21.15.175:0/2950163197 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2fec0717a0 0x7f2fec0ca170 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc00c400 tx=0x7f2fdc00c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:29.392 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.385+0000 7f2fe8ff9640 1 -- 172.21.15.175:0/2950163197 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc0052d0 con 0x7f2fec0717a0 2024-04-15T15:14:29.392 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.385+0000 7f2fe8ff9640 1 -- 172.21.15.175:0/2950163197 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fdc005490 con 0x7f2fec0717a0 2024-04-15T15:14:29.392 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.385+0000 7f2ff21cd640 1 -- 172.21.15.175:0/2950163197 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fec0cc2f0 con 0x7f2fec0717a0 2024-04-15T15:14:29.392 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.385+0000 7f2fe8ff9640 1 -- 172.21.15.175:0/2950163197 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc007260 con 0x7f2fec0717a0 2024-04-15T15:14:29.392 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.385+0000 7f2ff21cd640 1 -- 172.21.15.175:0/2950163197 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fec1f27e0 con 0x7f2fec0717a0 2024-04-15T15:14:29.392 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.385+0000 7f2fe8ff9640 1 -- 172.21.15.175:0/2950163197 <== mon.2 v2:172.21.15.175:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2fdc007400 con 0x7f2fec0717a0 2024-04-15T15:14:29.392 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.385+0000 7f2fe8ff9640 1 -- 172.21.15.175:0/2950163197 <== mon.2 v2:172.21.15.175:3301/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f2fdc011bf0 con 0x7f2fec0717a0 2024-04-15T15:14:29.392 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.386+0000 7f2ff21cd640 1 -- 172.21.15.175:0/2950163197 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fec075c50 con 0x7f2fec0717a0 2024-04-15T15:14:29.392 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.388+0000 7f2fe8ff9640 1 -- 172.21.15.175:0/2950163197 <== mon.2 v2:172.21.15.175:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f2fdc0035f0 con 0x7f2fec0717a0 2024-04-15T15:14:29.393 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-04-15T15:14:29.384+0000 7f8ede82d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-15T15:14:29.462 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.461+0000 7f2ff21cd640 1 -- 172.21.15.175:0/2950163197 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_command({"prefix": "osd new", "uuid": "c120037a-a16b-4504-847c-81cb859d31be", "id": 7} v 0) -- 0x7f2fec15e5c0 con 0x7f2fec0717a0 2024-04-15T15:14:29.466 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.465+0000 7f2fe8ff9640 1 -- 172.21.15.175:0/2950163197 <== mon.2 v2:172.21.15.175:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c120037a-a16b-4504-847c-81cb859d31be", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f2fdc003a00 con 0x7f2fec0717a0 2024-04-15T15:14:29.466 INFO:teuthology.orchestra.run.smithi175.stdout:7 2024-04-15T15:14:29.467 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.466+0000 7f2ff21cd640 1 -- 172.21.15.175:0/2950163197 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2fec0717a0 msgr2=0x7f2fec0ca170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:29.467 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.466+0000 7f2ff21cd640 1 --2- 172.21.15.175:0/2950163197 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2fec0717a0 0x7f2fec0ca170 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc00c400 tx=0x7f2fdc00c800 comp rx=0 tx=0).stop 2024-04-15T15:14:29.467 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.466+0000 7f2ff21cd640 1 -- 172.21.15.175:0/2950163197 shutdown_connections 2024-04-15T15:14:29.467 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.466+0000 7f2ff21cd640 1 --2- 172.21.15.175:0/2950163197 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2fec15cbe0 0x7f2fec1f22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:29.467 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.466+0000 7f2ff21cd640 1 --2- 172.21.15.175:0/2950163197 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2fec0ccfa0 0x7f2fec0cb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:29.467 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.466+0000 7f2ff21cd640 1 --2- 172.21.15.175:0/2950163197 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2fec0717a0 0x7f2fec0ca170 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:29.467 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.466+0000 7f2ff21cd640 1 -- 172.21.15.175:0/2950163197 >> 172.21.15.175:0/2950163197 conn(0x7f2fec0d24b0 msgr2=0x7f2fec0b50b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:29.467 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.466+0000 7f2ff21cd640 1 -- 172.21.15.175:0/2950163197 shutdown_connections 2024-04-15T15:14:29.468 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:14:29.466+0000 7f2ff21cd640 1 -- 172.21.15.175:0/2950163197 wait complete. 2024-04-15T15:14:29.477 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-15T15:14:29.477 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-15T15:14:29.480 INFO:tasks.ceph.osd.0:Started 2024-04-15T15:14:29.480 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-15T15:14:29.480 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-15T15:14:29.481 INFO:tasks.ceph.mgr.x.smithi175.stderr:2024-04-15T15:14:29.480+0000 7f4b2580f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-15T15:14:29.482 INFO:tasks.ceph.osd.1:Started 2024-04-15T15:14:29.482 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-15T15:14:29.482 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-15T15:14:29.484 INFO:tasks.ceph.osd.2:Started 2024-04-15T15:14:29.484 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-15T15:14:29.485 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-15T15:14:29.487 INFO:tasks.ceph.osd.3:Started 2024-04-15T15:14:29.487 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-15T15:14:29.487 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-15T15:14:29.489 INFO:tasks.ceph.osd.4:Started 2024-04-15T15:14:29.489 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-15T15:14:29.489 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-15T15:14:29.491 INFO:tasks.ceph.osd.5:Started 2024-04-15T15:14:29.491 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-15T15:14:29.492 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-15T15:14:29.493 INFO:tasks.ceph.osd.6:Started 2024-04-15T15:14:29.494 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-15T15:14:29.494 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-15T15:14:29.498 INFO:tasks.ceph.osd.7:Started 2024-04-15T15:14:29.499 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:14:29.502 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-04-15T15:14:29.500+0000 7f8ede82d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-15T15:14:29.538 INFO:tasks.ceph.mgr.x.smithi175.stderr:2024-04-15T15:14:29.537+0000 7f4b2580f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-15T15:14:29.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.618+0000 7f4285922640 1 Processor -- start 2024-04-15T15:14:29.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.618+0000 7f4285922640 1 -- start start 2024-04-15T15:14:29.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.619+0000 7f4285922640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f428015c480 0x7f428006ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:29.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.619+0000 7f4285922640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f42800ce670 0x7f42800cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:29.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.619+0000 7f4285922640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f42800cf300 0x7f42800cff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:29.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.619+0000 7f4285922640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f42800d1b30 con 0x7f42800cf300 2024-04-15T15:14:29.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.619+0000 7f4285922640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f42800cca00 con 0x7f42800ce670 2024-04-15T15:14:29.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.619+0000 7f4285922640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f42800d1d80 con 0x7f428015c480 2024-04-15T15:14:29.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.619+0000 7f427f7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f42800cf300 0x7f42800cff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:29.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.619+0000 7f427f7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f42800cf300 0x7f42800cff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54510/0 (socket says 172.21.15.72:54510) 2024-04-15T15:14:29.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.619+0000 7f427f7fe640 1 -- 172.21.15.72:0/631713092 learned_addr learned my addr 172.21.15.72:0/631713092 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:29.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.620+0000 7f427e7fc640 1 --2- 172.21.15.72:0/631713092 >> v2:172.21.15.175:3300/0 conn(0x7f42800ce670 0x7f42800cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:29.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.620+0000 7f427effd640 1 --2- 172.21.15.72:0/631713092 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f428015c480 0x7f428006ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:29.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.620+0000 7f427f7fe640 1 -- 172.21.15.72:0/631713092 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f428015c480 msgr2=0x7f428006ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:29.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.620+0000 7f427f7fe640 1 --2- 172.21.15.72:0/631713092 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f428015c480 0x7f428006ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:29.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.620+0000 7f427f7fe640 1 -- 172.21.15.72:0/631713092 >> v2:172.21.15.175:3300/0 conn(0x7f42800ce670 msgr2=0x7f42800cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:29.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.620+0000 7f427f7fe640 1 --2- 172.21.15.72:0/631713092 >> v2:172.21.15.175:3300/0 conn(0x7f42800ce670 0x7f42800cedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:29.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.620+0000 7f427f7fe640 1 -- 172.21.15.72:0/631713092 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42800d2060 con 0x7f42800cf300 2024-04-15T15:14:29.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.621+0000 7f427f7fe640 1 --2- 172.21.15.72:0/631713092 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42800cf300 0x7f42800cff80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f42800d3420 tx=0x7f42780302e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a329a14a77947397 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:29.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.621+0000 7f427dffb640 1 -- 172.21.15.72:0/631713092 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42780383f0 con 0x7f42800cf300 2024-04-15T15:14:29.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.621+0000 7f427dffb640 1 -- 172.21.15.72:0/631713092 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4278038590 con 0x7f42800cf300 2024-04-15T15:14:29.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.621+0000 7f427dffb640 1 -- 172.21.15.72:0/631713092 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4278038960 con 0x7f42800cf300 2024-04-15T15:14:29.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.621+0000 7f4285922640 1 -- 172.21.15.72:0/631713092 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42800cf300 msgr2=0x7f42800cff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:29.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.621+0000 7f4285922640 1 --2- 172.21.15.72:0/631713092 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42800cf300 0x7f42800cff80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f42800d3420 tx=0x7f42780302e0 comp rx=0 tx=0).stop 2024-04-15T15:14:29.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.621+0000 7f4285922640 1 -- 172.21.15.72:0/631713092 shutdown_connections 2024-04-15T15:14:29.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.621+0000 7f4285922640 1 --2- 172.21.15.72:0/631713092 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42800cf300 0x7f42800cff80 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:29.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.621+0000 7f4285922640 1 --2- 172.21.15.72:0/631713092 >> v2:172.21.15.175:3300/0 conn(0x7f42800ce670 0x7f42800cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:29.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.621+0000 7f4285922640 1 --2- 172.21.15.72:0/631713092 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f428015c480 0x7f428006ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:29.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.621+0000 7f4285922640 1 -- 172.21.15.72:0/631713092 >> 172.21.15.72:0/631713092 conn(0x7f42800b3fa0 msgr2=0x7f42800b43b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:29.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.621+0000 7f4285922640 1 -- 172.21.15.72:0/631713092 shutdown_connections 2024-04-15T15:14:29.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.622+0000 7f4285922640 1 -- 172.21.15.72:0/631713092 wait complete. 2024-04-15T15:14:29.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.623+0000 7f4285922640 1 Processor -- start 2024-04-15T15:14:29.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.623+0000 7f4285922640 1 -- start start 2024-04-15T15:14:29.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.623+0000 7f4285922640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f42800ce670 0x7f4280157b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:29.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.624+0000 7f4285922640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f42800cf300 0x7f42800cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:29.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.624+0000 7f4285922640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f428015c480 0x7f428006df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:29.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.624+0000 7f4285922640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f42800cd170 con 0x7f428015c480 2024-04-15T15:14:29.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.624+0000 7f4285922640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f42800cead0 con 0x7f42800ce670 2024-04-15T15:14:29.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.624+0000 7f4285922640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f428015c860 con 0x7f42800cf300 2024-04-15T15:14:29.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.624+0000 7f427f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f428015c480 0x7f428006df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:29.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.624+0000 7f427f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f428015c480 0x7f428006df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54516/0 (socket says 172.21.15.72:54516) 2024-04-15T15:14:29.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.624+0000 7f427f7fe640 1 -- 172.21.15.72:0/3576281997 learned_addr learned my addr 172.21.15.72:0/3576281997 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:29.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.624+0000 7f427e7fc640 1 --2- 172.21.15.72:0/3576281997 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f42800cf300 0x7f42800cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:29.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.624+0000 7f427effd640 1 --2- 172.21.15.72:0/3576281997 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f42800ce670 0x7f4280157b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:29.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.624+0000 7f427f7fe640 1 -- 172.21.15.72:0/3576281997 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f42800cf300 msgr2=0x7f42800cba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:29.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.624+0000 7f427f7fe640 1 --2- 172.21.15.72:0/3576281997 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f42800cf300 0x7f42800cba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:29.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.624+0000 7f427f7fe640 1 -- 172.21.15.72:0/3576281997 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f42800ce670 msgr2=0x7f4280157b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:29.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.624+0000 7f427f7fe640 1 --2- 172.21.15.72:0/3576281997 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f42800ce670 0x7f4280157b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:29.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.624+0000 7f427f7fe640 1 -- 172.21.15.72:0/3576281997 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4280152b80 con 0x7f428015c480 2024-04-15T15:14:29.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.624+0000 7f427f7fe640 1 --2- 172.21.15.72:0/3576281997 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f428015c480 0x7f428006df80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f427803a040 tx=0x7f4278004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:29.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.624+0000 7f4267fff640 1 -- 172.21.15.72:0/3576281997 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4278047020 con 0x7f428015c480 2024-04-15T15:14:29.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.624+0000 7f4267fff640 1 -- 172.21.15.72:0/3576281997 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4278007930 con 0x7f428015c480 2024-04-15T15:14:29.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.624+0000 7f4285922640 1 -- 172.21.15.72:0/3576281997 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42800cc3b0 con 0x7f428015c480 2024-04-15T15:14:29.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.624+0000 7f4267fff640 1 -- 172.21.15.72:0/3576281997 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42780383f0 con 0x7f428015c480 2024-04-15T15:14:29.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.624+0000 7f4285922640 1 -- 172.21.15.72:0/3576281997 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f428006d780 con 0x7f428015c480 2024-04-15T15:14:29.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.625+0000 7f4267fff640 1 -- 172.21.15.72:0/3576281997 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f427804a020 con 0x7f428015c480 2024-04-15T15:14:29.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.625+0000 7f4267fff640 1 -- 172.21.15.72:0/3576281997 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f427804c400 con 0x7f428015c480 2024-04-15T15:14:29.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.625+0000 7f4285922640 1 -- 172.21.15.72:0/3576281997 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42800690f0 con 0x7f428015c480 2024-04-15T15:14:29.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.627+0000 7f4267fff640 1 -- 172.21.15.72:0/3576281997 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f4278051030 con 0x7f428015c480 2024-04-15T15:14:29.631 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-04-15T15:14:29.630+0000 7f8ede82d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-15T15:14:29.671 INFO:tasks.ceph.mgr.x.smithi175.stderr:2024-04-15T15:14:29.670+0000 7f4b2580f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-15T15:14:29.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.689+0000 7f4285922640 1 -- 172.21.15.72:0/3576281997 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f42800cf7e0 con 0x7f428015c480 2024-04-15T15:14:29.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.689+0000 7f4267fff640 1 -- 172.21.15.72:0/3576281997 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f4278007d20 con 0x7f428015c480 2024-04-15T15:14:29.691 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:14:29.691 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":10,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:14:29.462079+0000","last_up_change":"0.000000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:14:29.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.690+0000 7f4285922640 1 -- 172.21.15.72:0/3576281997 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f428015c480 msgr2=0x7f428006df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:29.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.690+0000 7f4285922640 1 --2- 172.21.15.72:0/3576281997 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f428015c480 0x7f428006df80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f427803a040 tx=0x7f4278004a40 comp rx=0 tx=0).stop 2024-04-15T15:14:29.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.690+0000 7f4285922640 1 -- 172.21.15.72:0/3576281997 shutdown_connections 2024-04-15T15:14:29.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.690+0000 7f4285922640 1 --2- 172.21.15.72:0/3576281997 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f428015c480 0x7f428006df80 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:29.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.690+0000 7f4285922640 1 --2- 172.21.15.72:0/3576281997 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f42800cf300 0x7f42800cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:29.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.690+0000 7f4285922640 1 --2- 172.21.15.72:0/3576281997 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f42800ce670 0x7f4280157b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:29.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.690+0000 7f4285922640 1 -- 172.21.15.72:0/3576281997 >> 172.21.15.72:0/3576281997 conn(0x7f42800b3fa0 msgr2=0x7f42800b5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:29.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.690+0000 7f4285922640 1 -- 172.21.15.72:0/3576281997 shutdown_connections 2024-04-15T15:14:29.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:29.690+0000 7f4285922640 1 -- 172.21.15.72:0/3576281997 wait complete. 2024-04-15T15:14:29.702 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-15T15:14:29.703 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-15T15:14:29.765 INFO:tasks.ceph.mgr.x.smithi175.stderr:2024-04-15T15:14:29.764+0000 7f4b2580f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-15T15:14:29.792 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-04-15T15:14:29.790+0000 7f8ede82d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-15T15:14:29.834 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-04-15T15:14:29.833+0000 7f8ede82d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-15T15:14:29.921 INFO:tasks.ceph.mgr.x.smithi175.stderr:2024-04-15T15:14:29.920+0000 7f4b2580f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-15T15:14:29.964 INFO:tasks.ceph.mgr.x.smithi175.stderr:2024-04-15T15:14:29.963+0000 7f4b2580f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-15T15:14:29.991 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-04-15T15:14:29.990+0000 7f8ede82d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-15T15:14:30.087 INFO:tasks.ceph.osd.0.smithi072.stderr:2024-04-15T15:14:30.085+0000 7f5af2d2a640 -1 Falling back to public interface 2024-04-15T15:14:30.096 INFO:tasks.ceph.osd.5.smithi175.stderr:2024-04-15T15:14:30.095+0000 7ff074a0c640 -1 Falling back to public interface 2024-04-15T15:14:30.105 INFO:tasks.ceph.osd.6.smithi175.stderr:2024-04-15T15:14:30.104+0000 7f18aa811640 -1 Falling back to public interface 2024-04-15T15:14:30.105 INFO:tasks.ceph.osd.3.smithi072.stderr:2024-04-15T15:14:30.104+0000 7ff3686b5640 -1 Falling back to public interface 2024-04-15T15:14:30.108 INFO:tasks.ceph.osd.2.smithi072.stderr:2024-04-15T15:14:30.106+0000 7f1657773640 -1 Falling back to public interface 2024-04-15T15:14:30.110 INFO:tasks.ceph.osd.7.smithi175.stderr:2024-04-15T15:14:30.109+0000 7f6b7540c640 -1 Falling back to public interface 2024-04-15T15:14:30.112 INFO:tasks.ceph.osd.4.smithi175.stderr:2024-04-15T15:14:30.111+0000 7f9e531bd640 -1 Falling back to public interface 2024-04-15T15:14:30.119 INFO:tasks.ceph.osd.1.smithi072.stderr:2024-04-15T15:14:30.118+0000 7fc3b8f93640 -1 Falling back to public interface 2024-04-15T15:14:30.124 INFO:tasks.ceph.mgr.x.smithi175.stderr:2024-04-15T15:14:30.124+0000 7f4b2580f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-15T15:14:30.511 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-15T15:14:30.603 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.602+0000 7fa28e463640 1 Processor -- start 2024-04-15T15:14:30.604 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.602+0000 7fa28e463640 1 -- start start 2024-04-15T15:14:30.608 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.603+0000 7fa28e463640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa28815e950 0x7fa2880cb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:30.608 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.603+0000 7fa28e463640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2880d0ec0 con 0x7fa2880b1ea0 2024-04-15T15:14:30.608 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.603+0000 7fa28e463640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2880cd6f0 con 0x7fa28806b9e0 2024-04-15T15:14:30.609 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.603+0000 7fa28e463640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fa2880d0ca0 con 0x7fa28815e950 2024-04-15T15:14:30.609 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.603+0000 7fa2877fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fa2880b1ea0 0x7fa2880d0590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:55376/0 (socket says 172.21.15.72:55376) 2024-04-15T15:14:30.609 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.603+0000 7fa2877fe640 1 -- 172.21.15.72:0/3034464649 learned_addr learned my addr 172.21.15.72:0/3034464649 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:14:30.609 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.603+0000 7fa287fff640 1 --2- 172.21.15.72:0/3034464649 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa28815e950 0x7fa2880cb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:30.609 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.603+0000 7fa286ffd640 1 -- 172.21.15.72:0/3034464649 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 51844076 0 0) 0x7fa2880d0ec0 con 0x7fa2880b1ea0 2024-04-15T15:14:30.609 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.603+0000 7fa286ffd640 1 -- 172.21.15.72:0/3034464649 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa270003520 con 0x7fa2880b1ea0 2024-04-15T15:14:30.609 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.603+0000 7fa286ffd640 1 -- 172.21.15.72:0/3034464649 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 954340941 0 0) 0x7fa2880cd6f0 con 0x7fa28806b9e0 2024-04-15T15:14:30.609 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.603+0000 7fa286ffd640 1 -- 172.21.15.72:0/3034464649 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2880d0ec0 con 0x7fa28806b9e0 2024-04-15T15:14:30.609 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.603+0000 7fa286ffd640 1 -- 172.21.15.72:0/3034464649 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1778243967 0 0) 0x7fa270003520 con 0x7fa2880b1ea0 2024-04-15T15:14:30.609 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.603+0000 7fa286ffd640 1 -- 172.21.15.72:0/3034464649 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa28815e950 msgr2=0x7fa2880cb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:30.609 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.603+0000 7fa286ffd640 1 --2- 172.21.15.72:0/3034464649 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa28815e950 0x7fa2880cb9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:30.609 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.603+0000 7fa286ffd640 1 -- 172.21.15.72:0/3034464649 >> v1:172.21.15.175:6789/0 conn(0x7fa28806b9e0 legacy=0x7fa2880ce1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:30.610 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.603+0000 7fa286ffd640 1 -- 172.21.15.72:0/3034464649 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2880cd9d0 con 0x7fa2880b1ea0 2024-04-15T15:14:30.610 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.603+0000 7fa286ffd640 1 -- 172.21.15.72:0/3034464649 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7fa27c001670 con 0x7fa2880b1ea0 2024-04-15T15:14:30.610 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.603+0000 7fa287fff640 1 --2- 172.21.15.72:0/3034464649 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa28815e950 0x7fa2880cb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:14:30.610 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.603+0000 7fa286ffd640 1 -- 172.21.15.72:0/3034464649 >> v1:172.21.15.72:6789/0 conn(0x7fa2880b1ea0 legacy=0x7fa2880d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:30.610 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.603+0000 7fa286ffd640 1 --2- 172.21.15.72:0/3034464649 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa270003b70 0x7fa270003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:30.610 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.603+0000 7fa286ffd640 1 --2- 172.21.15.72:0/3034464649 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa28815e950 0x7fa270004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:30.610 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.603+0000 7fa286ffd640 1 --2- 172.21.15.72:0/3034464649 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa270004d40 0x7fa2700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:30.610 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.603+0000 7fa286ffd640 1 -- 172.21.15.72:0/3034464649 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fa2880d0ca0 con 0x7fa270003b70 2024-04-15T15:14:30.611 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.603+0000 7fa286ffd640 1 -- 172.21.15.72:0/3034464649 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fa27000ba70 con 0x7fa270004d40 2024-04-15T15:14:30.611 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.604+0000 7fa286ffd640 1 -- 172.21.15.72:0/3034464649 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fa27000bbf0 con 0x7fa28815e950 2024-04-15T15:14:30.611 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.604+0000 7fa287fff640 1 --2- 172.21.15.72:0/3034464649 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa270003b70 0x7fa270003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:30.611 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.604+0000 7fa28ca00640 1 --2- 172.21.15.72:0/3034464649 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa270004d40 0x7fa2700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:30.611 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.604+0000 7fa2877fe640 1 --2- 172.21.15.72:0/3034464649 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa28815e950 0x7fa270004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:30.611 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.604+0000 7fa287fff640 1 -- 172.21.15.72:0/3034464649 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa28815e950 msgr2=0x7fa270004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:30.611 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.604+0000 7fa287fff640 1 --2- 172.21.15.72:0/3034464649 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa28815e950 0x7fa270004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:30.611 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.604+0000 7fa287fff640 1 -- 172.21.15.72:0/3034464649 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa270004d40 msgr2=0x7fa2700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:30.611 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.604+0000 7fa287fff640 1 --2- 172.21.15.72:0/3034464649 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa270004d40 0x7fa2700051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:30.611 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.604+0000 7fa287fff640 1 -- 172.21.15.72:0/3034464649 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa270003520 con 0x7fa270003b70 2024-04-15T15:14:30.612 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.605+0000 7fa287fff640 1 --2- 172.21.15.72:0/3034464649 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa270003b70 0x7fa270003f70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa278009cc0 tx=0x7fa270005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=700ad75eef5999a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:30.612 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.605+0000 7fa286ffd640 1 -- 172.21.15.72:0/3034464649 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa278039070 con 0x7fa270003b70 2024-04-15T15:14:30.612 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.605+0000 7fa286ffd640 1 -- 172.21.15.72:0/3034464649 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa27802c8f0 con 0x7fa270003b70 2024-04-15T15:14:30.612 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.605+0000 7fa286ffd640 1 -- 172.21.15.72:0/3034464649 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa278009860 con 0x7fa270003b70 2024-04-15T15:14:30.612 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.605+0000 7fa28e463640 1 -- 172.21.15.72:0/3034464649 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa270003b70 msgr2=0x7fa270003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:30.612 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.605+0000 7fa28e463640 1 --2- 172.21.15.72:0/3034464649 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa270003b70 0x7fa270003f70 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa278009cc0 tx=0x7fa270005830 comp rx=0 tx=0).stop 2024-04-15T15:14:30.612 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.605+0000 7fa28ca00640 1 -- 172.21.15.72:0/3034464649 reap_dead start 2024-04-15T15:14:30.612 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.606+0000 7fa28e463640 1 -- 172.21.15.72:0/3034464649 shutdown_connections 2024-04-15T15:14:30.612 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.606+0000 7fa28e463640 1 -- 172.21.15.72:0/3034464649 >> 172.21.15.72:0/3034464649 conn(0x7fa2880b2f40 msgr2=0x7fa2880b3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:30.612 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.606+0000 7fa28e463640 1 -- 172.21.15.72:0/3034464649 shutdown_connections 2024-04-15T15:14:30.612 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.606+0000 7fa28e463640 1 -- 172.21.15.72:0/3034464649 wait complete. 2024-04-15T15:14:30.616 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.607+0000 7fa28e463640 1 Processor -- start 2024-04-15T15:14:30.616 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.608+0000 7fa28e463640 1 -- start start 2024-04-15T15:14:30.616 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.609+0000 7fa28e463640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa270003b70 0x7fa288157420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:30.616 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.609+0000 7fa28e463640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa2880c4ea0 0x7fa2880c6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:30.616 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.609+0000 7fa28e463640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa2880c8e50 0x7fa2880c9c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:30.616 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.609+0000 7fa28e463640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fa28815eeb0 con 0x7fa270003b70 2024-04-15T15:14:30.616 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.609+0000 7fa28e463640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fa28815ed30 con 0x7fa2880c8e50 2024-04-15T15:14:30.616 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.609+0000 7fa28e463640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fa2880ce8e0 con 0x7fa2880c4ea0 2024-04-15T15:14:30.616 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.609+0000 7fa287fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa270003b70 0x7fa288157420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:30.616 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.609+0000 7fa287fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa270003b70 0x7fa288157420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54530/0 (socket says 172.21.15.72:54530) 2024-04-15T15:14:30.617 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.609+0000 7fa287fff640 1 -- 172.21.15.72:0/3141776262 learned_addr learned my addr 172.21.15.72:0/3141776262 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:30.617 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.609+0000 7fa28ca00640 1 --2- 172.21.15.72:0/3141776262 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa2880c8e50 0x7fa2880c9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:30.617 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.609+0000 7fa2877fe640 1 --2- 172.21.15.72:0/3141776262 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa2880c4ea0 0x7fa2880c6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:30.617 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.609+0000 7fa287fff640 1 -- 172.21.15.72:0/3141776262 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa2880c4ea0 msgr2=0x7fa2880c6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:30.617 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.609+0000 7fa287fff640 1 --2- 172.21.15.72:0/3141776262 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa2880c4ea0 0x7fa2880c6440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:30.617 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.609+0000 7fa287fff640 1 -- 172.21.15.72:0/3141776262 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa2880c8e50 msgr2=0x7fa2880c9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:30.617 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.609+0000 7fa287fff640 1 --2- 172.21.15.72:0/3141776262 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa2880c8e50 0x7fa2880c9c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:30.617 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.609+0000 7fa287fff640 1 -- 172.21.15.72:0/3141776262 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2880c6980 con 0x7fa270003b70 2024-04-15T15:14:30.617 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.609+0000 7fa28ca00640 1 --2- 172.21.15.72:0/3141776262 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa2880c8e50 0x7fa2880c9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:14:30.617 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.609+0000 7fa287fff640 1 --2- 172.21.15.72:0/3141776262 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa270003b70 0x7fa288157420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa278009cc0 tx=0x7fa278004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:30.618 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.609+0000 7fa284ff9640 1 -- 172.21.15.72:0/3141776262 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa278039050 con 0x7fa270003b70 2024-04-15T15:14:30.618 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.610+0000 7fa284ff9640 1 -- 172.21.15.72:0/3141776262 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa278045070 con 0x7fa270003b70 2024-04-15T15:14:30.618 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.610+0000 7fa28e463640 1 -- 172.21.15.72:0/3141776262 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2880c53a0 con 0x7fa270003b70 2024-04-15T15:14:30.618 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.610+0000 7fa284ff9640 1 -- 172.21.15.72:0/3141776262 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa278009860 con 0x7fa270003b70 2024-04-15T15:14:30.618 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.610+0000 7fa28e463640 1 -- 172.21.15.72:0/3141776262 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2880c5960 con 0x7fa270003b70 2024-04-15T15:14:30.618 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.610+0000 7fa284ff9640 1 -- 172.21.15.72:0/3141776262 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa27804a460 con 0x7fa270003b70 2024-04-15T15:14:30.618 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.610+0000 7fa284ff9640 1 -- 172.21.15.72:0/3141776262 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fa278040b20 con 0x7fa270003b70 2024-04-15T15:14:30.618 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.610+0000 7fa28e463640 1 -- 172.21.15.72:0/3141776262 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa28806be50 con 0x7fa270003b70 2024-04-15T15:14:30.618 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.612+0000 7fa284ff9640 1 -- 172.21.15.72:0/3141776262 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fa28806be50 con 0x7fa270003b70 2024-04-15T15:14:30.676 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.674+0000 7fa28e463640 1 -- 172.21.15.72:0/3141776262 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa288069490 con 0x7fa270003b70 2024-04-15T15:14:30.676 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.675+0000 7fa284ff9640 1 -- 172.21.15.72:0/3141776262 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fa278040450 con 0x7fa270003b70 2024-04-15T15:14:30.677 INFO:teuthology.misc.health.smithi072.stdout: 2024-04-15T15:14:30.677 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":10,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:14:29.462079+0000","last_up_change":"0.000000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:14:30.678 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.676+0000 7fa28e463640 1 -- 172.21.15.72:0/3141776262 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa270003b70 msgr2=0x7fa288157420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:30.678 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.676+0000 7fa28e463640 1 --2- 172.21.15.72:0/3141776262 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa270003b70 0x7fa288157420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa278009cc0 tx=0x7fa278004730 comp rx=0 tx=0).stop 2024-04-15T15:14:30.678 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.676+0000 7fa28e463640 1 -- 172.21.15.72:0/3141776262 shutdown_connections 2024-04-15T15:14:30.678 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.676+0000 7fa28e463640 1 --2- 172.21.15.72:0/3141776262 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa2880c8e50 0x7fa2880c9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:30.678 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.676+0000 7fa28e463640 1 --2- 172.21.15.72:0/3141776262 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa2880c4ea0 0x7fa2880c6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:30.678 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.676+0000 7fa28e463640 1 --2- 172.21.15.72:0/3141776262 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa270003b70 0x7fa288157420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:30.678 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.676+0000 7fa28e463640 1 -- 172.21.15.72:0/3141776262 >> 172.21.15.72:0/3141776262 conn(0x7fa2880b2f40 msgr2=0x7fa2880b3350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:30.678 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.676+0000 7fa28e463640 1 -- 172.21.15.72:0/3141776262 shutdown_connections 2024-04-15T15:14:30.679 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:30.676+0000 7fa28e463640 1 -- 172.21.15.72:0/3141776262 wait complete. 2024-04-15T15:14:30.689 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-15T15:14:30.892 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-04-15T15:14:30.891+0000 7f8ede82d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-15T15:14:30.934 INFO:tasks.ceph.mgr.x.smithi175.stderr:2024-04-15T15:14:30.933+0000 7f4b2580f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-15T15:14:31.169 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-04-15T15:14:31.167+0000 7f8ede82d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-15T15:14:31.216 INFO:tasks.ceph.mgr.x.smithi175.stderr:2024-04-15T15:14:31.214+0000 7f4b2580f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-15T15:14:31.220 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-04-15T15:14:31.219+0000 7f8ede82d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-15T15:14:31.267 INFO:tasks.ceph.mgr.x.smithi175.stderr:2024-04-15T15:14:31.266+0000 7f4b2580f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-15T15:14:31.335 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-04-15T15:14:31.333+0000 7f8ede82d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-15T15:14:31.388 INFO:tasks.ceph.mgr.x.smithi175.stderr:2024-04-15T15:14:31.386+0000 7f4b2580f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-15T15:14:32.348 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-04-15T15:14:32.346+0000 7f8e7ecdc640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-15T15:14:32.348 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-04-15T15:14:32.347+0000 7f8e7ecdc640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-15T15:14:33.542 INFO:tasks.ceph.osd.0.smithi072.stderr:2024-04-15T15:14:33.539+0000 7f5af2d2a640 -1 osd.0 0 log_to_monitors true 2024-04-15T15:14:33.598 INFO:tasks.ceph.osd.7.smithi175.stderr:2024-04-15T15:14:33.596+0000 7f6b7540c640 -1 osd.7 0 log_to_monitors true 2024-04-15T15:14:33.621 INFO:tasks.ceph.osd.5.smithi175.stderr:2024-04-15T15:14:33.620+0000 7ff074a0c640 -1 osd.5 0 log_to_monitors true 2024-04-15T15:14:33.628 INFO:tasks.ceph.osd.6.smithi175.stderr:2024-04-15T15:14:33.626+0000 7f18aa811640 -1 osd.6 0 log_to_monitors true 2024-04-15T15:14:33.642 INFO:tasks.ceph.osd.4.smithi175.stderr:2024-04-15T15:14:33.640+0000 7f9e531bd640 -1 osd.4 0 log_to_monitors true 2024-04-15T15:14:33.647 INFO:tasks.ceph.osd.2.smithi072.stderr:2024-04-15T15:14:33.646+0000 7f1657773640 -1 osd.2 0 log_to_monitors true 2024-04-15T15:14:33.656 INFO:tasks.ceph.osd.1.smithi072.stderr:2024-04-15T15:14:33.655+0000 7fc3b8f93640 -1 osd.1 0 log_to_monitors true 2024-04-15T15:14:33.657 INFO:tasks.ceph.osd.3.smithi072.stderr:2024-04-15T15:14:33.656+0000 7ff3686b5640 -1 osd.3 0 log_to_monitors true 2024-04-15T15:14:37.414 INFO:tasks.ceph.osd.0.smithi072.stderr:2024-04-15T15:14:37.411+0000 7f5aeecc0640 -1 osd.0 0 waiting for initial osdmap 2024-04-15T15:14:37.434 INFO:tasks.ceph.osd.0.smithi072.stderr:2024-04-15T15:14:37.432+0000 7f5ae6ae0640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T15:14:37.441 INFO:tasks.ceph.osd.3.smithi072.stderr:2024-04-15T15:14:37.438+0000 7ff364e5f640 -1 osd.3 0 waiting for initial osdmap 2024-04-15T15:14:37.447 INFO:tasks.ceph.osd.4.smithi175.stderr:2024-04-15T15:14:37.444+0000 7f9e4f967640 -1 osd.4 0 waiting for initial osdmap 2024-04-15T15:14:37.456 INFO:tasks.ceph.osd.3.smithi072.stderr:2024-04-15T15:14:37.454+0000 7ff35c46b640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T15:14:37.457 INFO:tasks.ceph.osd.6.smithi175.stderr:2024-04-15T15:14:37.456+0000 7f18a6923640 -1 osd.6 0 waiting for initial osdmap 2024-04-15T15:14:37.461 INFO:tasks.ceph.osd.4.smithi175.stderr:2024-04-15T15:14:37.459+0000 7f9e46f73640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T15:14:37.466 INFO:tasks.ceph.osd.2.smithi072.stderr:2024-04-15T15:14:37.464+0000 7f1653f1d640 -1 osd.2 0 waiting for initial osdmap 2024-04-15T15:14:37.470 INFO:tasks.ceph.osd.1.smithi072.stderr:2024-04-15T15:14:37.467+0000 7fc3b4f29640 -1 osd.1 0 waiting for initial osdmap 2024-04-15T15:14:37.473 INFO:tasks.ceph.osd.6.smithi175.stderr:2024-04-15T15:14:37.471+0000 7f189df2f640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T15:14:37.478 INFO:tasks.ceph.osd.7.smithi175.stderr:2024-04-15T15:14:37.477+0000 7f6b714da640 -1 osd.7 0 waiting for initial osdmap 2024-04-15T15:14:37.485 INFO:tasks.ceph.osd.1.smithi072.stderr:2024-04-15T15:14:37.483+0000 7fc3acd49640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T15:14:37.486 INFO:tasks.ceph.osd.2.smithi072.stderr:2024-04-15T15:14:37.483+0000 7f164b529640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T15:14:37.490 INFO:tasks.ceph.osd.7.smithi175.stderr:2024-04-15T15:14:37.489+0000 7f6b68ae6640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T15:14:37.498 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-15T15:14:37.588 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.586+0000 7f8c133a3640 1 Processor -- start 2024-04-15T15:14:37.589 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.586+0000 7f8c133a3640 1 -- start start 2024-04-15T15:14:37.590 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.587+0000 7f8c133a3640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8c0c15e5c0 0x7f8c0c0d32d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:37.590 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.587+0000 7f8c133a3640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8c0c0b66f0 0x7f8c0c0d3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:37.590 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.587+0000 7f8c133a3640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c0c0cef00 con 0x7f8c0c0cffa0 2024-04-15T15:14:37.591 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.587+0000 7f8c133a3640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f8c0c0cf1e0 con 0x7f8c0c15e5c0 2024-04-15T15:14:37.591 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.587+0000 7f8c133a3640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f8c0c0cf4c0 con 0x7f8c0c0b66f0 2024-04-15T15:14:37.591 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.587+0000 7f8c1113f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8c0c15e5c0 0x7f8c0c0d32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:37.591 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.587+0000 7f8c1113f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8c0c15e5c0 0x7f8c0c0d32d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43198/0 (socket says 172.21.15.72:43198) 2024-04-15T15:14:37.591 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.587+0000 7f8c1113f640 1 -- 172.21.15.72:0/383354141 learned_addr learned my addr 172.21.15.72:0/383354141 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:37.591 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.587+0000 7f8c1093e640 1 --1- 172.21.15.72:0/383354141 >> v1:172.21.15.72:6789/0 conn(0x7f8c0c0cffa0 0x7f8c0c0b5fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:34688/0 (socket says 172.21.15.72:34688) 2024-04-15T15:14:37.591 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.587+0000 7f8c11940640 1 --2- 172.21.15.72:0/383354141 >> v2:172.21.15.175:3300/0 conn(0x7f8c0c0b66f0 0x7f8c0c0d3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:37.591 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.587+0000 7f8c1113f640 1 -- 172.21.15.72:0/383354141 >> v2:172.21.15.175:3300/0 conn(0x7f8c0c0b66f0 msgr2=0x7f8c0c0d3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:37.591 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.587+0000 7f8c1113f640 1 --2- 172.21.15.72:0/383354141 >> v2:172.21.15.175:3300/0 conn(0x7f8c0c0b66f0 0x7f8c0c0d3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:37.591 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.587+0000 7f8c1113f640 1 -- 172.21.15.72:0/383354141 >> v1:172.21.15.72:6789/0 conn(0x7f8c0c0cffa0 legacy=0x7f8c0c0b5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:37.591 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.587+0000 7f8c1113f640 1 -- 172.21.15.72:0/383354141 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c0c0cf7a0 con 0x7f8c0c15e5c0 2024-04-15T15:14:37.591 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.588+0000 7f8c1113f640 1 --2- 172.21.15.72:0/383354141 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8c0c15e5c0 0x7f8c0c0d32d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c0d08c0 tx=0x7f8c0802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51b2940eff63d3bc server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:37.591 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.588+0000 7f8c03fff640 1 -- 172.21.15.72:0/383354141 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c0802ca80 con 0x7f8c0c15e5c0 2024-04-15T15:14:37.592 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.588+0000 7f8c03fff640 1 -- 172.21.15.72:0/383354141 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c0802cc20 con 0x7f8c0c15e5c0 2024-04-15T15:14:37.592 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.588+0000 7f8c03fff640 1 -- 172.21.15.72:0/383354141 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c080343f0 con 0x7f8c0c15e5c0 2024-04-15T15:14:37.596 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.588+0000 7f8c133a3640 1 -- 172.21.15.72:0/383354141 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8c0c15e5c0 msgr2=0x7f8c0c0d32d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:37.596 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.588+0000 7f8c133a3640 1 --2- 172.21.15.72:0/383354141 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8c0c15e5c0 0x7f8c0c0d32d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c0d08c0 tx=0x7f8c0802bee0 comp rx=0 tx=0).stop 2024-04-15T15:14:37.596 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.589+0000 7f8c133a3640 1 -- 172.21.15.72:0/383354141 shutdown_connections 2024-04-15T15:14:37.596 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.589+0000 7f8c133a3640 1 --2- 172.21.15.72:0/383354141 >> v2:172.21.15.175:3300/0 conn(0x7f8c0c0b66f0 0x7f8c0c0d3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:37.596 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.589+0000 7f8c133a3640 1 --2- 172.21.15.72:0/383354141 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8c0c15e5c0 0x7f8c0c0d32d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:37.596 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.589+0000 7f8c133a3640 1 -- 172.21.15.72:0/383354141 >> 172.21.15.72:0/383354141 conn(0x7f8c0c08bfa0 msgr2=0x7f8c0c0d4560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:37.596 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.589+0000 7f8c133a3640 1 -- 172.21.15.72:0/383354141 shutdown_connections 2024-04-15T15:14:37.597 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.589+0000 7f8c133a3640 1 -- 172.21.15.72:0/383354141 wait complete. 2024-04-15T15:14:37.597 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.590+0000 7f8c133a3640 1 Processor -- start 2024-04-15T15:14:37.597 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.590+0000 7f8c133a3640 1 -- start start 2024-04-15T15:14:37.597 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.590+0000 7f8c133a3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8c0c0b66f0 0x7f8c0c0ce650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:37.597 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.591+0000 7f8c133a3640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8c0c0cffa0 0x7f8c0c15f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:37.597 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.591+0000 7f8c133a3640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8c0c15e5c0 0x7f8c0c15a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:37.597 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.591+0000 7f8c133a3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8c0c15e9a0 con 0x7f8c0c0b66f0 2024-04-15T15:14:37.597 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.591+0000 7f8c133a3640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f8c0c0cebb0 con 0x7f8c0c0cffa0 2024-04-15T15:14:37.597 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.591+0000 7f8c133a3640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f8c0c15eb20 con 0x7f8c0c15e5c0 2024-04-15T15:14:37.597 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.591+0000 7f8c1113f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8c0c0b66f0 0x7f8c0c0ce650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:37.597 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.591+0000 7f8c1113f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8c0c0b66f0 0x7f8c0c0ce650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43208/0 (socket says 172.21.15.72:43208) 2024-04-15T15:14:37.597 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.591+0000 7f8c1113f640 1 -- 172.21.15.72:0/1533076893 learned_addr learned my addr 172.21.15.72:0/1533076893 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:37.597 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.591+0000 7f8c11940640 1 --2- 172.21.15.72:0/1533076893 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8c0c15e5c0 0x7f8c0c15a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:37.597 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.591+0000 7f8c1093e640 1 --2- 172.21.15.72:0/1533076893 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8c0c0cffa0 0x7f8c0c15f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:37.598 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.591+0000 7f8c1113f640 1 -- 172.21.15.72:0/1533076893 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8c0c15e5c0 msgr2=0x7f8c0c15a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:37.598 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.591+0000 7f8c1113f640 1 --2- 172.21.15.72:0/1533076893 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8c0c15e5c0 0x7f8c0c15a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:37.598 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.591+0000 7f8c1113f640 1 -- 172.21.15.72:0/1533076893 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8c0c0cffa0 msgr2=0x7f8c0c15f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:37.598 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.591+0000 7f8c1113f640 1 --2- 172.21.15.72:0/1533076893 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8c0c0cffa0 0x7f8c0c15f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:37.598 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.591+0000 7f8c1113f640 1 -- 172.21.15.72:0/1533076893 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c0c160b70 con 0x7f8c0c0b66f0 2024-04-15T15:14:37.598 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.591+0000 7f8c1093e640 1 --2- 172.21.15.72:0/1533076893 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8c0c0cffa0 0x7f8c0c15f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:14:37.598 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.591+0000 7f8c11940640 1 --2- 172.21.15.72:0/1533076893 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8c0c15e5c0 0x7f8c0c15a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:14:37.598 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.591+0000 7f8c1113f640 1 --2- 172.21.15.72:0/1533076893 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8c0c0b66f0 0x7f8c0c0ce650 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8c08036040 tx=0x7f8c08004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:37.598 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.591+0000 7f8c01ffb640 1 -- 172.21.15.72:0/1533076893 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c08043020 con 0x7f8c0c0b66f0 2024-04-15T15:14:37.598 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.591+0000 7f8c01ffb640 1 -- 172.21.15.72:0/1533076893 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c0803e450 con 0x7f8c0c0b66f0 2024-04-15T15:14:37.598 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.591+0000 7f8c133a3640 1 -- 172.21.15.72:0/1533076893 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c0c160e50 con 0x7f8c0c0b66f0 2024-04-15T15:14:37.598 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.591+0000 7f8c01ffb640 1 -- 172.21.15.72:0/1533076893 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c08009510 con 0x7f8c0c0b66f0 2024-04-15T15:14:37.598 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.591+0000 7f8c133a3640 1 -- 172.21.15.72:0/1533076893 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c0c161410 con 0x7f8c0c0b66f0 2024-04-15T15:14:37.599 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.592+0000 7f8c133a3640 1 -- 172.21.15.72:0/1533076893 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c0c069490 con 0x7f8c0c0b66f0 2024-04-15T15:14:37.599 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.593+0000 7f8c01ffb640 1 -- 172.21.15.72:0/1533076893 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7f8c08046020 con 0x7f8c0c0b66f0 2024-04-15T15:14:37.599 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.593+0000 7f8c01ffb640 1 --2- 172.21.15.72:0/1533076893 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8bf006c860 0x7f8bf006ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:37.599 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.593+0000 7f8c01ffb640 1 -- 172.21.15.72:0/1533076893 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2231+0+0 (secure 0 0 0) 0x7f8c080ad4c0 con 0x7f8c0c0b66f0 2024-04-15T15:14:37.599 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.593+0000 7f8c1093e640 1 --2- 172.21.15.72:0/1533076893 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8bf006c860 0x7f8bf006ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:37.599 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.594+0000 7f8c1093e640 1 --2- 172.21.15.72:0/1533076893 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8bf006c860 0x7f8bf006ed20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc007fd0 tx=0x7f8bfc007d60 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:37.600 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.598+0000 7f8c01ffb640 1 -- 172.21.15.72:0/1533076893 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8c0804d030 con 0x7f8c0c0b66f0 2024-04-15T15:14:37.634 INFO:tasks.ceph.osd.5.smithi175.stderr:2024-04-15T15:14:37.632+0000 7ff07037d640 -1 osd.5 0 waiting for initial osdmap 2024-04-15T15:14:37.656 INFO:tasks.ceph.osd.5.smithi175.stderr:2024-04-15T15:14:37.654+0000 7ff06819d640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T15:14:37.776 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.773+0000 7f8c133a3640 1 -- 172.21.15.72:0/1533076893 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8c0c0b6d30 con 0x7f8c0c0b66f0 2024-04-15T15:14:37.776 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.774+0000 7f8c01ffb640 1 -- 172.21.15.72:0/1533076893 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) ==== 74+0+7058 (secure 0 0 0) 0x7f8c0807c7f0 con 0x7f8c0c0b66f0 2024-04-15T15:14:37.776 INFO:teuthology.misc.health.smithi072.stdout: 2024-04-15T15:14:37.776 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":12,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:14:35.347735+0000","last_up_change":"0.000000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:14:37.779 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.776+0000 7f8c133a3640 1 -- 172.21.15.72:0/1533076893 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8bf006c860 msgr2=0x7f8bf006ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:37.779 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.776+0000 7f8c133a3640 1 --2- 172.21.15.72:0/1533076893 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8bf006c860 0x7f8bf006ed20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8bfc007fd0 tx=0x7f8bfc007d60 comp rx=0 tx=0).stop 2024-04-15T15:14:37.779 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.776+0000 7f8c133a3640 1 -- 172.21.15.72:0/1533076893 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8c0c0b66f0 msgr2=0x7f8c0c0ce650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:37.779 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.776+0000 7f8c133a3640 1 --2- 172.21.15.72:0/1533076893 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8c0c0b66f0 0x7f8c0c0ce650 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8c08036040 tx=0x7f8c08004a40 comp rx=0 tx=0).stop 2024-04-15T15:14:37.779 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.776+0000 7f8c133a3640 1 -- 172.21.15.72:0/1533076893 shutdown_connections 2024-04-15T15:14:37.779 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.776+0000 7f8c133a3640 1 --2- 172.21.15.72:0/1533076893 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8bf006c860 0x7f8bf006ed20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:37.779 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.776+0000 7f8c133a3640 1 --2- 172.21.15.72:0/1533076893 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8c0c15e5c0 0x7f8c0c15a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:37.779 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.776+0000 7f8c133a3640 1 --2- 172.21.15.72:0/1533076893 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8c0c0cffa0 0x7f8c0c15f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:37.779 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.776+0000 7f8c133a3640 1 --2- 172.21.15.72:0/1533076893 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8c0c0b66f0 0x7f8c0c0ce650 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:37.779 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.776+0000 7f8c133a3640 1 -- 172.21.15.72:0/1533076893 >> 172.21.15.72:0/1533076893 conn(0x7f8c0c08bfa0 msgr2=0x7f8c0c06da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:37.779 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.776+0000 7f8c133a3640 1 -- 172.21.15.72:0/1533076893 shutdown_connections 2024-04-15T15:14:37.779 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:37.776+0000 7f8c133a3640 1 -- 172.21.15.72:0/1533076893 wait complete. 2024-04-15T15:14:37.788 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-15T15:14:44.596 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-15T15:14:44.698 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.695+0000 7fe54e3cd640 1 Processor -- start 2024-04-15T15:14:44.698 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.695+0000 7fe54e3cd640 1 -- start start 2024-04-15T15:14:44.700 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.696+0000 7fe54e3cd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe5480b66f0 0x7fe548169650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:44.700 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.696+0000 7fe54e3cd640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5480cef00 con 0x7fe54815e5c0 2024-04-15T15:14:44.700 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.696+0000 7fe54e3cd640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5480cf1e0 con 0x7fe5480cffa0 2024-04-15T15:14:44.700 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.696+0000 7fe54e3cd640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fe5480cf4c0 con 0x7fe5480b66f0 2024-04-15T15:14:44.700 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.696+0000 7fe54c96a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe5480b66f0 0x7fe548169650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:44.700 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.696+0000 7fe54c96a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe5480b66f0 0x7fe548169650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36230/0 (socket says 172.21.15.72:36230) 2024-04-15T15:14:44.700 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.696+0000 7fe54c96a640 1 -- 172.21.15.72:0/553713385 learned_addr learned my addr 172.21.15.72:0/553713385 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:44.700 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.696+0000 7fe547fff640 1 --1- 172.21.15.72:0/553713385 >> v1:172.21.15.72:6789/0 conn(0x7fe54815e5c0 0x7fe5480d32d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:33458/0 (socket says 172.21.15.72:33458) 2024-04-15T15:14:44.700 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.696+0000 7fe54c96a640 1 -- 172.21.15.72:0/553713385 >> v1:172.21.15.175:6789/0 conn(0x7fe5480cffa0 legacy=0x7fe5480b5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:44.700 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.696+0000 7fe54c96a640 1 -- 172.21.15.72:0/553713385 >> v1:172.21.15.72:6789/0 conn(0x7fe54815e5c0 legacy=0x7fe5480d32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:44.700 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.696+0000 7fe54c96a640 1 -- 172.21.15.72:0/553713385 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5480cf7a0 con 0x7fe5480b66f0 2024-04-15T15:14:44.700 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.697+0000 7fe54c96a640 1 --2- 172.21.15.72:0/553713385 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5480b66f0 0x7fe548169650 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe540009e50 tx=0x7fe54002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2c8c109e8d2af78 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:44.700 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.697+0000 7fe546ffd640 1 -- 172.21.15.72:0/553713385 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe54002ca80 con 0x7fe5480b66f0 2024-04-15T15:14:44.701 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.697+0000 7fe546ffd640 1 -- 172.21.15.72:0/553713385 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe54002cc20 con 0x7fe5480b66f0 2024-04-15T15:14:44.701 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.697+0000 7fe546ffd640 1 -- 172.21.15.72:0/553713385 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5400343f0 con 0x7fe5480b66f0 2024-04-15T15:14:44.702 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.698+0000 7fe54e3cd640 1 -- 172.21.15.72:0/553713385 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5480b66f0 msgr2=0x7fe548169650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:44.702 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.698+0000 7fe54e3cd640 1 --2- 172.21.15.72:0/553713385 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5480b66f0 0x7fe548169650 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe540009e50 tx=0x7fe54002bee0 comp rx=0 tx=0).stop 2024-04-15T15:14:44.702 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.698+0000 7fe54e3cd640 1 -- 172.21.15.72:0/553713385 shutdown_connections 2024-04-15T15:14:44.702 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.698+0000 7fe54e3cd640 1 --2- 172.21.15.72:0/553713385 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5480b66f0 0x7fe548169650 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:44.702 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.698+0000 7fe54e3cd640 1 -- 172.21.15.72:0/553713385 >> 172.21.15.72:0/553713385 conn(0x7fe54808bfa0 msgr2=0x7fe5480d4560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:44.702 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.698+0000 7fe54e3cd640 1 -- 172.21.15.72:0/553713385 shutdown_connections 2024-04-15T15:14:44.702 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.698+0000 7fe54e3cd640 1 -- 172.21.15.72:0/553713385 wait complete. 2024-04-15T15:14:44.702 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.699+0000 7fe54e3cd640 1 Processor -- start 2024-04-15T15:14:44.702 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.699+0000 7fe54e3cd640 1 -- start start 2024-04-15T15:14:44.704 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.700+0000 7fe54e3cd640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe5480b66f0 0x7fe5480b8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:44.705 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.700+0000 7fe54e3cd640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe5480cffa0 0x7fe5480b6f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:44.705 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.700+0000 7fe54e3cd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe54815e5c0 0x7fe5480b7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:44.705 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.700+0000 7fe54e3cd640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe5480d39e0 con 0x7fe54815e5c0 2024-04-15T15:14:44.705 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.700+0000 7fe54e3cd640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fe5480cebb0 con 0x7fe5480b66f0 2024-04-15T15:14:44.705 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.700+0000 7fe54e3cd640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fe5480d3b60 con 0x7fe5480cffa0 2024-04-15T15:14:44.705 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.700+0000 7fe54c96a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe54815e5c0 0x7fe5480b7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:44.705 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.700+0000 7fe54c96a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe54815e5c0 0x7fe5480b7490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36244/0 (socket says 172.21.15.72:36244) 2024-04-15T15:14:44.705 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.700+0000 7fe54c96a640 1 -- 172.21.15.72:0/2358265547 learned_addr learned my addr 172.21.15.72:0/2358265547 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:44.705 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.700+0000 7fe547fff640 1 --2- 172.21.15.72:0/2358265547 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe5480b66f0 0x7fe5480b8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:44.705 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.700+0000 7fe5477fe640 1 --2- 172.21.15.72:0/2358265547 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe5480cffa0 0x7fe5480b6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:44.705 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.700+0000 7fe54c96a640 1 -- 172.21.15.72:0/2358265547 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe5480cffa0 msgr2=0x7fe5480b6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:44.705 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.700+0000 7fe54c96a640 1 --2- 172.21.15.72:0/2358265547 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe5480cffa0 0x7fe5480b6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:44.705 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.700+0000 7fe54c96a640 1 -- 172.21.15.72:0/2358265547 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe5480b66f0 msgr2=0x7fe5480b8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:44.706 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.700+0000 7fe54c96a640 1 --2- 172.21.15.72:0/2358265547 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe5480b66f0 0x7fe5480b8e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:44.706 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.700+0000 7fe54c96a640 1 -- 172.21.15.72:0/2358265547 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe548154b90 con 0x7fe54815e5c0 2024-04-15T15:14:44.706 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.700+0000 7fe547fff640 1 --2- 172.21.15.72:0/2358265547 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe5480b66f0 0x7fe5480b8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:14:44.706 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.700+0000 7fe5477fe640 1 --2- 172.21.15.72:0/2358265547 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe5480cffa0 0x7fe5480b6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:14:44.706 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.701+0000 7fe54c96a640 1 --2- 172.21.15.72:0/2358265547 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe54815e5c0 0x7fe5480b7490 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe540009f70 tx=0x7fe540004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:44.706 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.701+0000 7fe544ff9640 1 -- 172.21.15.72:0/2358265547 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe540043020 con 0x7fe54815e5c0 2024-04-15T15:14:44.706 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.701+0000 7fe544ff9640 1 -- 172.21.15.72:0/2358265547 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe54003e450 con 0x7fe54815e5c0 2024-04-15T15:14:44.706 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.701+0000 7fe54e3cd640 1 -- 172.21.15.72:0/2358265547 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5480b96d0 con 0x7fe54815e5c0 2024-04-15T15:14:44.706 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.701+0000 7fe544ff9640 1 -- 172.21.15.72:0/2358265547 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe540009510 con 0x7fe54815e5c0 2024-04-15T15:14:44.706 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.701+0000 7fe54e3cd640 1 -- 172.21.15.72:0/2358265547 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5480b9c90 con 0x7fe54815e5c0 2024-04-15T15:14:44.706 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.702+0000 7fe54e3cd640 1 -- 172.21.15.72:0/2358265547 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5480ce290 con 0x7fe54815e5c0 2024-04-15T15:14:44.706 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.702+0000 7fe544ff9640 1 -- 172.21.15.72:0/2358265547 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe540046020 con 0x7fe54815e5c0 2024-04-15T15:14:44.707 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.703+0000 7fe544ff9640 1 --2- 172.21.15.72:0/2358265547 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe53406c8f0 0x7fe53406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:44.707 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.703+0000 7fe544ff9640 1 -- 172.21.15.72:0/2358265547 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fe5400ae0b0 con 0x7fe54815e5c0 2024-04-15T15:14:44.707 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.703+0000 7fe547fff640 1 --2- 172.21.15.72:0/2358265547 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe53406c8f0 0x7fe53406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:44.707 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.703+0000 7fe547fff640 1 --2- 172.21.15.72:0/2358265547 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe53406c8f0 0x7fe53406edb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe538000f60 tx=0x7fe538009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:44.710 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.707+0000 7fe544ff9640 1 -- 172.21.15.72:0/2358265547 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe5480ce290 con 0x7fe54815e5c0 2024-04-15T15:14:44.886 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.884+0000 7fe54e3cd640 1 -- 172.21.15.72:0/2358265547 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe548069490 con 0x7fe54815e5c0 2024-04-15T15:14:44.887 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.884+0000 7fe544ff9640 1 -- 172.21.15.72:0/2358265547 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12884 (secure 0 0 0) 0x7fe54004d030 con 0x7fe54815e5c0 2024-04-15T15:14:44.887 INFO:teuthology.misc.health.smithi072.stdout: 2024-04-15T15:14:44.887 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":16,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:14:41.421762+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:14:44.889 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.886+0000 7fe54e3cd640 1 -- 172.21.15.72:0/2358265547 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe53406c8f0 msgr2=0x7fe53406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:44.889 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.886+0000 7fe54e3cd640 1 --2- 172.21.15.72:0/2358265547 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe53406c8f0 0x7fe53406edb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe538000f60 tx=0x7fe538009040 comp rx=0 tx=0).stop 2024-04-15T15:14:44.889 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.886+0000 7fe54e3cd640 1 -- 172.21.15.72:0/2358265547 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe54815e5c0 msgr2=0x7fe5480b7490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:44.890 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.886+0000 7fe54e3cd640 1 --2- 172.21.15.72:0/2358265547 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe54815e5c0 0x7fe5480b7490 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe540009f70 tx=0x7fe540004a40 comp rx=0 tx=0).stop 2024-04-15T15:14:44.890 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.887+0000 7fe54e3cd640 1 -- 172.21.15.72:0/2358265547 shutdown_connections 2024-04-15T15:14:44.890 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.887+0000 7fe54e3cd640 1 --2- 172.21.15.72:0/2358265547 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe53406c8f0 0x7fe53406edb0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:44.890 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.887+0000 7fe54e3cd640 1 --2- 172.21.15.72:0/2358265547 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe54815e5c0 0x7fe5480b7490 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:44.890 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.887+0000 7fe54e3cd640 1 --2- 172.21.15.72:0/2358265547 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe5480cffa0 0x7fe5480b6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:44.890 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.887+0000 7fe54e3cd640 1 --2- 172.21.15.72:0/2358265547 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe5480b66f0 0x7fe5480b8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:44.890 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.887+0000 7fe54e3cd640 1 -- 172.21.15.72:0/2358265547 >> 172.21.15.72:0/2358265547 conn(0x7fe54808bfa0 msgr2=0x7fe5480cdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:44.890 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.887+0000 7fe54e3cd640 1 -- 172.21.15.72:0/2358265547 shutdown_connections 2024-04-15T15:14:44.890 INFO:teuthology.misc.health.smithi072.stderr:2024-04-15T15:14:44.887+0000 7fe54e3cd640 1 -- 172.21.15.72:0/2358265547 wait complete. 2024-04-15T15:14:44.899 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-15T15:14:44.899 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-15T15:14:44.899 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-15T15:14:44.899 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-15T15:14:44.901 INFO:tasks.ceph.mds.a:Started 2024-04-15T15:14:44.902 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-15T15:14:44.902 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-15T15:14:44.904 INFO:tasks.ceph.mds.c:Started 2024-04-15T15:14:44.904 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-15T15:14:44.904 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-15T15:14:44.907 INFO:tasks.ceph.mds.b:Started 2024-04-15T15:14:44.907 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-15T15:14:44.908 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-15T15:14:44.910 INFO:tasks.ceph.mds.d:Started 2024-04-15T15:14:44.910 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-04-15T15:14:44.910 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:14:44.979 INFO:tasks.ceph.mds.a.smithi072.stdout:starting mds.a at 2024-04-15T15:14:44.980 INFO:tasks.ceph.mds.c.smithi072.stdout:starting mds.c at 2024-04-15T15:14:44.996 INFO:tasks.ceph.mds.d.smithi175.stdout:starting mds.d at 2024-04-15T15:14:44.996 INFO:tasks.ceph.mds.b.smithi175.stdout:starting mds.b at 2024-04-15T15:14:45.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.016+0000 7f37d0d9d640 1 Processor -- start 2024-04-15T15:14:45.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.016+0000 7f37d0d9d640 1 -- start start 2024-04-15T15:14:45.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.017+0000 7f37d0d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f37cc15b2b0 0x7f37cc06fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:45.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.017+0000 7f37d0d9d640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f37cc06d200 0x7f37cc06d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:45.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.017+0000 7f37d0d9d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37cc070160 con 0x7f37cc0afd70 2024-04-15T15:14:45.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.017+0000 7f37d0d9d640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f37cc070430 con 0x7f37cc15b2b0 2024-04-15T15:14:45.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.017+0000 7f37d0d9d640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f37cc070710 con 0x7f37cc06d200 2024-04-15T15:14:45.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.017+0000 7f37cb7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f37cc15b2b0 0x7f37cc06fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:45.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.017+0000 7f37c2ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f37cc0afd70 0x7f37cc0b5420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:33480/0 (socket says 172.21.15.72:33480) 2024-04-15T15:14:45.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.017+0000 7f37c2ffd640 1 -- 172.21.15.72:0/1044310403 learned_addr learned my addr 172.21.15.72:0/1044310403 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:14:45.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.018+0000 7f37cb7fe640 1 -- 172.21.15.72:0/1044310403 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f37cc06d200 msgr2=0x7f37cc06d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:45.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.018+0000 7f37cb7fe640 1 --2- 172.21.15.72:0/1044310403 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f37cc06d200 0x7f37cc06d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.018+0000 7f37cb7fe640 1 -- 172.21.15.72:0/1044310403 >> v1:172.21.15.72:6789/0 conn(0x7f37cc0afd70 legacy=0x7f37cc0b5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:45.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.018+0000 7f37cb7fe640 1 -- 172.21.15.72:0/1044310403 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37cc0b4c20 con 0x7f37cc15b2b0 2024-04-15T15:14:45.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.018+0000 7f37caffd640 1 -- 172.21.15.72:0/1044310403 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3333195151 0 0) 0x7f37cc070160 con 0x7f37cc0afd70 2024-04-15T15:14:45.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.018+0000 7f37cb7fe640 1 --2- 172.21.15.72:0/1044310403 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37cc15b2b0 0x7f37cc06fc20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f37bc0097a0 tx=0x7f37bc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2da9b9030cbcb601 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:45.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.019+0000 7f37caffd640 1 -- 172.21.15.72:0/1044310403 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37bc0315a0 con 0x7f37cc15b2b0 2024-04-15T15:14:45.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.019+0000 7f37caffd640 1 -- 172.21.15.72:0/1044310403 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37bc037070 con 0x7f37cc15b2b0 2024-04-15T15:14:45.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.019+0000 7f37caffd640 1 -- 172.21.15.72:0/1044310403 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37bc031960 con 0x7f37cc15b2b0 2024-04-15T15:14:45.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.019+0000 7f37d0d9d640 1 -- 172.21.15.72:0/1044310403 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37cc15b2b0 msgr2=0x7f37cc06fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:45.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.019+0000 7f37d0d9d640 1 --2- 172.21.15.72:0/1044310403 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37cc15b2b0 0x7f37cc06fc20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f37bc0097a0 tx=0x7f37bc02b840 comp rx=0 tx=0).stop 2024-04-15T15:14:45.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.019+0000 7f37d0d9d640 1 -- 172.21.15.72:0/1044310403 shutdown_connections 2024-04-15T15:14:45.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.019+0000 7f37d0d9d640 1 --2- 172.21.15.72:0/1044310403 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f37cc06d200 0x7f37cc06d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.019+0000 7f37d0d9d640 1 --2- 172.21.15.72:0/1044310403 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37cc15b2b0 0x7f37cc06fc20 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.019+0000 7f37d0d9d640 1 -- 172.21.15.72:0/1044310403 >> 172.21.15.72:0/1044310403 conn(0x7f37cc072200 msgr2=0x7f37cc072610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:45.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.019+0000 7f37d0d9d640 1 -- 172.21.15.72:0/1044310403 shutdown_connections 2024-04-15T15:14:45.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.019+0000 7f37d0d9d640 1 -- 172.21.15.72:0/1044310403 wait complete. 2024-04-15T15:14:45.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.020+0000 7f37d0d9d640 1 Processor -- start 2024-04-15T15:14:45.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.020+0000 7f37d0d9d640 1 -- start start 2024-04-15T15:14:45.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.021+0000 7f37d0d9d640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37cc06d200 0x7f37cc14b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:45.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.021+0000 7f37d0d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37cc0afd70 0x7f37cc1578f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:45.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.021+0000 7f37d0d9d640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f37cc15b2b0 0x7f37cc15ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:45.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.021+0000 7f37d0d9d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f37cc15b810 con 0x7f37cc0afd70 2024-04-15T15:14:45.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.021+0000 7f37d0d9d640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f37cc15b690 con 0x7f37cc06d200 2024-04-15T15:14:45.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.021+0000 7f37d0d9d640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f37cc0ce2d0 con 0x7f37cc15b2b0 2024-04-15T15:14:45.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.021+0000 7f37c2ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37cc0afd70 0x7f37cc1578f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:45.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.021+0000 7f37c2ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37cc0afd70 0x7f37cc1578f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36276/0 (socket says 172.21.15.72:36276) 2024-04-15T15:14:45.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.021+0000 7f37c2ffd640 1 -- 172.21.15.72:0/3811428445 learned_addr learned my addr 172.21.15.72:0/3811428445 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:45.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.021+0000 7f37cb7fe640 1 --2- 172.21.15.72:0/3811428445 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37cc06d200 0x7f37cc14b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:45.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.021+0000 7f37cbfff640 1 --2- 172.21.15.72:0/3811428445 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f37cc15b2b0 0x7f37cc15ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:45.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.022+0000 7f37c2ffd640 1 -- 172.21.15.72:0/3811428445 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f37cc15b2b0 msgr2=0x7f37cc15ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:45.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.022+0000 7f37c2ffd640 1 --2- 172.21.15.72:0/3811428445 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f37cc15b2b0 0x7f37cc15ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.022+0000 7f37c2ffd640 1 -- 172.21.15.72:0/3811428445 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37cc06d200 msgr2=0x7f37cc14b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:45.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.022+0000 7f37c2ffd640 1 --2- 172.21.15.72:0/3811428445 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37cc06d200 0x7f37cc14b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.022+0000 7f37c2ffd640 1 -- 172.21.15.72:0/3811428445 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37cc158240 con 0x7f37cc0afd70 2024-04-15T15:14:45.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.022+0000 7f37c2ffd640 1 --2- 172.21.15.72:0/3811428445 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37cc0afd70 0x7f37cc1578f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f37b400c500 tx=0x7f37b400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:45.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.022+0000 7f37c8ff9640 1 -- 172.21.15.72:0/3811428445 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b4018070 con 0x7f37cc0afd70 2024-04-15T15:14:45.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.022+0000 7f37c8ff9640 1 -- 172.21.15.72:0/3811428445 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37b4007580 con 0x7f37cc0afd70 2024-04-15T15:14:45.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.022+0000 7f37c8ff9640 1 -- 172.21.15.72:0/3811428445 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b4004a20 con 0x7f37cc0afd70 2024-04-15T15:14:45.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.022+0000 7f37d0d9d640 1 -- 172.21.15.72:0/3811428445 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37cc158520 con 0x7f37cc0afd70 2024-04-15T15:14:45.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.022+0000 7f37d0d9d640 1 -- 172.21.15.72:0/3811428445 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37cc158ae0 con 0x7f37cc0afd70 2024-04-15T15:14:45.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.023+0000 7f37d0d9d640 1 -- 172.21.15.72:0/3811428445 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37cc075b40 con 0x7f37cc0afd70 2024-04-15T15:14:45.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.023+0000 7f37c8ff9640 1 -- 172.21.15.72:0/3811428445 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f37b4004bc0 con 0x7f37cc0afd70 2024-04-15T15:14:45.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.024+0000 7f37c8ff9640 1 --2- 172.21.15.72:0/3811428445 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f37b006c8f0 0x7f37b006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:45.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.024+0000 7f37c8ff9640 1 -- 172.21.15.72:0/3811428445 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f37b408aa50 con 0x7f37cc0afd70 2024-04-15T15:14:45.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.024+0000 7f37cb7fe640 1 --2- 172.21.15.72:0/3811428445 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f37b006c8f0 0x7f37b006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:45.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.024+0000 7f37cb7fe640 1 --2- 172.21.15.72:0/3811428445 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f37b006c8f0 0x7f37b006edb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f37bc036000 tx=0x7f37bc008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:45.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.028+0000 7f37c8ff9640 1 -- 172.21.15.72:0/3811428445 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f37b4058b90 con 0x7f37cc0afd70 2024-04-15T15:14:45.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.205+0000 7f37d0d9d640 1 -- 172.21.15.72:0/3811428445 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f37cc06bf40 con 0x7f37cc0afd70 2024-04-15T15:14:45.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.206+0000 7f37c8ff9640 1 -- 172.21.15.72:0/3811428445 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12884 (secure 0 0 0) 0x7f37b405ce90 con 0x7f37cc0afd70 2024-04-15T15:14:45.209 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:14:45.209 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":16,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:14:41.421762+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:14:45.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.208+0000 7f37d0d9d640 1 -- 172.21.15.72:0/3811428445 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f37b006c8f0 msgr2=0x7f37b006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:45.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.208+0000 7f37d0d9d640 1 --2- 172.21.15.72:0/3811428445 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f37b006c8f0 0x7f37b006edb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f37bc036000 tx=0x7f37bc008030 comp rx=0 tx=0).stop 2024-04-15T15:14:45.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.208+0000 7f37d0d9d640 1 -- 172.21.15.72:0/3811428445 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37cc0afd70 msgr2=0x7f37cc1578f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:45.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.208+0000 7f37d0d9d640 1 --2- 172.21.15.72:0/3811428445 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37cc0afd70 0x7f37cc1578f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f37b400c500 tx=0x7f37b400c9c0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.208+0000 7f37d0d9d640 1 -- 172.21.15.72:0/3811428445 shutdown_connections 2024-04-15T15:14:45.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.208+0000 7f37d0d9d640 1 --2- 172.21.15.72:0/3811428445 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f37b006c8f0 0x7f37b006edb0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.208+0000 7f37d0d9d640 1 --2- 172.21.15.72:0/3811428445 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f37cc15b2b0 0x7f37cc15ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.208+0000 7f37d0d9d640 1 --2- 172.21.15.72:0/3811428445 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37cc0afd70 0x7f37cc1578f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.208+0000 7f37d0d9d640 1 --2- 172.21.15.72:0/3811428445 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37cc06d200 0x7f37cc14b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.208+0000 7f37d0d9d640 1 -- 172.21.15.72:0/3811428445 >> 172.21.15.72:0/3811428445 conn(0x7f37cc072200 msgr2=0x7f37cc0b35f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:45.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.208+0000 7f37d0d9d640 1 -- 172.21.15.72:0/3811428445 shutdown_connections 2024-04-15T15:14:45.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.209+0000 7f37d0d9d640 1 -- 172.21.15.72:0/3811428445 wait complete. 2024-04-15T15:14:45.222 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:39.359102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:14:45.222 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:14:45.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.325+0000 7ffadcd11640 1 Processor -- start 2024-04-15T15:14:45.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.325+0000 7ffadcd11640 1 -- start start 2024-04-15T15:14:45.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.326+0000 7ffadcd11640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffad80b11d0 0x7ffad80d4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:45.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.326+0000 7ffadcd11640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffad80b65b0 con 0x7ffad80b6040 2024-04-15T15:14:45.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.326+0000 7ffadcd11640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffad80b6890 con 0x7ffad815e960 2024-04-15T15:14:45.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.326+0000 7ffadcd11640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7ffad80b6b70 con 0x7ffad80b11d0 2024-04-15T15:14:45.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.326+0000 7ffad6d9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ffad80b6040 0x7ffad80cffe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:33490/0 (socket says 172.21.15.72:33490) 2024-04-15T15:14:45.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.326+0000 7ffad6d9d640 1 -- 172.21.15.72:0/331398904 learned_addr learned my addr 172.21.15.72:0/331398904 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:14:45.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.326+0000 7ffad559a640 1 -- 172.21.15.72:0/331398904 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1641538617 0 0) 0x7ffad80b65b0 con 0x7ffad80b6040 2024-04-15T15:14:45.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.326+0000 7ffad5d9b640 1 --2- 172.21.15.72:0/331398904 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffad80b11d0 0x7ffad80d4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:45.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.326+0000 7ffad559a640 1 -- 172.21.15.72:0/331398904 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffac0003520 con 0x7ffad80b6040 2024-04-15T15:14:45.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.327+0000 7ffad559a640 1 -- 172.21.15.72:0/331398904 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3645884611 0 0) 0x7ffac0003520 con 0x7ffad80b6040 2024-04-15T15:14:45.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.327+0000 7ffad559a640 1 -- 172.21.15.72:0/331398904 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffad80b11d0 msgr2=0x7ffad80d4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:45.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.327+0000 7ffad559a640 1 --2- 172.21.15.72:0/331398904 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffad80b11d0 0x7ffad80d4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.327+0000 7ffad559a640 1 -- 172.21.15.72:0/331398904 >> v1:172.21.15.175:6789/0 conn(0x7ffad815e960 legacy=0x7ffad80d1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:45.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.327+0000 7ffad559a640 1 -- 172.21.15.72:0/331398904 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffad80b6e50 con 0x7ffad80b6040 2024-04-15T15:14:45.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.327+0000 7ffad559a640 1 -- 172.21.15.72:0/331398904 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7ffad0001670 con 0x7ffad80b6040 2024-04-15T15:14:45.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.327+0000 7ffad559a640 1 -- 172.21.15.72:0/331398904 >> v1:172.21.15.72:6789/0 conn(0x7ffad80b6040 legacy=0x7ffad80cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:45.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.327+0000 7ffad559a640 1 --2- 172.21.15.72:0/331398904 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffac00039e0 0x7ffac0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:45.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.327+0000 7ffad559a640 1 --2- 172.21.15.72:0/331398904 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffad80b11d0 0x7ffac0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:45.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.327+0000 7ffad559a640 1 --2- 172.21.15.72:0/331398904 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffac0004d40 0x7ffac00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:45.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.327+0000 7ffad559a640 1 -- 172.21.15.72:0/331398904 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ffad80b6b70 con 0x7ffac00039e0 2024-04-15T15:14:45.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.327+0000 7ffad559a640 1 -- 172.21.15.72:0/331398904 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7ffac000b9c0 con 0x7ffac0004d40 2024-04-15T15:14:45.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.327+0000 7ffad559a640 1 -- 172.21.15.72:0/331398904 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7ffac000bbc0 con 0x7ffad80b11d0 2024-04-15T15:14:45.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.327+0000 7ffad659c640 1 --2- 172.21.15.72:0/331398904 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffac00039e0 0x7ffac0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:45.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.327+0000 7ffad5d9b640 1 --2- 172.21.15.72:0/331398904 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffad80b11d0 0x7ffac0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:45.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.327+0000 7ffad6d9d640 1 --2- 172.21.15.72:0/331398904 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffac0004d40 0x7ffac00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:45.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.327+0000 7ffad659c640 1 -- 172.21.15.72:0/331398904 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffad80b11d0 msgr2=0x7ffac0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:45.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.327+0000 7ffad659c640 1 --2- 172.21.15.72:0/331398904 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffad80b11d0 0x7ffac0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.327+0000 7ffad659c640 1 -- 172.21.15.72:0/331398904 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffac0004d40 msgr2=0x7ffac00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:45.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.327+0000 7ffad659c640 1 --2- 172.21.15.72:0/331398904 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffac0004d40 0x7ffac00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.327+0000 7ffad659c640 1 -- 172.21.15.72:0/331398904 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffac0003520 con 0x7ffac00039e0 2024-04-15T15:14:45.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.328+0000 7ffad659c640 1 --2- 172.21.15.72:0/331398904 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffac00039e0 0x7ffac0003dc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ffac8001f60 tx=0x7ffac0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=202d0058d4300762 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:45.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.328+0000 7ffad559a640 1 -- 172.21.15.72:0/331398904 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac8001b30 con 0x7ffac00039e0 2024-04-15T15:14:45.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.328+0000 7ffad559a640 1 -- 172.21.15.72:0/331398904 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffac802d690 con 0x7ffac00039e0 2024-04-15T15:14:45.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.328+0000 7ffad559a640 1 -- 172.21.15.72:0/331398904 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac802da80 con 0x7ffac00039e0 2024-04-15T15:14:45.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.329+0000 7ffadcd11640 1 -- 172.21.15.72:0/331398904 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffac00039e0 msgr2=0x7ffac0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:45.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.329+0000 7ffadcd11640 1 --2- 172.21.15.72:0/331398904 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffac00039e0 0x7ffac0003dc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ffac8001f60 tx=0x7ffac0005830 comp rx=0 tx=0).stop 2024-04-15T15:14:45.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.329+0000 7ffad6d9d640 1 -- 172.21.15.72:0/331398904 reap_dead start 2024-04-15T15:14:45.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.329+0000 7ffadcd11640 1 -- 172.21.15.72:0/331398904 shutdown_connections 2024-04-15T15:14:45.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.329+0000 7ffadcd11640 1 -- 172.21.15.72:0/331398904 >> 172.21.15.72:0/331398904 conn(0x7ffad80b9580 msgr2=0x7ffad80b9990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:45.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.329+0000 7ffadcd11640 1 -- 172.21.15.72:0/331398904 shutdown_connections 2024-04-15T15:14:45.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.329+0000 7ffadcd11640 1 -- 172.21.15.72:0/331398904 wait complete. 2024-04-15T15:14:45.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.330+0000 7ffadcd11640 1 Processor -- start 2024-04-15T15:14:45.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.330+0000 7ffadcd11640 1 -- start start 2024-04-15T15:14:45.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.331+0000 7ffadcd11640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffac00039e0 0x7ffad815cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:45.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.331+0000 7ffadcd11640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffad80cc0a0 0x7ffad806cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:45.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.331+0000 7ffadcd11640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffad806d270 0x7ffad80cd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:45.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.331+0000 7ffadcd11640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ffad80d0870 con 0x7ffad806d270 2024-04-15T15:14:45.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.331+0000 7ffadcd11640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7ffad80d06f0 con 0x7ffac00039e0 2024-04-15T15:14:45.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.331+0000 7ffadcd11640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7ffad80d09f0 con 0x7ffad80cc0a0 2024-04-15T15:14:45.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.331+0000 7ffad6d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffad806d270 0x7ffad80cd3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:45.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.331+0000 7ffad6d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffad806d270 0x7ffad80cd3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36282/0 (socket says 172.21.15.72:36282) 2024-04-15T15:14:45.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.331+0000 7ffad6d9d640 1 -- 172.21.15.72:0/2310580016 learned_addr learned my addr 172.21.15.72:0/2310580016 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:45.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.331+0000 7ffad659c640 1 --2- 172.21.15.72:0/2310580016 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffac00039e0 0x7ffad815cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:45.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.331+0000 7ffad5d9b640 1 --2- 172.21.15.72:0/2310580016 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffad80cc0a0 0x7ffad806cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:45.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.331+0000 7ffad6d9d640 1 -- 172.21.15.72:0/2310580016 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffad80cc0a0 msgr2=0x7ffad806cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:45.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.331+0000 7ffad6d9d640 1 --2- 172.21.15.72:0/2310580016 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffad80cc0a0 0x7ffad806cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.331+0000 7ffad6d9d640 1 -- 172.21.15.72:0/2310580016 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffac00039e0 msgr2=0x7ffad815cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:45.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.331+0000 7ffad6d9d640 1 --2- 172.21.15.72:0/2310580016 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffac00039e0 0x7ffad815cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.331+0000 7ffad6d9d640 1 -- 172.21.15.72:0/2310580016 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffad806bc90 con 0x7ffad806d270 2024-04-15T15:14:45.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.331+0000 7ffad6d9d640 1 --2- 172.21.15.72:0/2310580016 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffad806d270 0x7ffad80cd3e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ffad0000c00 tx=0x7ffad0007660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:45.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.331+0000 7ffac6ffd640 1 -- 172.21.15.72:0/2310580016 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad0001640 con 0x7ffad806d270 2024-04-15T15:14:45.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.331+0000 7ffac6ffd640 1 -- 172.21.15.72:0/2310580016 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffad0018070 con 0x7ffad806d270 2024-04-15T15:14:45.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.331+0000 7ffadcd11640 1 -- 172.21.15.72:0/2310580016 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffad806bf70 con 0x7ffad806d270 2024-04-15T15:14:45.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.331+0000 7ffac6ffd640 1 -- 172.21.15.72:0/2310580016 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad000faf0 con 0x7ffad806d270 2024-04-15T15:14:45.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.332+0000 7ffadcd11640 1 -- 172.21.15.72:0/2310580016 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffad806c530 con 0x7ffad806d270 2024-04-15T15:14:45.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.333+0000 7ffadcd11640 1 -- 172.21.15.72:0/2310580016 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffad8075c50 con 0x7ffad806d270 2024-04-15T15:14:45.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.333+0000 7ffac6ffd640 1 -- 172.21.15.72:0/2310580016 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ffad000fc90 con 0x7ffad806d270 2024-04-15T15:14:45.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.333+0000 7ffac6ffd640 1 --2- 172.21.15.72:0/2310580016 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ffab806c8f0 0x7ffab806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:45.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.333+0000 7ffac6ffd640 1 -- 172.21.15.72:0/2310580016 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7ffad008b450 con 0x7ffad806d270 2024-04-15T15:14:45.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.333+0000 7ffad659c640 1 --2- 172.21.15.72:0/2310580016 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ffab806c8f0 0x7ffab806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:45.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.334+0000 7ffad659c640 1 --2- 172.21.15.72:0/2310580016 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ffab806c8f0 0x7ffab806edb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ffac8036000 tx=0x7ffac8031000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:45.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.338+0000 7ffac6ffd640 1 -- 172.21.15.72:0/2310580016 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ffad0059c60 con 0x7ffad806d270 2024-04-15T15:14:45.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.513+0000 7ffadcd11640 1 -- 172.21.15.72:0/2310580016 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ffad815e140 con 0x7ffad806d270 2024-04-15T15:14:45.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.514+0000 7ffac6ffd640 1 -- 172.21.15.72:0/2310580016 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7ffad0059560 con 0x7ffad806d270 2024-04-15T15:14:45.517 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-15T15:14:45.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.516+0000 7ffadcd11640 1 -- 172.21.15.72:0/2310580016 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ffab806c8f0 msgr2=0x7ffab806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:45.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.516+0000 7ffadcd11640 1 --2- 172.21.15.72:0/2310580016 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ffab806c8f0 0x7ffab806edb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ffac8036000 tx=0x7ffac8031000 comp rx=0 tx=0).stop 2024-04-15T15:14:45.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.516+0000 7ffadcd11640 1 -- 172.21.15.72:0/2310580016 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffad806d270 msgr2=0x7ffad80cd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:45.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.516+0000 7ffadcd11640 1 --2- 172.21.15.72:0/2310580016 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffad806d270 0x7ffad80cd3e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ffad0000c00 tx=0x7ffad0007660 comp rx=0 tx=0).stop 2024-04-15T15:14:45.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.516+0000 7ffadcd11640 1 -- 172.21.15.72:0/2310580016 shutdown_connections 2024-04-15T15:14:45.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.516+0000 7ffadcd11640 1 --2- 172.21.15.72:0/2310580016 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ffab806c8f0 0x7ffab806edb0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.516+0000 7ffadcd11640 1 --2- 172.21.15.72:0/2310580016 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffad806d270 0x7ffad80cd3e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.516+0000 7ffadcd11640 1 --2- 172.21.15.72:0/2310580016 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffad80cc0a0 0x7ffad806cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.516+0000 7ffadcd11640 1 --2- 172.21.15.72:0/2310580016 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffac00039e0 0x7ffad815cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.516+0000 7ffadcd11640 1 -- 172.21.15.72:0/2310580016 >> 172.21.15.72:0/2310580016 conn(0x7ffad80b9580 msgr2=0x7ffad80d4900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:45.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.516+0000 7ffadcd11640 1 -- 172.21.15.72:0/2310580016 shutdown_connections 2024-04-15T15:14:45.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.516+0000 7ffadcd11640 1 -- 172.21.15.72:0/2310580016 wait complete. 2024-04-15T15:14:45.529 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:14:45.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.629+0000 7fbb95bf2640 1 Processor -- start 2024-04-15T15:14:45.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.629+0000 7fbb95bf2640 1 -- start start 2024-04-15T15:14:45.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.630+0000 7fbb95bf2640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbb901557f0 0x7fbb900b52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:45.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.630+0000 7fbb95bf2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fbb900b57e0 0x7fbb900cca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:45.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.630+0000 7fbb95bf2640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb90155bd0 con 0x7fbb900ccf40 2024-04-15T15:14:45.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.630+0000 7fbb95bf2640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fbb900d0120 con 0x7fbb900b57e0 2024-04-15T15:14:45.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.630+0000 7fbb95bf2640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fbb900d0400 con 0x7fbb901557f0 2024-04-15T15:14:45.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.630+0000 7fbb8effd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fbb900b57e0 0x7fbb900cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:45.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.630+0000 7fbb8effd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fbb900b57e0 0x7fbb900cca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36292/0 (socket says 172.21.15.72:36292) 2024-04-15T15:14:45.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.630+0000 7fbb8ffff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fbb900ccf40 0x7fbb9015f020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:33498/0 (socket says 172.21.15.72:33498) 2024-04-15T15:14:45.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.630+0000 7fbb8effd640 1 -- 172.21.15.72:0/2691261840 learned_addr learned my addr 172.21.15.72:0/2691261840 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:45.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.631+0000 7fbb8f7fe640 1 --2- 172.21.15.72:0/2691261840 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbb901557f0 0x7fbb900b52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:45.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.631+0000 7fbb8effd640 1 -- 172.21.15.72:0/2691261840 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbb901557f0 msgr2=0x7fbb900b52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:45.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.631+0000 7fbb8effd640 1 --2- 172.21.15.72:0/2691261840 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbb901557f0 0x7fbb900b52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.631+0000 7fbb8effd640 1 -- 172.21.15.72:0/2691261840 >> v1:172.21.15.72:6789/0 conn(0x7fbb900ccf40 legacy=0x7fbb9015f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:45.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.631+0000 7fbb8effd640 1 -- 172.21.15.72:0/2691261840 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb900d06e0 con 0x7fbb900b57e0 2024-04-15T15:14:45.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.631+0000 7fbb8e7fc640 1 -- 172.21.15.72:0/2691261840 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2005343299 0 0) 0x7fbb90155bd0 con 0x7fbb900ccf40 2024-04-15T15:14:45.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.631+0000 7fbb8f7fe640 1 --2- 172.21.15.72:0/2691261840 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbb901557f0 0x7fbb900b52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:14:45.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.631+0000 7fbb8effd640 1 --2- 172.21.15.72:0/2691261840 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbb900b57e0 0x7fbb900cca00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbb84009870 tx=0x7fbb8402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa72b12195cf2b37 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:45.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.632+0000 7fbb8e7fc640 1 -- 172.21.15.72:0/2691261840 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb84034070 con 0x7fbb900b57e0 2024-04-15T15:14:45.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.632+0000 7fbb8e7fc640 1 -- 172.21.15.72:0/2691261840 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb84030840 con 0x7fbb900b57e0 2024-04-15T15:14:45.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.632+0000 7fbb8e7fc640 1 -- 172.21.15.72:0/2691261840 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb84030c10 con 0x7fbb900b57e0 2024-04-15T15:14:45.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.632+0000 7fbb95bf2640 1 -- 172.21.15.72:0/2691261840 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbb900b57e0 msgr2=0x7fbb900cca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:45.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.632+0000 7fbb95bf2640 1 --2- 172.21.15.72:0/2691261840 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbb900b57e0 0x7fbb900cca00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbb84009870 tx=0x7fbb8402b980 comp rx=0 tx=0).stop 2024-04-15T15:14:45.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.632+0000 7fbb95bf2640 1 -- 172.21.15.72:0/2691261840 shutdown_connections 2024-04-15T15:14:45.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.632+0000 7fbb95bf2640 1 --2- 172.21.15.72:0/2691261840 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbb900b57e0 0x7fbb900cca00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.632+0000 7fbb95bf2640 1 --2- 172.21.15.72:0/2691261840 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbb901557f0 0x7fbb900b52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.632+0000 7fbb95bf2640 1 -- 172.21.15.72:0/2691261840 >> 172.21.15.72:0/2691261840 conn(0x7fbb900b3fa0 msgr2=0x7fbb900b43b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:45.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.632+0000 7fbb95bf2640 1 -- 172.21.15.72:0/2691261840 shutdown_connections 2024-04-15T15:14:45.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.632+0000 7fbb95bf2640 1 -- 172.21.15.72:0/2691261840 wait complete. 2024-04-15T15:14:45.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.633+0000 7fbb95bf2640 1 Processor -- start 2024-04-15T15:14:45.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.633+0000 7fbb95bf2640 1 -- start start 2024-04-15T15:14:45.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.634+0000 7fbb95bf2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbb900b57e0 0x7fbb90154630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:45.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.634+0000 7fbb95bf2640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbb900ccf40 0x7fbb9015bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:45.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.634+0000 7fbb95bf2640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbb901557f0 0x7fbb9015c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:45.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.634+0000 7fbb95bf2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fbb9015f730 con 0x7fbb900b57e0 2024-04-15T15:14:45.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.634+0000 7fbb95bf2640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fbb900d0ca0 con 0x7fbb901557f0 2024-04-15T15:14:45.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.634+0000 7fbb95bf2640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fbb9015f8b0 con 0x7fbb900ccf40 2024-04-15T15:14:45.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.634+0000 7fbb8f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbb900b57e0 0x7fbb90154630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:45.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.634+0000 7fbb8f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbb900b57e0 0x7fbb90154630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36308/0 (socket says 172.21.15.72:36308) 2024-04-15T15:14:45.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.634+0000 7fbb8f7fe640 1 -- 172.21.15.72:0/934500504 learned_addr learned my addr 172.21.15.72:0/934500504 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:45.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.634+0000 7fbb8ffff640 1 --2- 172.21.15.72:0/934500504 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbb901557f0 0x7fbb9015c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:45.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.634+0000 7fbb8f7fe640 1 -- 172.21.15.72:0/934500504 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbb900ccf40 msgr2=0x7fbb9015bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:45.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.634+0000 7fbb8f7fe640 1 --2- 172.21.15.72:0/934500504 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbb900ccf40 0x7fbb9015bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.635+0000 7fbb8f7fe640 1 -- 172.21.15.72:0/934500504 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbb901557f0 msgr2=0x7fbb9015c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:45.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.635+0000 7fbb8f7fe640 1 --2- 172.21.15.72:0/934500504 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbb901557f0 0x7fbb9015c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.635+0000 7fbb8f7fe640 1 -- 172.21.15.72:0/934500504 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb900ca4e0 con 0x7fbb900b57e0 2024-04-15T15:14:45.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.635+0000 7fbb8ffff640 1 --2- 172.21.15.72:0/934500504 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbb901557f0 0x7fbb9015c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:14:45.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.635+0000 7fbb8f7fe640 1 --2- 172.21.15.72:0/934500504 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbb900b57e0 0x7fbb90154630 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbb8000c7f0 tx=0x7fbb8000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:45.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.635+0000 7fbb77fff640 1 -- 172.21.15.72:0/934500504 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb80015070 con 0x7fbb900b57e0 2024-04-15T15:14:45.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.635+0000 7fbb77fff640 1 -- 172.21.15.72:0/934500504 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb80007820 con 0x7fbb900b57e0 2024-04-15T15:14:45.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.635+0000 7fbb95bf2640 1 -- 172.21.15.72:0/934500504 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb900ca7c0 con 0x7fbb900b57e0 2024-04-15T15:14:45.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.635+0000 7fbb77fff640 1 -- 172.21.15.72:0/934500504 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb80007bd0 con 0x7fbb900b57e0 2024-04-15T15:14:45.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.635+0000 7fbb95bf2640 1 -- 172.21.15.72:0/934500504 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb900cad80 con 0x7fbb900b57e0 2024-04-15T15:14:45.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.636+0000 7fbb95bf2640 1 -- 172.21.15.72:0/934500504 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb900690f0 con 0x7fbb900b57e0 2024-04-15T15:14:45.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.636+0000 7fbb77fff640 1 -- 172.21.15.72:0/934500504 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbb80018070 con 0x7fbb900b57e0 2024-04-15T15:14:45.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.637+0000 7fbb77fff640 1 --2- 172.21.15.72:0/934500504 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fbb7c06c8f0 0x7fbb7c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:45.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.637+0000 7fbb8effd640 1 --2- 172.21.15.72:0/934500504 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fbb7c06c8f0 0x7fbb7c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:45.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.637+0000 7fbb77fff640 1 -- 172.21.15.72:0/934500504 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fbb8008b970 con 0x7fbb900b57e0 2024-04-15T15:14:45.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.637+0000 7fbb8effd640 1 --2- 172.21.15.72:0/934500504 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fbb7c06c8f0 0x7fbb7c06edb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbb84036000 tx=0x7fbb84030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:45.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.641+0000 7fbb77fff640 1 -- 172.21.15.72:0/934500504 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbb8005a0f0 con 0x7fbb900b57e0 2024-04-15T15:14:45.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.852+0000 7fbb95bf2640 1 -- 172.21.15.72:0/934500504 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb900d1ac0 con 0x7fbb900b57e0 2024-04-15T15:14:45.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.853+0000 7fbb77fff640 1 -- 172.21.15.72:0/934500504 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1392 (secure 0 0 0) 0x7fbb800599f0 con 0x7fbb900b57e0 2024-04-15T15:14:45.856 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:14:45.856 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4296,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/4151548261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4151548261},{"type":"v1","addr":"172.21.15.72:6836","nonce":4151548261}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-15T15:14:45.856 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 2 2024-04-15T15:14:45.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.855+0000 7fbb95bf2640 1 -- 172.21.15.72:0/934500504 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fbb7c06c8f0 msgr2=0x7fbb7c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:45.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.855+0000 7fbb95bf2640 1 --2- 172.21.15.72:0/934500504 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fbb7c06c8f0 0x7fbb7c06edb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbb84036000 tx=0x7fbb84030000 comp rx=0 tx=0).stop 2024-04-15T15:14:45.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.855+0000 7fbb95bf2640 1 -- 172.21.15.72:0/934500504 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbb900b57e0 msgr2=0x7fbb90154630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:45.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.855+0000 7fbb95bf2640 1 --2- 172.21.15.72:0/934500504 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbb900b57e0 0x7fbb90154630 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbb8000c7f0 tx=0x7fbb8000ccb0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.855+0000 7fbb95bf2640 1 -- 172.21.15.72:0/934500504 shutdown_connections 2024-04-15T15:14:45.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.855+0000 7fbb95bf2640 1 --2- 172.21.15.72:0/934500504 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fbb7c06c8f0 0x7fbb7c06edb0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.855+0000 7fbb95bf2640 1 --2- 172.21.15.72:0/934500504 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbb901557f0 0x7fbb9015c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.855+0000 7fbb95bf2640 1 --2- 172.21.15.72:0/934500504 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbb900ccf40 0x7fbb9015bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.855+0000 7fbb95bf2640 1 --2- 172.21.15.72:0/934500504 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbb900b57e0 0x7fbb90154630 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.855+0000 7fbb95bf2640 1 -- 172.21.15.72:0/934500504 >> 172.21.15.72:0/934500504 conn(0x7fbb900b3fa0 msgr2=0x7fbb90161cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:45.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.855+0000 7fbb95bf2640 1 -- 172.21.15.72:0/934500504 shutdown_connections 2024-04-15T15:14:45.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:45.855+0000 7fbb95bf2640 1 -- 172.21.15.72:0/934500504 wait complete. 2024-04-15T15:14:47.869 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:14:47.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.982+0000 7f3e0ebaf640 1 Processor -- start 2024-04-15T15:14:47.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.982+0000 7f3e0ebaf640 1 -- start start 2024-04-15T15:14:47.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.983+0000 7f3e0ebaf640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3e08161fb0 0x7f3e08162450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:47.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.983+0000 7f3e0ebaf640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3e080c9570 0x7f3e08162990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:47.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.983+0000 7f3e0ebaf640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e080ccd40 con 0x7f3e080d12f0 2024-04-15T15:14:47.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.983+0000 7f3e0ebaf640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f3e080ccaf0 con 0x7f3e08161fb0 2024-04-15T15:14:47.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.983+0000 7f3e0ebaf640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f3e080cd020 con 0x7f3e080c9570 2024-04-15T15:14:47.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.983+0000 7f3e0c94b640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f3e080d12f0 0x7f3e081618a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:33510/0 (socket says 172.21.15.72:33510) 2024-04-15T15:14:47.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.983+0000 7f3e0c94b640 1 -- 172.21.15.72:0/3696390374 learned_addr learned my addr 172.21.15.72:0/3696390374 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:14:47.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.983+0000 7f3dfffff640 1 --2- 172.21.15.72:0/3696390374 >> v2:172.21.15.72:3300/0 conn(0x7f3e08161fb0 0x7f3e08162450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:47.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.983+0000 7f3dfffff640 1 -- 172.21.15.72:0/3696390374 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3e080c9570 msgr2=0x7f3e08162990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:47.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.983+0000 7f3e0d14c640 1 --2- 172.21.15.72:0/3696390374 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3e080c9570 0x7f3e08162990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:47.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.983+0000 7f3dff7fe640 1 -- 172.21.15.72:0/3696390374 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1224567040 0 0) 0x7f3e080ccd40 con 0x7f3e080d12f0 2024-04-15T15:14:47.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.983+0000 7f3dfffff640 1 --2- 172.21.15.72:0/3696390374 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3e080c9570 0x7f3e08162990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.983+0000 7f3dfffff640 1 -- 172.21.15.72:0/3696390374 >> v1:172.21.15.72:6789/0 conn(0x7f3e080d12f0 legacy=0x7f3e081618a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:47.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.983+0000 7f3dfffff640 1 -- 172.21.15.72:0/3696390374 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e080cd300 con 0x7f3e08161fb0 2024-04-15T15:14:47.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.984+0000 7f3dfffff640 1 --2- 172.21.15.72:0/3696390374 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e08161fb0 0x7f3e08162450 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3df80097d0 tx=0x7f3df802b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f52f484611cda5c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:47.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.984+0000 7f3dff7fe640 1 -- 172.21.15.72:0/3696390374 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df8035020 con 0x7f3e08161fb0 2024-04-15T15:14:47.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.984+0000 7f3dff7fe640 1 -- 172.21.15.72:0/3696390374 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3df8031800 con 0x7f3e08161fb0 2024-04-15T15:14:47.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.984+0000 7f3dff7fe640 1 -- 172.21.15.72:0/3696390374 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df8031bd0 con 0x7f3e08161fb0 2024-04-15T15:14:47.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.985+0000 7f3e0ebaf640 1 -- 172.21.15.72:0/3696390374 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e08161fb0 msgr2=0x7f3e08162450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:47.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.985+0000 7f3e0ebaf640 1 --2- 172.21.15.72:0/3696390374 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e08161fb0 0x7f3e08162450 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3df80097d0 tx=0x7f3df802b8a0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.985+0000 7f3e0ebaf640 1 -- 172.21.15.72:0/3696390374 shutdown_connections 2024-04-15T15:14:47.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.985+0000 7f3e0ebaf640 1 --2- 172.21.15.72:0/3696390374 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3e080c9570 0x7f3e08162990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.985+0000 7f3e0ebaf640 1 --2- 172.21.15.72:0/3696390374 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e08161fb0 0x7f3e08162450 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.985+0000 7f3e0ebaf640 1 -- 172.21.15.72:0/3696390374 >> 172.21.15.72:0/3696390374 conn(0x7f3e080b3f90 msgr2=0x7f3e080b43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:47.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.985+0000 7f3e0ebaf640 1 -- 172.21.15.72:0/3696390374 shutdown_connections 2024-04-15T15:14:47.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.985+0000 7f3e0ebaf640 1 -- 172.21.15.72:0/3696390374 wait complete. 2024-04-15T15:14:47.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.986+0000 7f3e0ebaf640 1 Processor -- start 2024-04-15T15:14:47.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.986+0000 7f3e0ebaf640 1 -- start start 2024-04-15T15:14:47.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.987+0000 7f3e0ebaf640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3e080c9570 0x7f3e0806d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:47.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.987+0000 7f3e0ebaf640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e080d12f0 0x7f3e080ae2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:47.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.987+0000 7f3e0ebaf640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e08161fb0 0x7f3e081f3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:47.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.987+0000 7f3e0ebaf640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f3e080d2210 con 0x7f3e080d12f0 2024-04-15T15:14:47.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.987+0000 7f3e0ebaf640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f3e080d2090 con 0x7f3e08161fb0 2024-04-15T15:14:47.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.987+0000 7f3e0ebaf640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f3e080b68e0 con 0x7f3e080c9570 2024-04-15T15:14:47.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.987+0000 7f3dfffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e080d12f0 0x7f3e080ae2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:47.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.987+0000 7f3dfffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e080d12f0 0x7f3e080ae2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36332/0 (socket says 172.21.15.72:36332) 2024-04-15T15:14:47.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.987+0000 7f3dfffff640 1 -- 172.21.15.72:0/414960052 learned_addr learned my addr 172.21.15.72:0/414960052 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:47.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.987+0000 7f3e0c94b640 1 --2- 172.21.15.72:0/414960052 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3e080c9570 0x7f3e0806d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:47.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.987+0000 7f3e0d14c640 1 --2- 172.21.15.72:0/414960052 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e08161fb0 0x7f3e081f3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:47.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.987+0000 7f3dfffff640 1 -- 172.21.15.72:0/414960052 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3e080c9570 msgr2=0x7f3e0806d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:47.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.987+0000 7f3dfffff640 1 --2- 172.21.15.72:0/414960052 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3e080c9570 0x7f3e0806d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.987+0000 7f3dfffff640 1 -- 172.21.15.72:0/414960052 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e08161fb0 msgr2=0x7f3e081f3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:47.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.987+0000 7f3dfffff640 1 --2- 172.21.15.72:0/414960052 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e08161fb0 0x7f3e081f3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.987+0000 7f3dfffff640 1 -- 172.21.15.72:0/414960052 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e080af7e0 con 0x7f3e080d12f0 2024-04-15T15:14:47.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.988+0000 7f3dfffff640 1 --2- 172.21.15.72:0/414960052 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e080d12f0 0x7f3e080ae2e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3df8034040 tx=0x7f3df80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:47.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.988+0000 7f3dfd7fa640 1 -- 172.21.15.72:0/414960052 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df8041070 con 0x7f3e080d12f0 2024-04-15T15:14:47.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.988+0000 7f3dfd7fa640 1 -- 172.21.15.72:0/414960052 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3df803ad00 con 0x7f3e080d12f0 2024-04-15T15:14:47.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.988+0000 7f3e0ebaf640 1 -- 172.21.15.72:0/414960052 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e080aea30 con 0x7f3e080d12f0 2024-04-15T15:14:47.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.988+0000 7f3dfd7fa640 1 -- 172.21.15.72:0/414960052 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df8035040 con 0x7f3e080d12f0 2024-04-15T15:14:47.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.988+0000 7f3e0ebaf640 1 -- 172.21.15.72:0/414960052 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e080aeff0 con 0x7f3e080d12f0 2024-04-15T15:14:47.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.989+0000 7f3e0ebaf640 1 -- 172.21.15.72:0/414960052 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e080689e0 con 0x7f3e080d12f0 2024-04-15T15:14:47.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.989+0000 7f3dfd7fa640 1 -- 172.21.15.72:0/414960052 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3df8038020 con 0x7f3e080d12f0 2024-04-15T15:14:47.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.990+0000 7f3dfd7fa640 1 --2- 172.21.15.72:0/414960052 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3df406c8f0 0x7f3df406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:47.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.990+0000 7f3dfd7fa640 1 -- 172.21.15.72:0/414960052 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f3df80ad540 con 0x7f3e080d12f0 2024-04-15T15:14:47.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.990+0000 7f3e0c94b640 1 --2- 172.21.15.72:0/414960052 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3df406c8f0 0x7f3df406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:47.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.990+0000 7f3e0c94b640 1 --2- 172.21.15.72:0/414960052 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3df406c8f0 0x7f3df406edb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3e04000f60 tx=0x7f3e04007b50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:47.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:47.994+0000 7f3dfd7fa640 1 -- 172.21.15.72:0/414960052 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3df8043350 con 0x7f3e080d12f0 2024-04-15T15:14:48.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.206+0000 7f3e0ebaf640 1 -- 172.21.15.72:0/414960052 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3e080b4990 con 0x7f3e080d12f0 2024-04-15T15:14:48.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.207+0000 7f3dfd7fa640 1 -- 172.21.15.72:0/414960052 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3870 (secure 0 0 0) 0x7f3df807bcc0 con 0x7f3e080d12f0 2024-04-15T15:14:48.208 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:14:48.208 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4267,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6833/1659343919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1659343919},{"type":"v1","addr":"172.21.15.175:6833","nonce":1659343919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6835/4113588141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6834","nonce":4113588141},{"type":"v1","addr":"172.21.15.175:6835","nonce":4113588141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4296,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/4151548261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4151548261},{"type":"v1","addr":"172.21.15.72:6836","nonce":4151548261}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4299,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/3458067804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3458067804},{"type":"v1","addr":"172.21.15.72:6837","nonce":3458067804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-15T15:14:48.209 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 3 2024-04-15T15:14:48.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.208+0000 7f3e0ebaf640 1 -- 172.21.15.72:0/414960052 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3df406c8f0 msgr2=0x7f3df406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:48.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.208+0000 7f3e0ebaf640 1 --2- 172.21.15.72:0/414960052 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3df406c8f0 0x7f3df406edb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3e04000f60 tx=0x7f3e04007b50 comp rx=0 tx=0).stop 2024-04-15T15:14:48.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.208+0000 7f3e0ebaf640 1 -- 172.21.15.72:0/414960052 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e080d12f0 msgr2=0x7f3e080ae2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:48.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.208+0000 7f3e0ebaf640 1 --2- 172.21.15.72:0/414960052 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e080d12f0 0x7f3e080ae2e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3df8034040 tx=0x7f3df80046c0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.209+0000 7f3e0ebaf640 1 -- 172.21.15.72:0/414960052 shutdown_connections 2024-04-15T15:14:48.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.209+0000 7f3e0ebaf640 1 --2- 172.21.15.72:0/414960052 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3df406c8f0 0x7f3df406edb0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.209+0000 7f3e0ebaf640 1 --2- 172.21.15.72:0/414960052 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e08161fb0 0x7f3e081f3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.209+0000 7f3e0ebaf640 1 --2- 172.21.15.72:0/414960052 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e080d12f0 0x7f3e080ae2e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.209+0000 7f3e0ebaf640 1 --2- 172.21.15.72:0/414960052 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3e080c9570 0x7f3e0806d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.209+0000 7f3e0ebaf640 1 -- 172.21.15.72:0/414960052 >> 172.21.15.72:0/414960052 conn(0x7f3e080b3f90 msgr2=0x7f3e080ca760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:48.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.209+0000 7f3e0ebaf640 1 -- 172.21.15.72:0/414960052 shutdown_connections 2024-04-15T15:14:48.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.209+0000 7f3e0ebaf640 1 -- 172.21.15.72:0/414960052 wait complete. 2024-04-15T15:14:48.221 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:14:48.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.323+0000 7f0d63b22640 1 Processor -- start 2024-04-15T15:14:48.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.323+0000 7f0d63b22640 1 -- start start 2024-04-15T15:14:48.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.324+0000 7f0d63b22640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0d5c161fb0 0x7f0d5c162450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:48.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.324+0000 7f0d63b22640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d5c0ccd40 con 0x7f0d5c0d12f0 2024-04-15T15:14:48.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.324+0000 7f0d63b22640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d5c0cd020 con 0x7f0d5c0c9570 2024-04-15T15:14:48.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.324+0000 7f0d63b22640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f0d5c0ccaf0 con 0x7f0d5c161fb0 2024-04-15T15:14:48.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.324+0000 7f0d618be640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f0d5c0d12f0 0x7f0d5c1618a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:33516/0 (socket says 172.21.15.72:33516) 2024-04-15T15:14:48.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.324+0000 7f0d618be640 1 -- 172.21.15.72:0/4138783673 learned_addr learned my addr 172.21.15.72:0/4138783673 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:14:48.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.324+0000 7f0d608bc640 1 -- 172.21.15.72:0/4138783673 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3048225871 0 0) 0x7f0d5c0ccd40 con 0x7f0d5c0d12f0 2024-04-15T15:14:48.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.324+0000 7f0d610bd640 1 --2- 172.21.15.72:0/4138783673 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0d5c161fb0 0x7f0d5c162450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:48.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.324+0000 7f0d608bc640 1 -- 172.21.15.72:0/4138783673 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d48003520 con 0x7f0d5c0d12f0 2024-04-15T15:14:48.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.324+0000 7f0d608bc640 1 -- 172.21.15.72:0/4138783673 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 747558551 0 0) 0x7f0d48003520 con 0x7f0d5c0d12f0 2024-04-15T15:14:48.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.324+0000 7f0d608bc640 1 -- 172.21.15.72:0/4138783673 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0d5c161fb0 msgr2=0x7f0d5c162450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:48.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.324+0000 7f0d608bc640 1 --2- 172.21.15.72:0/4138783673 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0d5c161fb0 0x7f0d5c162450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.324+0000 7f0d608bc640 1 -- 172.21.15.72:0/4138783673 >> v1:172.21.15.175:6789/0 conn(0x7f0d5c0c9570 legacy=0x7f0d5c162990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:48.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.324+0000 7f0d608bc640 1 -- 172.21.15.72:0/4138783673 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d5c0cd300 con 0x7f0d5c0d12f0 2024-04-15T15:14:48.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.324+0000 7f0d608bc640 1 -- 172.21.15.72:0/4138783673 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f0d58001670 con 0x7f0d5c0d12f0 2024-04-15T15:14:48.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.324+0000 7f0d610bd640 1 --2- 172.21.15.72:0/4138783673 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0d5c161fb0 0x7f0d5c162450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:14:48.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.324+0000 7f0d608bc640 1 -- 172.21.15.72:0/4138783673 >> v1:172.21.15.72:6789/0 conn(0x7f0d5c0d12f0 legacy=0x7f0d5c1618a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:48.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.324+0000 7f0d608bc640 1 --2- 172.21.15.72:0/4138783673 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0d480039e0 0x7f0d48003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:48.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.324+0000 7f0d608bc640 1 --2- 172.21.15.72:0/4138783673 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d48004800 0x7f0d48004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:48.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.324+0000 7f0d608bc640 1 --2- 172.21.15.72:0/4138783673 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0d5c161fb0 0x7f0d480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:48.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.324+0000 7f0d608bc640 1 -- 172.21.15.72:0/4138783673 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f0d5c0ccaf0 con 0x7f0d48004800 2024-04-15T15:14:48.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.324+0000 7f0d608bc640 1 -- 172.21.15.72:0/4138783673 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f0d4800baf0 con 0x7f0d480039e0 2024-04-15T15:14:48.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.324+0000 7f0d608bc640 1 -- 172.21.15.72:0/4138783673 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f0d4800bcf0 con 0x7f0d5c161fb0 2024-04-15T15:14:48.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.324+0000 7f0d610bd640 1 --2- 172.21.15.72:0/4138783673 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d48004800 0x7f0d48004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:48.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.325+0000 7f0d618be640 1 --2- 172.21.15.72:0/4138783673 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0d480039e0 0x7f0d48003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:48.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.325+0000 7f0d620bf640 1 --2- 172.21.15.72:0/4138783673 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0d5c161fb0 0x7f0d480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:48.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.325+0000 7f0d610bd640 1 -- 172.21.15.72:0/4138783673 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0d5c161fb0 msgr2=0x7f0d480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:48.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.325+0000 7f0d610bd640 1 --2- 172.21.15.72:0/4138783673 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0d5c161fb0 0x7f0d480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.325+0000 7f0d610bd640 1 -- 172.21.15.72:0/4138783673 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0d480039e0 msgr2=0x7f0d48003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:48.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.325+0000 7f0d610bd640 1 --2- 172.21.15.72:0/4138783673 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0d480039e0 0x7f0d48003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.325+0000 7f0d610bd640 1 -- 172.21.15.72:0/4138783673 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d48003520 con 0x7f0d48004800 2024-04-15T15:14:48.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.325+0000 7f0d610bd640 1 --2- 172.21.15.72:0/4138783673 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d48004800 0x7f0d48004c80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c1623b0 tx=0x7f0d48005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5535e51735faed5 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:48.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.326+0000 7f0d608bc640 1 -- 172.21.15.72:0/4138783673 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c00f040 con 0x7f0d48004800 2024-04-15T15:14:48.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.326+0000 7f0d608bc640 1 -- 172.21.15.72:0/4138783673 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d4c004030 con 0x7f0d48004800 2024-04-15T15:14:48.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.326+0000 7f0d608bc640 1 -- 172.21.15.72:0/4138783673 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c004400 con 0x7f0d48004800 2024-04-15T15:14:48.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.326+0000 7f0d63b22640 1 -- 172.21.15.72:0/4138783673 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d48004800 msgr2=0x7f0d48004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:48.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.326+0000 7f0d63b22640 1 --2- 172.21.15.72:0/4138783673 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d48004800 0x7f0d48004c80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c1623b0 tx=0x7f0d48005830 comp rx=0 tx=0).stop 2024-04-15T15:14:48.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.326+0000 7f0d620bf640 1 -- 172.21.15.72:0/4138783673 reap_dead start 2024-04-15T15:14:48.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.326+0000 7f0d63b22640 1 -- 172.21.15.72:0/4138783673 shutdown_connections 2024-04-15T15:14:48.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.326+0000 7f0d63b22640 1 -- 172.21.15.72:0/4138783673 >> 172.21.15.72:0/4138783673 conn(0x7f0d5c0b3f90 msgr2=0x7f0d5c0b43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:48.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.326+0000 7f0d63b22640 1 -- 172.21.15.72:0/4138783673 shutdown_connections 2024-04-15T15:14:48.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.326+0000 7f0d63b22640 1 -- 172.21.15.72:0/4138783673 wait complete. 2024-04-15T15:14:48.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.327+0000 7f0d63b22640 1 Processor -- start 2024-04-15T15:14:48.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.327+0000 7f0d63b22640 1 -- start start 2024-04-15T15:14:48.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.328+0000 7f0d63b22640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d48004800 0x7f0d5c15cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:48.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.328+0000 7f0d63b22640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0d5c152a20 0x7f0d5c0b1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:48.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.328+0000 7f0d63b22640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0d5c0cda00 0x7f0d5c0cde90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:48.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.328+0000 7f0d63b22640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f0d5c0d2210 con 0x7f0d48004800 2024-04-15T15:14:48.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.328+0000 7f0d63b22640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f0d5c0d2090 con 0x7f0d5c0cda00 2024-04-15T15:14:48.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.328+0000 7f0d63b22640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f0d5c1630a0 con 0x7f0d5c152a20 2024-04-15T15:14:48.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.328+0000 7f0d618be640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d48004800 0x7f0d5c15cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:48.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.328+0000 7f0d618be640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d48004800 0x7f0d5c15cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36356/0 (socket says 172.21.15.72:36356) 2024-04-15T15:14:48.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.328+0000 7f0d618be640 1 -- 172.21.15.72:0/1477179843 learned_addr learned my addr 172.21.15.72:0/1477179843 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:48.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.328+0000 7f0d620bf640 1 --2- 172.21.15.72:0/1477179843 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0d5c0cda00 0x7f0d5c0cde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:48.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.328+0000 7f0d610bd640 1 --2- 172.21.15.72:0/1477179843 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0d5c152a20 0x7f0d5c0b1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:48.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.328+0000 7f0d618be640 1 -- 172.21.15.72:0/1477179843 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0d5c152a20 msgr2=0x7f0d5c0b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:48.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.328+0000 7f0d618be640 1 --2- 172.21.15.72:0/1477179843 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0d5c152a20 0x7f0d5c0b1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.328+0000 7f0d618be640 1 -- 172.21.15.72:0/1477179843 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0d5c0cda00 msgr2=0x7f0d5c0cde90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:48.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.328+0000 7f0d618be640 1 --2- 172.21.15.72:0/1477179843 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0d5c0cda00 0x7f0d5c0cde90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.328+0000 7f0d618be640 1 -- 172.21.15.72:0/1477179843 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d5c0adb20 con 0x7f0d48004800 2024-04-15T15:14:48.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.328+0000 7f0d620bf640 1 --2- 172.21.15.72:0/1477179843 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0d5c0cda00 0x7f0d5c0cde90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:14:48.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.329+0000 7f0d618be640 1 --2- 172.21.15.72:0/1477179843 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d48004800 0x7f0d5c15cad0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0d580072c0 tx=0x7f0d580076c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:48.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.329+0000 7f0d527fc640 1 -- 172.21.15.72:0/1477179843 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d5800f960 con 0x7f0d48004800 2024-04-15T15:14:48.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.329+0000 7f0d527fc640 1 -- 172.21.15.72:0/1477179843 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d58018070 con 0x7f0d48004800 2024-04-15T15:14:48.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.329+0000 7f0d63b22640 1 -- 172.21.15.72:0/1477179843 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d5c0ade00 con 0x7f0d48004800 2024-04-15T15:14:48.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.329+0000 7f0d527fc640 1 -- 172.21.15.72:0/1477179843 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d5800fd60 con 0x7f0d48004800 2024-04-15T15:14:48.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.329+0000 7f0d63b22640 1 -- 172.21.15.72:0/1477179843 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d5c0ae3c0 con 0x7f0d48004800 2024-04-15T15:14:48.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.330+0000 7f0d63b22640 1 -- 172.21.15.72:0/1477179843 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d5c0ae5c0 con 0x7f0d48004800 2024-04-15T15:14:48.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.330+0000 7f0d527fc640 1 -- 172.21.15.72:0/1477179843 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0d5801d460 con 0x7f0d48004800 2024-04-15T15:14:48.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.331+0000 7f0d527fc640 1 --2- 172.21.15.72:0/1477179843 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0d4006c920 0x7f0d4006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:48.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.331+0000 7f0d527fc640 1 -- 172.21.15.72:0/1477179843 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f0d5808b8b0 con 0x7f0d48004800 2024-04-15T15:14:48.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.331+0000 7f0d610bd640 1 --2- 172.21.15.72:0/1477179843 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0d4006c920 0x7f0d4006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:48.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.331+0000 7f0d610bd640 1 --2- 172.21.15.72:0/1477179843 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0d4006c920 0x7f0d4006ede0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c035000 tx=0x7f0d4c03a040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:48.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.335+0000 7f0d527fc640 1 -- 172.21.15.72:0/1477179843 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0d5c0ae5c0 con 0x7f0d48004800 2024-04-15T15:14:48.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.512+0000 7f0d63b22640 1 -- 172.21.15.72:0/1477179843 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0d5c0689e0 con 0x7f0d48004800 2024-04-15T15:14:48.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.513+0000 7f0d527fc640 1 -- 172.21.15.72:0/1477179843 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12884 (secure 0 0 0) 0x7f0d5805a030 con 0x7f0d48004800 2024-04-15T15:14:48.514 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:14:48.515 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":16,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:14:41.421762+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:14:48.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.515+0000 7f0d63b22640 1 -- 172.21.15.72:0/1477179843 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0d4006c920 msgr2=0x7f0d4006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:48.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.515+0000 7f0d63b22640 1 --2- 172.21.15.72:0/1477179843 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0d4006c920 0x7f0d4006ede0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c035000 tx=0x7f0d4c03a040 comp rx=0 tx=0).stop 2024-04-15T15:14:48.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.515+0000 7f0d63b22640 1 -- 172.21.15.72:0/1477179843 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d48004800 msgr2=0x7f0d5c15cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:48.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.515+0000 7f0d63b22640 1 --2- 172.21.15.72:0/1477179843 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d48004800 0x7f0d5c15cad0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0d580072c0 tx=0x7f0d580076c0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.515+0000 7f0d63b22640 1 -- 172.21.15.72:0/1477179843 shutdown_connections 2024-04-15T15:14:48.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.515+0000 7f0d63b22640 1 --2- 172.21.15.72:0/1477179843 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0d4006c920 0x7f0d4006ede0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.515+0000 7f0d63b22640 1 --2- 172.21.15.72:0/1477179843 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0d5c0cda00 0x7f0d5c0cde90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.515+0000 7f0d63b22640 1 --2- 172.21.15.72:0/1477179843 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0d5c152a20 0x7f0d5c0b1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.515+0000 7f0d63b22640 1 --2- 172.21.15.72:0/1477179843 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d48004800 0x7f0d5c15cad0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.515+0000 7f0d63b22640 1 -- 172.21.15.72:0/1477179843 >> 172.21.15.72:0/1477179843 conn(0x7f0d5c0b3f90 msgr2=0x7f0d5c0b2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:48.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.515+0000 7f0d63b22640 1 -- 172.21.15.72:0/1477179843 shutdown_connections 2024-04-15T15:14:48.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.515+0000 7f0d63b22640 1 -- 172.21.15.72:0/1477179843 wait complete. 2024-04-15T15:14:48.527 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:39.359102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:14:48.528 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:14:48.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.632+0000 7f50d6eb1640 1 Processor -- start 2024-04-15T15:14:48.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.632+0000 7f50d6eb1640 1 -- start start 2024-04-15T15:14:48.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.633+0000 7f50d6eb1640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f50d015c5f0 0x7f50d00d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:48.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.633+0000 7f50d6eb1640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50d00cd990 con 0x7f50d006f510 2024-04-15T15:14:48.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.633+0000 7f50d6eb1640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50d0160ff0 con 0x7f50d00ce190 2024-04-15T15:14:48.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.633+0000 7f50d6eb1640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f50d0160da0 con 0x7f50d015c5f0 2024-04-15T15:14:48.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.633+0000 7f50d544e640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f50d006f510 0x7f50d00cd280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:33526/0 (socket says 172.21.15.72:33526) 2024-04-15T15:14:48.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.633+0000 7f50d544e640 1 -- 172.21.15.72:0/3347402870 learned_addr learned my addr 172.21.15.72:0/3347402870 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:14:48.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.633+0000 7f50d4c4d640 1 --2- 172.21.15.72:0/3347402870 >> v2:172.21.15.175:3300/0 conn(0x7f50d015c5f0 0x7f50d00d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:48.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.633+0000 7f50cf7fe640 1 -- 172.21.15.72:0/3347402870 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2053740115 0 0) 0x7f50d00cd990 con 0x7f50d006f510 2024-04-15T15:14:48.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.633+0000 7f50cf7fe640 1 -- 172.21.15.72:0/3347402870 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50b8003520 con 0x7f50d006f510 2024-04-15T15:14:48.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.633+0000 7f50cf7fe640 1 -- 172.21.15.72:0/3347402870 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3266333837 0 0) 0x7f50d0160ff0 con 0x7f50d00ce190 2024-04-15T15:14:48.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.633+0000 7f50cf7fe640 1 -- 172.21.15.72:0/3347402870 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50d00cd990 con 0x7f50d00ce190 2024-04-15T15:14:48.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.633+0000 7f50d4c4d640 1 -- 172.21.15.72:0/3347402870 >> v1:172.21.15.175:6789/0 conn(0x7f50d00ce190 legacy=0x7f50d00ccb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:48.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.633+0000 7f50d4c4d640 1 -- 172.21.15.72:0/3347402870 >> v1:172.21.15.72:6789/0 conn(0x7f50d006f510 legacy=0x7f50d00cd280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:48.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.634+0000 7f50d4c4d640 1 -- 172.21.15.72:0/3347402870 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50d00b5840 con 0x7f50d015c5f0 2024-04-15T15:14:48.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.634+0000 7f50d4c4d640 1 --2- 172.21.15.72:0/3347402870 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f50d015c5f0 0x7f50d00d1100 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f50c00097a0 tx=0x7f50c002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb147c6f2f9c25e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:48.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.634+0000 7f50cf7fe640 1 -- 172.21.15.72:0/3347402870 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c0035020 con 0x7f50d015c5f0 2024-04-15T15:14:48.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.635+0000 7f50cf7fe640 1 -- 172.21.15.72:0/3347402870 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50c0031830 con 0x7f50d015c5f0 2024-04-15T15:14:48.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.635+0000 7f50cf7fe640 1 -- 172.21.15.72:0/3347402870 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c0031be0 con 0x7f50d015c5f0 2024-04-15T15:14:48.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.635+0000 7f50d6eb1640 1 -- 172.21.15.72:0/3347402870 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f50d015c5f0 msgr2=0x7f50d00d1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:48.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.635+0000 7f50d6eb1640 1 --2- 172.21.15.72:0/3347402870 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f50d015c5f0 0x7f50d00d1100 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f50c00097a0 tx=0x7f50c002b8d0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.635+0000 7f50d6eb1640 1 -- 172.21.15.72:0/3347402870 shutdown_connections 2024-04-15T15:14:48.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.635+0000 7f50d6eb1640 1 --2- 172.21.15.72:0/3347402870 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f50d015c5f0 0x7f50d00d1100 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.635+0000 7f50d6eb1640 1 -- 172.21.15.72:0/3347402870 >> 172.21.15.72:0/3347402870 conn(0x7f50d00b4010 msgr2=0x7f50d00b4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:48.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.635+0000 7f50d6eb1640 1 -- 172.21.15.72:0/3347402870 shutdown_connections 2024-04-15T15:14:48.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.635+0000 7f50d6eb1640 1 -- 172.21.15.72:0/3347402870 wait complete. 2024-04-15T15:14:48.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.636+0000 7f50d6eb1640 1 Processor -- start 2024-04-15T15:14:48.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.636+0000 7f50d6eb1640 1 -- start start 2024-04-15T15:14:48.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.637+0000 7f50d6eb1640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f50d006f510 0x7f50d00c8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:48.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.637+0000 7f50d6eb1640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f50d015c5f0 0x7f50d014e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:48.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.637+0000 7f50d6eb1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50d014ec30 0x7f50d00b1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:48.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.637+0000 7f50d6eb1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f50d00b5bb0 con 0x7f50d014ec30 2024-04-15T15:14:48.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.637+0000 7f50d6eb1640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f50d01612d0 con 0x7f50d015c5f0 2024-04-15T15:14:48.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.637+0000 7f50d6eb1640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f50d015c9d0 con 0x7f50d006f510 2024-04-15T15:14:48.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.637+0000 7f50d544e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50d014ec30 0x7f50d00b1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:48.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.637+0000 7f50d544e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50d014ec30 0x7f50d00b1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36360/0 (socket says 172.21.15.72:36360) 2024-04-15T15:14:48.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.637+0000 7f50d544e640 1 -- 172.21.15.72:0/76590138 learned_addr learned my addr 172.21.15.72:0/76590138 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:48.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.637+0000 7f50cffff640 1 --2- 172.21.15.72:0/76590138 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f50d015c5f0 0x7f50d014e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:48.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.637+0000 7f50d544e640 1 -- 172.21.15.72:0/76590138 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f50d006f510 msgr2=0x7f50d00c8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:48.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.637+0000 7f50d4c4d640 1 --2- 172.21.15.72:0/76590138 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f50d006f510 0x7f50d00c8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:48.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.638+0000 7f50d544e640 1 --2- 172.21.15.72:0/76590138 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f50d006f510 0x7f50d00c8360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.638+0000 7f50d544e640 1 -- 172.21.15.72:0/76590138 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f50d015c5f0 msgr2=0x7f50d014e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:48.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.638+0000 7f50d544e640 1 --2- 172.21.15.72:0/76590138 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f50d015c5f0 0x7f50d014e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.638+0000 7f50d544e640 1 -- 172.21.15.72:0/76590138 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50d01f28d0 con 0x7f50d014ec30 2024-04-15T15:14:48.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.638+0000 7f50cffff640 1 --2- 172.21.15.72:0/76590138 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f50d015c5f0 0x7f50d014e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:14:48.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.638+0000 7f50d544e640 1 --2- 172.21.15.72:0/76590138 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50d014ec30 0x7f50d00b1f80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f50c800cbc0 tx=0x7f50c800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:48.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.638+0000 7f50cd7fa640 1 -- 172.21.15.72:0/76590138 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c8017070 con 0x7f50d014ec30 2024-04-15T15:14:48.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.638+0000 7f50cd7fa640 1 -- 172.21.15.72:0/76590138 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50c800ace0 con 0x7f50d014ec30 2024-04-15T15:14:48.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.638+0000 7f50d6eb1640 1 -- 172.21.15.72:0/76590138 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50d01f2bb0 con 0x7f50d014ec30 2024-04-15T15:14:48.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.638+0000 7f50cd7fa640 1 -- 172.21.15.72:0/76590138 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c8012600 con 0x7f50d014ec30 2024-04-15T15:14:48.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.638+0000 7f50d6eb1640 1 -- 172.21.15.72:0/76590138 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50d01f3170 con 0x7f50d014ec30 2024-04-15T15:14:48.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.639+0000 7f50d6eb1640 1 -- 172.21.15.72:0/76590138 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50d0075c50 con 0x7f50d014ec30 2024-04-15T15:14:48.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.639+0000 7f50cd7fa640 1 -- 172.21.15.72:0/76590138 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f50c801a070 con 0x7f50d014ec30 2024-04-15T15:14:48.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.640+0000 7f50cd7fa640 1 --2- 172.21.15.72:0/76590138 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f50bc06c8f0 0x7f50bc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:48.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.640+0000 7f50cd7fa640 1 -- 172.21.15.72:0/76590138 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f50c808b4d0 con 0x7f50d014ec30 2024-04-15T15:14:48.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.640+0000 7f50d4c4d640 1 --2- 172.21.15.72:0/76590138 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f50bc06c8f0 0x7f50bc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:48.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.640+0000 7f50d4c4d640 1 --2- 172.21.15.72:0/76590138 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f50bc06c8f0 0x7f50bc06edb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f50c0036000 tx=0x7f50c0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:48.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.644+0000 7f50cd7fa640 1 -- 172.21.15.72:0/76590138 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f50c8059600 con 0x7f50d014ec30 2024-04-15T15:14:48.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.818+0000 7f50d6eb1640 1 -- 172.21.15.72:0/76590138 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f50d00d1e50 con 0x7f50d014ec30 2024-04-15T15:14:48.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.818+0000 7f50cd7fa640 1 -- 172.21.15.72:0/76590138 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7f50c805d900 con 0x7f50d014ec30 2024-04-15T15:14:48.820 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-15T15:14:48.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.820+0000 7f50d6eb1640 1 -- 172.21.15.72:0/76590138 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f50bc06c8f0 msgr2=0x7f50bc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:48.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.820+0000 7f50d6eb1640 1 --2- 172.21.15.72:0/76590138 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f50bc06c8f0 0x7f50bc06edb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f50c0036000 tx=0x7f50c0008030 comp rx=0 tx=0).stop 2024-04-15T15:14:48.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.820+0000 7f50d6eb1640 1 -- 172.21.15.72:0/76590138 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50d014ec30 msgr2=0x7f50d00b1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:48.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.820+0000 7f50d6eb1640 1 --2- 172.21.15.72:0/76590138 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50d014ec30 0x7f50d00b1f80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f50c800cbc0 tx=0x7f50c800a430 comp rx=0 tx=0).stop 2024-04-15T15:14:48.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.820+0000 7f50d6eb1640 1 -- 172.21.15.72:0/76590138 shutdown_connections 2024-04-15T15:14:48.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.820+0000 7f50d6eb1640 1 --2- 172.21.15.72:0/76590138 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f50bc06c8f0 0x7f50bc06edb0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.820+0000 7f50d6eb1640 1 --2- 172.21.15.72:0/76590138 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50d014ec30 0x7f50d00b1f80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.820+0000 7f50d6eb1640 1 --2- 172.21.15.72:0/76590138 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f50d015c5f0 0x7f50d014e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.820+0000 7f50d6eb1640 1 --2- 172.21.15.72:0/76590138 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f50d006f510 0x7f50d00c8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.820+0000 7f50d6eb1640 1 -- 172.21.15.72:0/76590138 >> 172.21.15.72:0/76590138 conn(0x7f50d00b4010 msgr2=0x7f50d015b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:48.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.821+0000 7f50d6eb1640 1 -- 172.21.15.72:0/76590138 shutdown_connections 2024-04-15T15:14:48.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.821+0000 7f50d6eb1640 1 -- 172.21.15.72:0/76590138 wait complete. 2024-04-15T15:14:48.833 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-15T15:14:48.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.933+0000 7fd251132640 1 Processor -- start 2024-04-15T15:14:48.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.933+0000 7fd251132640 1 -- start start 2024-04-15T15:14:48.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.934+0000 7fd251132640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd24c15c500 0x7fd24c15f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:48.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.934+0000 7fd251132640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd24c0cc0b0 0x7fd24c0c68d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:48.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.934+0000 7fd251132640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd24c1624e0 con 0x7fd24c15c120 2024-04-15T15:14:48.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.934+0000 7fd251132640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fd24c1627c0 con 0x7fd24c0cc0b0 2024-04-15T15:14:48.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.934+0000 7fd251132640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fd24c162aa0 con 0x7fd24c15c500 2024-04-15T15:14:48.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.934+0000 7fd24b59e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd24c0cc0b0 0x7fd24c0c68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:48.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.934+0000 7fd24b59e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd24c0cc0b0 0x7fd24c0c68d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36368/0 (socket says 172.21.15.72:36368) 2024-04-15T15:14:48.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.934+0000 7fd24b59e640 1 -- 172.21.15.72:0/3143254503 learned_addr learned my addr 172.21.15.72:0/3143254503 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:48.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.934+0000 7fd243fff640 1 --2- 172.21.15.72:0/3143254503 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd24c15c500 0x7fd24c15f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:48.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.934+0000 7fd24b59e640 1 -- 172.21.15.72:0/3143254503 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd24c15c500 msgr2=0x7fd24c15f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:48.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.934+0000 7fd24b59e640 1 --2- 172.21.15.72:0/3143254503 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd24c15c500 0x7fd24c15f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.934+0000 7fd24b59e640 1 -- 172.21.15.72:0/3143254503 >> v1:172.21.15.175:6789/0 conn(0x7fd24c15c120 legacy=0x7fd24c0cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:48.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.934+0000 7fd24b59e640 1 -- 172.21.15.72:0/3143254503 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd24c162d80 con 0x7fd24c0cc0b0 2024-04-15T15:14:48.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.934+0000 7fd243fff640 1 --2- 172.21.15.72:0/3143254503 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd24c15c500 0x7fd24c15f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:14:48.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.935+0000 7fd24b59e640 1 --2- 172.21.15.72:0/3143254503 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd24c0cc0b0 0x7fd24c0c68d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd244009e50 tx=0x7fd24402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35bfbce02713f32e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:48.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.935+0000 7fd24a59c640 1 -- 172.21.15.72:0/3143254503 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd24402cb00 con 0x7fd24c0cc0b0 2024-04-15T15:14:48.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.935+0000 7fd24a59c640 1 -- 172.21.15.72:0/3143254503 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd24402cca0 con 0x7fd24c0cc0b0 2024-04-15T15:14:48.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.935+0000 7fd24a59c640 1 -- 172.21.15.72:0/3143254503 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd244034610 con 0x7fd24c0cc0b0 2024-04-15T15:14:48.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.935+0000 7fd251132640 1 -- 172.21.15.72:0/3143254503 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd24c0cc0b0 msgr2=0x7fd24c0c68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:48.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.935+0000 7fd251132640 1 --2- 172.21.15.72:0/3143254503 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd24c0cc0b0 0x7fd24c0c68d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd244009e50 tx=0x7fd24402bf60 comp rx=0 tx=0).stop 2024-04-15T15:14:48.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.936+0000 7fd251132640 1 -- 172.21.15.72:0/3143254503 shutdown_connections 2024-04-15T15:14:48.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.936+0000 7fd251132640 1 --2- 172.21.15.72:0/3143254503 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd24c0cc0b0 0x7fd24c0c68d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.936+0000 7fd251132640 1 --2- 172.21.15.72:0/3143254503 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd24c15c500 0x7fd24c15f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.936+0000 7fd251132640 1 -- 172.21.15.72:0/3143254503 >> 172.21.15.72:0/3143254503 conn(0x7fd24c0cd4a0 msgr2=0x7fd24c0cd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:48.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.936+0000 7fd251132640 1 -- 172.21.15.72:0/3143254503 shutdown_connections 2024-04-15T15:14:48.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.936+0000 7fd251132640 1 -- 172.21.15.72:0/3143254503 wait complete. 2024-04-15T15:14:48.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.937+0000 7fd251132640 1 Processor -- start 2024-04-15T15:14:48.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.937+0000 7fd251132640 1 -- start start 2024-04-15T15:14:48.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.938+0000 7fd251132640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd24c0cc0b0 0x7fd24c1615c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:48.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.938+0000 7fd251132640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd24c15c120 0x7fd24c14b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:48.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.938+0000 7fd251132640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd24c15c500 0x7fd24c1591a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:48.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.938+0000 7fd251132640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd24c0708e0 con 0x7fd24c0cc0b0 2024-04-15T15:14:48.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.938+0000 7fd251132640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fd24c070760 con 0x7fd24c15c500 2024-04-15T15:14:48.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.938+0000 7fd251132640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fd24c070a60 con 0x7fd24c15c120 2024-04-15T15:14:48.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.938+0000 7fd24ad9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd24c0cc0b0 0x7fd24c1615c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:48.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.938+0000 7fd24ad9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd24c0cc0b0 0x7fd24c1615c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36374/0 (socket says 172.21.15.72:36374) 2024-04-15T15:14:48.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.938+0000 7fd24ad9d640 1 -- 172.21.15.72:0/2605945795 learned_addr learned my addr 172.21.15.72:0/2605945795 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:48.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.938+0000 7fd24b59e640 1 --2- 172.21.15.72:0/2605945795 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd24c15c500 0x7fd24c1591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:48.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.938+0000 7fd243fff640 1 --2- 172.21.15.72:0/2605945795 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd24c15c120 0x7fd24c14b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:48.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.938+0000 7fd24ad9d640 1 -- 172.21.15.72:0/2605945795 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd24c15c120 msgr2=0x7fd24c14b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:48.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.938+0000 7fd24ad9d640 1 --2- 172.21.15.72:0/2605945795 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd24c15c120 0x7fd24c14b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.938+0000 7fd24ad9d640 1 -- 172.21.15.72:0/2605945795 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd24c15c500 msgr2=0x7fd24c1591a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:48.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.938+0000 7fd24ad9d640 1 --2- 172.21.15.72:0/2605945795 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd24c15c500 0x7fd24c1591a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.938+0000 7fd24ad9d640 1 -- 172.21.15.72:0/2605945795 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd24c1598f0 con 0x7fd24c0cc0b0 2024-04-15T15:14:48.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.938+0000 7fd243fff640 1 --2- 172.21.15.72:0/2605945795 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd24c15c120 0x7fd24c14b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:14:48.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.938+0000 7fd24ad9d640 1 --2- 172.21.15.72:0/2605945795 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd24c0cc0b0 0x7fd24c1615c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd23c00c430 tx=0x7fd23c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:48.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.938+0000 7fd2437fe640 1 -- 172.21.15.72:0/2605945795 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd23c0052d0 con 0x7fd24c0cc0b0 2024-04-15T15:14:48.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.938+0000 7fd2437fe640 1 -- 172.21.15.72:0/2605945795 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd23c005490 con 0x7fd24c0cc0b0 2024-04-15T15:14:48.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.938+0000 7fd251132640 1 -- 172.21.15.72:0/2605945795 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd24c159bd0 con 0x7fd24c0cc0b0 2024-04-15T15:14:48.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.938+0000 7fd2437fe640 1 -- 172.21.15.72:0/2605945795 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd23c007260 con 0x7fd24c0cc0b0 2024-04-15T15:14:48.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.938+0000 7fd251132640 1 -- 172.21.15.72:0/2605945795 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd24c15a190 con 0x7fd24c0cc0b0 2024-04-15T15:14:48.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.939+0000 7fd251132640 1 -- 172.21.15.72:0/2605945795 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd24c0689e0 con 0x7fd24c0cc0b0 2024-04-15T15:14:48.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.940+0000 7fd2437fe640 1 -- 172.21.15.72:0/2605945795 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd23c007400 con 0x7fd24c0cc0b0 2024-04-15T15:14:48.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.940+0000 7fd2437fe640 1 --2- 172.21.15.72:0/2605945795 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd22c06c8f0 0x7fd22c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:48.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.940+0000 7fd2437fe640 1 -- 172.21.15.72:0/2605945795 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fd23c08ac20 con 0x7fd24c0cc0b0 2024-04-15T15:14:48.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.940+0000 7fd243fff640 1 --2- 172.21.15.72:0/2605945795 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd22c06c8f0 0x7fd22c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:48.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.941+0000 7fd243fff640 1 --2- 172.21.15.72:0/2605945795 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd22c06c8f0 0x7fd22c06edb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd24c0c6ce0 tx=0x7fd234008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:48.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:48.944+0000 7fd2437fe640 1 -- 172.21.15.72:0/2605945795 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd23c058de0 con 0x7fd24c0cc0b0 2024-04-15T15:14:49.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.124+0000 7fd251132640 1 -- 172.21.15.72:0/2605945795 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fd24c0cec90 con 0x7fd24c0cc0b0 2024-04-15T15:14:49.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.124+0000 7fd2437fe640 1 -- 172.21.15.72:0/2605945795 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) ==== 84+0+65 (secure 0 0 0) 0x7fd23c05d0e0 con 0x7fd24c0cc0b0 2024-04-15T15:14:49.126 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:14:49.126 INFO:teuthology.orchestra.run.smithi072.stdout:[ 2024-04-15T15:14:49.126 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:14:49.127 INFO:teuthology.orchestra.run.smithi072.stdout: "poolnum": 1, 2024-04-15T15:14:49.127 INFO:teuthology.orchestra.run.smithi072.stdout: "poolname": ".mgr" 2024-04-15T15:14:49.127 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:14:49.127 INFO:teuthology.orchestra.run.smithi072.stdout:] 2024-04-15T15:14:49.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.126+0000 7fd251132640 1 -- 172.21.15.72:0/2605945795 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd22c06c8f0 msgr2=0x7fd22c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:49.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.126+0000 7fd251132640 1 --2- 172.21.15.72:0/2605945795 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd22c06c8f0 0x7fd22c06edb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd24c0c6ce0 tx=0x7fd234008040 comp rx=0 tx=0).stop 2024-04-15T15:14:49.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.126+0000 7fd251132640 1 -- 172.21.15.72:0/2605945795 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd24c0cc0b0 msgr2=0x7fd24c1615c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:49.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.126+0000 7fd251132640 1 --2- 172.21.15.72:0/2605945795 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd24c0cc0b0 0x7fd24c1615c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd23c00c430 tx=0x7fd23c00c8f0 comp rx=0 tx=0).stop 2024-04-15T15:14:49.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.127+0000 7fd251132640 1 -- 172.21.15.72:0/2605945795 shutdown_connections 2024-04-15T15:14:49.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.127+0000 7fd251132640 1 --2- 172.21.15.72:0/2605945795 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd22c06c8f0 0x7fd22c06edb0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:49.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.127+0000 7fd251132640 1 --2- 172.21.15.72:0/2605945795 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd24c15c500 0x7fd24c1591a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:49.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.127+0000 7fd251132640 1 --2- 172.21.15.72:0/2605945795 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd24c15c120 0x7fd24c14b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:49.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.127+0000 7fd251132640 1 --2- 172.21.15.72:0/2605945795 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd24c0cc0b0 0x7fd24c1615c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:49.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.127+0000 7fd251132640 1 -- 172.21.15.72:0/2605945795 >> 172.21.15.72:0/2605945795 conn(0x7fd24c0cd4a0 msgr2=0x7fd24c0b1330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:49.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.127+0000 7fd251132640 1 -- 172.21.15.72:0/2605945795 shutdown_connections 2024-04-15T15:14:49.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.127+0000 7fd251132640 1 -- 172.21.15.72:0/2605945795 wait complete. 2024-04-15T15:14:49.139 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-15T15:14:49.139 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-15T15:14:49.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.244+0000 7fcdc98d9640 1 Processor -- start 2024-04-15T15:14:49.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.244+0000 7fcdc98d9640 1 -- start start 2024-04-15T15:14:49.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.245+0000 7fcdc98d9640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fcdc415d300 0x7fcdc40d0e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:49.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.245+0000 7fcdc98d9640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcdc40d13b0 0x7fcdc40b3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:49.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.245+0000 7fcdc98d9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdc40d00b0 con 0x7fcdc40b4460 2024-04-15T15:14:49.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.245+0000 7fcdc98d9640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fcdc40b4a70 con 0x7fcdc415d300 2024-04-15T15:14:49.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.245+0000 7fcdc98d9640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fcdc40d0390 con 0x7fcdc40d13b0 2024-04-15T15:14:49.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.245+0000 7fcdc37fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fcdc40b4460 0x7fcdc40cca00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:33538/0 (socket says 172.21.15.72:33538) 2024-04-15T15:14:49.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.245+0000 7fcdc37fe640 1 -- 172.21.15.72:0/3276062416 learned_addr learned my addr 172.21.15.72:0/3276062416 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:14:49.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.246+0000 7fcdc27fc640 1 --2- 172.21.15.72:0/3276062416 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcdc40d13b0 0x7fcdc40b3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:49.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.246+0000 7fcdc2ffd640 1 --2- 172.21.15.72:0/3276062416 >> v2:172.21.15.175:3300/0 conn(0x7fcdc415d300 0x7fcdc40d0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:49.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.246+0000 7fcdc1ffb640 1 -- 172.21.15.72:0/3276062416 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3409632353 0 0) 0x7fcdc40d00b0 con 0x7fcdc40b4460 2024-04-15T15:14:49.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.246+0000 7fcdc1ffb640 1 -- 172.21.15.72:0/3276062416 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcdb0003520 con 0x7fcdc40b4460 2024-04-15T15:14:49.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.246+0000 7fcdc1ffb640 1 -- 172.21.15.72:0/3276062416 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3445719844 0 0) 0x7fcdb0003520 con 0x7fcdc40b4460 2024-04-15T15:14:49.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.246+0000 7fcdc1ffb640 1 -- 172.21.15.72:0/3276062416 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcdc40d13b0 msgr2=0x7fcdc40b3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:49.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.246+0000 7fcdc1ffb640 1 --2- 172.21.15.72:0/3276062416 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcdc40d13b0 0x7fcdc40b3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:49.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.246+0000 7fcdc1ffb640 1 -- 172.21.15.72:0/3276062416 >> v2:172.21.15.175:3300/0 conn(0x7fcdc415d300 msgr2=0x7fcdc40d0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:49.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.246+0000 7fcdc1ffb640 1 --2- 172.21.15.72:0/3276062416 >> v2:172.21.15.175:3300/0 conn(0x7fcdc415d300 0x7fcdc40d0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:49.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.246+0000 7fcdc1ffb640 1 -- 172.21.15.72:0/3276062416 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdc40d0670 con 0x7fcdc40b4460 2024-04-15T15:14:49.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.246+0000 7fcdc1ffb640 1 -- 172.21.15.72:0/3276062416 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7fcdbc001670 con 0x7fcdc40b4460 2024-04-15T15:14:49.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.246+0000 7fcdc2ffd640 1 --2- 172.21.15.72:0/3276062416 >> v2:172.21.15.175:3300/0 conn(0x7fcdc415d300 0x7fcdc40d0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:14:49.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.246+0000 7fcdc1ffb640 1 -- 172.21.15.72:0/3276062416 >> v1:172.21.15.72:6789/0 conn(0x7fcdc40b4460 legacy=0x7fcdc40cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:49.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.246+0000 7fcdc1ffb640 1 --2- 172.21.15.72:0/3276062416 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcdc40d13b0 0x7fcdb00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:49.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.246+0000 7fcdc1ffb640 1 --2- 172.21.15.72:0/3276062416 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcdb0003f30 0x7fcdb0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:49.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.246+0000 7fcdc1ffb640 1 --2- 172.21.15.72:0/3276062416 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdb0004d40 0x7fcdb00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:49.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.246+0000 7fcdc1ffb640 1 -- 172.21.15.72:0/3276062416 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc40b4a70 con 0x7fcdb0004d40 2024-04-15T15:14:49.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.246+0000 7fcdc1ffb640 1 -- 172.21.15.72:0/3276062416 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc40d0390 con 0x7fcdb0003f30 2024-04-15T15:14:49.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.246+0000 7fcdc1ffb640 1 -- 172.21.15.72:0/3276062416 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fcdb000ba10 con 0x7fcdc40d13b0 2024-04-15T15:14:49.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.246+0000 7fcdc37fe640 1 --2- 172.21.15.72:0/3276062416 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdb0004d40 0x7fcdb00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:49.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.246+0000 7fcdc2ffd640 1 --2- 172.21.15.72:0/3276062416 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcdc40d13b0 0x7fcdb00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:49.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.246+0000 7fcdc27fc640 1 --2- 172.21.15.72:0/3276062416 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcdb0003f30 0x7fcdb0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:49.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.246+0000 7fcdc37fe640 1 -- 172.21.15.72:0/3276062416 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcdc40d13b0 msgr2=0x7fcdb00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:49.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.246+0000 7fcdc37fe640 1 --2- 172.21.15.72:0/3276062416 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcdc40d13b0 0x7fcdb00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:49.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.246+0000 7fcdc37fe640 1 -- 172.21.15.72:0/3276062416 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcdb0003f30 msgr2=0x7fcdb0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:49.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.246+0000 7fcdc37fe640 1 --2- 172.21.15.72:0/3276062416 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcdb0003f30 0x7fcdb0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:49.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.246+0000 7fcdc37fe640 1 -- 172.21.15.72:0/3276062416 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdb0003520 con 0x7fcdb0004d40 2024-04-15T15:14:49.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.247+0000 7fcdc37fe640 1 --2- 172.21.15.72:0/3276062416 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdb0004d40 0x7fcdb00051c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc00c2c0 tx=0x7fcdb0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43c394bcaba6c018 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:49.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.247+0000 7fcdc1ffb640 1 -- 172.21.15.72:0/3276062416 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc007d50 con 0x7fcdb0004d40 2024-04-15T15:14:49.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.247+0000 7fcdc1ffb640 1 -- 172.21.15.72:0/3276062416 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdbc030070 con 0x7fcdb0004d40 2024-04-15T15:14:49.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.247+0000 7fcdc1ffb640 1 -- 172.21.15.72:0/3276062416 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc02cda0 con 0x7fcdb0004d40 2024-04-15T15:14:49.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.248+0000 7fcdc98d9640 1 -- 172.21.15.72:0/3276062416 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdb0004d40 msgr2=0x7fcdb00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:49.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.248+0000 7fcdc98d9640 1 --2- 172.21.15.72:0/3276062416 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdb0004d40 0x7fcdb00051c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc00c2c0 tx=0x7fcdb0005830 comp rx=0 tx=0).stop 2024-04-15T15:14:49.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.248+0000 7fcdc37fe640 1 -- 172.21.15.72:0/3276062416 reap_dead start 2024-04-15T15:14:49.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.248+0000 7fcdc98d9640 1 -- 172.21.15.72:0/3276062416 shutdown_connections 2024-04-15T15:14:49.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.248+0000 7fcdc98d9640 1 -- 172.21.15.72:0/3276062416 >> 172.21.15.72:0/3276062416 conn(0x7fcdc40d2440 msgr2=0x7fcdc40d2820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:49.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.248+0000 7fcdc98d9640 1 -- 172.21.15.72:0/3276062416 shutdown_connections 2024-04-15T15:14:49.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.248+0000 7fcdc98d9640 1 -- 172.21.15.72:0/3276062416 wait complete. 2024-04-15T15:14:49.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.249+0000 7fcdc98d9640 1 Processor -- start 2024-04-15T15:14:49.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.249+0000 7fcdc98d9640 1 -- start start 2024-04-15T15:14:49.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.250+0000 7fcdc98d9640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcdb0004d40 0x7fcdc406ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:49.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.250+0000 7fcdc98d9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdc414d1f0 0x7fcdc406d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:49.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.250+0000 7fcdc98d9640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcdc406d840 0x7fcdc4151840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:49.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.250+0000 7fcdc98d9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc40cd290 con 0x7fcdc414d1f0 2024-04-15T15:14:49.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.250+0000 7fcdc98d9640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc40cd110 con 0x7fcdc406d840 2024-04-15T15:14:49.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.250+0000 7fcdc98d9640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fcdc40b3f20 con 0x7fcdb0004d40 2024-04-15T15:14:49.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.250+0000 7fcdc27fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdc414d1f0 0x7fcdc406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:49.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.250+0000 7fcdc27fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdc414d1f0 0x7fcdc406d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36390/0 (socket says 172.21.15.72:36390) 2024-04-15T15:14:49.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.250+0000 7fcdc27fc640 1 -- 172.21.15.72:0/781372606 learned_addr learned my addr 172.21.15.72:0/781372606 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:49.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.250+0000 7fcdc37fe640 1 --2- 172.21.15.72:0/781372606 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcdc406d840 0x7fcdc4151840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:49.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.250+0000 7fcdc2ffd640 1 --2- 172.21.15.72:0/781372606 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcdb0004d40 0x7fcdc406ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:49.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.250+0000 7fcdc27fc640 1 -- 172.21.15.72:0/781372606 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcdb0004d40 msgr2=0x7fcdc406ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:49.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.250+0000 7fcdc27fc640 1 --2- 172.21.15.72:0/781372606 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcdb0004d40 0x7fcdc406ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:49.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.250+0000 7fcdc27fc640 1 -- 172.21.15.72:0/781372606 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcdc406d840 msgr2=0x7fcdc4151840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:49.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.250+0000 7fcdc27fc640 1 --2- 172.21.15.72:0/781372606 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcdc406d840 0x7fcdc4151840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:49.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.250+0000 7fcdc27fc640 1 -- 172.21.15.72:0/781372606 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdc406ddc0 con 0x7fcdc414d1f0 2024-04-15T15:14:49.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.250+0000 7fcdc37fe640 1 --2- 172.21.15.72:0/781372606 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcdc406d840 0x7fcdc4151840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:14:49.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.251+0000 7fcdc27fc640 1 --2- 172.21.15.72:0/781372606 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdc414d1f0 0x7fcdc406d300 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcdac011910 tx=0x7fcdac011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:49.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.251+0000 7fcdbbfff640 1 -- 172.21.15.72:0/781372606 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdac005ba0 con 0x7fcdc414d1f0 2024-04-15T15:14:49.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.251+0000 7fcdbbfff640 1 -- 172.21.15.72:0/781372606 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdac005d40 con 0x7fcdc414d1f0 2024-04-15T15:14:49.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.251+0000 7fcdbbfff640 1 -- 172.21.15.72:0/781372606 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdac00fb00 con 0x7fcdc414d1f0 2024-04-15T15:14:49.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.251+0000 7fcdc98d9640 1 -- 172.21.15.72:0/781372606 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdc406e0a0 con 0x7fcdc414d1f0 2024-04-15T15:14:49.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.251+0000 7fcdc98d9640 1 -- 172.21.15.72:0/781372606 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdc406e660 con 0x7fcdc414d1f0 2024-04-15T15:14:49.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.252+0000 7fcdc98d9640 1 -- 172.21.15.72:0/781372606 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdc40690f0 con 0x7fcdc414d1f0 2024-04-15T15:14:49.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.252+0000 7fcdbbfff640 1 -- 172.21.15.72:0/781372606 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcdac012040 con 0x7fcdc414d1f0 2024-04-15T15:14:49.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.253+0000 7fcdbbfff640 1 --2- 172.21.15.72:0/781372606 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcda406c920 0x7fcda406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:49.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.253+0000 7fcdc2ffd640 1 --2- 172.21.15.72:0/781372606 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcda406c920 0x7fcda406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:49.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.253+0000 7fcdbbfff640 1 -- 172.21.15.72:0/781372606 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fcdac08be30 con 0x7fcdc414d1f0 2024-04-15T15:14:49.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.253+0000 7fcdc2ffd640 1 --2- 172.21.15.72:0/781372606 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcda406c920 0x7fcda406ede0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4007620 tx=0x7fcdb400b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:49.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.257+0000 7fcdbbfff640 1 -- 172.21.15.72:0/781372606 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcdac018030 con 0x7fcdc414d1f0 2024-04-15T15:14:49.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:49.434+0000 7fcdc98d9640 1 -- 172.21.15.72:0/781372606 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fcdc40ceca0 con 0x7fcdc414d1f0 2024-04-15T15:14:50.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.049+0000 7fcdbbfff640 1 -- 172.21.15.72:0/781372606 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) ==== 138+0+0 (secure 0 0 0) 0x7fcdac01b360 con 0x7fcdc414d1f0 2024-04-15T15:14:50.051 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_metadata' created 2024-04-15T15:14:50.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.051+0000 7fcdc98d9640 1 -- 172.21.15.72:0/781372606 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcda406c920 msgr2=0x7fcda406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:50.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.051+0000 7fcdc98d9640 1 --2- 172.21.15.72:0/781372606 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcda406c920 0x7fcda406ede0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4007620 tx=0x7fcdb400b040 comp rx=0 tx=0).stop 2024-04-15T15:14:50.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.051+0000 7fcdc98d9640 1 -- 172.21.15.72:0/781372606 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdc414d1f0 msgr2=0x7fcdc406d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:50.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.051+0000 7fcdc98d9640 1 --2- 172.21.15.72:0/781372606 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdc414d1f0 0x7fcdc406d300 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fcdac011910 tx=0x7fcdac011dd0 comp rx=0 tx=0).stop 2024-04-15T15:14:50.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.051+0000 7fcdc98d9640 1 -- 172.21.15.72:0/781372606 shutdown_connections 2024-04-15T15:14:50.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.051+0000 7fcdc98d9640 1 --2- 172.21.15.72:0/781372606 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcda406c920 0x7fcda406ede0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:50.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.051+0000 7fcdc98d9640 1 --2- 172.21.15.72:0/781372606 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcdc406d840 0x7fcdc4151840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:50.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.051+0000 7fcdc98d9640 1 --2- 172.21.15.72:0/781372606 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcdc414d1f0 0x7fcdc406d300 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:50.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.051+0000 7fcdc98d9640 1 --2- 172.21.15.72:0/781372606 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcdb0004d40 0x7fcdc406ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:50.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.051+0000 7fcdc98d9640 1 -- 172.21.15.72:0/781372606 >> 172.21.15.72:0/781372606 conn(0x7fcdc40d2440 msgr2=0x7fcdc415fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:50.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.052+0000 7fcdc98d9640 1 -- 172.21.15.72:0/781372606 shutdown_connections 2024-04-15T15:14:50.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.052+0000 7fcdc98d9640 1 -- 172.21.15.72:0/781372606 wait complete. 2024-04-15T15:14:50.068 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-15T15:14:50.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.176+0000 7f31d7ad3640 1 Processor -- start 2024-04-15T15:14:50.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.177+0000 7f31d7ad3640 1 -- start start 2024-04-15T15:14:50.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.177+0000 7f31d7ad3640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f31d00cf6e0 0x7f31d00cfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:50.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.177+0000 7f31d7ad3640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31d00b62b0 con 0x7f31d015e720 2024-04-15T15:14:50.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.177+0000 7f31d7ad3640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31d006d7d0 con 0x7f31d00d00b0 2024-04-15T15:14:50.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.177+0000 7f31d7ad3640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f31d00b5510 con 0x7f31d00cf6e0 2024-04-15T15:14:50.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.178+0000 7f31d586f640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f31d015e720 0x7f31d00b4e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:33548/0 (socket says 172.21.15.72:33548) 2024-04-15T15:14:50.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.178+0000 7f31d586f640 1 -- 172.21.15.72:0/186325719 learned_addr learned my addr 172.21.15.72:0/186325719 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:14:50.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.178+0000 7f31d486d640 1 -- 172.21.15.72:0/186325719 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 148686285 0 0) 0x7f31d00b62b0 con 0x7f31d015e720 2024-04-15T15:14:50.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.178+0000 7f31d486d640 1 -- 172.21.15.72:0/186325719 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f31bc003520 con 0x7f31d015e720 2024-04-15T15:14:50.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.178+0000 7f31d506e640 1 --2- 172.21.15.72:0/186325719 >> v2:172.21.15.175:3300/0 conn(0x7f31d00cf6e0 0x7f31d00cfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:50.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.178+0000 7f31d486d640 1 -- 172.21.15.72:0/186325719 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1381138612 0 0) 0x7f31bc003520 con 0x7f31d015e720 2024-04-15T15:14:50.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.178+0000 7f31d486d640 1 -- 172.21.15.72:0/186325719 >> v2:172.21.15.175:3300/0 conn(0x7f31d00cf6e0 msgr2=0x7f31d00cfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:50.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.178+0000 7f31d486d640 1 --2- 172.21.15.72:0/186325719 >> v2:172.21.15.175:3300/0 conn(0x7f31d00cf6e0 0x7f31d00cfb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:50.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.178+0000 7f31d486d640 1 -- 172.21.15.72:0/186325719 >> v1:172.21.15.175:6789/0 conn(0x7f31d00d00b0 legacy=0x7f31d006c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:50.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.178+0000 7f31d486d640 1 -- 172.21.15.72:0/186325719 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31d00b5730 con 0x7f31d015e720 2024-04-15T15:14:50.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.178+0000 7f31d486d640 1 -- 172.21.15.72:0/186325719 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f31cc0019f0 con 0x7f31d015e720 2024-04-15T15:14:50.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.178+0000 7f31d486d640 1 -- 172.21.15.72:0/186325719 >> v1:172.21.15.72:6789/0 conn(0x7f31d015e720 legacy=0x7f31d00b4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:50.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.178+0000 7f31d486d640 1 --2- 172.21.15.72:0/186325719 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31bc0039b0 0x7f31bc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:50.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.178+0000 7f31d486d640 1 --2- 172.21.15.72:0/186325719 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31bc004800 0x7f31bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:50.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.178+0000 7f31d486d640 1 --2- 172.21.15.72:0/186325719 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f31bc0052e0 0x7f31bc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:50.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.178+0000 7f31d486d640 1 -- 172.21.15.72:0/186325719 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f31d00b5510 con 0x7f31bc0039b0 2024-04-15T15:14:50.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.178+0000 7f31d486d640 1 -- 172.21.15.72:0/186325719 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f31bc00c1b0 con 0x7f31bc004800 2024-04-15T15:14:50.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.178+0000 7f31d486d640 1 -- 172.21.15.72:0/186325719 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f31bc00c3b0 con 0x7f31bc0052e0 2024-04-15T15:14:50.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.178+0000 7f31d586f640 1 --2- 172.21.15.72:0/186325719 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31bc0039b0 0x7f31bc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:50.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.178+0000 7f31d506e640 1 --2- 172.21.15.72:0/186325719 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31bc004800 0x7f31bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:50.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.178+0000 7f31d6070640 1 --2- 172.21.15.72:0/186325719 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f31bc0052e0 0x7f31bc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:50.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.178+0000 7f31d586f640 1 -- 172.21.15.72:0/186325719 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f31bc0052e0 msgr2=0x7f31bc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:50.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.178+0000 7f31d586f640 1 --2- 172.21.15.72:0/186325719 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f31bc0052e0 0x7f31bc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:50.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.178+0000 7f31d586f640 1 -- 172.21.15.72:0/186325719 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31bc004800 msgr2=0x7f31bc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:50.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.178+0000 7f31d586f640 1 --2- 172.21.15.72:0/186325719 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31bc004800 0x7f31bc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:50.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.178+0000 7f31d6070640 1 -- 172.21.15.72:0/186325719 reap_dead start 2024-04-15T15:14:50.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.178+0000 7f31d586f640 1 -- 172.21.15.72:0/186325719 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31bc003520 con 0x7f31bc0039b0 2024-04-15T15:14:50.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.178+0000 7f31d506e640 1 --2- 172.21.15.72:0/186325719 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31bc004800 0x7f31bc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:14:50.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.179+0000 7f31d586f640 1 --2- 172.21.15.72:0/186325719 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31bc0039b0 0x7f31bc003dd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00a830 tx=0x7f31bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1af54a495026f18 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:50.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.179+0000 7f31d486d640 1 -- 172.21.15.72:0/186325719 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31cc00a480 con 0x7f31bc0039b0 2024-04-15T15:14:50.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.179+0000 7f31d486d640 1 -- 172.21.15.72:0/186325719 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31cc034630 con 0x7f31bc0039b0 2024-04-15T15:14:50.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.179+0000 7f31d486d640 1 -- 172.21.15.72:0/186325719 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31cc034aa0 con 0x7f31bc0039b0 2024-04-15T15:14:50.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.180+0000 7f31d7ad3640 1 -- 172.21.15.72:0/186325719 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31bc0039b0 msgr2=0x7f31bc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:50.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.180+0000 7f31d7ad3640 1 --2- 172.21.15.72:0/186325719 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31bc0039b0 0x7f31bc003dd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f31cc00a830 tx=0x7f31bc005830 comp rx=0 tx=0).stop 2024-04-15T15:14:50.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.180+0000 7f31d7ad3640 1 -- 172.21.15.72:0/186325719 shutdown_connections 2024-04-15T15:14:50.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.180+0000 7f31d7ad3640 1 --2- 172.21.15.72:0/186325719 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31bc0039b0 0x7f31bc003dd0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:50.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.180+0000 7f31d7ad3640 1 -- 172.21.15.72:0/186325719 >> 172.21.15.72:0/186325719 conn(0x7f31d0070540 msgr2=0x7f31d0070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:50.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.180+0000 7f31d7ad3640 1 -- 172.21.15.72:0/186325719 shutdown_connections 2024-04-15T15:14:50.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.180+0000 7f31d7ad3640 1 -- 172.21.15.72:0/186325719 wait complete. 2024-04-15T15:14:50.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.181+0000 7f31d7ad3640 1 Processor -- start 2024-04-15T15:14:50.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.181+0000 7f31d7ad3640 1 -- start start 2024-04-15T15:14:50.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.182+0000 7f31d7ad3640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f31bc0039b0 0x7f31d0157680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:50.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.182+0000 7f31d7ad3640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31d0154bd0 0x7f31d0153c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:50.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.182+0000 7f31d7ad3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31d0160cd0 0x7f31d0161150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:50.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.182+0000 7f31d7ad3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f31d00b5cf0 con 0x7f31d0160cd0 2024-04-15T15:14:50.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.182+0000 7f31d7ad3640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f31d00d0500 con 0x7f31d0154bd0 2024-04-15T15:14:50.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.182+0000 7f31d7ad3640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f31d00b5fd0 con 0x7f31bc0039b0 2024-04-15T15:14:50.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.182+0000 7f31d6070640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31d0160cd0 0x7f31d0161150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:50.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.182+0000 7f31d6070640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31d0160cd0 0x7f31d0161150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36402/0 (socket says 172.21.15.72:36402) 2024-04-15T15:14:50.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.182+0000 7f31d6070640 1 -- 172.21.15.72:0/820688813 learned_addr learned my addr 172.21.15.72:0/820688813 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:50.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.182+0000 7f31d586f640 1 --2- 172.21.15.72:0/820688813 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f31bc0039b0 0x7f31d0157680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:50.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.182+0000 7f31d506e640 1 --2- 172.21.15.72:0/820688813 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31d0154bd0 0x7f31d0153c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:50.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.182+0000 7f31d6070640 1 -- 172.21.15.72:0/820688813 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f31bc0039b0 msgr2=0x7f31d0157680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:50.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.182+0000 7f31d6070640 1 --2- 172.21.15.72:0/820688813 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f31bc0039b0 0x7f31d0157680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:50.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.182+0000 7f31d6070640 1 -- 172.21.15.72:0/820688813 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31d0154bd0 msgr2=0x7f31d0153c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:50.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.182+0000 7f31d6070640 1 --2- 172.21.15.72:0/820688813 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31d0154bd0 0x7f31d0153c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:50.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.182+0000 7f31d6070640 1 -- 172.21.15.72:0/820688813 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31d00cb1a0 con 0x7f31d0160cd0 2024-04-15T15:14:50.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.182+0000 7f31d506e640 1 --2- 172.21.15.72:0/820688813 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31d0154bd0 0x7f31d0153c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:14:50.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.182+0000 7f31d6070640 1 --2- 172.21.15.72:0/820688813 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31d0160cd0 0x7f31d0161150 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f31d00d0e70 tx=0x7f31c800e8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:50.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.182+0000 7f31c67fc640 1 -- 172.21.15.72:0/820688813 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c8019070 con 0x7f31d0160cd0 2024-04-15T15:14:50.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.182+0000 7f31c67fc640 1 -- 172.21.15.72:0/820688813 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31c80052e0 con 0x7f31d0160cd0 2024-04-15T15:14:50.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.182+0000 7f31d7ad3640 1 -- 172.21.15.72:0/820688813 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31d00cb480 con 0x7f31d0160cd0 2024-04-15T15:14:50.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.182+0000 7f31c67fc640 1 -- 172.21.15.72:0/820688813 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31c8005490 con 0x7f31d0160cd0 2024-04-15T15:14:50.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.183+0000 7f31d7ad3640 1 -- 172.21.15.72:0/820688813 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31d00cba40 con 0x7f31d0160cd0 2024-04-15T15:14:50.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.184+0000 7f31d7ad3640 1 -- 172.21.15.72:0/820688813 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31d0075c50 con 0x7f31d0160cd0 2024-04-15T15:14:50.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.184+0000 7f31c67fc640 1 -- 172.21.15.72:0/820688813 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f31c801c070 con 0x7f31d0160cd0 2024-04-15T15:14:50.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.184+0000 7f31c67fc640 1 --2- 172.21.15.72:0/820688813 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f31b406c920 0x7f31b406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:50.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.184+0000 7f31d586f640 1 --2- 172.21.15.72:0/820688813 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f31b406c920 0x7f31b406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:50.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.184+0000 7f31c67fc640 1 -- 172.21.15.72:0/820688813 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 5227+0+0 (secure 0 0 0) 0x7f31c808ad70 con 0x7f31d0160cd0 2024-04-15T15:14:50.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.185+0000 7f31d586f640 1 --2- 172.21.15.72:0/820688813 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f31b406c920 0x7f31b406ede0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f31cc034000 tx=0x7f31cc02f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:50.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.189+0000 7f31c67fc640 1 -- 172.21.15.72:0/820688813 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f31c8059380 con 0x7f31d0160cd0 2024-04-15T15:14:50.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:50.364+0000 7f31d7ad3640 1 -- 172.21.15.72:0/820688813 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f31d00cfc70 con 0x7f31d0160cd0 2024-04-15T15:14:50.780 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:14:50.782 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:14:51.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.052+0000 7f31c67fc640 1 -- 172.21.15.72:0/820688813 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v18) ==== 171+0+0 (secure 0 0 0) 0x7f31c8058c80 con 0x7f31d0160cd0 2024-04-15T15:14:51.053 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_data' created 2024-04-15T15:14:51.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.054+0000 7f31d7ad3640 1 -- 172.21.15.72:0/820688813 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f31b406c920 msgr2=0x7f31b406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:51.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.054+0000 7f31d7ad3640 1 --2- 172.21.15.72:0/820688813 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f31b406c920 0x7f31b406ede0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f31cc034000 tx=0x7f31cc02f000 comp rx=0 tx=0).stop 2024-04-15T15:14:51.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.054+0000 7f31d7ad3640 1 -- 172.21.15.72:0/820688813 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31d0160cd0 msgr2=0x7f31d0161150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:51.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.054+0000 7f31d7ad3640 1 --2- 172.21.15.72:0/820688813 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31d0160cd0 0x7f31d0161150 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f31d00d0e70 tx=0x7f31c800e8a0 comp rx=0 tx=0).stop 2024-04-15T15:14:51.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.055+0000 7f31d7ad3640 1 -- 172.21.15.72:0/820688813 shutdown_connections 2024-04-15T15:14:51.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.055+0000 7f31d7ad3640 1 --2- 172.21.15.72:0/820688813 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f31b406c920 0x7f31b406ede0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:51.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.055+0000 7f31d7ad3640 1 --2- 172.21.15.72:0/820688813 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31d0160cd0 0x7f31d0161150 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:51.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.055+0000 7f31d7ad3640 1 --2- 172.21.15.72:0/820688813 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31d0154bd0 0x7f31d0153c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:51.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.055+0000 7f31d7ad3640 1 --2- 172.21.15.72:0/820688813 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f31bc0039b0 0x7f31d0157680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:51.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.055+0000 7f31d7ad3640 1 -- 172.21.15.72:0/820688813 >> 172.21.15.72:0/820688813 conn(0x7f31d0070540 msgr2=0x7f31d0072ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:51.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.055+0000 7f31d7ad3640 1 -- 172.21.15.72:0/820688813 shutdown_connections 2024-04-15T15:14:51.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.055+0000 7f31d7ad3640 1 -- 172.21.15.72:0/820688813 wait complete. 2024-04-15T15:14:51.078 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-15T15:14:51.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.185+0000 7f90274b1640 1 Processor -- start 2024-04-15T15:14:51.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.185+0000 7f90274b1640 1 -- start start 2024-04-15T15:14:51.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.186+0000 7f90274b1640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f90200b11d0 0x7f90200d4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:51.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.186+0000 7f90274b1640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f90200b6040 0x7f90200d4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:51.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.186+0000 7f90274b1640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90200b65b0 con 0x7f902015e960 2024-04-15T15:14:51.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.186+0000 7f90274b1640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f90200b6890 con 0x7f90200b6040 2024-04-15T15:14:51.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.186+0000 7f90274b1640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f90200b6b70 con 0x7f90200b11d0 2024-04-15T15:14:51.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.186+0000 7f902524d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f902015e960 0x7f90200d1920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:33556/0 (socket says 172.21.15.72:33556) 2024-04-15T15:14:51.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.186+0000 7f902524d640 1 -- 172.21.15.72:0/2185828103 learned_addr learned my addr 172.21.15.72:0/2185828103 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:14:51.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.186+0000 7f9017fff640 1 -- 172.21.15.72:0/2185828103 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1276580159 0 0) 0x7f90200b65b0 con 0x7f902015e960 2024-04-15T15:14:51.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.186+0000 7f9025a4e640 1 --2- 172.21.15.72:0/2185828103 >> v2:172.21.15.175:3300/0 conn(0x7f90200b6040 0x7f90200d4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:51.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.186+0000 7f9017fff640 1 -- 172.21.15.72:0/2185828103 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f900c003520 con 0x7f902015e960 2024-04-15T15:14:51.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.186+0000 7f9024a4c640 1 --2- 172.21.15.72:0/2185828103 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f90200b11d0 0x7f90200d4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:51.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.186+0000 7f9017fff640 1 -- 172.21.15.72:0/2185828103 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3271517403 0 0) 0x7f900c003520 con 0x7f902015e960 2024-04-15T15:14:51.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.186+0000 7f9017fff640 1 -- 172.21.15.72:0/2185828103 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f90200b11d0 msgr2=0x7f90200d4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:51.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.186+0000 7f9017fff640 1 --2- 172.21.15.72:0/2185828103 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f90200b11d0 0x7f90200d4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:51.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.186+0000 7f9017fff640 1 -- 172.21.15.72:0/2185828103 >> v2:172.21.15.175:3300/0 conn(0x7f90200b6040 msgr2=0x7f90200d4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:51.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.186+0000 7f9017fff640 1 --2- 172.21.15.72:0/2185828103 >> v2:172.21.15.175:3300/0 conn(0x7f90200b6040 0x7f90200d4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:51.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.186+0000 7f9017fff640 1 -- 172.21.15.72:0/2185828103 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90200b6e50 con 0x7f902015e960 2024-04-15T15:14:51.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.186+0000 7f9017fff640 1 -- 172.21.15.72:0/2185828103 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f901c0019f0 con 0x7f902015e960 2024-04-15T15:14:51.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.186+0000 7f9017fff640 1 -- 172.21.15.72:0/2185828103 >> v1:172.21.15.72:6789/0 conn(0x7f902015e960 legacy=0x7f90200d1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:51.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.186+0000 7f9017fff640 1 --2- 172.21.15.72:0/2185828103 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f900c0039b0 0x7f900c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:51.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.186+0000 7f9017fff640 1 --2- 172.21.15.72:0/2185828103 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f90200b11d0 0x7f900c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:51.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.187+0000 7f9017fff640 1 --2- 172.21.15.72:0/2185828103 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f900c004d40 0x7f900c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:51.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.187+0000 7f9017fff640 1 -- 172.21.15.72:0/2185828103 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f90200b6890 con 0x7f900c0039b0 2024-04-15T15:14:51.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.187+0000 7f9017fff640 1 -- 172.21.15.72:0/2185828103 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f90200b6b70 con 0x7f900c004d40 2024-04-15T15:14:51.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.187+0000 7f9017fff640 1 -- 172.21.15.72:0/2185828103 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f900c00b9c0 con 0x7f90200b11d0 2024-04-15T15:14:51.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.187+0000 7f902524d640 1 --2- 172.21.15.72:0/2185828103 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f900c0039b0 0x7f900c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:51.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.187+0000 7f9025a4e640 1 --2- 172.21.15.72:0/2185828103 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f900c004d40 0x7f900c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:51.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.187+0000 7f9024a4c640 1 --2- 172.21.15.72:0/2185828103 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f90200b11d0 0x7f900c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:51.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.187+0000 7f902524d640 1 -- 172.21.15.72:0/2185828103 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f90200b11d0 msgr2=0x7f900c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:51.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.187+0000 7f902524d640 1 --2- 172.21.15.72:0/2185828103 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f90200b11d0 0x7f900c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:51.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.187+0000 7f902524d640 1 -- 172.21.15.72:0/2185828103 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f900c004d40 msgr2=0x7f900c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:51.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.187+0000 7f902524d640 1 --2- 172.21.15.72:0/2185828103 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f900c004d40 0x7f900c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:51.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.187+0000 7f902524d640 1 -- 172.21.15.72:0/2185828103 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f900c003520 con 0x7f900c0039b0 2024-04-15T15:14:51.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.188+0000 7f902524d640 1 --2- 172.21.15.72:0/2185828103 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f900c0039b0 0x7f900c003dd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f901c00a870 tx=0x7f900c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba6452b3b754d06a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:51.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.188+0000 7f9017fff640 1 -- 172.21.15.72:0/2185828103 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901c00a4c0 con 0x7f900c0039b0 2024-04-15T15:14:51.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.188+0000 7f9017fff640 1 -- 172.21.15.72:0/2185828103 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f901c034630 con 0x7f900c0039b0 2024-04-15T15:14:51.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.188+0000 7f9017fff640 1 -- 172.21.15.72:0/2185828103 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901c034a20 con 0x7f900c0039b0 2024-04-15T15:14:51.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.188+0000 7f90274b1640 1 -- 172.21.15.72:0/2185828103 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f900c0039b0 msgr2=0x7f900c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:51.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.188+0000 7f90274b1640 1 --2- 172.21.15.72:0/2185828103 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f900c0039b0 0x7f900c003dd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f901c00a870 tx=0x7f900c005830 comp rx=0 tx=0).stop 2024-04-15T15:14:51.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.188+0000 7f9025a4e640 1 -- 172.21.15.72:0/2185828103 reap_dead start 2024-04-15T15:14:51.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.188+0000 7f90274b1640 1 -- 172.21.15.72:0/2185828103 shutdown_connections 2024-04-15T15:14:51.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.188+0000 7f90274b1640 1 -- 172.21.15.72:0/2185828103 >> 172.21.15.72:0/2185828103 conn(0x7f90200b9580 msgr2=0x7f90200b9990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:51.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.188+0000 7f90274b1640 1 -- 172.21.15.72:0/2185828103 shutdown_connections 2024-04-15T15:14:51.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.188+0000 7f90274b1640 1 -- 172.21.15.72:0/2185828103 wait complete. 2024-04-15T15:14:51.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.189+0000 7f90274b1640 1 Processor -- start 2024-04-15T15:14:51.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.190+0000 7f90274b1640 1 -- start start 2024-04-15T15:14:51.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.190+0000 7f90274b1640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f900c0039b0 0x7f9020161a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:51.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.190+0000 7f90274b1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f902015c720 0x7f90200cbb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:51.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.190+0000 7f90274b1640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f90200cc0a0 0x7f902009f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:51.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.190+0000 7f90274b1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f90200d4a80 con 0x7f902015c720 2024-04-15T15:14:51.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.190+0000 7f90274b1640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f90200d4900 con 0x7f900c0039b0 2024-04-15T15:14:51.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.190+0000 7f90274b1640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f90200d4c00 con 0x7f90200cc0a0 2024-04-15T15:14:51.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.190+0000 7f9024a4c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f902015c720 0x7f90200cbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:51.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.190+0000 7f9024a4c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f902015c720 0x7f90200cbb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36426/0 (socket says 172.21.15.72:36426) 2024-04-15T15:14:51.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.190+0000 7f9024a4c640 1 -- 172.21.15.72:0/1066529162 learned_addr learned my addr 172.21.15.72:0/1066529162 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:51.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.191+0000 7f902524d640 1 --2- 172.21.15.72:0/1066529162 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f900c0039b0 0x7f9020161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:51.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.191+0000 7f9025a4e640 1 --2- 172.21.15.72:0/1066529162 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f90200cc0a0 0x7f902009f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:51.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.191+0000 7f9024a4c640 1 -- 172.21.15.72:0/1066529162 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f90200cc0a0 msgr2=0x7f902009f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:51.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.191+0000 7f9024a4c640 1 --2- 172.21.15.72:0/1066529162 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f90200cc0a0 0x7f902009f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:51.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.191+0000 7f9024a4c640 1 -- 172.21.15.72:0/1066529162 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f900c0039b0 msgr2=0x7f9020161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:51.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.191+0000 7f9024a4c640 1 --2- 172.21.15.72:0/1066529162 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f900c0039b0 0x7f9020161a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:51.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.191+0000 7f9024a4c640 1 -- 172.21.15.72:0/1066529162 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90200ce3d0 con 0x7f902015c720 2024-04-15T15:14:51.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.191+0000 7f9024a4c640 1 --2- 172.21.15.72:0/1066529162 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f902015c720 0x7f90200cbb60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9010011940 tx=0x7f9010011e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:51.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.191+0000 7f9015ffb640 1 -- 172.21.15.72:0/1066529162 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9010019070 con 0x7f902015c720 2024-04-15T15:14:51.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.191+0000 7f9015ffb640 1 -- 172.21.15.72:0/1066529162 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9010004240 con 0x7f902015c720 2024-04-15T15:14:51.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.191+0000 7f90274b1640 1 -- 172.21.15.72:0/1066529162 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f902015f9a0 con 0x7f902015c720 2024-04-15T15:14:51.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.191+0000 7f9015ffb640 1 -- 172.21.15.72:0/1066529162 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f901000f8e0 con 0x7f902015c720 2024-04-15T15:14:51.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.191+0000 7f90274b1640 1 -- 172.21.15.72:0/1066529162 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f902015ce20 con 0x7f902015c720 2024-04-15T15:14:51.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.192+0000 7f90274b1640 1 -- 172.21.15.72:0/1066529162 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9020075c50 con 0x7f902015c720 2024-04-15T15:14:51.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.193+0000 7f9015ffb640 1 -- 172.21.15.72:0/1066529162 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f901001c070 con 0x7f902015c720 2024-04-15T15:14:51.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.193+0000 7f9015ffb640 1 --2- 172.21.15.72:0/1066529162 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f900406c8f0 0x7f900406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:51.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.193+0000 7f902524d640 1 --2- 172.21.15.72:0/1066529162 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f900406c8f0 0x7f900406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:51.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.193+0000 7f9015ffb640 1 -- 172.21.15.72:0/1066529162 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5609+0+0 (secure 0 0 0) 0x7f901008bfd0 con 0x7f902015c720 2024-04-15T15:14:51.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.194+0000 7f902524d640 1 --2- 172.21.15.72:0/1066529162 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f900406c8f0 0x7f900406edb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f901c034000 tx=0x7f901c009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:51.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.198+0000 7f9015ffb640 1 -- 172.21.15.72:0/1066529162 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f901005a470 con 0x7f902015c720 2024-04-15T15:14:51.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:51.410+0000 7f90274b1640 1 -- 172.21.15.72:0/1066529162 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f90200cc480 con 0x7f902015c720 2024-04-15T15:14:52.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.067+0000 7f9015ffb640 1 -- 172.21.15.72:0/1066529162 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7f9010059d70 con 0x7f902015c720 2024-04-15T15:14:52.072 INFO:teuthology.orchestra.run.smithi072.stderr:new fs with metadata pool 2 and data pool 3 2024-04-15T15:14:52.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.082+0000 7f90274b1640 1 -- 172.21.15.72:0/1066529162 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f900406c8f0 msgr2=0x7f900406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:52.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.082+0000 7f90274b1640 1 --2- 172.21.15.72:0/1066529162 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f900406c8f0 0x7f900406edb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f901c034000 tx=0x7f901c009030 comp rx=0 tx=0).stop 2024-04-15T15:14:52.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.082+0000 7f90274b1640 1 -- 172.21.15.72:0/1066529162 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f902015c720 msgr2=0x7f90200cbb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:52.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.082+0000 7f90274b1640 1 --2- 172.21.15.72:0/1066529162 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f902015c720 0x7f90200cbb60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9010011940 tx=0x7f9010011e00 comp rx=0 tx=0).stop 2024-04-15T15:14:52.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.083+0000 7f90274b1640 1 -- 172.21.15.72:0/1066529162 shutdown_connections 2024-04-15T15:14:52.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.083+0000 7f90274b1640 1 --2- 172.21.15.72:0/1066529162 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f900406c8f0 0x7f900406edb0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:52.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.083+0000 7f90274b1640 1 --2- 172.21.15.72:0/1066529162 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f90200cc0a0 0x7f902009f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:52.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.083+0000 7f90274b1640 1 --2- 172.21.15.72:0/1066529162 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f902015c720 0x7f90200cbb60 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:52.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.083+0000 7f90274b1640 1 --2- 172.21.15.72:0/1066529162 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f900c0039b0 0x7f9020161a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:52.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.083+0000 7f90274b1640 1 -- 172.21.15.72:0/1066529162 >> 172.21.15.72:0/1066529162 conn(0x7f90200b9580 msgr2=0x7f90200d2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:52.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.084+0000 7f90274b1640 1 -- 172.21.15.72:0/1066529162 shutdown_connections 2024-04-15T15:14:52.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.085+0000 7f90274b1640 1 -- 172.21.15.72:0/1066529162 wait complete. 2024-04-15T15:14:52.103 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-15T15:14:52.103 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-15T15:14:52.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.210+0000 7f6e4da55640 1 Processor -- start 2024-04-15T15:14:52.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.210+0000 7f6e4da55640 1 -- start start 2024-04-15T15:14:52.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.211+0000 7f6e4da55640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6e480d1cf0 0x7f6e480b24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:52.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.211+0000 7f6e4da55640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e480b1d60 con 0x7f6e4815c4d0 2024-04-15T15:14:52.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.211+0000 7f6e4da55640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e4815d2d0 con 0x7f6e480b2a30 2024-04-15T15:14:52.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.211+0000 7f6e4da55640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f6e480cd560 con 0x7f6e480d1cf0 2024-04-15T15:14:52.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.211+0000 7f6e46ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f6e4815c4d0 0x7f6e4815c8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37178/0 (socket says 172.21.15.72:37178) 2024-04-15T15:14:52.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.211+0000 7f6e46ffd640 1 -- 172.21.15.72:0/2147492873 learned_addr learned my addr 172.21.15.72:0/2147492873 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:14:52.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.212+0000 7f6e45ffb640 1 -- 172.21.15.72:0/2147492873 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1453509839 0 0) 0x7f6e480b1d60 con 0x7f6e4815c4d0 2024-04-15T15:14:52.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.212+0000 7f6e467fc640 1 --2- 172.21.15.72:0/2147492873 >> v2:172.21.15.175:3300/0 conn(0x7f6e480d1cf0 0x7f6e480b24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:52.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.212+0000 7f6e45ffb640 1 -- 172.21.15.72:0/2147492873 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e34003520 con 0x7f6e4815c4d0 2024-04-15T15:14:52.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.212+0000 7f6e45ffb640 1 -- 172.21.15.72:0/2147492873 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2298739499 0 0) 0x7f6e34003520 con 0x7f6e4815c4d0 2024-04-15T15:14:52.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.212+0000 7f6e45ffb640 1 -- 172.21.15.72:0/2147492873 >> v2:172.21.15.175:3300/0 conn(0x7f6e480d1cf0 msgr2=0x7f6e480b24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:52.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.212+0000 7f6e45ffb640 1 --2- 172.21.15.72:0/2147492873 >> v2:172.21.15.175:3300/0 conn(0x7f6e480d1cf0 0x7f6e480b24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:52.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.212+0000 7f6e45ffb640 1 -- 172.21.15.72:0/2147492873 >> v1:172.21.15.175:6789/0 conn(0x7f6e480b2a30 legacy=0x7f6e480d01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:52.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.212+0000 7f6e45ffb640 1 -- 172.21.15.72:0/2147492873 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e4815d5b0 con 0x7f6e4815c4d0 2024-04-15T15:14:52.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.212+0000 7f6e45ffb640 1 -- 172.21.15.72:0/2147492873 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f6e380019f0 con 0x7f6e4815c4d0 2024-04-15T15:14:52.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.212+0000 7f6e45ffb640 1 -- 172.21.15.72:0/2147492873 >> v1:172.21.15.72:6789/0 conn(0x7f6e4815c4d0 legacy=0x7f6e4815c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:52.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.212+0000 7f6e45ffb640 1 --2- 172.21.15.72:0/2147492873 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e340039b0 0x7f6e34003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:52.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.212+0000 7f6e45ffb640 1 --2- 172.21.15.72:0/2147492873 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6e34004800 0x7f6e34004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:52.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.212+0000 7f6e45ffb640 1 --2- 172.21.15.72:0/2147492873 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6e340051c0 0x7f6e3400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:52.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.212+0000 7f6e45ffb640 1 -- 172.21.15.72:0/2147492873 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6e480cd560 con 0x7f6e34004800 2024-04-15T15:14:52.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.212+0000 7f6e45ffb640 1 -- 172.21.15.72:0/2147492873 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f6e3400c160 con 0x7f6e340039b0 2024-04-15T15:14:52.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.212+0000 7f6e45ffb640 1 -- 172.21.15.72:0/2147492873 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f6e3400c360 con 0x7f6e340051c0 2024-04-15T15:14:52.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.212+0000 7f6e467fc640 1 --2- 172.21.15.72:0/2147492873 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6e34004800 0x7f6e34004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:52.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.212+0000 7f6e46ffd640 1 --2- 172.21.15.72:0/2147492873 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e340039b0 0x7f6e34003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:52.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.212+0000 7f6e477fe640 1 --2- 172.21.15.72:0/2147492873 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6e340051c0 0x7f6e3400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:52.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.212+0000 7f6e467fc640 1 -- 172.21.15.72:0/2147492873 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6e340051c0 msgr2=0x7f6e3400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:52.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.212+0000 7f6e467fc640 1 --2- 172.21.15.72:0/2147492873 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6e340051c0 0x7f6e3400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:52.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.212+0000 7f6e467fc640 1 -- 172.21.15.72:0/2147492873 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e340039b0 msgr2=0x7f6e34003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:52.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.212+0000 7f6e467fc640 1 --2- 172.21.15.72:0/2147492873 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e340039b0 0x7f6e34003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:52.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.212+0000 7f6e467fc640 1 -- 172.21.15.72:0/2147492873 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e34003520 con 0x7f6e34004800 2024-04-15T15:14:52.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.212+0000 7f6e477fe640 1 -- 172.21.15.72:0/2147492873 reap_dead start 2024-04-15T15:14:52.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.213+0000 7f6e467fc640 1 --2- 172.21.15.72:0/2147492873 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6e34004800 0x7f6e34004c80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6e30009ac0 tx=0x7f6e34005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=533a198ec9e0ec4e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:52.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.213+0000 7f6e45ffb640 1 -- 172.21.15.72:0/2147492873 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e30039070 con 0x7f6e34004800 2024-04-15T15:14:52.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.213+0000 7f6e45ffb640 1 -- 172.21.15.72:0/2147492873 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e3002c6b0 con 0x7f6e34004800 2024-04-15T15:14:52.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.213+0000 7f6e45ffb640 1 -- 172.21.15.72:0/2147492873 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e30009690 con 0x7f6e34004800 2024-04-15T15:14:52.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.214+0000 7f6e4da55640 1 -- 172.21.15.72:0/2147492873 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6e34004800 msgr2=0x7f6e34004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:52.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.214+0000 7f6e4da55640 1 --2- 172.21.15.72:0/2147492873 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6e34004800 0x7f6e34004c80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6e30009ac0 tx=0x7f6e34005830 comp rx=0 tx=0).stop 2024-04-15T15:14:52.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.214+0000 7f6e4da55640 1 -- 172.21.15.72:0/2147492873 shutdown_connections 2024-04-15T15:14:52.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.214+0000 7f6e4da55640 1 --2- 172.21.15.72:0/2147492873 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6e34004800 0x7f6e34004c80 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:52.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.214+0000 7f6e4da55640 1 -- 172.21.15.72:0/2147492873 >> 172.21.15.72:0/2147492873 conn(0x7f6e480d23e0 msgr2=0x7f6e480d27f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:52.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.214+0000 7f6e4da55640 1 -- 172.21.15.72:0/2147492873 shutdown_connections 2024-04-15T15:14:52.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.214+0000 7f6e4da55640 1 -- 172.21.15.72:0/2147492873 wait complete. 2024-04-15T15:14:52.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.215+0000 7f6e4da55640 1 Processor -- start 2024-04-15T15:14:52.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.215+0000 7f6e4da55640 1 -- start start 2024-04-15T15:14:52.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.216+0000 7f6e4da55640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6e34004800 0x7f6e4814d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:52.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.216+0000 7f6e4da55640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e481f6ab0 0x7f6e481f5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:52.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.216+0000 7f6e4da55640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6e481f6330 0x7f6e480b5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:52.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.216+0000 7f6e4da55640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6e480b24f0 con 0x7f6e481f6330 2024-04-15T15:14:52.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.216+0000 7f6e4da55640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f6e480b1fd0 con 0x7f6e481f6ab0 2024-04-15T15:14:52.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.216+0000 7f6e4da55640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f6e480b2670 con 0x7f6e34004800 2024-04-15T15:14:52.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.216+0000 7f6e477fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6e481f6330 0x7f6e480b5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:52.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.216+0000 7f6e477fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6e481f6330 0x7f6e480b5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45032/0 (socket says 172.21.15.72:45032) 2024-04-15T15:14:52.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.216+0000 7f6e477fe640 1 -- 172.21.15.72:0/2035570985 learned_addr learned my addr 172.21.15.72:0/2035570985 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:52.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.216+0000 7f6e46ffd640 1 --2- 172.21.15.72:0/2035570985 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6e34004800 0x7f6e4814d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:52.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.216+0000 7f6e467fc640 1 --2- 172.21.15.72:0/2035570985 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e481f6ab0 0x7f6e481f5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:52.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.216+0000 7f6e477fe640 1 -- 172.21.15.72:0/2035570985 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6e34004800 msgr2=0x7f6e4814d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:52.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.216+0000 7f6e477fe640 1 --2- 172.21.15.72:0/2035570985 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6e34004800 0x7f6e4814d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:52.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.216+0000 7f6e477fe640 1 -- 172.21.15.72:0/2035570985 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e481f6ab0 msgr2=0x7f6e481f5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:52.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.216+0000 7f6e477fe640 1 --2- 172.21.15.72:0/2035570985 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e481f6ab0 0x7f6e481f5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:52.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.216+0000 7f6e477fe640 1 -- 172.21.15.72:0/2035570985 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e480cae30 con 0x7f6e481f6330 2024-04-15T15:14:52.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.216+0000 7f6e477fe640 1 --2- 172.21.15.72:0/2035570985 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6e481f6330 0x7f6e480b5de0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6e480cdf10 tx=0x7f6e4000e820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:52.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.216+0000 7f6e3ffff640 1 -- 172.21.15.72:0/2035570985 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e40019070 con 0x7f6e481f6330 2024-04-15T15:14:52.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.216+0000 7f6e3ffff640 1 -- 172.21.15.72:0/2035570985 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e400052e0 con 0x7f6e481f6330 2024-04-15T15:14:52.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.216+0000 7f6e3ffff640 1 -- 172.21.15.72:0/2035570985 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e40003830 con 0x7f6e481f6330 2024-04-15T15:14:52.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.216+0000 7f6e4da55640 1 -- 172.21.15.72:0/2035570985 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e480b5020 con 0x7f6e481f6330 2024-04-15T15:14:52.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.217+0000 7f6e4da55640 1 -- 172.21.15.72:0/2035570985 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e480b55e0 con 0x7f6e481f6330 2024-04-15T15:14:52.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.217+0000 7f6e4da55640 1 -- 172.21.15.72:0/2035570985 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e480690f0 con 0x7f6e481f6330 2024-04-15T15:14:52.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.218+0000 7f6e3ffff640 1 -- 172.21.15.72:0/2035570985 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6e4001c070 con 0x7f6e481f6330 2024-04-15T15:14:52.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.218+0000 7f6e3ffff640 1 --2- 172.21.15.72:0/2035570985 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6e2806c8f0 0x7f6e2806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:52.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.218+0000 7f6e46ffd640 1 --2- 172.21.15.72:0/2035570985 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6e2806c8f0 0x7f6e2806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:52.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.218+0000 7f6e3ffff640 1 -- 172.21.15.72:0/2035570985 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7f6e4008ade0 con 0x7f6e481f6330 2024-04-15T15:14:52.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.219+0000 7f6e46ffd640 1 --2- 172.21.15.72:0/2035570985 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6e2806c8f0 0x7f6e2806edb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6e38001560 tx=0x7f6e38006040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:52.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.223+0000 7f6e3ffff640 1 -- 172.21.15.72:0/2035570985 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6e40058bc0 con 0x7f6e481f6330 2024-04-15T15:14:52.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:52.405+0000 7f6e4da55640 1 -- 172.21.15.72:0/2035570985 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f6e480d16e0 con 0x7f6e481f6330 2024-04-15T15:14:53.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.073+0000 7f6e3ffff640 1 -- 172.21.15.72:0/2035570985 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v20) ==== 157+0+0 (secure 0 0 0) 0x7f6e4005cec0 con 0x7f6e481f6330 2024-04-15T15:14:53.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.076+0000 7f6e4da55640 1 -- 172.21.15.72:0/2035570985 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6e2806c8f0 msgr2=0x7f6e2806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:53.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.076+0000 7f6e4da55640 1 --2- 172.21.15.72:0/2035570985 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6e2806c8f0 0x7f6e2806edb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6e38001560 tx=0x7f6e38006040 comp rx=0 tx=0).stop 2024-04-15T15:14:53.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.076+0000 7f6e4da55640 1 -- 172.21.15.72:0/2035570985 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6e481f6330 msgr2=0x7f6e480b5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:53.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.076+0000 7f6e4da55640 1 --2- 172.21.15.72:0/2035570985 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6e481f6330 0x7f6e480b5de0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6e480cdf10 tx=0x7f6e4000e820 comp rx=0 tx=0).stop 2024-04-15T15:14:53.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.076+0000 7f6e4da55640 1 -- 172.21.15.72:0/2035570985 shutdown_connections 2024-04-15T15:14:53.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.076+0000 7f6e4da55640 1 --2- 172.21.15.72:0/2035570985 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6e2806c8f0 0x7f6e2806edb0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:53.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.076+0000 7f6e4da55640 1 --2- 172.21.15.72:0/2035570985 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6e481f6330 0x7f6e480b5de0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:53.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.076+0000 7f6e4da55640 1 --2- 172.21.15.72:0/2035570985 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6e481f6ab0 0x7f6e481f5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:53.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.076+0000 7f6e4da55640 1 --2- 172.21.15.72:0/2035570985 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6e34004800 0x7f6e4814d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:53.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.076+0000 7f6e4da55640 1 -- 172.21.15.72:0/2035570985 >> 172.21.15.72:0/2035570985 conn(0x7f6e480d23e0 msgr2=0x7f6e48160650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:53.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.076+0000 7f6e4da55640 1 -- 172.21.15.72:0/2035570985 shutdown_connections 2024-04-15T15:14:53.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.076+0000 7f6e4da55640 1 -- 172.21.15.72:0/2035570985 wait complete. 2024-04-15T15:14:53.091 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-15T15:14:53.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.206+0000 7fbd7d57c640 1 Processor -- start 2024-04-15T15:14:53.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.206+0000 7fbd7d57c640 1 -- start start 2024-04-15T15:14:53.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.206+0000 7fbd7d57c640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbd780d08f0 0x7fbd780ced20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:53.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.206+0000 7fbd7d57c640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fbd780cf260 0x7fbd780ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:53.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.206+0000 7fbd7d57c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd780b2130 con 0x7fbd7815c400 2024-04-15T15:14:53.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.206+0000 7fbd7d57c640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fbd780b2410 con 0x7fbd780cf260 2024-04-15T15:14:53.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.206+0000 7fbd7d57c640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fbd780b26f0 con 0x7fbd780d08f0 2024-04-15T15:14:53.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.207+0000 7fbd76ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fbd7815c400 0x7fbd780d01e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37180/0 (socket says 172.21.15.72:37180) 2024-04-15T15:14:53.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.207+0000 7fbd76ffd640 1 -- 172.21.15.72:0/902912635 learned_addr learned my addr 172.21.15.72:0/902912635 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:14:53.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.207+0000 7fbd767fc640 1 -- 172.21.15.72:0/902912635 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 486822648 0 0) 0x7fbd780b2130 con 0x7fbd7815c400 2024-04-15T15:14:53.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.207+0000 7fbd767fc640 1 -- 172.21.15.72:0/902912635 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd64003520 con 0x7fbd7815c400 2024-04-15T15:14:53.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.207+0000 7fbd777fe640 1 --2- 172.21.15.72:0/902912635 >> v2:172.21.15.175:3300/0 conn(0x7fbd780cf260 0x7fbd780ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:53.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.207+0000 7fbd6e7fc640 1 --2- 172.21.15.72:0/902912635 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbd780d08f0 0x7fbd780ced20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:53.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.207+0000 7fbd767fc640 1 -- 172.21.15.72:0/902912635 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2484925721 0 0) 0x7fbd64003520 con 0x7fbd7815c400 2024-04-15T15:14:53.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.207+0000 7fbd767fc640 1 -- 172.21.15.72:0/902912635 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbd780d08f0 msgr2=0x7fbd780ced20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:53.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.207+0000 7fbd767fc640 1 --2- 172.21.15.72:0/902912635 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbd780d08f0 0x7fbd780ced20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:53.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.207+0000 7fbd767fc640 1 -- 172.21.15.72:0/902912635 >> v2:172.21.15.175:3300/0 conn(0x7fbd780cf260 msgr2=0x7fbd780ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:53.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.207+0000 7fbd767fc640 1 --2- 172.21.15.72:0/902912635 >> v2:172.21.15.175:3300/0 conn(0x7fbd780cf260 0x7fbd780ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:53.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.207+0000 7fbd767fc640 1 -- 172.21.15.72:0/902912635 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd780b29d0 con 0x7fbd7815c400 2024-04-15T15:14:53.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.207+0000 7fbd767fc640 1 -- 172.21.15.72:0/902912635 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7fbd68001670 con 0x7fbd7815c400 2024-04-15T15:14:53.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.207+0000 7fbd767fc640 1 -- 172.21.15.72:0/902912635 >> v1:172.21.15.72:6789/0 conn(0x7fbd7815c400 legacy=0x7fbd780d01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:53.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.207+0000 7fbd767fc640 1 --2- 172.21.15.72:0/902912635 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbd640039b0 0x7fbd64003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:53.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.207+0000 7fbd767fc640 1 --2- 172.21.15.72:0/902912635 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbd780d08f0 0x7fbd64004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:53.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.207+0000 7fbd767fc640 1 --2- 172.21.15.72:0/902912635 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd64004d40 0x7fbd640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:53.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.207+0000 7fbd767fc640 1 -- 172.21.15.72:0/902912635 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fbd780b2410 con 0x7fbd64004d40 2024-04-15T15:14:53.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.207+0000 7fbd767fc640 1 -- 172.21.15.72:0/902912635 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fbd780b26f0 con 0x7fbd640039b0 2024-04-15T15:14:53.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.207+0000 7fbd767fc640 1 -- 172.21.15.72:0/902912635 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fbd6400ba70 con 0x7fbd780d08f0 2024-04-15T15:14:53.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.207+0000 7fbd777fe640 1 --2- 172.21.15.72:0/902912635 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd64004d40 0x7fbd640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:53.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.207+0000 7fbd6e7fc640 1 --2- 172.21.15.72:0/902912635 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbd780d08f0 0x7fbd64004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:53.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.207+0000 7fbd76ffd640 1 --2- 172.21.15.72:0/902912635 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbd640039b0 0x7fbd64003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:53.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.207+0000 7fbd777fe640 1 -- 172.21.15.72:0/902912635 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbd780d08f0 msgr2=0x7fbd64004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:53.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.207+0000 7fbd777fe640 1 --2- 172.21.15.72:0/902912635 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbd780d08f0 0x7fbd64004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:53.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.207+0000 7fbd777fe640 1 -- 172.21.15.72:0/902912635 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbd640039b0 msgr2=0x7fbd64003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:53.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.207+0000 7fbd777fe640 1 --2- 172.21.15.72:0/902912635 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbd640039b0 0x7fbd64003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:53.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.208+0000 7fbd777fe640 1 -- 172.21.15.72:0/902912635 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd64003520 con 0x7fbd64004d40 2024-04-15T15:14:53.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.208+0000 7fbd777fe640 1 --2- 172.21.15.72:0/902912635 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd64004d40 0x7fbd640051c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbd70007390 tx=0x7fbd64005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63374c7bfa213ddc server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:53.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.208+0000 7fbd767fc640 1 -- 172.21.15.72:0/902912635 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd70039070 con 0x7fbd64004d40 2024-04-15T15:14:53.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.208+0000 7fbd767fc640 1 -- 172.21.15.72:0/902912635 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd7002ccf0 con 0x7fbd64004d40 2024-04-15T15:14:53.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.208+0000 7fbd767fc640 1 -- 172.21.15.72:0/902912635 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd70009be0 con 0x7fbd64004d40 2024-04-15T15:14:53.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.209+0000 7fbd7d57c640 1 -- 172.21.15.72:0/902912635 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd64004d40 msgr2=0x7fbd640051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:53.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.209+0000 7fbd7d57c640 1 --2- 172.21.15.72:0/902912635 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd64004d40 0x7fbd640051c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbd70007390 tx=0x7fbd64005830 comp rx=0 tx=0).stop 2024-04-15T15:14:53.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.209+0000 7fbd777fe640 1 -- 172.21.15.72:0/902912635 reap_dead start 2024-04-15T15:14:53.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.209+0000 7fbd7d57c640 1 -- 172.21.15.72:0/902912635 shutdown_connections 2024-04-15T15:14:53.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.209+0000 7fbd7d57c640 1 -- 172.21.15.72:0/902912635 >> 172.21.15.72:0/902912635 conn(0x7fbd780b4010 msgr2=0x7fbd780b43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:53.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.209+0000 7fbd7d57c640 1 -- 172.21.15.72:0/902912635 shutdown_connections 2024-04-15T15:14:53.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.209+0000 7fbd7d57c640 1 -- 172.21.15.72:0/902912635 wait complete. 2024-04-15T15:14:53.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.210+0000 7fbd7d57c640 1 Processor -- start 2024-04-15T15:14:53.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.210+0000 7fbd7d57c640 1 -- start start 2024-04-15T15:14:53.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.211+0000 7fbd7d57c640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbd64004d40 0x7fbd780c8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:53.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.211+0000 7fbd7d57c640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbd78158500 0x7fbd7815b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:53.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.211+0000 7fbd7d57c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd78158980 0x7fbd7814a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:53.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.211+0000 7fbd7d57c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fbd7815c7e0 con 0x7fbd78158980 2024-04-15T15:14:53.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.211+0000 7fbd7d57c640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fbd780b1df0 con 0x7fbd78158500 2024-04-15T15:14:53.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.211+0000 7fbd7d57c640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fbd7815c960 con 0x7fbd64004d40 2024-04-15T15:14:53.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.211+0000 7fbd777fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd78158980 0x7fbd7814a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:53.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.211+0000 7fbd777fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd78158980 0x7fbd7814a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45050/0 (socket says 172.21.15.72:45050) 2024-04-15T15:14:53.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.211+0000 7fbd777fe640 1 -- 172.21.15.72:0/3565407509 learned_addr learned my addr 172.21.15.72:0/3565407509 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:53.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.211+0000 7fbd76ffd640 1 --2- 172.21.15.72:0/3565407509 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbd64004d40 0x7fbd780c8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:53.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.211+0000 7fbd6e7fc640 1 --2- 172.21.15.72:0/3565407509 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbd78158500 0x7fbd7815b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:53.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.211+0000 7fbd777fe640 1 -- 172.21.15.72:0/3565407509 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbd64004d40 msgr2=0x7fbd780c8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:53.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.211+0000 7fbd777fe640 1 --2- 172.21.15.72:0/3565407509 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbd64004d40 0x7fbd780c8cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:53.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.211+0000 7fbd777fe640 1 -- 172.21.15.72:0/3565407509 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbd78158500 msgr2=0x7fbd7815b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:53.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.211+0000 7fbd777fe640 1 --2- 172.21.15.72:0/3565407509 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbd78158500 0x7fbd7815b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:53.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.211+0000 7fbd777fe640 1 -- 172.21.15.72:0/3565407509 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd7815a3d0 con 0x7fbd78158980 2024-04-15T15:14:53.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.211+0000 7fbd6e7fc640 1 --2- 172.21.15.72:0/3565407509 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbd78158500 0x7fbd7815b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:14:53.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.212+0000 7fbd777fe640 1 --2- 172.21.15.72:0/3565407509 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd78158980 0x7fbd7814a950 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbd70007390 tx=0x7fbd70004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:53.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.212+0000 7fbd6ffff640 1 -- 172.21.15.72:0/3565407509 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd70039070 con 0x7fbd78158980 2024-04-15T15:14:53.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.212+0000 7fbd6ffff640 1 -- 172.21.15.72:0/3565407509 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd70044070 con 0x7fbd78158980 2024-04-15T15:14:53.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.212+0000 7fbd7d57c640 1 -- 172.21.15.72:0/3565407509 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd7815a6b0 con 0x7fbd78158980 2024-04-15T15:14:53.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.212+0000 7fbd6ffff640 1 -- 172.21.15.72:0/3565407509 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd70009be0 con 0x7fbd78158980 2024-04-15T15:14:53.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.212+0000 7fbd7d57c640 1 -- 172.21.15.72:0/3565407509 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd7815ac70 con 0x7fbd78158980 2024-04-15T15:14:53.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.213+0000 7fbd7d57c640 1 -- 172.21.15.72:0/3565407509 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd780690f0 con 0x7fbd78158980 2024-04-15T15:14:53.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.213+0000 7fbd6ffff640 1 -- 172.21.15.72:0/3565407509 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbd7003e830 con 0x7fbd78158980 2024-04-15T15:14:53.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.214+0000 7fbd6ffff640 1 --2- 172.21.15.72:0/3565407509 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fbd5806c920 0x7fbd5806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:53.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.214+0000 7fbd76ffd640 1 --2- 172.21.15.72:0/3565407509 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fbd5806c920 0x7fbd5806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:53.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.214+0000 7fbd6ffff640 1 -- 172.21.15.72:0/3565407509 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 6007+0+0 (secure 0 0 0) 0x7fbd700add30 con 0x7fbd78158980 2024-04-15T15:14:53.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.214+0000 7fbd76ffd640 1 --2- 172.21.15.72:0/3565407509 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fbd5806c920 0x7fbd5806ede0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbd68000c00 tx=0x7fbd68009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:53.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.218+0000 7fbd6ffff640 1 -- 172.21.15.72:0/3565407509 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbd7007bfa0 con 0x7fbd78158980 2024-04-15T15:14:53.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:53.396+0000 7fbd7d57c640 1 -- 172.21.15.72:0/3565407509 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fbd780b4580 con 0x7fbd78158980 2024-04-15T15:14:55.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.076+0000 7fbd6ffff640 1 -- 172.21.15.72:0/3565407509 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v22) ==== 228+0+0 (secure 0 0 0) 0x7fbd7007b8a0 con 0x7fbd78158980 2024-04-15T15:14:55.078 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_data_ec' created 2024-04-15T15:14:55.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.080+0000 7fbd7d57c640 1 -- 172.21.15.72:0/3565407509 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fbd5806c920 msgr2=0x7fbd5806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:55.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.080+0000 7fbd7d57c640 1 --2- 172.21.15.72:0/3565407509 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fbd5806c920 0x7fbd5806ede0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbd68000c00 tx=0x7fbd68009040 comp rx=0 tx=0).stop 2024-04-15T15:14:55.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.080+0000 7fbd7d57c640 1 -- 172.21.15.72:0/3565407509 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd78158980 msgr2=0x7fbd7814a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:55.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.080+0000 7fbd7d57c640 1 --2- 172.21.15.72:0/3565407509 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd78158980 0x7fbd7814a950 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fbd70007390 tx=0x7fbd70004ab0 comp rx=0 tx=0).stop 2024-04-15T15:14:55.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.080+0000 7fbd7d57c640 1 -- 172.21.15.72:0/3565407509 shutdown_connections 2024-04-15T15:14:55.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.080+0000 7fbd7d57c640 1 --2- 172.21.15.72:0/3565407509 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fbd5806c920 0x7fbd5806ede0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:55.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.080+0000 7fbd7d57c640 1 --2- 172.21.15.72:0/3565407509 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbd78158980 0x7fbd7814a950 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:55.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.080+0000 7fbd7d57c640 1 --2- 172.21.15.72:0/3565407509 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbd78158500 0x7fbd7815b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:55.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.080+0000 7fbd7d57c640 1 --2- 172.21.15.72:0/3565407509 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbd64004d40 0x7fbd780c8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:55.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.080+0000 7fbd7d57c640 1 -- 172.21.15.72:0/3565407509 >> 172.21.15.72:0/3565407509 conn(0x7fbd780b4010 msgr2=0x7fbd780cca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:55.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.080+0000 7fbd7d57c640 1 -- 172.21.15.72:0/3565407509 shutdown_connections 2024-04-15T15:14:55.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.080+0000 7fbd7d57c640 1 -- 172.21.15.72:0/3565407509 wait complete. 2024-04-15T15:14:55.101 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-15T15:14:55.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.210+0000 7fa0ef828640 1 Processor -- start 2024-04-15T15:14:55.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.210+0000 7fa0ef828640 1 -- start start 2024-04-15T15:14:55.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.211+0000 7fa0ef828640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa0e80d47d0 0x7fa0e80d2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:55.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.211+0000 7fa0ef828640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa0e806f110 0x7fa0e8161b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:55.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.211+0000 7fa0ef828640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0e80d33f0 con 0x7fa0e80d4bb0 2024-04-15T15:14:55.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.211+0000 7fa0ef828640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fa0e80d28c0 con 0x7fa0e806f110 2024-04-15T15:14:55.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.211+0000 7fa0ef828640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fa0e80d31a0 con 0x7fa0e80d47d0 2024-04-15T15:14:55.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.211+0000 7fa0eddc5640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fa0e80d4bb0 0x7fa0e80b6050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37182/0 (socket says 172.21.15.72:37182) 2024-04-15T15:14:55.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.211+0000 7fa0eddc5640 1 -- 172.21.15.72:0/881559699 learned_addr learned my addr 172.21.15.72:0/881559699 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:14:55.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.211+0000 7fa0ecdc3640 1 --2- 172.21.15.72:0/881559699 >> v2:172.21.15.175:3300/0 conn(0x7fa0e806f110 0x7fa0e8161b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:55.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.211+0000 7fa0ed5c4640 1 --2- 172.21.15.72:0/881559699 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa0e80d47d0 0x7fa0e80d2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:55.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.211+0000 7fa0dffff640 1 -- 172.21.15.72:0/881559699 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2925069032 0 0) 0x7fa0e80d33f0 con 0x7fa0e80d4bb0 2024-04-15T15:14:55.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.211+0000 7fa0dffff640 1 -- 172.21.15.72:0/881559699 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0d4003520 con 0x7fa0e80d4bb0 2024-04-15T15:14:55.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.211+0000 7fa0dffff640 1 -- 172.21.15.72:0/881559699 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4183357191 0 0) 0x7fa0d4003520 con 0x7fa0e80d4bb0 2024-04-15T15:14:55.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.211+0000 7fa0dffff640 1 -- 172.21.15.72:0/881559699 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa0e80d47d0 msgr2=0x7fa0e80d2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:55.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.211+0000 7fa0dffff640 1 --2- 172.21.15.72:0/881559699 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa0e80d47d0 0x7fa0e80d2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:55.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.211+0000 7fa0dffff640 1 -- 172.21.15.72:0/881559699 >> v2:172.21.15.175:3300/0 conn(0x7fa0e806f110 msgr2=0x7fa0e8161b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:55.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.211+0000 7fa0dffff640 1 --2- 172.21.15.72:0/881559699 >> v2:172.21.15.175:3300/0 conn(0x7fa0e806f110 0x7fa0e8161b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:55.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.211+0000 7fa0dffff640 1 -- 172.21.15.72:0/881559699 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0e80d36d0 con 0x7fa0e80d4bb0 2024-04-15T15:14:55.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.211+0000 7fa0dffff640 1 -- 172.21.15.72:0/881559699 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7fa0e00019f0 con 0x7fa0e80d4bb0 2024-04-15T15:14:55.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.211+0000 7fa0ed5c4640 1 --2- 172.21.15.72:0/881559699 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa0e80d47d0 0x7fa0e80d2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:14:55.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.211+0000 7fa0dffff640 1 -- 172.21.15.72:0/881559699 >> v1:172.21.15.72:6789/0 conn(0x7fa0e80d4bb0 legacy=0x7fa0e80b6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:55.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.211+0000 7fa0dffff640 1 --2- 172.21.15.72:0/881559699 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa0e80d47d0 0x7fa0d40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:55.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.211+0000 7fa0dffff640 1 --2- 172.21.15.72:0/881559699 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0d4003ef0 0x7fa0d4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:55.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.211+0000 7fa0dffff640 1 --2- 172.21.15.72:0/881559699 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa0d4004d40 0x7fa0d40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:55.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.211+0000 7fa0dffff640 1 -- 172.21.15.72:0/881559699 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e80d28c0 con 0x7fa0d4003ef0 2024-04-15T15:14:55.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.211+0000 7fa0dffff640 1 -- 172.21.15.72:0/881559699 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e80d31a0 con 0x7fa0d4004d40 2024-04-15T15:14:55.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.211+0000 7fa0dffff640 1 -- 172.21.15.72:0/881559699 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fa0d400b9c0 con 0x7fa0e80d47d0 2024-04-15T15:14:55.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.211+0000 7fa0ecdc3640 1 --2- 172.21.15.72:0/881559699 >> v2:172.21.15.175:3300/0 conn(0x7fa0e806f110 0x7fa0e8161b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:14:55.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.212+0000 7fa0ed5c4640 1 --2- 172.21.15.72:0/881559699 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa0e80d47d0 0x7fa0d40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:55.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.212+0000 7fa0ecdc3640 1 --2- 172.21.15.72:0/881559699 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0d4003ef0 0x7fa0d4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:55.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.212+0000 7fa0eddc5640 1 --2- 172.21.15.72:0/881559699 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa0d4004d40 0x7fa0d40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:55.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.212+0000 7fa0ed5c4640 1 -- 172.21.15.72:0/881559699 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa0d4004d40 msgr2=0x7fa0d40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:55.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.212+0000 7fa0ed5c4640 1 --2- 172.21.15.72:0/881559699 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa0d4004d40 0x7fa0d40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:55.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.212+0000 7fa0ed5c4640 1 -- 172.21.15.72:0/881559699 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0d4003ef0 msgr2=0x7fa0d4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:55.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.212+0000 7fa0ed5c4640 1 --2- 172.21.15.72:0/881559699 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0d4003ef0 0x7fa0d4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:55.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.212+0000 7fa0ed5c4640 1 -- 172.21.15.72:0/881559699 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0d4003520 con 0x7fa0e80d47d0 2024-04-15T15:14:55.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.212+0000 7fa0ecdc3640 1 --2- 172.21.15.72:0/881559699 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0d4003ef0 0x7fa0d4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:14:55.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.213+0000 7fa0ed5c4640 1 --2- 172.21.15.72:0/881559699 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa0e80d47d0 0x7fa0d40039b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa0e4009c90 tx=0x7fa0d4005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf03e4ca35369329 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:55.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.213+0000 7fa0dffff640 1 -- 172.21.15.72:0/881559699 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e4039070 con 0x7fa0e80d47d0 2024-04-15T15:14:55.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.213+0000 7fa0dffff640 1 -- 172.21.15.72:0/881559699 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0e402c980 con 0x7fa0e80d47d0 2024-04-15T15:14:55.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.213+0000 7fa0dffff640 1 -- 172.21.15.72:0/881559699 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e4009860 con 0x7fa0e80d47d0 2024-04-15T15:14:55.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.213+0000 7fa0ef828640 1 -- 172.21.15.72:0/881559699 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa0e80d47d0 msgr2=0x7fa0d40039b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:55.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.213+0000 7fa0ef828640 1 --2- 172.21.15.72:0/881559699 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa0e80d47d0 0x7fa0d40039b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa0e4009c90 tx=0x7fa0d4005830 comp rx=0 tx=0).stop 2024-04-15T15:14:55.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.213+0000 7fa0eddc5640 1 -- 172.21.15.72:0/881559699 reap_dead start 2024-04-15T15:14:55.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.213+0000 7fa0ef828640 1 -- 172.21.15.72:0/881559699 shutdown_connections 2024-04-15T15:14:55.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.213+0000 7fa0ef828640 1 -- 172.21.15.72:0/881559699 >> 172.21.15.72:0/881559699 conn(0x7fa0e809d950 msgr2=0x7fa0e80b7520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:55.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.213+0000 7fa0ef828640 1 -- 172.21.15.72:0/881559699 shutdown_connections 2024-04-15T15:14:55.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.213+0000 7fa0ef828640 1 -- 172.21.15.72:0/881559699 wait complete. 2024-04-15T15:14:55.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.215+0000 7fa0ef828640 1 Processor -- start 2024-04-15T15:14:55.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.215+0000 7fa0ef828640 1 -- start start 2024-04-15T15:14:55.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.215+0000 7fa0ef828640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa0e80d47d0 0x7fa0e80cc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:55.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.215+0000 7fa0ef828640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa0e80cd450 0x7fa0e806e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:55.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.215+0000 7fa0ef828640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0e80cd8d0 0x7fa0e814d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:55.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.215+0000 7fa0ef828640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e80b1240 con 0x7fa0e80cd8d0 2024-04-15T15:14:55.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.215+0000 7fa0ef828640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fa0e80d3c90 con 0x7fa0e80cd450 2024-04-15T15:14:55.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.215+0000 7fa0ef828640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fa0e80b13c0 con 0x7fa0e80d47d0 2024-04-15T15:14:55.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.215+0000 7fa0eddc5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0e80cd8d0 0x7fa0e814d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:55.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.215+0000 7fa0eddc5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0e80cd8d0 0x7fa0e814d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45076/0 (socket says 172.21.15.72:45076) 2024-04-15T15:14:55.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.215+0000 7fa0eddc5640 1 -- 172.21.15.72:0/1854179434 learned_addr learned my addr 172.21.15.72:0/1854179434 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:55.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.216+0000 7fa0ecdc3640 1 --2- 172.21.15.72:0/1854179434 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa0e80cd450 0x7fa0e806e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:55.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.216+0000 7fa0ed5c4640 1 --2- 172.21.15.72:0/1854179434 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa0e80d47d0 0x7fa0e80cc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:55.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.216+0000 7fa0eddc5640 1 -- 172.21.15.72:0/1854179434 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa0e80d47d0 msgr2=0x7fa0e80cc6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:55.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.216+0000 7fa0eddc5640 1 --2- 172.21.15.72:0/1854179434 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa0e80d47d0 0x7fa0e80cc6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:55.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.216+0000 7fa0eddc5640 1 -- 172.21.15.72:0/1854179434 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa0e80cd450 msgr2=0x7fa0e806e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:55.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.216+0000 7fa0eddc5640 1 --2- 172.21.15.72:0/1854179434 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa0e80cd450 0x7fa0e806e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:55.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.216+0000 7fa0eddc5640 1 -- 172.21.15.72:0/1854179434 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0e80cb8f0 con 0x7fa0e80cd8d0 2024-04-15T15:14:55.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.216+0000 7fa0eddc5640 1 --2- 172.21.15.72:0/1854179434 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0e80cd8d0 0x7fa0e814d770 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa0e80d4f90 tx=0x7fa0e0007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:55.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.216+0000 7fa0ddffb640 1 -- 172.21.15.72:0/1854179434 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0d8009840 con 0x7fa0e80cd8d0 2024-04-15T15:14:55.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.216+0000 7fa0ddffb640 1 -- 172.21.15.72:0/1854179434 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0e0018070 con 0x7fa0e80cd8d0 2024-04-15T15:14:55.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.216+0000 7fa0ef828640 1 -- 172.21.15.72:0/1854179434 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0e80cbbd0 con 0x7fa0e80cd8d0 2024-04-15T15:14:55.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.216+0000 7fa0ddffb640 1 -- 172.21.15.72:0/1854179434 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e000ada0 con 0x7fa0e80cd8d0 2024-04-15T15:14:55.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.216+0000 7fa0ef828640 1 -- 172.21.15.72:0/1854179434 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0e80cc190 con 0x7fa0e80cd8d0 2024-04-15T15:14:55.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.217+0000 7fa0ef828640 1 -- 172.21.15.72:0/1854179434 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0e80cc390 con 0x7fa0e80cd8d0 2024-04-15T15:14:55.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.217+0000 7fa0ddffb640 1 -- 172.21.15.72:0/1854179434 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa0e001d460 con 0x7fa0e80cd8d0 2024-04-15T15:14:55.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.218+0000 7fa0ddffb640 1 --2- 172.21.15.72:0/1854179434 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa0cc06c8f0 0x7fa0cc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:55.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.218+0000 7fa0ddffb640 1 -- 172.21.15.72:0/1854179434 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 6500+0+0 (secure 0 0 0) 0x7fa0e008c280 con 0x7fa0e80cd8d0 2024-04-15T15:14:55.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.218+0000 7fa0ed5c4640 1 --2- 172.21.15.72:0/1854179434 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa0cc06c8f0 0x7fa0cc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:55.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.218+0000 7fa0ed5c4640 1 --2- 172.21.15.72:0/1854179434 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa0cc06c8f0 0x7fa0cc06edb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa0e4036000 tx=0x7fa0e4030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:55.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.223+0000 7fa0ddffb640 1 -- 172.21.15.72:0/1854179434 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa0e80cc390 con 0x7fa0e80cd8d0 2024-04-15T15:14:55.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:55.397+0000 7fa0ef828640 1 -- 172.21.15.72:0/1854179434 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fa0e80689e0 con 0x7fa0e80cd8d0 2024-04-15T15:14:55.991 INFO:tasks.ceph.mds.c.smithi072.stderr:2024-04-15T15:14:55.989+0000 7f7075518640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-15T15:14:56.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.078+0000 7fa0ddffb640 1 -- 172.21.15.72:0/1854179434 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v23) ==== 169+0+0 (secure 0 0 0) 0x7fa0e005a420 con 0x7fa0e80cd8d0 2024-04-15T15:14:56.080 INFO:teuthology.orchestra.run.smithi072.stderr:set pool 4 allow_ec_overwrites to true 2024-04-15T15:14:56.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.083+0000 7fa0ef828640 1 -- 172.21.15.72:0/1854179434 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa0cc06c8f0 msgr2=0x7fa0cc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:56.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.083+0000 7fa0ef828640 1 --2- 172.21.15.72:0/1854179434 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa0cc06c8f0 0x7fa0cc06edb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa0e4036000 tx=0x7fa0e4030000 comp rx=0 tx=0).stop 2024-04-15T15:14:56.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.084+0000 7fa0ef828640 1 -- 172.21.15.72:0/1854179434 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0e80cd8d0 msgr2=0x7fa0e814d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:56.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.084+0000 7fa0ef828640 1 --2- 172.21.15.72:0/1854179434 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0e80cd8d0 0x7fa0e814d770 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa0e80d4f90 tx=0x7fa0e0007a90 comp rx=0 tx=0).stop 2024-04-15T15:14:56.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.084+0000 7fa0ef828640 1 -- 172.21.15.72:0/1854179434 shutdown_connections 2024-04-15T15:14:56.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.084+0000 7fa0ef828640 1 --2- 172.21.15.72:0/1854179434 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa0cc06c8f0 0x7fa0cc06edb0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:56.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.084+0000 7fa0ef828640 1 --2- 172.21.15.72:0/1854179434 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa0e80cd8d0 0x7fa0e814d770 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:56.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.084+0000 7fa0ef828640 1 --2- 172.21.15.72:0/1854179434 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa0e80cd450 0x7fa0e806e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:56.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.084+0000 7fa0ef828640 1 --2- 172.21.15.72:0/1854179434 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa0e80d47d0 0x7fa0e80cc6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:56.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.084+0000 7fa0ef828640 1 -- 172.21.15.72:0/1854179434 >> 172.21.15.72:0/1854179434 conn(0x7fa0e809d950 msgr2=0x7fa0e8156990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:56.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.085+0000 7fa0ef828640 1 -- 172.21.15.72:0/1854179434 shutdown_connections 2024-04-15T15:14:56.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.087+0000 7fa0ef828640 1 -- 172.21.15.72:0/1854179434 wait complete. 2024-04-15T15:14:56.115 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-15T15:14:56.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.219+0000 7fd1200b2640 1 Processor -- start 2024-04-15T15:14:56.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.219+0000 7fd1200b2640 1 -- start start 2024-04-15T15:14:56.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.220+0000 7fd1200b2640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd118155790 0x7fd1180b5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:56.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.220+0000 7fd1200b2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd1181622f0 0x7fd118162760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:56.222 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.220+0000 7fd1200b2640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd118155b70 con 0x7fd11815ed00 2024-04-15T15:14:56.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.220+0000 7fd1200b2640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fd118162ca0 con 0x7fd1181622f0 2024-04-15T15:14:56.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.220+0000 7fd1200b2640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fd11806ecb0 con 0x7fd118155790 2024-04-15T15:14:56.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.220+0000 7fd11d64d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd1181622f0 0x7fd118162760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:56.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.220+0000 7fd11d64d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd1181622f0 0x7fd118162760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45086/0 (socket says 172.21.15.72:45086) 2024-04-15T15:14:56.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.220+0000 7fd11e64f640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fd11815ed00 0x7fd11815f0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37196/0 (socket says 172.21.15.72:37196) 2024-04-15T15:14:56.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.220+0000 7fd11d64d640 1 -- 172.21.15.72:0/3302591057 learned_addr learned my addr 172.21.15.72:0/3302591057 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:56.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.221+0000 7fd11de4e640 1 --2- 172.21.15.72:0/3302591057 >> v2:172.21.15.175:3300/0 conn(0x7fd118155790 0x7fd1180b5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:56.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.221+0000 7fd11d64d640 1 -- 172.21.15.72:0/3302591057 >> v2:172.21.15.175:3300/0 conn(0x7fd118155790 msgr2=0x7fd1180b5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:56.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.221+0000 7fd11d64d640 1 --2- 172.21.15.72:0/3302591057 >> v2:172.21.15.175:3300/0 conn(0x7fd118155790 0x7fd1180b5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:56.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.221+0000 7fd11d64d640 1 -- 172.21.15.72:0/3302591057 >> v1:172.21.15.72:6789/0 conn(0x7fd11815ed00 legacy=0x7fd11815f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:56.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.221+0000 7fd11ce4c640 1 -- 172.21.15.72:0/3302591057 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 368780249 0 0) 0x7fd118155b70 con 0x7fd11815ed00 2024-04-15T15:14:56.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.221+0000 7fd11d64d640 1 -- 172.21.15.72:0/3302591057 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd11806ef00 con 0x7fd1181622f0 2024-04-15T15:14:56.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.221+0000 7fd11d64d640 1 --2- 172.21.15.72:0/3302591057 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1181622f0 0x7fd118162760 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd11806c950 tx=0x7fd10802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82c2f2760e6ec9c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:56.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.222+0000 7fd11ce4c640 1 -- 172.21.15.72:0/3302591057 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd108034070 con 0x7fd1181622f0 2024-04-15T15:14:56.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.222+0000 7fd11ce4c640 1 -- 172.21.15.72:0/3302591057 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd108030860 con 0x7fd1181622f0 2024-04-15T15:14:56.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.222+0000 7fd11ce4c640 1 -- 172.21.15.72:0/3302591057 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd108030c30 con 0x7fd1181622f0 2024-04-15T15:14:56.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.222+0000 7fd1200b2640 1 -- 172.21.15.72:0/3302591057 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1181622f0 msgr2=0x7fd118162760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:56.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.222+0000 7fd1200b2640 1 --2- 172.21.15.72:0/3302591057 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1181622f0 0x7fd118162760 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd11806c950 tx=0x7fd10802b9a0 comp rx=0 tx=0).stop 2024-04-15T15:14:56.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.222+0000 7fd1200b2640 1 -- 172.21.15.72:0/3302591057 shutdown_connections 2024-04-15T15:14:56.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.222+0000 7fd1200b2640 1 --2- 172.21.15.72:0/3302591057 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd1181622f0 0x7fd118162760 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:56.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.222+0000 7fd1200b2640 1 --2- 172.21.15.72:0/3302591057 >> v2:172.21.15.175:3300/0 conn(0x7fd118155790 0x7fd1180b5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:56.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.222+0000 7fd1200b2640 1 -- 172.21.15.72:0/3302591057 >> 172.21.15.72:0/3302591057 conn(0x7fd1180d24d0 msgr2=0x7fd1180d28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:56.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.222+0000 7fd1200b2640 1 -- 172.21.15.72:0/3302591057 shutdown_connections 2024-04-15T15:14:56.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.222+0000 7fd1200b2640 1 -- 172.21.15.72:0/3302591057 wait complete. 2024-04-15T15:14:56.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.223+0000 7fd1200b2640 1 Processor -- start 2024-04-15T15:14:56.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.223+0000 7fd1200b2640 1 -- start start 2024-04-15T15:14:56.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.224+0000 7fd1200b2640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd118155790 0x7fd1181f3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:56.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.224+0000 7fd1200b2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd11815ed00 0x7fd11814d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:56.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.224+0000 7fd1200b2640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd1181622f0 0x7fd11814dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:56.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.224+0000 7fd1200b2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd1180b5380 con 0x7fd11815ed00 2024-04-15T15:14:56.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.224+0000 7fd1200b2640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fd1180b59d0 con 0x7fd1181622f0 2024-04-15T15:14:56.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.224+0000 7fd1200b2640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fd1180b5660 con 0x7fd118155790 2024-04-15T15:14:56.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.224+0000 7fd11d64d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd11815ed00 0x7fd11814d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:56.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.224+0000 7fd11d64d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd11815ed00 0x7fd11814d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45090/0 (socket says 172.21.15.72:45090) 2024-04-15T15:14:56.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.224+0000 7fd11d64d640 1 -- 172.21.15.72:0/2987711984 learned_addr learned my addr 172.21.15.72:0/2987711984 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:56.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.224+0000 7fd11d64d640 1 -- 172.21.15.72:0/2987711984 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd118155790 msgr2=0x7fd1181f3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:56.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.224+0000 7fd11d64d640 1 --2- 172.21.15.72:0/2987711984 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd118155790 0x7fd1181f3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:56.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.224+0000 7fd11e64f640 1 --2- 172.21.15.72:0/2987711984 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd1181622f0 0x7fd11814dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:56.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.225+0000 7fd11d64d640 1 -- 172.21.15.72:0/2987711984 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd1181622f0 msgr2=0x7fd11814dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:56.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.225+0000 7fd11d64d640 1 --2- 172.21.15.72:0/2987711984 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd1181622f0 0x7fd11814dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:56.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.225+0000 7fd11d64d640 1 -- 172.21.15.72:0/2987711984 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd11814faa0 con 0x7fd11815ed00 2024-04-15T15:14:56.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.225+0000 7fd11d64d640 1 --2- 172.21.15.72:0/2987711984 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd11815ed00 0x7fd11814d580 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd10802b9e0 tx=0x7fd1080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:56.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.225+0000 7fd10e7fc640 1 -- 172.21.15.72:0/2987711984 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd108034040 con 0x7fd11815ed00 2024-04-15T15:14:56.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.225+0000 7fd10e7fc640 1 -- 172.21.15.72:0/2987711984 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1080433f0 con 0x7fd11815ed00 2024-04-15T15:14:56.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.225+0000 7fd1200b2640 1 -- 172.21.15.72:0/2987711984 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1180cd330 con 0x7fd11815ed00 2024-04-15T15:14:56.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.225+0000 7fd10e7fc640 1 -- 172.21.15.72:0/2987711984 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1080437e0 con 0x7fd11815ed00 2024-04-15T15:14:56.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.225+0000 7fd1200b2640 1 -- 172.21.15.72:0/2987711984 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1180cd8f0 con 0x7fd11815ed00 2024-04-15T15:14:56.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.226+0000 7fd1200b2640 1 -- 172.21.15.72:0/2987711984 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd118075c50 con 0x7fd11815ed00 2024-04-15T15:14:56.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.226+0000 7fd10e7fc640 1 -- 172.21.15.72:0/2987711984 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd10804a060 con 0x7fd11815ed00 2024-04-15T15:14:56.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.227+0000 7fd10e7fc640 1 --2- 172.21.15.72:0/2987711984 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd0fc06c920 0x7fd0fc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:56.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.227+0000 7fd10e7fc640 1 -- 172.21.15.72:0/2987711984 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 6500+0+0 (secure 0 0 0) 0x7fd1080ae150 con 0x7fd11815ed00 2024-04-15T15:14:56.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.227+0000 7fd11de4e640 1 --2- 172.21.15.72:0/2987711984 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd0fc06c920 0x7fd0fc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:56.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.227+0000 7fd11de4e640 1 --2- 172.21.15.72:0/2987711984 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd0fc06c920 0x7fd0fc06ede0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd114007e60 tx=0x7fd114007390 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:56.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.231+0000 7fd10e7fc640 1 -- 172.21.15.72:0/2987711984 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd11814faa0 con 0x7fd11815ed00 2024-04-15T15:14:56.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:56.450+0000 7fd1200b2640 1 -- 172.21.15.72:0/2987711984 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7fd1181611b0 con 0x7fd11815ed00 2024-04-15T15:14:57.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.110+0000 7fd10e7fc640 1 -- 172.21.15.72:0/2987711984 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v7) ==== 137+0+0 (secure 0 0 0) 0x7fd10803e070 con 0x7fd11815ed00 2024-04-15T15:14:57.112 INFO:teuthology.orchestra.run.smithi072.stderr:added data pool 4 to fsmap 2024-04-15T15:14:57.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.112+0000 7fd1200b2640 1 -- 172.21.15.72:0/2987711984 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd0fc06c920 msgr2=0x7fd0fc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:57.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.112+0000 7fd1200b2640 1 --2- 172.21.15.72:0/2987711984 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd0fc06c920 0x7fd0fc06ede0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd114007e60 tx=0x7fd114007390 comp rx=0 tx=0).stop 2024-04-15T15:14:57.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.112+0000 7fd1200b2640 1 -- 172.21.15.72:0/2987711984 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd11815ed00 msgr2=0x7fd11814d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:57.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.112+0000 7fd1200b2640 1 --2- 172.21.15.72:0/2987711984 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd11815ed00 0x7fd11814d580 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd10802b9e0 tx=0x7fd1080046c0 comp rx=0 tx=0).stop 2024-04-15T15:14:57.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.112+0000 7fd1200b2640 1 -- 172.21.15.72:0/2987711984 shutdown_connections 2024-04-15T15:14:57.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.112+0000 7fd1200b2640 1 --2- 172.21.15.72:0/2987711984 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd0fc06c920 0x7fd0fc06ede0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:57.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.112+0000 7fd1200b2640 1 --2- 172.21.15.72:0/2987711984 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd1181622f0 0x7fd11814dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:57.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.112+0000 7fd1200b2640 1 --2- 172.21.15.72:0/2987711984 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd11815ed00 0x7fd11814d580 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:57.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.112+0000 7fd1200b2640 1 --2- 172.21.15.72:0/2987711984 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd118155790 0x7fd1181f3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:57.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.112+0000 7fd1200b2640 1 -- 172.21.15.72:0/2987711984 >> 172.21.15.72:0/2987711984 conn(0x7fd1180d24d0 msgr2=0x7fd118070e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:57.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.112+0000 7fd1200b2640 1 -- 172.21.15.72:0/2987711984 shutdown_connections 2024-04-15T15:14:57.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.112+0000 7fd1200b2640 1 -- 172.21.15.72:0/2987711984 wait complete. 2024-04-15T15:14:57.127 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:14:57.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.232+0000 7fa3e72c8640 1 Processor -- start 2024-04-15T15:14:57.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.233+0000 7fa3e72c8640 1 -- start start 2024-04-15T15:14:57.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.233+0000 7fa3e72c8640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa3e015c120 0x7fa3e015f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:57.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.233+0000 7fa3e72c8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa3e015c500 0x7fa3e00cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:57.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.233+0000 7fa3e72c8640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3e01624e0 con 0x7fa3e0070760 2024-04-15T15:14:57.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.233+0000 7fa3e72c8640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fa3e01627c0 con 0x7fa3e015c500 2024-04-15T15:14:57.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.233+0000 7fa3e72c8640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fa3e0162aa0 con 0x7fa3e015c120 2024-04-15T15:14:57.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.234+0000 7fa3e4863640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa3e015c500 0x7fa3e00cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:57.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.234+0000 7fa3e4863640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa3e015c500 0x7fa3e00cb9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45094/0 (socket says 172.21.15.72:45094) 2024-04-15T15:14:57.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.234+0000 7fa3e4863640 1 -- 172.21.15.72:0/2168206291 learned_addr learned my addr 172.21.15.72:0/2168206291 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:57.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.234+0000 7fa3e5064640 1 --2- 172.21.15.72:0/2168206291 >> v2:172.21.15.175:3300/0 conn(0x7fa3e015c120 0x7fa3e015f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:57.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.234+0000 7fa3e4863640 1 -- 172.21.15.72:0/2168206291 >> v2:172.21.15.175:3300/0 conn(0x7fa3e015c120 msgr2=0x7fa3e015f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:57.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.234+0000 7fa3e4863640 1 --2- 172.21.15.72:0/2168206291 >> v2:172.21.15.175:3300/0 conn(0x7fa3e015c120 0x7fa3e015f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:57.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.234+0000 7fa3e4863640 1 -- 172.21.15.72:0/2168206291 >> v1:172.21.15.175:6789/0 conn(0x7fa3e0070760 legacy=0x7fa3e00c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:57.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.234+0000 7fa3e4863640 1 -- 172.21.15.72:0/2168206291 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3e0162d80 con 0x7fa3e015c500 2024-04-15T15:14:57.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.235+0000 7fa3e4863640 1 --2- 172.21.15.72:0/2168206291 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3e015c500 0x7fa3e00cb9a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa3e00ce3c0 tx=0x7fa3d002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86d3d888e10cc6cc server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:57.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.235+0000 7fa3d7fff640 1 -- 172.21.15.72:0/2168206291 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3d0035020 con 0x7fa3e015c500 2024-04-15T15:14:57.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.235+0000 7fa3d7fff640 1 -- 172.21.15.72:0/2168206291 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3d0031870 con 0x7fa3e015c500 2024-04-15T15:14:57.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.235+0000 7fa3d7fff640 1 -- 172.21.15.72:0/2168206291 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3d0031c40 con 0x7fa3e015c500 2024-04-15T15:14:57.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.235+0000 7fa3e72c8640 1 -- 172.21.15.72:0/2168206291 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3e015c500 msgr2=0x7fa3e00cb9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:57.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.235+0000 7fa3e72c8640 1 --2- 172.21.15.72:0/2168206291 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3e015c500 0x7fa3e00cb9a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa3e00ce3c0 tx=0x7fa3d002b910 comp rx=0 tx=0).stop 2024-04-15T15:14:57.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.235+0000 7fa3e72c8640 1 -- 172.21.15.72:0/2168206291 shutdown_connections 2024-04-15T15:14:57.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.235+0000 7fa3e72c8640 1 --2- 172.21.15.72:0/2168206291 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3e015c500 0x7fa3e00cb9a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:57.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.235+0000 7fa3e72c8640 1 --2- 172.21.15.72:0/2168206291 >> v2:172.21.15.175:3300/0 conn(0x7fa3e015c120 0x7fa3e015f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:57.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.235+0000 7fa3e72c8640 1 -- 172.21.15.72:0/2168206291 >> 172.21.15.72:0/2168206291 conn(0x7fa3e00cd4a0 msgr2=0x7fa3e00cd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:57.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.235+0000 7fa3e72c8640 1 -- 172.21.15.72:0/2168206291 shutdown_connections 2024-04-15T15:14:57.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.235+0000 7fa3e72c8640 1 -- 172.21.15.72:0/2168206291 wait complete. 2024-04-15T15:14:57.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.237+0000 7fa3e72c8640 1 Processor -- start 2024-04-15T15:14:57.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.237+0000 7fa3e72c8640 1 -- start start 2024-04-15T15:14:57.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.237+0000 7fa3e72c8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3e0070760 0x7fa3e00c7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:57.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.237+0000 7fa3e72c8640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa3e015c120 0x7fa3e014d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:57.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.237+0000 7fa3e72c8640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa3e015c500 0x7fa3e00ca370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:57.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.237+0000 7fa3e72c8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fa3e00cc060 con 0x7fa3e0070760 2024-04-15T15:14:57.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.237+0000 7fa3e72c8640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fa3e00cbee0 con 0x7fa3e015c120 2024-04-15T15:14:57.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.237+0000 7fa3e72c8640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fa3e00cc1e0 con 0x7fa3e015c500 2024-04-15T15:14:57.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.237+0000 7fa3e5064640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3e0070760 0x7fa3e00c7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:57.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.237+0000 7fa3e5064640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3e0070760 0x7fa3e00c7af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45104/0 (socket says 172.21.15.72:45104) 2024-04-15T15:14:57.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.237+0000 7fa3e5064640 1 -- 172.21.15.72:0/1256583595 learned_addr learned my addr 172.21.15.72:0/1256583595 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:57.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.237+0000 7fa3e4863640 1 --2- 172.21.15.72:0/1256583595 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa3e015c120 0x7fa3e014d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:57.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.237+0000 7fa3e5865640 1 --2- 172.21.15.72:0/1256583595 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa3e015c500 0x7fa3e00ca370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:57.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.238+0000 7fa3e5064640 1 -- 172.21.15.72:0/1256583595 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa3e015c500 msgr2=0x7fa3e00ca370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:57.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.238+0000 7fa3e5064640 1 --2- 172.21.15.72:0/1256583595 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa3e015c500 0x7fa3e00ca370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:57.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.238+0000 7fa3e5064640 1 -- 172.21.15.72:0/1256583595 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa3e015c120 msgr2=0x7fa3e014d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:57.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.238+0000 7fa3e5064640 1 --2- 172.21.15.72:0/1256583595 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa3e015c120 0x7fa3e014d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:57.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.238+0000 7fa3e5064640 1 -- 172.21.15.72:0/1256583595 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3e014dac0 con 0x7fa3e0070760 2024-04-15T15:14:57.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.238+0000 7fa3e5064640 1 --2- 172.21.15.72:0/1256583595 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3e0070760 0x7fa3e00c7af0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc007f20 tx=0x7fa3dc00e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:57.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.238+0000 7fa3d5ffb640 1 -- 172.21.15.72:0/1256583595 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc014620 con 0x7fa3e0070760 2024-04-15T15:14:57.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.238+0000 7fa3d5ffb640 1 -- 172.21.15.72:0/1256583595 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3dc019070 con 0x7fa3e0070760 2024-04-15T15:14:57.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.238+0000 7fa3e72c8640 1 -- 172.21.15.72:0/1256583595 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3e014dff0 con 0x7fa3e0070760 2024-04-15T15:14:57.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.238+0000 7fa3d5ffb640 1 -- 172.21.15.72:0/1256583595 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3dc014a00 con 0x7fa3e0070760 2024-04-15T15:14:57.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.238+0000 7fa3e72c8640 1 -- 172.21.15.72:0/1256583595 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3e015faa0 con 0x7fa3e0070760 2024-04-15T15:14:57.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.239+0000 7fa3e72c8640 1 -- 172.21.15.72:0/1256583595 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3e00689e0 con 0x7fa3e0070760 2024-04-15T15:14:57.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.239+0000 7fa3d5ffb640 1 -- 172.21.15.72:0/1256583595 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa3dc014ba0 con 0x7fa3e0070760 2024-04-15T15:14:57.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.240+0000 7fa3d5ffb640 1 --2- 172.21.15.72:0/1256583595 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa3c406c8f0 0x7fa3c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:57.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.240+0000 7fa3e4863640 1 --2- 172.21.15.72:0/1256583595 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa3c406c8f0 0x7fa3c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:57.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.240+0000 7fa3d5ffb640 1 -- 172.21.15.72:0/1256583595 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa3dc08c610 con 0x7fa3e0070760 2024-04-15T15:14:57.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.240+0000 7fa3e4863640 1 --2- 172.21.15.72:0/1256583595 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa3c406c8f0 0x7fa3c406edb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0036000 tx=0x7fa3d003c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:57.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.244+0000 7fa3d5ffb640 1 -- 172.21.15.72:0/1256583595 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa3dc05a140 con 0x7fa3e0070760 2024-04-15T15:14:57.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.457+0000 7fa3e72c8640 1 -- 172.21.15.72:0/1256583595 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa3e00cade0 con 0x7fa3e0070760 2024-04-15T15:14:57.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.457+0000 7fa3d5ffb640 1 -- 172.21.15.72:0/1256583595 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5183 (secure 0 0 0) 0x7fa3dc05e440 con 0x7fa3e0070760 2024-04-15T15:14:57.459 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:14:57.459 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4267,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6833/1659343919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1659343919},{"type":"v1","addr":"172.21.15.175:6833","nonce":1659343919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6835/4113588141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6834","nonce":4113588141},{"type":"v1","addr":"172.21.15.175:6835","nonce":4113588141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4296,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/4151548261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4151548261},{"type":"v1","addr":"172.21.15.72:6836","nonce":4151548261}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:14:51.411091+0000","modified":"2024-04-15T15:14:56.451317+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/3458067804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3458067804},{"type":"v1","addr":"172.21.15.72:6837","nonce":3458067804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4299,"qdb_cluster":[4299]},"id":1}]} 2024-04-15T15:14:57.460 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 7 2024-04-15T15:14:57.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.459+0000 7fa3e72c8640 1 -- 172.21.15.72:0/1256583595 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa3c406c8f0 msgr2=0x7fa3c406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:57.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.459+0000 7fa3e72c8640 1 --2- 172.21.15.72:0/1256583595 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa3c406c8f0 0x7fa3c406edb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0036000 tx=0x7fa3d003c040 comp rx=0 tx=0).stop 2024-04-15T15:14:57.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.459+0000 7fa3e72c8640 1 -- 172.21.15.72:0/1256583595 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3e0070760 msgr2=0x7fa3e00c7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:57.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.459+0000 7fa3e72c8640 1 --2- 172.21.15.72:0/1256583595 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3e0070760 0x7fa3e00c7af0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc007f20 tx=0x7fa3dc00e570 comp rx=0 tx=0).stop 2024-04-15T15:14:57.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.459+0000 7fa3e72c8640 1 -- 172.21.15.72:0/1256583595 shutdown_connections 2024-04-15T15:14:57.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.459+0000 7fa3e72c8640 1 --2- 172.21.15.72:0/1256583595 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa3c406c8f0 0x7fa3c406edb0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:57.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.459+0000 7fa3e72c8640 1 --2- 172.21.15.72:0/1256583595 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa3e015c500 0x7fa3e00ca370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:57.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.459+0000 7fa3e72c8640 1 --2- 172.21.15.72:0/1256583595 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa3e015c120 0x7fa3e014d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:57.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.459+0000 7fa3e72c8640 1 --2- 172.21.15.72:0/1256583595 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3e0070760 0x7fa3e00c7af0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:57.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.459+0000 7fa3e72c8640 1 -- 172.21.15.72:0/1256583595 >> 172.21.15.72:0/1256583595 conn(0x7fa3e00cd4a0 msgr2=0x7fa3e00b1070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:57.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.459+0000 7fa3e72c8640 1 -- 172.21.15.72:0/1256583595 shutdown_connections 2024-04-15T15:14:57.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.459+0000 7fa3e72c8640 1 -- 172.21.15.72:0/1256583595 wait complete. 2024-04-15T15:14:57.472 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:14:57.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.573+0000 7f3e59ec8640 1 Processor -- start 2024-04-15T15:14:57.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.573+0000 7f3e59ec8640 1 -- start start 2024-04-15T15:14:57.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.574+0000 7f3e59ec8640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f3e5415eba0 0x7f3e5406f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:57.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.574+0000 7f3e59ec8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3e540ccb90 0x7f3e540ccfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:57.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.574+0000 7f3e59ec8640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e540afb10 con 0x7f3e5406f720 2024-04-15T15:14:57.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.574+0000 7f3e59ec8640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f3e540cd520 con 0x7f3e540ccb90 2024-04-15T15:14:57.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.574+0000 7f3e59ec8640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f3e5415ef80 con 0x7f3e5415eba0 2024-04-15T15:14:57.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.574+0000 7f3e53fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3e540ccb90 0x7f3e540ccfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:57.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.574+0000 7f3e53fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3e540ccb90 0x7f3e540ccfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45114/0 (socket says 172.21.15.72:45114) 2024-04-15T15:14:57.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.574+0000 7f3e53fff640 1 -- 172.21.15.72:0/1415531944 learned_addr learned my addr 172.21.15.72:0/1415531944 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:57.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.574+0000 7f3e52ffd640 1 --1- 172.21.15.72:0/1415531944 >> v1:172.21.15.72:6789/0 conn(0x7f3e5406f720 0x7f3e540af400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37210/0 (socket says 172.21.15.72:37210) 2024-04-15T15:14:57.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.574+0000 7f3e53fff640 1 -- 172.21.15.72:0/1415531944 >> v2:172.21.15.175:3300/0 conn(0x7f3e5415eba0 msgr2=0x7f3e5406f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:57.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.574+0000 7f3e53fff640 1 --2- 172.21.15.72:0/1415531944 >> v2:172.21.15.175:3300/0 conn(0x7f3e5415eba0 0x7f3e5406f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:57.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.574+0000 7f3e53fff640 1 -- 172.21.15.72:0/1415531944 >> v1:172.21.15.72:6789/0 conn(0x7f3e5406f720 legacy=0x7f3e540af400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:57.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.574+0000 7f3e53fff640 1 -- 172.21.15.72:0/1415531944 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e540ce2b0 con 0x7f3e540ccb90 2024-04-15T15:14:57.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.574+0000 7f3e527fc640 1 -- 172.21.15.72:0/1415531944 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4020362554 0 0) 0x7f3e540afb10 con 0x7f3e5406f720 2024-04-15T15:14:57.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.575+0000 7f3e53fff640 1 --2- 172.21.15.72:0/1415531944 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e540ccb90 0x7f3e540ccfe0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3e540b2e90 tx=0x7f3e4c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14086f489b629aa8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:57.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.575+0000 7f3e527fc640 1 -- 172.21.15.72:0/1415531944 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c02ca80 con 0x7f3e540ccb90 2024-04-15T15:14:57.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.575+0000 7f3e527fc640 1 -- 172.21.15.72:0/1415531944 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e4c02cc20 con 0x7f3e540ccb90 2024-04-15T15:14:57.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.575+0000 7f3e527fc640 1 -- 172.21.15.72:0/1415531944 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c0343f0 con 0x7f3e540ccb90 2024-04-15T15:14:57.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.575+0000 7f3e59ec8640 1 -- 172.21.15.72:0/1415531944 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e540ccb90 msgr2=0x7f3e540ccfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:57.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.575+0000 7f3e59ec8640 1 --2- 172.21.15.72:0/1415531944 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e540ccb90 0x7f3e540ccfe0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3e540b2e90 tx=0x7f3e4c02bee0 comp rx=0 tx=0).stop 2024-04-15T15:14:57.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.575+0000 7f3e59ec8640 1 -- 172.21.15.72:0/1415531944 shutdown_connections 2024-04-15T15:14:57.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.575+0000 7f3e59ec8640 1 --2- 172.21.15.72:0/1415531944 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e540ccb90 0x7f3e540ccfe0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:57.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.575+0000 7f3e59ec8640 1 --2- 172.21.15.72:0/1415531944 >> v2:172.21.15.175:3300/0 conn(0x7f3e5415eba0 0x7f3e5406f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:57.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.575+0000 7f3e59ec8640 1 -- 172.21.15.72:0/1415531944 >> 172.21.15.72:0/1415531944 conn(0x7f3e540b1dd0 msgr2=0x7f3e540b21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:57.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.575+0000 7f3e59ec8640 1 -- 172.21.15.72:0/1415531944 shutdown_connections 2024-04-15T15:14:57.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.576+0000 7f3e59ec8640 1 -- 172.21.15.72:0/1415531944 wait complete. 2024-04-15T15:14:57.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.577+0000 7f3e59ec8640 1 Processor -- start 2024-04-15T15:14:57.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.577+0000 7f3e59ec8640 1 -- start start 2024-04-15T15:14:57.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.577+0000 7f3e59ec8640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3e540ccb90 0x7f3e54153120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:57.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.577+0000 7f3e59ec8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e5415eba0 0x7f3e540727c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:57.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.578+0000 7f3e59ec8640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e54142820 0x7f3e54072d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:57.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.578+0000 7f3e59ec8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5406ecc0 con 0x7f3e5415eba0 2024-04-15T15:14:57.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.578+0000 7f3e59ec8640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f3e540ce870 con 0x7f3e54142820 2024-04-15T15:14:57.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.578+0000 7f3e59ec8640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f3e540b2b80 con 0x7f3e540ccb90 2024-04-15T15:14:57.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.578+0000 7f3e52ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e5415eba0 0x7f3e540727c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:57.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.578+0000 7f3e52ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e5415eba0 0x7f3e540727c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45116/0 (socket says 172.21.15.72:45116) 2024-04-15T15:14:57.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.578+0000 7f3e52ffd640 1 -- 172.21.15.72:0/192950978 learned_addr learned my addr 172.21.15.72:0/192950978 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:57.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.578+0000 7f3e53fff640 1 --2- 172.21.15.72:0/192950978 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e54142820 0x7f3e54072d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:57.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.578+0000 7f3e52ffd640 1 -- 172.21.15.72:0/192950978 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3e540ccb90 msgr2=0x7f3e54153120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:57.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.578+0000 7f3e537fe640 1 --2- 172.21.15.72:0/192950978 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3e540ccb90 0x7f3e54153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:57.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.578+0000 7f3e52ffd640 1 --2- 172.21.15.72:0/192950978 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3e540ccb90 0x7f3e54153120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:57.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.578+0000 7f3e52ffd640 1 -- 172.21.15.72:0/192950978 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e54142820 msgr2=0x7f3e54072d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:57.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.578+0000 7f3e52ffd640 1 --2- 172.21.15.72:0/192950978 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e54142820 0x7f3e54072d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:57.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.578+0000 7f3e52ffd640 1 -- 172.21.15.72:0/192950978 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e541591b0 con 0x7f3e5415eba0 2024-04-15T15:14:57.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.578+0000 7f3e53fff640 1 --2- 172.21.15.72:0/192950978 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e54142820 0x7f3e54072d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:14:57.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.578+0000 7f3e537fe640 1 --2- 172.21.15.72:0/192950978 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3e540ccb90 0x7f3e54153120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:14:57.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.578+0000 7f3e52ffd640 1 --2- 172.21.15.72:0/192950978 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e5415eba0 0x7f3e540727c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c00c430 tx=0x7f3e3c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:57.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.578+0000 7f3e4bfff640 1 -- 172.21.15.72:0/192950978 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c0074e0 con 0x7f3e5415eba0 2024-04-15T15:14:57.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.578+0000 7f3e4bfff640 1 -- 172.21.15.72:0/192950978 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e3c0047e0 con 0x7f3e5415eba0 2024-04-15T15:14:57.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.578+0000 7f3e59ec8640 1 -- 172.21.15.72:0/192950978 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e54071ce0 con 0x7f3e5415eba0 2024-04-15T15:14:57.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.578+0000 7f3e4bfff640 1 -- 172.21.15.72:0/192950978 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c004bd0 con 0x7f3e5415eba0 2024-04-15T15:14:57.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.578+0000 7f3e59ec8640 1 -- 172.21.15.72:0/192950978 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e54159490 con 0x7f3e5415eba0 2024-04-15T15:14:57.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.579+0000 7f3e59ec8640 1 -- 172.21.15.72:0/192950978 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e540722f0 con 0x7f3e5415eba0 2024-04-15T15:14:57.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.580+0000 7f3e4bfff640 1 -- 172.21.15.72:0/192950978 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3e3c011070 con 0x7f3e5415eba0 2024-04-15T15:14:57.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.580+0000 7f3e4bfff640 1 --2- 172.21.15.72:0/192950978 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3e3406c920 0x7f3e3406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:57.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.580+0000 7f3e4bfff640 1 -- 172.21.15.72:0/192950978 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f3e3c08bba0 con 0x7f3e5415eba0 2024-04-15T15:14:57.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.580+0000 7f3e537fe640 1 --2- 172.21.15.72:0/192950978 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3e3406c920 0x7f3e3406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:57.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.580+0000 7f3e537fe640 1 --2- 172.21.15.72:0/192950978 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3e3406c920 0x7f3e3406ede0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3e44006fd0 tx=0x7f3e44008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:57.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.584+0000 7f3e4bfff640 1 -- 172.21.15.72:0/192950978 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3e3c059c00 con 0x7f3e5415eba0 2024-04-15T15:14:57.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.760+0000 7f3e59ec8640 1 -- 172.21.15.72:0/192950978 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3e540689e0 con 0x7f3e5415eba0 2024-04-15T15:14:57.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.761+0000 7f3e4bfff640 1 -- 172.21.15.72:0/192950978 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7f3e3c08b020 con 0x7f3e5415eba0 2024-04-15T15:14:57.763 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:14:57.763 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":24,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:14:56.451113+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:14:49.438366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:14:50.368379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:14:54.081157+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:14:57.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.763+0000 7f3e59ec8640 1 -- 172.21.15.72:0/192950978 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3e3406c920 msgr2=0x7f3e3406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:57.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.763+0000 7f3e59ec8640 1 --2- 172.21.15.72:0/192950978 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3e3406c920 0x7f3e3406ede0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3e44006fd0 tx=0x7f3e44008040 comp rx=0 tx=0).stop 2024-04-15T15:14:57.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.763+0000 7f3e59ec8640 1 -- 172.21.15.72:0/192950978 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e5415eba0 msgr2=0x7f3e540727c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:57.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.763+0000 7f3e59ec8640 1 --2- 172.21.15.72:0/192950978 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e5415eba0 0x7f3e540727c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c00c430 tx=0x7f3e3c00c8f0 comp rx=0 tx=0).stop 2024-04-15T15:14:57.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.763+0000 7f3e59ec8640 1 -- 172.21.15.72:0/192950978 shutdown_connections 2024-04-15T15:14:57.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.763+0000 7f3e59ec8640 1 --2- 172.21.15.72:0/192950978 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3e3406c920 0x7f3e3406ede0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:57.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.763+0000 7f3e59ec8640 1 --2- 172.21.15.72:0/192950978 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3e54142820 0x7f3e54072d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:57.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.763+0000 7f3e59ec8640 1 --2- 172.21.15.72:0/192950978 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3e5415eba0 0x7f3e540727c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:57.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.763+0000 7f3e59ec8640 1 --2- 172.21.15.72:0/192950978 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3e540ccb90 0x7f3e54153120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:57.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.763+0000 7f3e59ec8640 1 -- 172.21.15.72:0/192950978 >> 172.21.15.72:0/192950978 conn(0x7f3e540b1dd0 msgr2=0x7f3e540afeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:57.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.763+0000 7f3e59ec8640 1 -- 172.21.15.72:0/192950978 shutdown_connections 2024-04-15T15:14:57.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.763+0000 7f3e59ec8640 1 -- 172.21.15.72:0/192950978 wait complete. 2024-04-15T15:14:57.776 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:14:57.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.877+0000 7f204e5ba640 1 Processor -- start 2024-04-15T15:14:57.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.877+0000 7f204e5ba640 1 -- start start 2024-04-15T15:14:57.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.877+0000 7f204e5ba640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f204815fd90 0x7f20480cd250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:57.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.877+0000 7f204e5ba640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f20480b6d70 0x7f20480d1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:57.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.877+0000 7f204e5ba640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20480ce5d0 con 0x7f20480cd790 2024-04-15T15:14:57.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.877+0000 7f204e5ba640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f20480ccae0 con 0x7f204815fd90 2024-04-15T15:14:57.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.877+0000 7f204e5ba640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f20480ce8b0 con 0x7f20480b6d70 2024-04-15T15:14:57.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.878+0000 7f2047fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f204815fd90 0x7f20480cd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:57.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.878+0000 7f204cb57640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f20480cd790 0x7f204806f510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37220/0 (socket says 172.21.15.72:37220) 2024-04-15T15:14:57.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.878+0000 7f204cb57640 1 -- 172.21.15.72:0/1295059027 learned_addr learned my addr 172.21.15.72:0/1295059027 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:14:57.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.878+0000 7f203ffff640 1 --2- 172.21.15.72:0/1295059027 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f20480b6d70 0x7f20480d1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:57.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.878+0000 7f2047fff640 1 -- 172.21.15.72:0/1295059027 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f20480b6d70 msgr2=0x7f20480d1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:57.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.878+0000 7f2047fff640 1 --2- 172.21.15.72:0/1295059027 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f20480b6d70 0x7f20480d1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:57.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.878+0000 7f2047fff640 1 -- 172.21.15.72:0/1295059027 >> v1:172.21.15.72:6789/0 conn(0x7f20480cd790 legacy=0x7f204806f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:57.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.878+0000 7f20477fe640 1 -- 172.21.15.72:0/1295059027 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1420697305 0 0) 0x7f20480ce5d0 con 0x7f20480cd790 2024-04-15T15:14:57.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.878+0000 7f2047fff640 1 -- 172.21.15.72:0/1295059027 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20480ceb90 con 0x7f204815fd90 2024-04-15T15:14:57.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.879+0000 7f2047fff640 1 --2- 172.21.15.72:0/1295059027 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f204815fd90 0x7f20480cd250 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f20380097a0 tx=0x7f203802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81c7411e79edbefb server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:57.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.879+0000 7f20477fe640 1 -- 172.21.15.72:0/1295059027 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2038035020 con 0x7f204815fd90 2024-04-15T15:14:57.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.879+0000 7f20477fe640 1 -- 172.21.15.72:0/1295059027 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2038031830 con 0x7f204815fd90 2024-04-15T15:14:57.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.879+0000 7f20477fe640 1 -- 172.21.15.72:0/1295059027 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2038031c00 con 0x7f204815fd90 2024-04-15T15:14:57.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.879+0000 7f204e5ba640 1 -- 172.21.15.72:0/1295059027 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f204815fd90 msgr2=0x7f20480cd250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:57.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.879+0000 7f204e5ba640 1 --2- 172.21.15.72:0/1295059027 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f204815fd90 0x7f20480cd250 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f20380097a0 tx=0x7f203802b8d0 comp rx=0 tx=0).stop 2024-04-15T15:14:57.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.879+0000 7f204e5ba640 1 -- 172.21.15.72:0/1295059027 shutdown_connections 2024-04-15T15:14:57.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.879+0000 7f204e5ba640 1 --2- 172.21.15.72:0/1295059027 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f20480b6d70 0x7f20480d1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:57.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.879+0000 7f204e5ba640 1 --2- 172.21.15.72:0/1295059027 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f204815fd90 0x7f20480cd250 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:57.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.879+0000 7f204e5ba640 1 -- 172.21.15.72:0/1295059027 >> 172.21.15.72:0/1295059027 conn(0x7f20480cef10 msgr2=0x7f20480cf320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:57.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.879+0000 7f204e5ba640 1 -- 172.21.15.72:0/1295059027 shutdown_connections 2024-04-15T15:14:57.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.879+0000 7f204e5ba640 1 -- 172.21.15.72:0/1295059027 wait complete. 2024-04-15T15:14:57.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.881+0000 7f204e5ba640 1 Processor -- start 2024-04-15T15:14:57.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.881+0000 7f204e5ba640 1 -- start start 2024-04-15T15:14:57.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.881+0000 7f204e5ba640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f20480b6d70 0x7f2048151a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:57.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.881+0000 7f204e5ba640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f204815fd90 0x7f20480ca5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:57.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.881+0000 7f204e5ba640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20481f78c0 0x7f20481f4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:57.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.881+0000 7f204e5ba640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f20480b5810 con 0x7f20481f78c0 2024-04-15T15:14:57.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.881+0000 7f204e5ba640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f2048160170 con 0x7f204815fd90 2024-04-15T15:14:57.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.881+0000 7f204e5ba640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f20480b5990 con 0x7f20480b6d70 2024-04-15T15:14:57.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.881+0000 7f204cb57640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20481f78c0 0x7f20481f4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:57.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.881+0000 7f204cb57640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20481f78c0 0x7f20481f4280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45134/0 (socket says 172.21.15.72:45134) 2024-04-15T15:14:57.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.881+0000 7f204cb57640 1 -- 172.21.15.72:0/1673849102 learned_addr learned my addr 172.21.15.72:0/1673849102 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:57.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.882+0000 7f203ffff640 1 --2- 172.21.15.72:0/1673849102 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f204815fd90 0x7f20480ca5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:57.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.882+0000 7f204cb57640 1 -- 172.21.15.72:0/1673849102 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f20480b6d70 msgr2=0x7f2048151a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:57.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.882+0000 7f2047fff640 1 --2- 172.21.15.72:0/1673849102 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f20480b6d70 0x7f2048151a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:57.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.882+0000 7f204cb57640 1 --2- 172.21.15.72:0/1673849102 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f20480b6d70 0x7f2048151a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:57.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.882+0000 7f204cb57640 1 -- 172.21.15.72:0/1673849102 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f204815fd90 msgr2=0x7f20480ca5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:57.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.882+0000 7f204cb57640 1 --2- 172.21.15.72:0/1673849102 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f204815fd90 0x7f20480ca5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:57.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.882+0000 7f204cb57640 1 -- 172.21.15.72:0/1673849102 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f204806deb0 con 0x7f20481f78c0 2024-04-15T15:14:57.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.882+0000 7f203ffff640 1 --2- 172.21.15.72:0/1673849102 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f204815fd90 0x7f20480ca5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:14:57.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.882+0000 7f204cb57640 1 --2- 172.21.15.72:0/1673849102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20481f78c0 0x7f20481f4280 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f204000c730 tx=0x7f204000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:57.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.882+0000 7f20457fa640 1 -- 172.21.15.72:0/1673849102 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2040018070 con 0x7f20481f78c0 2024-04-15T15:14:57.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.882+0000 7f20457fa640 1 -- 172.21.15.72:0/1673849102 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f204000ce40 con 0x7f20481f78c0 2024-04-15T15:14:57.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.882+0000 7f204e5ba640 1 -- 172.21.15.72:0/1673849102 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20481f4ab0 con 0x7f20481f78c0 2024-04-15T15:14:57.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.882+0000 7f20457fa640 1 -- 172.21.15.72:0/1673849102 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f204000a800 con 0x7f20481f78c0 2024-04-15T15:14:57.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.882+0000 7f204e5ba640 1 -- 172.21.15.72:0/1673849102 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20481f5070 con 0x7f20481f78c0 2024-04-15T15:14:57.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.883+0000 7f204e5ba640 1 -- 172.21.15.72:0/1673849102 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20481f5270 con 0x7f20481f78c0 2024-04-15T15:14:57.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.884+0000 7f20457fa640 1 -- 172.21.15.72:0/1673849102 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f204001b070 con 0x7f20481f78c0 2024-04-15T15:14:57.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.884+0000 7f20457fa640 1 --2- 172.21.15.72:0/1673849102 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f202806c920 0x7f202806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:57.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.884+0000 7f2047fff640 1 --2- 172.21.15.72:0/1673849102 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f202806c920 0x7f202806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:57.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.884+0000 7f20457fa640 1 -- 172.21.15.72:0/1673849102 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f204008b3b0 con 0x7f20481f78c0 2024-04-15T15:14:57.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.884+0000 7f2047fff640 1 --2- 172.21.15.72:0/1673849102 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f202806c920 0x7f202806ede0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2038036000 tx=0x7f2038008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:57.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:57.888+0000 7f20457fa640 1 -- 172.21.15.72:0/1673849102 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f20400594a0 con 0x7f20481f78c0 2024-04-15T15:14:58.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.062+0000 7f204e5ba640 1 -- 172.21.15.72:0/1673849102 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f20480690f0 con 0x7f20481f78c0 2024-04-15T15:14:58.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.063+0000 7f20457fa640 1 -- 172.21.15.72:0/1673849102 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7f2040058da0 con 0x7f20481f78c0 2024-04-15T15:14:58.065 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:14:58.065 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":24,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:14:56.451113+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:14:49.438366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:14:50.368379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:14:54.081157+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:14:58.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.065+0000 7f204e5ba640 1 -- 172.21.15.72:0/1673849102 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f202806c920 msgr2=0x7f202806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:58.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.065+0000 7f204e5ba640 1 --2- 172.21.15.72:0/1673849102 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f202806c920 0x7f202806ede0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2038036000 tx=0x7f2038008030 comp rx=0 tx=0).stop 2024-04-15T15:14:58.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.065+0000 7f204e5ba640 1 -- 172.21.15.72:0/1673849102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20481f78c0 msgr2=0x7f20481f4280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:58.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.065+0000 7f204e5ba640 1 --2- 172.21.15.72:0/1673849102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20481f78c0 0x7f20481f4280 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f204000c730 tx=0x7f204000cbf0 comp rx=0 tx=0).stop 2024-04-15T15:14:58.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.065+0000 7f204e5ba640 1 -- 172.21.15.72:0/1673849102 shutdown_connections 2024-04-15T15:14:58.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.065+0000 7f204e5ba640 1 --2- 172.21.15.72:0/1673849102 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f202806c920 0x7f202806ede0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:58.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.065+0000 7f204e5ba640 1 --2- 172.21.15.72:0/1673849102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20481f78c0 0x7f20481f4280 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:58.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.065+0000 7f204e5ba640 1 --2- 172.21.15.72:0/1673849102 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f204815fd90 0x7f20480ca5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:58.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.065+0000 7f204e5ba640 1 --2- 172.21.15.72:0/1673849102 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f20480b6d70 0x7f2048151a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:58.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.065+0000 7f204e5ba640 1 -- 172.21.15.72:0/1673849102 >> 172.21.15.72:0/1673849102 conn(0x7f20480cef10 msgr2=0x7f204806f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:58.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.066+0000 7f204e5ba640 1 -- 172.21.15.72:0/1673849102 shutdown_connections 2024-04-15T15:14:58.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.066+0000 7f204e5ba640 1 -- 172.21.15.72:0/1673849102 wait complete. 2024-04-15T15:14:58.094 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:14:58.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.197+0000 7f0ae84eb640 1 Processor -- start 2024-04-15T15:14:58.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.197+0000 7f0ae84eb640 1 -- start start 2024-04-15T15:14:58.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.198+0000 7f0ae84eb640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0ae00d1790 0x7f0ae00b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:58.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.198+0000 7f0ae84eb640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0ae00b5790 0x7f0ae00cd950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:58.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.198+0000 7f0ae84eb640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ae015caf0 con 0x7f0ae015c710 2024-04-15T15:14:58.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.198+0000 7f0ae84eb640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f0ae006f510 con 0x7f0ae00b5790 2024-04-15T15:14:58.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.198+0000 7f0ae84eb640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f0ae00ccb90 con 0x7f0ae00d1790 2024-04-15T15:14:58.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.198+0000 7f0ae6a88640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0ae00b5790 0x7f0ae00cd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:58.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.198+0000 7f0ae6287640 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f0ae015c710 0x7f0ae00d1080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.72:59078/0 (socket says 172.21.15.72:59078) 2024-04-15T15:14:58.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.198+0000 7f0ae5a86640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0ae00d1790 0x7f0ae00b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:58.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.198+0000 7f0ae6287640 1 -- 172.21.15.72:0/2440586963 learned_addr learned my addr 172.21.15.72:0/2440586963 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:14:58.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.198+0000 7f0ae6a88640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0ae00b5790 0x7f0ae00cd950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.72:42694/0 (socket says 172.21.15.72:42694) 2024-04-15T15:14:58.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.199+0000 7f0ae5a86640 1 -- 172.21.15.72:0/2440586963 >> v2:172.21.15.175:3300/0 conn(0x7f0ae00b5790 msgr2=0x7f0ae00cd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:58.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.199+0000 7f0ae5a86640 1 --2- 172.21.15.72:0/2440586963 >> v2:172.21.15.175:3300/0 conn(0x7f0ae00b5790 0x7f0ae00cd950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:58.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.199+0000 7f0ae5a86640 1 -- 172.21.15.72:0/2440586963 >> v1:172.21.15.175:6789/0 conn(0x7f0ae015c710 legacy=0x7f0ae00d1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:58.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.199+0000 7f0ae5285640 1 -- 172.21.15.72:0/2440586963 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4205702354 0 0) 0x7f0ae015caf0 con 0x7f0ae015c710 2024-04-15T15:14:58.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.199+0000 7f0ae5a86640 1 -- 172.21.15.72:0/2440586963 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ae00cce70 con 0x7f0ae00d1790 2024-04-15T15:14:58.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.199+0000 7f0ae6a88640 1 --2- 172.21.15.72:0/2440586963 >> v2:172.21.15.175:3300/0 conn(0x7f0ae00b5790 0x7f0ae00cd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:14:58.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.199+0000 7f0ae5a86640 1 --2- 172.21.15.72:0/2440586963 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0ae00d1790 0x7f0ae00b66b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0ad00097a0 tx=0x7f0ad002b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eee7000ac3cf29b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:58.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.200+0000 7f0ae5285640 1 -- 172.21.15.72:0/2440586963 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad0034070 con 0x7f0ae00d1790 2024-04-15T15:14:58.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.200+0000 7f0ae5285640 1 -- 172.21.15.72:0/2440586963 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ad0030610 con 0x7f0ae00d1790 2024-04-15T15:14:58.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.200+0000 7f0ae5285640 1 -- 172.21.15.72:0/2440586963 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad00309e0 con 0x7f0ae00d1790 2024-04-15T15:14:58.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.200+0000 7f0ae84eb640 1 -- 172.21.15.72:0/2440586963 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0ae00d1790 msgr2=0x7f0ae00b66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:58.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.200+0000 7f0ae84eb640 1 --2- 172.21.15.72:0/2440586963 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0ae00d1790 0x7f0ae00b66b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0ad00097a0 tx=0x7f0ad002b960 comp rx=0 tx=0).stop 2024-04-15T15:14:58.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.200+0000 7f0ae84eb640 1 -- 172.21.15.72:0/2440586963 shutdown_connections 2024-04-15T15:14:58.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.200+0000 7f0ae84eb640 1 --2- 172.21.15.72:0/2440586963 >> v2:172.21.15.175:3300/0 conn(0x7f0ae00b5790 0x7f0ae00cd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:58.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.200+0000 7f0ae84eb640 1 --2- 172.21.15.72:0/2440586963 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0ae00d1790 0x7f0ae00b66b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:58.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.200+0000 7f0ae84eb640 1 -- 172.21.15.72:0/2440586963 >> 172.21.15.72:0/2440586963 conn(0x7f0ae00b3f90 msgr2=0x7f0ae00b43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:58.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.200+0000 7f0ae84eb640 1 -- 172.21.15.72:0/2440586963 shutdown_connections 2024-04-15T15:14:58.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.200+0000 7f0ae84eb640 1 -- 172.21.15.72:0/2440586963 wait complete. 2024-04-15T15:14:58.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.201+0000 7f0ae84eb640 1 Processor -- start 2024-04-15T15:14:58.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.201+0000 7f0ae84eb640 1 -- start start 2024-04-15T15:14:58.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.202+0000 7f0ae84eb640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0ae00b5790 0x7f0ae0158e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:58.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.202+0000 7f0ae84eb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0ae00d1790 0x7f0ae00cb550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:58.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.202+0000 7f0ae84eb640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0ae015c710 0x7f0ae015a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:58.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.202+0000 7f0ae84eb640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae00d1cd0 con 0x7f0ae00d1790 2024-04-15T15:14:58.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.202+0000 7f0ae84eb640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae00cd430 con 0x7f0ae015c710 2024-04-15T15:14:58.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.202+0000 7f0ae84eb640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f0ae00b6cd0 con 0x7f0ae00b5790 2024-04-15T15:14:58.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.202+0000 7f0ae5a86640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0ae00d1790 0x7f0ae00cb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:58.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.202+0000 7f0ae5a86640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0ae00d1790 0x7f0ae00cb550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45140/0 (socket says 172.21.15.72:45140) 2024-04-15T15:14:58.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.202+0000 7f0ae5a86640 1 -- 172.21.15.72:0/2552354769 learned_addr learned my addr 172.21.15.72:0/2552354769 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:58.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.202+0000 7f0ae6a88640 1 --2- 172.21.15.72:0/2552354769 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0ae015c710 0x7f0ae015a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:58.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.202+0000 7f0ae6287640 1 --2- 172.21.15.72:0/2552354769 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0ae00b5790 0x7f0ae0158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:58.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.203+0000 7f0ae5a86640 1 -- 172.21.15.72:0/2552354769 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0ae00b5790 msgr2=0x7f0ae0158e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:58.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.203+0000 7f0ae5a86640 1 --2- 172.21.15.72:0/2552354769 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0ae00b5790 0x7f0ae0158e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:58.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.203+0000 7f0ae5a86640 1 -- 172.21.15.72:0/2552354769 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0ae015c710 msgr2=0x7f0ae015a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:58.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.203+0000 7f0ae5a86640 1 --2- 172.21.15.72:0/2552354769 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0ae015c710 0x7f0ae015a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:58.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.203+0000 7f0ae5a86640 1 -- 172.21.15.72:0/2552354769 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ae00cba90 con 0x7f0ae00d1790 2024-04-15T15:14:58.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.203+0000 7f0ae5a86640 1 --2- 172.21.15.72:0/2552354769 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0ae00d1790 0x7f0ae00cb550 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0ad002b9a0 tx=0x7f0ad00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:58.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.203+0000 7f0ad6ffd640 1 -- 172.21.15.72:0/2552354769 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad0034040 con 0x7f0ae00d1790 2024-04-15T15:14:58.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.203+0000 7f0ad6ffd640 1 -- 172.21.15.72:0/2552354769 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ad0030410 con 0x7f0ae00d1790 2024-04-15T15:14:58.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.203+0000 7f0ae84eb640 1 -- 172.21.15.72:0/2552354769 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ae006e260 con 0x7f0ae00d1790 2024-04-15T15:14:58.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.203+0000 7f0ad6ffd640 1 -- 172.21.15.72:0/2552354769 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad0006e10 con 0x7f0ae00d1790 2024-04-15T15:14:58.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.203+0000 7f0ae84eb640 1 -- 172.21.15.72:0/2552354769 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ae006d9c0 con 0x7f0ae00d1790 2024-04-15T15:14:58.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.204+0000 7f0ae84eb640 1 -- 172.21.15.72:0/2552354769 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ae00689e0 con 0x7f0ae00d1790 2024-04-15T15:14:58.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.204+0000 7f0ad6ffd640 1 -- 172.21.15.72:0/2552354769 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0ad004c060 con 0x7f0ae00d1790 2024-04-15T15:14:58.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.205+0000 7f0ad6ffd640 1 --2- 172.21.15.72:0/2552354769 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0ac406c8f0 0x7f0ac406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:58.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.205+0000 7f0ae6287640 1 --2- 172.21.15.72:0/2552354769 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0ac406c8f0 0x7f0ac406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:58.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.205+0000 7f0ad6ffd640 1 -- 172.21.15.72:0/2552354769 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f0ad00adae0 con 0x7f0ae00d1790 2024-04-15T15:14:58.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.205+0000 7f0ae6287640 1 --2- 172.21.15.72:0/2552354769 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0ac406c8f0 0x7f0ac406edb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0adc000f60 tx=0x7f0adc007d50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:58.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.209+0000 7f0ad6ffd640 1 -- 172.21.15.72:0/2552354769 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0ae00cba90 con 0x7f0ae00d1790 2024-04-15T15:14:58.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.421+0000 7f0ae84eb640 1 -- 172.21.15.72:0/2552354769 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0ae00c9020 con 0x7f0ae00d1790 2024-04-15T15:14:58.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.421+0000 7f0ad6ffd640 1 -- 172.21.15.72:0/2552354769 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5183 (secure 0 0 0) 0x7f0ad00305e0 con 0x7f0ae00d1790 2024-04-15T15:14:58.423 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:14:58.423 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4267,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6833/1659343919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1659343919},{"type":"v1","addr":"172.21.15.175:6833","nonce":1659343919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6835/4113588141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6834","nonce":4113588141},{"type":"v1","addr":"172.21.15.175:6835","nonce":4113588141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4296,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/4151548261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4151548261},{"type":"v1","addr":"172.21.15.72:6836","nonce":4151548261}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:14:51.411091+0000","modified":"2024-04-15T15:14:56.451317+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/3458067804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3458067804},{"type":"v1","addr":"172.21.15.72:6837","nonce":3458067804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4299,"qdb_cluster":[4299]},"id":1}]} 2024-04-15T15:14:58.424 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 7 2024-04-15T15:14:58.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.424+0000 7f0ae84eb640 1 -- 172.21.15.72:0/2552354769 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0ac406c8f0 msgr2=0x7f0ac406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:58.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.424+0000 7f0ae84eb640 1 --2- 172.21.15.72:0/2552354769 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0ac406c8f0 0x7f0ac406edb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0adc000f60 tx=0x7f0adc007d50 comp rx=0 tx=0).stop 2024-04-15T15:14:58.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.424+0000 7f0ae84eb640 1 -- 172.21.15.72:0/2552354769 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0ae00d1790 msgr2=0x7f0ae00cb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:58.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.424+0000 7f0ae84eb640 1 --2- 172.21.15.72:0/2552354769 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0ae00d1790 0x7f0ae00cb550 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0ad002b9a0 tx=0x7f0ad00046c0 comp rx=0 tx=0).stop 2024-04-15T15:14:58.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.424+0000 7f0ae84eb640 1 -- 172.21.15.72:0/2552354769 shutdown_connections 2024-04-15T15:14:58.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.424+0000 7f0ae84eb640 1 --2- 172.21.15.72:0/2552354769 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0ac406c8f0 0x7f0ac406edb0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:58.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.424+0000 7f0ae84eb640 1 --2- 172.21.15.72:0/2552354769 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0ae015c710 0x7f0ae015a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:58.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.424+0000 7f0ae84eb640 1 --2- 172.21.15.72:0/2552354769 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0ae00d1790 0x7f0ae00cb550 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:58.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.424+0000 7f0ae84eb640 1 --2- 172.21.15.72:0/2552354769 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0ae00b5790 0x7f0ae0158e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:58.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.424+0000 7f0ae84eb640 1 -- 172.21.15.72:0/2552354769 >> 172.21.15.72:0/2552354769 conn(0x7f0ae00b3f90 msgr2=0x7f0ae0071660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:58.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.424+0000 7f0ae84eb640 1 -- 172.21.15.72:0/2552354769 shutdown_connections 2024-04-15T15:14:58.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.424+0000 7f0ae84eb640 1 -- 172.21.15.72:0/2552354769 wait complete. 2024-04-15T15:14:58.436 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:14:58.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.537+0000 7fe16398f640 1 Processor -- start 2024-04-15T15:14:58.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.537+0000 7fe16398f640 1 -- start start 2024-04-15T15:14:58.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.538+0000 7fe16398f640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe15c0d4aa0 0x7fe15c0d4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:58.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.538+0000 7fe16398f640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe15c0cfb50 0x7fe15c0cffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:58.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.538+0000 7fe16398f640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe15c0b60b0 con 0x7fe15c15e9b0 2024-04-15T15:14:58.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.538+0000 7fe16398f640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fe15c15ed90 con 0x7fe15c0cfb50 2024-04-15T15:14:58.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.538+0000 7fe16398f640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fe15c0d5420 con 0x7fe15c0d4aa0 2024-04-15T15:14:58.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.538+0000 7fe16172b640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fe15c15e9b0 0x7fe15c0d3420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37222/0 (socket says 172.21.15.72:37222) 2024-04-15T15:14:58.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.538+0000 7fe16172b640 1 -- 172.21.15.72:0/3907847082 learned_addr learned my addr 172.21.15.72:0/3907847082 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:14:58.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.538+0000 7fe15bfff640 1 -- 172.21.15.72:0/3907847082 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1670141965 0 0) 0x7fe15c0b60b0 con 0x7fe15c15e9b0 2024-04-15T15:14:58.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.538+0000 7fe15bfff640 1 -- 172.21.15.72:0/3907847082 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe144003520 con 0x7fe15c15e9b0 2024-04-15T15:14:58.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.538+0000 7fe161f2c640 1 --2- 172.21.15.72:0/3907847082 >> v2:172.21.15.175:3300/0 conn(0x7fe15c0cfb50 0x7fe15c0cffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:58.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.538+0000 7fe160f2a640 1 --2- 172.21.15.72:0/3907847082 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe15c0d4aa0 0x7fe15c0d4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:58.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.538+0000 7fe15bfff640 1 -- 172.21.15.72:0/3907847082 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3978551731 0 0) 0x7fe144003520 con 0x7fe15c15e9b0 2024-04-15T15:14:58.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.538+0000 7fe15bfff640 1 -- 172.21.15.72:0/3907847082 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe15c0d4aa0 msgr2=0x7fe15c0d4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:58.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.538+0000 7fe15bfff640 1 --2- 172.21.15.72:0/3907847082 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe15c0d4aa0 0x7fe15c0d4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:58.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.538+0000 7fe15bfff640 1 -- 172.21.15.72:0/3907847082 >> v2:172.21.15.175:3300/0 conn(0x7fe15c0cfb50 msgr2=0x7fe15c0cffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:58.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.538+0000 7fe15bfff640 1 --2- 172.21.15.72:0/3907847082 >> v2:172.21.15.175:3300/0 conn(0x7fe15c0cfb50 0x7fe15c0cffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:58.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.538+0000 7fe15bfff640 1 -- 172.21.15.72:0/3907847082 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe15c0d3ea0 con 0x7fe15c15e9b0 2024-04-15T15:14:58.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.538+0000 7fe15bfff640 1 -- 172.21.15.72:0/3907847082 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7fe14c001690 con 0x7fe15c15e9b0 2024-04-15T15:14:58.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.538+0000 7fe15bfff640 1 -- 172.21.15.72:0/3907847082 >> v1:172.21.15.72:6789/0 conn(0x7fe15c15e9b0 legacy=0x7fe15c0d3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:58.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.539+0000 7fe15bfff640 1 --2- 172.21.15.72:0/3907847082 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe15c0d4aa0 0x7fe1440039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:58.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.539+0000 7fe15bfff640 1 --2- 172.21.15.72:0/3907847082 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe144003f30 0x7fe144004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:58.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.539+0000 7fe15bfff640 1 --2- 172.21.15.72:0/3907847082 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe144004d40 0x7fe1440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:58.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.539+0000 7fe15bfff640 1 -- 172.21.15.72:0/3907847082 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe15c15ed90 con 0x7fe144003f30 2024-04-15T15:14:58.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.539+0000 7fe15bfff640 1 -- 172.21.15.72:0/3907847082 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fe15c0d5420 con 0x7fe144004d40 2024-04-15T15:14:58.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.539+0000 7fe15bfff640 1 -- 172.21.15.72:0/3907847082 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fe14400b9f0 con 0x7fe15c0d4aa0 2024-04-15T15:14:58.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.539+0000 7fe160f2a640 1 --2- 172.21.15.72:0/3907847082 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe144003f30 0x7fe144004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:58.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.539+0000 7fe161f2c640 1 --2- 172.21.15.72:0/3907847082 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe144004d40 0x7fe1440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:58.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.539+0000 7fe16172b640 1 --2- 172.21.15.72:0/3907847082 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe15c0d4aa0 0x7fe1440039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:58.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.539+0000 7fe160f2a640 1 -- 172.21.15.72:0/3907847082 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe15c0d4aa0 msgr2=0x7fe1440039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:58.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.539+0000 7fe160f2a640 1 --2- 172.21.15.72:0/3907847082 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe15c0d4aa0 0x7fe1440039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:58.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.539+0000 7fe160f2a640 1 -- 172.21.15.72:0/3907847082 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe144004d40 msgr2=0x7fe1440051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:58.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.539+0000 7fe160f2a640 1 --2- 172.21.15.72:0/3907847082 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe144004d40 0x7fe1440051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:58.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.539+0000 7fe160f2a640 1 -- 172.21.15.72:0/3907847082 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe144003520 con 0x7fe144003f30 2024-04-15T15:14:58.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.539+0000 7fe16172b640 1 --2- 172.21.15.72:0/3907847082 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe15c0d4aa0 0x7fe1440039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:14:58.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.540+0000 7fe160f2a640 1 --2- 172.21.15.72:0/3907847082 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe144003f30 0x7fe144004800 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe150009f90 tx=0x7fe144005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79b670e30a828fb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:58.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.540+0000 7fe15bfff640 1 -- 172.21.15.72:0/3907847082 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe150039070 con 0x7fe144003f30 2024-04-15T15:14:58.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.540+0000 7fe15bfff640 1 -- 172.21.15.72:0/3907847082 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe15002cbd0 con 0x7fe144003f30 2024-04-15T15:14:58.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.540+0000 7fe15bfff640 1 -- 172.21.15.72:0/3907847082 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe150009c10 con 0x7fe144003f30 2024-04-15T15:14:58.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.540+0000 7fe16398f640 1 -- 172.21.15.72:0/3907847082 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe144003f30 msgr2=0x7fe144004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:58.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.540+0000 7fe16398f640 1 --2- 172.21.15.72:0/3907847082 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe144003f30 0x7fe144004800 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe150009f90 tx=0x7fe144005830 comp rx=0 tx=0).stop 2024-04-15T15:14:58.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.540+0000 7fe161f2c640 1 -- 172.21.15.72:0/3907847082 reap_dead start 2024-04-15T15:14:58.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.540+0000 7fe16398f640 1 -- 172.21.15.72:0/3907847082 shutdown_connections 2024-04-15T15:14:58.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.540+0000 7fe16398f640 1 -- 172.21.15.72:0/3907847082 >> 172.21.15.72:0/3907847082 conn(0x7fe15c09db30 msgr2=0x7fe15c0b95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:58.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.540+0000 7fe16398f640 1 -- 172.21.15.72:0/3907847082 shutdown_connections 2024-04-15T15:14:58.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.540+0000 7fe16398f640 1 -- 172.21.15.72:0/3907847082 wait complete. 2024-04-15T15:14:58.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.541+0000 7fe16398f640 1 Processor -- start 2024-04-15T15:14:58.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.542+0000 7fe16398f640 1 -- start start 2024-04-15T15:14:58.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.542+0000 7fe16398f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe144003f30 0x7fe15c1544f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:58.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.542+0000 7fe16398f640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe15c151a50 0x7fe15c0d2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:58.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.542+0000 7fe16398f640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe15c152ff0 0x7fe15c153480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:58.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.542+0000 7fe16398f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe15c0b6910 con 0x7fe144003f30 2024-04-15T15:14:58.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.542+0000 7fe16398f640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fe15c0b6630 con 0x7fe15c152ff0 2024-04-15T15:14:58.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.542+0000 7fe16398f640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fe15c0b6e30 con 0x7fe15c151a50 2024-04-15T15:14:58.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.542+0000 7fe16172b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe144003f30 0x7fe15c1544f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:58.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.542+0000 7fe16172b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe144003f30 0x7fe15c1544f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45152/0 (socket says 172.21.15.72:45152) 2024-04-15T15:14:58.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.542+0000 7fe16172b640 1 -- 172.21.15.72:0/2993259948 learned_addr learned my addr 172.21.15.72:0/2993259948 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:58.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.542+0000 7fe160f2a640 1 --2- 172.21.15.72:0/2993259948 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe15c151a50 0x7fe15c0d2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:58.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.542+0000 7fe161f2c640 1 --2- 172.21.15.72:0/2993259948 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe15c152ff0 0x7fe15c153480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:58.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.543+0000 7fe16172b640 1 -- 172.21.15.72:0/2993259948 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe15c151a50 msgr2=0x7fe15c0d2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:58.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.543+0000 7fe16172b640 1 --2- 172.21.15.72:0/2993259948 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe15c151a50 0x7fe15c0d2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:58.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.543+0000 7fe16172b640 1 -- 172.21.15.72:0/2993259948 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe15c152ff0 msgr2=0x7fe15c153480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:58.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.543+0000 7fe16172b640 1 --2- 172.21.15.72:0/2993259948 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe15c152ff0 0x7fe15c153480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:58.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.543+0000 7fe16172b640 1 -- 172.21.15.72:0/2993259948 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe15c154a30 con 0x7fe144003f30 2024-04-15T15:14:58.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.543+0000 7fe160f2a640 1 --2- 172.21.15.72:0/2993259948 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe15c151a50 0x7fe15c0d2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:14:58.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.543+0000 7fe161f2c640 1 --2- 172.21.15.72:0/2993259948 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe15c152ff0 0x7fe15c153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:14:58.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.543+0000 7fe16172b640 1 --2- 172.21.15.72:0/2993259948 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe144003f30 0x7fe15c1544f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe14c00e8a0 tx=0x7fe14c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:58.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.543+0000 7fe159ffb640 1 -- 172.21.15.72:0/2993259948 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe14c00c8d0 con 0x7fe144003f30 2024-04-15T15:14:58.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.543+0000 7fe159ffb640 1 -- 172.21.15.72:0/2993259948 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe14c00ca70 con 0x7fe144003f30 2024-04-15T15:14:58.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.543+0000 7fe16398f640 1 -- 172.21.15.72:0/2993259948 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe15c151f50 con 0x7fe144003f30 2024-04-15T15:14:58.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.543+0000 7fe159ffb640 1 -- 172.21.15.72:0/2993259948 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe14c00cc20 con 0x7fe144003f30 2024-04-15T15:14:58.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.543+0000 7fe16398f640 1 -- 172.21.15.72:0/2993259948 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe15c152510 con 0x7fe144003f30 2024-04-15T15:14:58.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.544+0000 7fe16398f640 1 -- 172.21.15.72:0/2993259948 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe15c0689e0 con 0x7fe144003f30 2024-04-15T15:14:58.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.544+0000 7fe159ffb640 1 -- 172.21.15.72:0/2993259948 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe14c01a020 con 0x7fe144003f30 2024-04-15T15:14:58.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.545+0000 7fe159ffb640 1 --2- 172.21.15.72:0/2993259948 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe14806c8f0 0x7fe14806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:58.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.545+0000 7fe160f2a640 1 --2- 172.21.15.72:0/2993259948 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe14806c8f0 0x7fe14806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:58.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.545+0000 7fe159ffb640 1 -- 172.21.15.72:0/2993259948 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fe14c08b920 con 0x7fe144003f30 2024-04-15T15:14:58.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.545+0000 7fe160f2a640 1 --2- 172.21.15.72:0/2993259948 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe14806c8f0 0x7fe14806edb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe150036000 tx=0x7fe15002e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:58.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.549+0000 7fe159ffb640 1 -- 172.21.15.72:0/2993259948 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe15c154a30 con 0x7fe144003f30 2024-04-15T15:14:58.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.728+0000 7fe16398f640 1 -- 172.21.15.72:0/2993259948 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe15c0b3920 con 0x7fe144003f30 2024-04-15T15:14:58.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.729+0000 7fe159ffb640 1 -- 172.21.15.72:0/2993259948 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7fe14c059980 con 0x7fe144003f30 2024-04-15T15:14:58.731 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:14:58.731 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":24,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:14:56.451113+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:14:49.438366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:14:50.368379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:14:54.081157+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:14:58.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.731+0000 7fe16398f640 1 -- 172.21.15.72:0/2993259948 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe14806c8f0 msgr2=0x7fe14806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:58.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.731+0000 7fe16398f640 1 --2- 172.21.15.72:0/2993259948 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe14806c8f0 0x7fe14806edb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe150036000 tx=0x7fe15002e000 comp rx=0 tx=0).stop 2024-04-15T15:14:58.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.731+0000 7fe16398f640 1 -- 172.21.15.72:0/2993259948 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe144003f30 msgr2=0x7fe15c1544f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:58.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.731+0000 7fe16398f640 1 --2- 172.21.15.72:0/2993259948 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe144003f30 0x7fe15c1544f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe14c00e8a0 tx=0x7fe14c00ed60 comp rx=0 tx=0).stop 2024-04-15T15:14:58.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.731+0000 7fe16398f640 1 -- 172.21.15.72:0/2993259948 shutdown_connections 2024-04-15T15:14:58.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.731+0000 7fe16398f640 1 --2- 172.21.15.72:0/2993259948 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe14806c8f0 0x7fe14806edb0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:58.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.731+0000 7fe16398f640 1 --2- 172.21.15.72:0/2993259948 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe15c152ff0 0x7fe15c153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:58.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.731+0000 7fe16398f640 1 --2- 172.21.15.72:0/2993259948 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe15c151a50 0x7fe15c0d2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:58.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.731+0000 7fe16398f640 1 --2- 172.21.15.72:0/2993259948 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe144003f30 0x7fe15c1544f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:58.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.731+0000 7fe16398f640 1 -- 172.21.15.72:0/2993259948 >> 172.21.15.72:0/2993259948 conn(0x7fe15c09db30 msgr2=0x7fe15c0b1240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:58.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.732+0000 7fe16398f640 1 -- 172.21.15.72:0/2993259948 shutdown_connections 2024-04-15T15:14:58.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:58.732+0000 7fe16398f640 1 -- 172.21.15.72:0/2993259948 wait complete. 2024-04-15T15:14:59.147 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-15T15:14:51.411091+0000', 'modified': '2024-04-15T15:14:56.451317+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4299}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4299': {'gid': 4299, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/3458067804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 3458067804}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 3458067804}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 4299, 'qdb_cluster': [4299]} 2024-04-15T15:14:59.147 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-15T15:14:59.147 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-15T15:14:59.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.250+0000 7f6a46f00640 1 Processor -- start 2024-04-15T15:14:59.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.250+0000 7f6a46f00640 1 -- start start 2024-04-15T15:14:59.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.251+0000 7f6a46f00640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6a400ce2f0 0x7f6a4015c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:59.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.251+0000 7f6a46f00640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6a400d08f0 0x7f6a400cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:59.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.251+0000 7f6a46f00640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a4006bbb0 con 0x7f6a400ce6d0 2024-04-15T15:14:59.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.251+0000 7f6a46f00640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f6a4006b960 con 0x7f6a400ce2f0 2024-04-15T15:14:59.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.251+0000 7f6a46f00640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f6a4006be90 con 0x7f6a400d08f0 2024-04-15T15:14:59.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.251+0000 7f6a4549d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f6a400ce6d0 0x7f6a40155600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37232/0 (socket says 172.21.15.72:37232) 2024-04-15T15:14:59.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.251+0000 7f6a4549d640 1 -- 172.21.15.72:0/2964724487 learned_addr learned my addr 172.21.15.72:0/2964724487 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:14:59.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.251+0000 7f6a44c9c640 1 --2- 172.21.15.72:0/2964724487 >> v2:172.21.15.175:3300/0 conn(0x7f6a400ce2f0 0x7f6a4015c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:59.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.251+0000 7f6a37fff640 1 --2- 172.21.15.72:0/2964724487 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6a400d08f0 0x7f6a400cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:59.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.251+0000 7f6a377fe640 1 -- 172.21.15.72:0/2964724487 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3068602235 0 0) 0x7f6a4006bbb0 con 0x7f6a400ce6d0 2024-04-15T15:14:59.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.251+0000 7f6a377fe640 1 -- 172.21.15.72:0/2964724487 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a28003520 con 0x7f6a400ce6d0 2024-04-15T15:14:59.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.252+0000 7f6a377fe640 1 -- 172.21.15.72:0/2964724487 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 928044263 0 0) 0x7f6a28003520 con 0x7f6a400ce6d0 2024-04-15T15:14:59.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.252+0000 7f6a377fe640 1 -- 172.21.15.72:0/2964724487 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6a400d08f0 msgr2=0x7f6a400cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:59.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.252+0000 7f6a377fe640 1 --2- 172.21.15.72:0/2964724487 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6a400d08f0 0x7f6a400cc0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.252+0000 7f6a377fe640 1 -- 172.21.15.72:0/2964724487 >> v2:172.21.15.175:3300/0 conn(0x7f6a400ce2f0 msgr2=0x7f6a4015c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:59.256 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.252+0000 7f6a377fe640 1 --2- 172.21.15.72:0/2964724487 >> v2:172.21.15.175:3300/0 conn(0x7f6a400ce2f0 0x7f6a4015c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.252+0000 7f6a377fe640 1 -- 172.21.15.72:0/2964724487 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a4006c170 con 0x7f6a400ce6d0 2024-04-15T15:14:59.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.252+0000 7f6a377fe640 1 -- 172.21.15.72:0/2964724487 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f6a380019f0 con 0x7f6a400ce6d0 2024-04-15T15:14:59.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.252+0000 7f6a377fe640 1 -- 172.21.15.72:0/2964724487 >> v1:172.21.15.72:6789/0 conn(0x7f6a400ce6d0 legacy=0x7f6a40155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:59.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.252+0000 7f6a377fe640 1 --2- 172.21.15.72:0/2964724487 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6a400d08f0 0x7f6a280039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:59.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.252+0000 7f6a377fe640 1 --2- 172.21.15.72:0/2964724487 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6a28003f30 0x7f6a28004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:59.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.252+0000 7f6a377fe640 1 --2- 172.21.15.72:0/2964724487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a28004d40 0x7f6a280051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:59.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.252+0000 7f6a377fe640 1 -- 172.21.15.72:0/2964724487 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4006b960 con 0x7f6a28004d40 2024-04-15T15:14:59.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.252+0000 7f6a377fe640 1 -- 172.21.15.72:0/2964724487 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4006be90 con 0x7f6a28003f30 2024-04-15T15:14:59.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.252+0000 7f6a377fe640 1 -- 172.21.15.72:0/2964724487 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f6a2800b9c0 con 0x7f6a400d08f0 2024-04-15T15:14:59.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.252+0000 7f6a4549d640 1 --2- 172.21.15.72:0/2964724487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a28004d40 0x7f6a280051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:59.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.252+0000 7f6a44c9c640 1 --2- 172.21.15.72:0/2964724487 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6a400d08f0 0x7f6a280039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:59.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.252+0000 7f6a37fff640 1 --2- 172.21.15.72:0/2964724487 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6a28003f30 0x7f6a28004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:59.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.252+0000 7f6a4549d640 1 -- 172.21.15.72:0/2964724487 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6a400d08f0 msgr2=0x7f6a280039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:59.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.252+0000 7f6a4549d640 1 --2- 172.21.15.72:0/2964724487 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6a400d08f0 0x7f6a280039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.257 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.252+0000 7f6a4549d640 1 -- 172.21.15.72:0/2964724487 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6a28003f30 msgr2=0x7f6a28004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:59.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.252+0000 7f6a4549d640 1 --2- 172.21.15.72:0/2964724487 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6a28003f30 0x7f6a28004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.252+0000 7f6a4549d640 1 -- 172.21.15.72:0/2964724487 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a28003520 con 0x7f6a28004d40 2024-04-15T15:14:59.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.253+0000 7f6a4549d640 1 --2- 172.21.15.72:0/2964724487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a28004d40 0x7f6a280051c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6a380013d0 tx=0x7f6a28005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=173958c68f01965a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:59.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.253+0000 7f6a377fe640 1 -- 172.21.15.72:0/2964724487 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3800a4e0 con 0x7f6a28004d40 2024-04-15T15:14:59.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.253+0000 7f6a377fe640 1 -- 172.21.15.72:0/2964724487 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a38034650 con 0x7f6a28004d40 2024-04-15T15:14:59.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.253+0000 7f6a377fe640 1 -- 172.21.15.72:0/2964724487 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a38034a40 con 0x7f6a28004d40 2024-04-15T15:14:59.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.253+0000 7f6a46f00640 1 -- 172.21.15.72:0/2964724487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a28004d40 msgr2=0x7f6a280051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:59.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.253+0000 7f6a46f00640 1 --2- 172.21.15.72:0/2964724487 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a28004d40 0x7f6a280051c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6a380013d0 tx=0x7f6a28005830 comp rx=0 tx=0).stop 2024-04-15T15:14:59.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.254+0000 7f6a4549d640 1 -- 172.21.15.72:0/2964724487 reap_dead start 2024-04-15T15:14:59.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.254+0000 7f6a46f00640 1 -- 172.21.15.72:0/2964724487 shutdown_connections 2024-04-15T15:14:59.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.254+0000 7f6a46f00640 1 -- 172.21.15.72:0/2964724487 >> 172.21.15.72:0/2964724487 conn(0x7f6a400b31d0 msgr2=0x7f6a400b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:59.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.254+0000 7f6a46f00640 1 -- 172.21.15.72:0/2964724487 shutdown_connections 2024-04-15T15:14:59.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.254+0000 7f6a46f00640 1 -- 172.21.15.72:0/2964724487 wait complete. 2024-04-15T15:14:59.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.255+0000 7f6a46f00640 1 Processor -- start 2024-04-15T15:14:59.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.255+0000 7f6a46f00640 1 -- start start 2024-04-15T15:14:59.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.256+0000 7f6a46f00640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6a28004d40 0x7f6a400cd680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:59.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.256+0000 7f6a46f00640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6a4009a1c0 0x7f6a40146e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:59.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.256+0000 7f6a46f00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a400b1fc0 0x7f6a400b2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:59.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.256+0000 7f6a46f00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6a40155d10 con 0x7f6a400b1fc0 2024-04-15T15:14:59.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.256+0000 7f6a46f00640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4015ca70 con 0x7f6a28004d40 2024-04-15T15:14:59.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.256+0000 7f6a46f00640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f6a40155e90 con 0x7f6a4009a1c0 2024-04-15T15:14:59.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.256+0000 7f6a4549d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a400b1fc0 0x7f6a400b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:59.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.256+0000 7f6a4549d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a400b1fc0 0x7f6a400b2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45160/0 (socket says 172.21.15.72:45160) 2024-04-15T15:14:59.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.256+0000 7f6a4549d640 1 -- 172.21.15.72:0/1948800733 learned_addr learned my addr 172.21.15.72:0/1948800733 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:59.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.256+0000 7f6a44c9c640 1 --2- 172.21.15.72:0/1948800733 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6a28004d40 0x7f6a400cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:59.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.256+0000 7f6a37fff640 1 --2- 172.21.15.72:0/1948800733 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6a4009a1c0 0x7f6a40146e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:59.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.256+0000 7f6a4549d640 1 -- 172.21.15.72:0/1948800733 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6a4009a1c0 msgr2=0x7f6a40146e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:59.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.256+0000 7f6a4549d640 1 --2- 172.21.15.72:0/1948800733 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6a4009a1c0 0x7f6a40146e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.256+0000 7f6a4549d640 1 -- 172.21.15.72:0/1948800733 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6a28004d40 msgr2=0x7f6a400cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:59.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.256+0000 7f6a4549d640 1 --2- 172.21.15.72:0/1948800733 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6a28004d40 0x7f6a400cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.256+0000 7f6a4549d640 1 -- 172.21.15.72:0/1948800733 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a40147380 con 0x7f6a400b1fc0 2024-04-15T15:14:59.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.256+0000 7f6a44c9c640 1 --2- 172.21.15.72:0/1948800733 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6a28004d40 0x7f6a400cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:14:59.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.256+0000 7f6a4549d640 1 --2- 172.21.15.72:0/1948800733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a400b1fc0 0x7f6a400b2440 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6a3800a0b0 tx=0x7f6a38001360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:59.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.256+0000 7f6a357fa640 1 -- 172.21.15.72:0/1948800733 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a38043070 con 0x7f6a400b1fc0 2024-04-15T15:14:59.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.256+0000 7f6a357fa640 1 -- 172.21.15.72:0/1948800733 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a380019b0 con 0x7f6a400b1fc0 2024-04-15T15:14:59.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.256+0000 7f6a357fa640 1 -- 172.21.15.72:0/1948800733 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a38039040 con 0x7f6a400b1fc0 2024-04-15T15:14:59.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.256+0000 7f6a46f00640 1 -- 172.21.15.72:0/1948800733 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a4009a6f0 con 0x7f6a400b1fc0 2024-04-15T15:14:59.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.257+0000 7f6a46f00640 1 -- 172.21.15.72:0/1948800733 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a401446b0 con 0x7f6a400b1fc0 2024-04-15T15:14:59.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.258+0000 7f6a357fa640 1 -- 172.21.15.72:0/1948800733 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6a38046070 con 0x7f6a400b1fc0 2024-04-15T15:14:59.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.259+0000 7f6a357fa640 1 --2- 172.21.15.72:0/1948800733 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6a2c06c920 0x7f6a2c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:59.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.259+0000 7f6a44c9c640 1 --2- 172.21.15.72:0/1948800733 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6a2c06c920 0x7f6a2c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:59.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.259+0000 7f6a357fa640 1 -- 172.21.15.72:0/1948800733 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f6a380ad140 con 0x7f6a400b1fc0 2024-04-15T15:14:59.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.259+0000 7f6a44c9c640 1 --2- 172.21.15.72:0/1948800733 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6a2c06c920 0x7f6a2c06ede0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6a401563e0 tx=0x7f6a3c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:59.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.260+0000 7f6a49bc1740 1 Processor -- start 2024-04-15T15:14:59.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.260+0000 7f6a49bc1740 1 -- start start 2024-04-15T15:14:59.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.260+0000 7f6a49bc1740 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55cc6fb0fe10 0x55cc6fae2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:59.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.260+0000 7f6a49bc1740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cc6fb101f0 0x55cc6fb12ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:59.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.260+0000 7f6a49bc1740 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55cc6fae32f0 0x55cc6fb130e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:59.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.260+0000 7f6a49bc1740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55cc6fa4d9c0 con 0x55cc6fb101f0 2024-04-15T15:14:59.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.260+0000 7f6a49bc1740 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x55cc6f9f4650 con 0x55cc6fb0fe10 2024-04-15T15:14:59.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.260+0000 7f6a49bc1740 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x55cc6fa49980 con 0x55cc6fae32f0 2024-04-15T15:14:59.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.260+0000 7f6a37fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cc6fb101f0 0x55cc6fb12ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:59.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.260+0000 7f6a37fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cc6fb101f0 0x55cc6fb12ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45176/0 (socket says 172.21.15.72:45176) 2024-04-15T15:14:59.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.260+0000 7f6a37fff640 1 -- 172.21.15.72:0/3819897623 learned_addr learned my addr 172.21.15.72:0/3819897623 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:59.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.261+0000 7f6a44c9c640 1 --2- 172.21.15.72:0/3819897623 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55cc6fb0fe10 0x55cc6fae2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:59.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.261+0000 7f6a4549d640 1 --2- 172.21.15.72:0/3819897623 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55cc6fae32f0 0x55cc6fb130e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:59.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.261+0000 7f6a37fff640 1 -- 172.21.15.72:0/3819897623 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55cc6fae32f0 msgr2=0x55cc6fb130e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:59.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.261+0000 7f6a37fff640 1 --2- 172.21.15.72:0/3819897623 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55cc6fae32f0 0x55cc6fb130e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.261+0000 7f6a37fff640 1 -- 172.21.15.72:0/3819897623 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55cc6fb0fe10 msgr2=0x55cc6fae2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:59.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.261+0000 7f6a37fff640 1 --2- 172.21.15.72:0/3819897623 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55cc6fb0fe10 0x55cc6fae2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.261+0000 7f6a37fff640 1 -- 172.21.15.72:0/3819897623 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc6f7602d0 con 0x55cc6fb101f0 2024-04-15T15:14:59.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.261+0000 7f6a44c9c640 1 --2- 172.21.15.72:0/3819897623 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55cc6fb0fe10 0x55cc6fae2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:14:59.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.261+0000 7f6a4549d640 1 --2- 172.21.15.72:0/3819897623 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55cc6fae32f0 0x55cc6fb130e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:14:59.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.261+0000 7f6a37fff640 1 --2- 172.21.15.72:0/3819897623 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cc6fb101f0 0x55cc6fb12ba0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6a30009a50 tx=0x7f6a30009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4fbd3548df20a59 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:59.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.261+0000 7f6a26ffd640 1 -- 172.21.15.72:0/3819897623 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3001b020 con 0x55cc6fb101f0 2024-04-15T15:14:59.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.261+0000 7f6a26ffd640 1 -- 172.21.15.72:0/3819897623 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a30004300 con 0x55cc6fb101f0 2024-04-15T15:14:59.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.261+0000 7f6a26ffd640 1 -- 172.21.15.72:0/3819897623 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a30013bb0 con 0x55cc6fb101f0 2024-04-15T15:14:59.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.261+0000 7f6a49bc1740 1 -- 172.21.15.72:0/3819897623 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cc6fb101f0 msgr2=0x55cc6fb12ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:59.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.261+0000 7f6a49bc1740 1 --2- 172.21.15.72:0/3819897623 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cc6fb101f0 0x55cc6fb12ba0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6a30009a50 tx=0x7f6a30009190 comp rx=0 tx=0).stop 2024-04-15T15:14:59.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.262+0000 7f6a49bc1740 1 -- 172.21.15.72:0/3819897623 shutdown_connections 2024-04-15T15:14:59.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.262+0000 7f6a49bc1740 1 --2- 172.21.15.72:0/3819897623 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55cc6fae32f0 0x55cc6fb130e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.262+0000 7f6a49bc1740 1 --2- 172.21.15.72:0/3819897623 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cc6fb101f0 0x55cc6fb12ba0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.262+0000 7f6a49bc1740 1 --2- 172.21.15.72:0/3819897623 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55cc6fb0fe10 0x55cc6fae2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.262+0000 7f6a49bc1740 1 -- 172.21.15.72:0/3819897623 >> 172.21.15.72:0/3819897623 conn(0x55cc6fae5020 msgr2=0x55cc6fb2dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:59.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.262+0000 7f6a49bc1740 1 -- 172.21.15.72:0/3819897623 shutdown_connections 2024-04-15T15:14:59.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.262+0000 7f6a49bc1740 1 -- 172.21.15.72:0/3819897623 wait complete. 2024-04-15T15:14:59.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.262+0000 7f6a49bc1740 1 -- start start 2024-04-15T15:14:59.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.262+0000 7f6a49bc1740 1 Processor -- start 2024-04-15T15:14:59.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.263+0000 7f6a49bc1740 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55cc6fb0fe10 0x55cc6fb011d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:59.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.263+0000 7f6a49bc1740 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55cc6fb101f0 0x55cc6fb253f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:59.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.263+0000 7f6a49bc1740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cc6fb27ca0 0x55cc6fb270d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:59.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.263+0000 7f6a49bc1740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55cc6fae75c0 con 0x55cc6fb27ca0 2024-04-15T15:14:59.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.263+0000 7f6a49bc1740 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x55cc6fa07a10 con 0x55cc6fb101f0 2024-04-15T15:14:59.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.263+0000 7f6a49bc1740 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x55cc6fae69b0 con 0x55cc6fb0fe10 2024-04-15T15:14:59.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.263+0000 7f6a49bc1740 10 client.0 _ll_register_callbacks cb 0x55cc6fa43250 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:14:59.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.263+0000 7f6a4549d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cc6fb27ca0 0x55cc6fb270d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:59.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.263+0000 7f6a4549d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cc6fb27ca0 0x55cc6fb270d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45180/0 (socket says 172.21.15.72:45180) 2024-04-15T15:14:59.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.263+0000 7f6a4549d640 1 -- 172.21.15.72:0/1337674760 learned_addr learned my addr 172.21.15.72:0/1337674760 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:59.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.263+0000 7f6a44c9c640 1 --2- 172.21.15.72:0/1337674760 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55cc6fb0fe10 0x55cc6fb011d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:59.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.263+0000 7f6a37fff640 1 --2- 172.21.15.72:0/1337674760 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55cc6fb101f0 0x55cc6fb253f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:59.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.263+0000 7f6a4549d640 1 -- 172.21.15.72:0/1337674760 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55cc6fb0fe10 msgr2=0x55cc6fb011d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:59.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.263+0000 7f6a4549d640 1 --2- 172.21.15.72:0/1337674760 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55cc6fb0fe10 0x55cc6fb011d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.263+0000 7f6a4549d640 1 -- 172.21.15.72:0/1337674760 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55cc6fb101f0 msgr2=0x55cc6fb253f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:59.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.263+0000 7f6a4549d640 1 --2- 172.21.15.72:0/1337674760 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55cc6fb101f0 0x55cc6fb253f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.263+0000 7f6a4549d640 1 -- 172.21.15.72:0/1337674760 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc6f82a6b0 con 0x55cc6fb27ca0 2024-04-15T15:14:59.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.264+0000 7f6a4549d640 1 --2- 172.21.15.72:0/1337674760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cc6fb27ca0 0x55cc6fb270d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6a38077050 tx=0x7f6a3807acb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89fa6c2807be14a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:59.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.264+0000 7f6a1f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-04-15T15:14:59.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.264+0000 7f6a1f7fe640 1 -- 172.21.15.72:0/1337674760 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3809a0c0 con 0x55cc6fb27ca0 2024-04-15T15:14:59.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.264+0000 7f6a1f7fe640 1 -- 172.21.15.72:0/1337674760 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f6a38077260 con 0x55cc6fb27ca0 2024-04-15T15:14:59.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.264+0000 7f6a49bc1740 1 -- 172.21.15.72:0/1337674760 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a30004300 con 0x55cc6fb27ca0 2024-04-15T15:14:59.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.264+0000 7f6a1f7fe640 1 -- 172.21.15.72:0/1337674760 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7f6a3807a040 con 0x55cc6fb27ca0 2024-04-15T15:14:59.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.264+0000 7f6a49bc1740 10 client.4452 fetch_fsmap learned FSMap version 7 2024-04-15T15:14:59.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.264+0000 7f6a49bc1740 1 -- 172.21.15.72:0/1337674760 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x55cc6fb26910 con 0x55cc6fb27ca0 2024-04-15T15:14:59.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.264+0000 7f6a1f7fe640 1 -- 172.21.15.72:0/1337674760 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a30004300 con 0x55cc6fb27ca0 2024-04-15T15:14:59.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.264+0000 7f6a1f7fe640 1 -- 172.21.15.72:0/1337674760 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 7) ==== 3502+0+0 (secure 0 0 0) 0x7f6a3807a1e0 con 0x55cc6fb27ca0 2024-04-15T15:14:59.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.264+0000 7f6a49bc1740 10 client.4452 fetch_fsmap finished waiting for FSMap version 7 2024-04-15T15:14:59.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.264+0000 7f6a49bc1740 10 client.4452 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-15T15:14:59.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.264+0000 7f6a49bc1740 20 client.4452 populate_metadata read hostname 'smithi072' 2024-04-15T15:14:59.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.264+0000 7f6a49bc1740 1 --2- 172.21.15.72:0/1337674760 >> [v2:172.21.15.72:6835/3458067804,v1:172.21.15.72:6837/3458067804] conn(0x55cc6fb151e0 0x55cc6fb155c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:59.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.264+0000 7f6a49bc1740 4 client.4452 mds_command: new command op to 4299 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-15T15:14:59.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.264+0000 7f6a49bc1740 1 -- 172.21.15.72:0/1337674760 --> [v2:172.21.15.72:6835/3458067804,v1:172.21.15.72:6837/3458067804] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cc6fa5b3e0 con 0x55cc6fb151e0 2024-04-15T15:14:59.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.264+0000 7f6a44c9c640 1 --2- 172.21.15.72:0/1337674760 >> [v2:172.21.15.72:6835/3458067804,v1:172.21.15.72:6837/3458067804] conn(0x55cc6fb151e0 0x55cc6fb155c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:59.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.265+0000 7f6a1f7fe640 10 client.4452 ms_handle_connect on v2:172.21.15.72:6835/3458067804 2024-04-15T15:14:59.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.265+0000 7f6a44c9c640 1 --2- 172.21.15.72:0/1337674760 >> [v2:172.21.15.72:6835/3458067804,v1:172.21.15.72:6837/3458067804] conn(0x55cc6fb151e0 0x55cc6fb155c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99fbbc43d41da126 server_cookie=6c7e94f222d09b7c in_seq=0 out_seq=0 2024-04-15T15:14:59.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.265+0000 7f6a1f7fe640 1 -- 172.21.15.72:0/1337674760 <== mds.0 v2:172.21.15.72:6835/3458067804 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cc6fa5b3e0 con 0x55cc6fb151e0 2024-04-15T15:14:59.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.265+0000 7f6a1f7fe640 10 client.4452 handle_command_reply: tid=0 2024-04-15T15:14:59.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.265+0000 7f6a49bc1740 1 client.4452 shutdown 2024-04-15T15:14:59.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.265+0000 7f6a49bc1740 1 -- 172.21.15.72:0/1337674760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cc6fb27ca0 msgr2=0x55cc6fb270d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:59.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.265+0000 7f6a49bc1740 1 --2- 172.21.15.72:0/1337674760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cc6fb27ca0 0x55cc6fb270d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6a38077050 tx=0x7f6a3807acb0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.265+0000 7f6a49bc1740 1 -- 172.21.15.72:0/1337674760 shutdown_connections 2024-04-15T15:14:59.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.265+0000 7f6a49bc1740 1 --2- 172.21.15.72:0/1337674760 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cc6fb27ca0 0x55cc6fb270d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.265+0000 7f6a49bc1740 1 --2- 172.21.15.72:0/1337674760 >> [v2:172.21.15.72:6835/3458067804,v1:172.21.15.72:6837/3458067804] conn(0x55cc6fb151e0 0x55cc6fb155c0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.265+0000 7f6a49bc1740 1 --2- 172.21.15.72:0/1337674760 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55cc6fb101f0 0x55cc6fb253f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.265+0000 7f6a49bc1740 1 --2- 172.21.15.72:0/1337674760 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55cc6fb0fe10 0x55cc6fb011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.265+0000 7f6a49bc1740 1 -- 172.21.15.72:0/1337674760 >> 172.21.15.72:0/1337674760 conn(0x55cc6fae32f0 msgr2=0x55cc6fb12b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:59.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.265+0000 7f6a1f7fe640 0 client.4452 ms_handle_reset on v2:172.21.15.72:6835/3458067804 2024-04-15T15:14:59.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.265+0000 7f6a49bc1740 1 -- 172.21.15.72:0/1337674760 shutdown_connections 2024-04-15T15:14:59.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.265+0000 7f6a49bc1740 1 -- 172.21.15.72:0/1337674760 wait complete. 2024-04-15T15:14:59.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.266+0000 7f6a49bc1740 20 client.4452 trim_cache size 0 max 16384 2024-04-15T15:14:59.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.283+0000 7f6a49bc1740 1 Processor -- start 2024-04-15T15:14:59.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.283+0000 7f6a49bc1740 1 -- start start 2024-04-15T15:14:59.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.284+0000 7f6a49bc1740 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55cc6fb0fe10 0x55cc6fb00fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:59.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.284+0000 7f6a49bc1740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cc6fb101f0 0x55cc6fb276a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:59.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.284+0000 7f6a49bc1740 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55cc6fb27ca0 0x55cc6fb2e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:59.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.284+0000 7f6a49bc1740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6a401ff720 con 0x55cc6fb101f0 2024-04-15T15:14:59.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.284+0000 7f6a49bc1740 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f6a401460d0 con 0x55cc6fb0fe10 2024-04-15T15:14:59.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.284+0000 7f6a49bc1740 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f6a401ff0d0 con 0x55cc6fb27ca0 2024-04-15T15:14:59.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.284+0000 7f6a37fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cc6fb101f0 0x55cc6fb276a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:59.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.284+0000 7f6a37fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cc6fb101f0 0x55cc6fb276a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45194/0 (socket says 172.21.15.72:45194) 2024-04-15T15:14:59.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.284+0000 7f6a37fff640 1 -- 172.21.15.72:0/427552307 learned_addr learned my addr 172.21.15.72:0/427552307 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:59.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.284+0000 7f6a44c9c640 1 --2- 172.21.15.72:0/427552307 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55cc6fb0fe10 0x55cc6fb00fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:59.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.284+0000 7f6a37fff640 1 -- 172.21.15.72:0/427552307 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55cc6fb27ca0 msgr2=0x55cc6fb2e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:59.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.284+0000 7f6a37fff640 1 --2- 172.21.15.72:0/427552307 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55cc6fb27ca0 0x55cc6fb2e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.285+0000 7f6a37fff640 1 -- 172.21.15.72:0/427552307 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55cc6fb0fe10 msgr2=0x55cc6fb00fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:59.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.285+0000 7f6a37fff640 1 --2- 172.21.15.72:0/427552307 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55cc6fb0fe10 0x55cc6fb00fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.285+0000 7f6a37fff640 1 -- 172.21.15.72:0/427552307 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc6fb00740 con 0x55cc6fb101f0 2024-04-15T15:14:59.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.285+0000 7f6a44c9c640 1 --2- 172.21.15.72:0/427552307 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55cc6fb0fe10 0x55cc6fb00fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:14:59.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.285+0000 7f6a37fff640 1 --2- 172.21.15.72:0/427552307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cc6fb101f0 0x55cc6fb276a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6a30009a20 tx=0x7f6a30010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31c9b4a69bc021ff server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:59.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.285+0000 7f6a267fc640 1 -- 172.21.15.72:0/427552307 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a30030070 con 0x55cc6fb101f0 2024-04-15T15:14:59.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.285+0000 7f6a267fc640 1 -- 172.21.15.72:0/427552307 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a30004500 con 0x55cc6fb101f0 2024-04-15T15:14:59.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.285+0000 7f6a267fc640 1 -- 172.21.15.72:0/427552307 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3001b040 con 0x55cc6fb101f0 2024-04-15T15:14:59.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.285+0000 7f6a49bc1740 1 -- 172.21.15.72:0/427552307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cc6fb101f0 msgr2=0x55cc6fb276a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:59.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.285+0000 7f6a49bc1740 1 --2- 172.21.15.72:0/427552307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cc6fb101f0 0x55cc6fb276a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6a30009a20 tx=0x7f6a30010000 comp rx=0 tx=0).stop 2024-04-15T15:14:59.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.285+0000 7f6a49bc1740 1 -- 172.21.15.72:0/427552307 shutdown_connections 2024-04-15T15:14:59.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.285+0000 7f6a49bc1740 1 --2- 172.21.15.72:0/427552307 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55cc6fb27ca0 0x55cc6fb2e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.285+0000 7f6a49bc1740 1 --2- 172.21.15.72:0/427552307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cc6fb101f0 0x55cc6fb276a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.285+0000 7f6a49bc1740 1 --2- 172.21.15.72:0/427552307 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55cc6fb0fe10 0x55cc6fb00fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.285+0000 7f6a49bc1740 1 -- 172.21.15.72:0/427552307 >> 172.21.15.72:0/427552307 conn(0x55cc6fae32f0 msgr2=0x55cc6fae9dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:59.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.285+0000 7f6a49bc1740 1 -- 172.21.15.72:0/427552307 shutdown_connections 2024-04-15T15:14:59.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.285+0000 7f6a49bc1740 1 -- 172.21.15.72:0/427552307 wait complete. 2024-04-15T15:14:59.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.286+0000 7f6a49bc1740 1 -- start start 2024-04-15T15:14:59.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.286+0000 7f6a49bc1740 1 Processor -- start 2024-04-15T15:14:59.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.287+0000 7f6a49bc1740 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55cc6fb0fe10 0x55cc6fb15fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:59.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.287+0000 7f6a49bc1740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cc6fb101f0 0x55cc6fb24010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:59.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.287+0000 7f6a49bc1740 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55cc6fb27ca0 0x55cc6fb24550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:59.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.287+0000 7f6a49bc1740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55cc6fae75c0 con 0x55cc6fb101f0 2024-04-15T15:14:59.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.287+0000 7f6a49bc1740 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x55cc6fa07a10 con 0x55cc6fb0fe10 2024-04-15T15:14:59.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.287+0000 7f6a49bc1740 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x55cc6fb26110 con 0x55cc6fb27ca0 2024-04-15T15:14:59.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.287+0000 7f6a49bc1740 10 client.0 _ll_register_callbacks cb 0x7f6a3807a800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:14:59.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.287+0000 7f6a37fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cc6fb101f0 0x55cc6fb24010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:59.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.287+0000 7f6a37fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cc6fb101f0 0x55cc6fb24010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45200/0 (socket says 172.21.15.72:45200) 2024-04-15T15:14:59.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.287+0000 7f6a37fff640 1 -- 172.21.15.72:0/2522765263 learned_addr learned my addr 172.21.15.72:0/2522765263 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:59.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.287+0000 7f6a44c9c640 1 --2- 172.21.15.72:0/2522765263 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55cc6fb0fe10 0x55cc6fb15fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:59.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.287+0000 7f6a37fff640 1 -- 172.21.15.72:0/2522765263 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55cc6fb27ca0 msgr2=0x55cc6fb24550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:59.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.287+0000 7f6a37fff640 1 --2- 172.21.15.72:0/2522765263 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55cc6fb27ca0 0x55cc6fb24550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.287+0000 7f6a37fff640 1 -- 172.21.15.72:0/2522765263 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55cc6fb0fe10 msgr2=0x55cc6fb15fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:59.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.287+0000 7f6a37fff640 1 --2- 172.21.15.72:0/2522765263 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55cc6fb0fe10 0x55cc6fb15fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.287+0000 7f6a37fff640 1 -- 172.21.15.72:0/2522765263 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc6fb16500 con 0x55cc6fb101f0 2024-04-15T15:14:59.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.287+0000 7f6a44c9c640 1 --2- 172.21.15.72:0/2522765263 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55cc6fb0fe10 0x55cc6fb15fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:14:59.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.287+0000 7f6a37fff640 1 --2- 172.21.15.72:0/2522765263 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cc6fb101f0 0x55cc6fb24010 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6a3002e040 tx=0x7f6a300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=823f8fea2d06a8c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:59.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.287+0000 7f6a1f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-04-15T15:14:59.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.287+0000 7f6a1f7fe640 1 -- 172.21.15.72:0/2522765263 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a30030040 con 0x55cc6fb101f0 2024-04-15T15:14:59.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.287+0000 7f6a1f7fe640 1 -- 172.21.15.72:0/2522765263 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f6a30009200 con 0x55cc6fb101f0 2024-04-15T15:14:59.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.287+0000 7f6a49bc1740 1 -- 172.21.15.72:0/2522765263 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55cc6fa55250 con 0x55cc6fb101f0 2024-04-15T15:14:59.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.288+0000 7f6a1f7fe640 1 -- 172.21.15.72:0/2522765263 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) ==== 24+0+0 (secure 0 0 0) 0x7f6a30043050 con 0x55cc6fb101f0 2024-04-15T15:14:59.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.288+0000 7f6a49bc1740 10 client.4458 fetch_fsmap learned FSMap version 7 2024-04-15T15:14:59.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.288+0000 7f6a49bc1740 1 -- 172.21.15.72:0/2522765263 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) -- 0x55cc6fb23af0 con 0x55cc6fb101f0 2024-04-15T15:14:59.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.288+0000 7f6a1f7fe640 1 -- 172.21.15.72:0/2522765263 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x55cc6fa55250 con 0x55cc6fb101f0 2024-04-15T15:14:59.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.288+0000 7f6a1f7fe640 1 -- 172.21.15.72:0/2522765263 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 7) ==== 3502+0+0 (secure 0 0 0) 0x7f6a30046020 con 0x55cc6fb101f0 2024-04-15T15:14:59.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.288+0000 7f6a49bc1740 10 client.4458 fetch_fsmap finished waiting for FSMap version 7 2024-04-15T15:14:59.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.288+0000 7f6a49bc1740 10 client.4458 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-15T15:14:59.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.288+0000 7f6a49bc1740 20 client.4458 populate_metadata read hostname 'smithi072' 2024-04-15T15:14:59.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.288+0000 7f6a49bc1740 1 --2- 172.21.15.72:0/2522765263 >> [v2:172.21.15.72:6835/3458067804,v1:172.21.15.72:6837/3458067804] conn(0x55cc6fb3f9c0 0x55cc6fb3fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:59.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.288+0000 7f6a49bc1740 4 client.4458 mds_command: new command op to 4299 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-15T15:14:59.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.288+0000 7f6a49bc1740 1 -- 172.21.15.72:0/2522765263 --> [v2:172.21.15.72:6835/3458067804,v1:172.21.15.72:6837/3458067804] -- command(tid 0: {"prefix": "status"}) -- 0x55cc6f9d1910 con 0x55cc6fb3f9c0 2024-04-15T15:14:59.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.288+0000 7f6a44c9c640 1 --2- 172.21.15.72:0/2522765263 >> [v2:172.21.15.72:6835/3458067804,v1:172.21.15.72:6837/3458067804] conn(0x55cc6fb3f9c0 0x55cc6fb3fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:59.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.288+0000 7f6a1f7fe640 10 client.4458 ms_handle_connect on v2:172.21.15.72:6835/3458067804 2024-04-15T15:14:59.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.288+0000 7f6a44c9c640 1 --2- 172.21.15.72:0/2522765263 >> [v2:172.21.15.72:6835/3458067804,v1:172.21.15.72:6837/3458067804] conn(0x55cc6fb3f9c0 0x55cc6fb3fda0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c458e5a23f37a0b8 server_cookie=c022a86d4637cc14 in_seq=0 out_seq=0 2024-04-15T15:14:59.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.288+0000 7f6a1f7fe640 1 -- 172.21.15.72:0/2522765263 <== mds.0 v2:172.21.15.72:6835/3458067804 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x55cc6f9d1910 con 0x55cc6fb3f9c0 2024-04-15T15:14:59.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.288+0000 7f6a1f7fe640 10 client.4458 handle_command_reply: tid=0 2024-04-15T15:14:59.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.288+0000 7f6a49bc1740 1 client.4458 shutdown 2024-04-15T15:14:59.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.288+0000 7f6a49bc1740 1 -- 172.21.15.72:0/2522765263 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cc6fb101f0 msgr2=0x55cc6fb24010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:59.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.288+0000 7f6a49bc1740 1 --2- 172.21.15.72:0/2522765263 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cc6fb101f0 0x55cc6fb24010 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6a3002e040 tx=0x7f6a300046c0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.288+0000 7f6a49bc1740 1 -- 172.21.15.72:0/2522765263 shutdown_connections 2024-04-15T15:14:59.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.288+0000 7f6a49bc1740 1 --2- 172.21.15.72:0/2522765263 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55cc6fb27ca0 0x55cc6fb24550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.288+0000 7f6a49bc1740 1 --2- 172.21.15.72:0/2522765263 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55cc6fb101f0 0x55cc6fb24010 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.288+0000 7f6a49bc1740 1 --2- 172.21.15.72:0/2522765263 >> [v2:172.21.15.72:6835/3458067804,v1:172.21.15.72:6837/3458067804] conn(0x55cc6fb3f9c0 0x55cc6fb3fda0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.288+0000 7f6a49bc1740 1 --2- 172.21.15.72:0/2522765263 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55cc6fb0fe10 0x55cc6fb15fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.288+0000 7f6a1f7fe640 0 client.4458 ms_handle_reset on v2:172.21.15.72:6835/3458067804 2024-04-15T15:14:59.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.288+0000 7f6a49bc1740 1 -- 172.21.15.72:0/2522765263 >> 172.21.15.72:0/2522765263 conn(0x55cc6fae32f0 msgr2=0x55cc6fb2e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:59.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.289+0000 7f6a49bc1740 1 -- 172.21.15.72:0/2522765263 shutdown_connections 2024-04-15T15:14:59.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.289+0000 7f6a49bc1740 1 -- 172.21.15.72:0/2522765263 wait complete. 2024-04-15T15:14:59.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.289+0000 7f6a49bc1740 20 client.4458 trim_cache size 0 max 16384 2024-04-15T15:14:59.298 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-04-15T15:14:59.298 INFO:teuthology.orchestra.run.smithi072.stdout: "cluster_fsid": "05091e39-b18a-4545-8b68-c0ae4699e116", 2024-04-15T15:14:59.299 INFO:teuthology.orchestra.run.smithi072.stdout: "whoami": 0, 2024-04-15T15:14:59.299 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 4299, 2024-04-15T15:14:59.299 INFO:teuthology.orchestra.run.smithi072.stdout: "want_state": "up:active", 2024-04-15T15:14:59.299 INFO:teuthology.orchestra.run.smithi072.stdout: "state": "up:active", 2024-04-15T15:14:59.299 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs", 2024-04-15T15:14:59.299 INFO:teuthology.orchestra.run.smithi072.stdout: "rank_uptime": 7.2187390579999997, 2024-04-15T15:14:59.299 INFO:teuthology.orchestra.run.smithi072.stdout: "mdsmap_epoch": 7, 2024-04-15T15:14:59.299 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch": 24, 2024-04-15T15:14:59.299 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch_barrier": 24, 2024-04-15T15:14:59.299 INFO:teuthology.orchestra.run.smithi072.stdout: "uptime": 14.308456249000001 2024-04-15T15:14:59.299 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-04-15T15:14:59.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.289+0000 7f6a46f00640 1 -- 172.21.15.72:0/1948800733 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6a2c06c920 msgr2=0x7f6a2c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:59.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.289+0000 7f6a46f00640 1 --2- 172.21.15.72:0/1948800733 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6a2c06c920 0x7f6a2c06ede0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6a401563e0 tx=0x7f6a3c008040 comp rx=0 tx=0).stop 2024-04-15T15:14:59.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.289+0000 7f6a46f00640 1 -- 172.21.15.72:0/1948800733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a400b1fc0 msgr2=0x7f6a400b2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:59.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.289+0000 7f6a46f00640 1 --2- 172.21.15.72:0/1948800733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a400b1fc0 0x7f6a400b2440 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6a3800a0b0 tx=0x7f6a38001360 comp rx=0 tx=0).stop 2024-04-15T15:14:59.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.289+0000 7f6a46f00640 1 -- 172.21.15.72:0/1948800733 shutdown_connections 2024-04-15T15:14:59.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.289+0000 7f6a46f00640 1 --2- 172.21.15.72:0/1948800733 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6a2c06c920 0x7f6a2c06ede0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.289+0000 7f6a46f00640 1 --2- 172.21.15.72:0/1948800733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6a400b1fc0 0x7f6a400b2440 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.289+0000 7f6a46f00640 1 --2- 172.21.15.72:0/1948800733 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6a4009a1c0 0x7f6a40146e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.289+0000 7f6a46f00640 1 --2- 172.21.15.72:0/1948800733 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6a28004d40 0x7f6a400cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.289+0000 7f6a46f00640 1 -- 172.21.15.72:0/1948800733 >> 172.21.15.72:0/1948800733 conn(0x7f6a400b31d0 msgr2=0x7f6a4015e210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:59.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.289+0000 7f6a46f00640 1 -- 172.21.15.72:0/1948800733 shutdown_connections 2024-04-15T15:14:59.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.289+0000 7f6a46f00640 1 -- 172.21.15.72:0/1948800733 wait complete. 2024-04-15T15:14:59.302 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-15T15:14:59.302 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-15T15:14:59.303 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:14:59.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.404+0000 7f69eebf4640 1 Processor -- start 2024-04-15T15:14:59.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.404+0000 7f69eebf4640 1 -- start start 2024-04-15T15:14:59.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.405+0000 7f69eebf4640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f69e80d12f0 0x7f69e81618a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:59.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.405+0000 7f69eebf4640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f69e80c9570 0x7f69e8161de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:59.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.405+0000 7f69eebf4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69e80ccd40 con 0x7f69e80b68e0 2024-04-15T15:14:59.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.405+0000 7f69eebf4640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f69e80ccaf0 con 0x7f69e80d12f0 2024-04-15T15:14:59.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.405+0000 7f69eebf4640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f69e80cd020 con 0x7f69e80c9570 2024-04-15T15:14:59.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.405+0000 7f69ed191640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f69e80b68e0 0x7f69e80cdb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37242/0 (socket says 172.21.15.72:37242) 2024-04-15T15:14:59.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.405+0000 7f69ed191640 1 -- 172.21.15.72:0/964178947 learned_addr learned my addr 172.21.15.72:0/964178947 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:14:59.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.405+0000 7f69ec990640 1 --2- 172.21.15.72:0/964178947 >> v2:172.21.15.175:3300/0 conn(0x7f69e80d12f0 0x7f69e81618a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:59.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.405+0000 7f69e7fff640 1 --2- 172.21.15.72:0/964178947 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f69e80c9570 0x7f69e8161de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:59.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.405+0000 7f69e77fe640 1 -- 172.21.15.72:0/964178947 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3768385113 0 0) 0x7f69e80ccd40 con 0x7f69e80b68e0 2024-04-15T15:14:59.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.405+0000 7f69e77fe640 1 -- 172.21.15.72:0/964178947 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69d4003520 con 0x7f69e80b68e0 2024-04-15T15:14:59.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.406+0000 7f69e77fe640 1 -- 172.21.15.72:0/964178947 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3836736364 0 0) 0x7f69d4003520 con 0x7f69e80b68e0 2024-04-15T15:14:59.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.406+0000 7f69e77fe640 1 -- 172.21.15.72:0/964178947 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f69e80c9570 msgr2=0x7f69e8161de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:59.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.406+0000 7f69e77fe640 1 --2- 172.21.15.72:0/964178947 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f69e80c9570 0x7f69e8161de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.406+0000 7f69e77fe640 1 -- 172.21.15.72:0/964178947 >> v2:172.21.15.175:3300/0 conn(0x7f69e80d12f0 msgr2=0x7f69e81618a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:59.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.406+0000 7f69e77fe640 1 --2- 172.21.15.72:0/964178947 >> v2:172.21.15.175:3300/0 conn(0x7f69e80d12f0 0x7f69e81618a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.406+0000 7f69e77fe640 1 -- 172.21.15.72:0/964178947 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69e80cd300 con 0x7f69e80b68e0 2024-04-15T15:14:59.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.406+0000 7f69e77fe640 1 -- 172.21.15.72:0/964178947 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f69d0001670 con 0x7f69e80b68e0 2024-04-15T15:14:59.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.406+0000 7f69e77fe640 1 -- 172.21.15.72:0/964178947 >> v1:172.21.15.72:6789/0 conn(0x7f69e80b68e0 legacy=0x7f69e80cdb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:59.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.406+0000 7f69e77fe640 1 --2- 172.21.15.72:0/964178947 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69d40039b0 0x7f69d4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:59.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.406+0000 7f69e77fe640 1 --2- 172.21.15.72:0/964178947 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69d4004800 0x7f69d4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:59.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.406+0000 7f69e77fe640 1 --2- 172.21.15.72:0/964178947 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f69e80c9570 0x7f69d40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:59.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.406+0000 7f69e77fe640 1 -- 172.21.15.72:0/964178947 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f69e80ccaf0 con 0x7f69d4004800 2024-04-15T15:14:59.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.406+0000 7f69e77fe640 1 -- 172.21.15.72:0/964178947 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f69e80cd020 con 0x7f69d40039b0 2024-04-15T15:14:59.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.406+0000 7f69e77fe640 1 -- 172.21.15.72:0/964178947 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f69d400b9c0 con 0x7f69e80c9570 2024-04-15T15:14:59.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.406+0000 7f69e7fff640 1 --2- 172.21.15.72:0/964178947 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69d4004800 0x7f69d4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:59.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.406+0000 7f69ec990640 1 --2- 172.21.15.72:0/964178947 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69d40039b0 0x7f69d4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:59.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.406+0000 7f69ed191640 1 --2- 172.21.15.72:0/964178947 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f69e80c9570 0x7f69d40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:59.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.406+0000 7f69e7fff640 1 -- 172.21.15.72:0/964178947 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f69e80c9570 msgr2=0x7f69d40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:59.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.406+0000 7f69e7fff640 1 --2- 172.21.15.72:0/964178947 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f69e80c9570 0x7f69d40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.406+0000 7f69e7fff640 1 -- 172.21.15.72:0/964178947 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69d40039b0 msgr2=0x7f69d4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:59.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.406+0000 7f69e7fff640 1 --2- 172.21.15.72:0/964178947 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69d40039b0 0x7f69d4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.406+0000 7f69e7fff640 1 -- 172.21.15.72:0/964178947 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69d4003520 con 0x7f69d4004800 2024-04-15T15:14:59.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.406+0000 7f69ed191640 1 --2- 172.21.15.72:0/964178947 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f69e80c9570 0x7f69d40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:14:59.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.406+0000 7f69ec990640 1 --2- 172.21.15.72:0/964178947 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69d40039b0 0x7f69d4003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:14:59.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.407+0000 7f69e7fff640 1 --2- 172.21.15.72:0/964178947 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69d4004800 0x7f69d4004c80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f69d8009fc0 tx=0x7f69d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7cc91750f9458aa server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:59.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.407+0000 7f69e77fe640 1 -- 172.21.15.72:0/964178947 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d8039070 con 0x7f69d4004800 2024-04-15T15:14:59.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.407+0000 7f69e77fe640 1 -- 172.21.15.72:0/964178947 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69d802cc20 con 0x7f69d4004800 2024-04-15T15:14:59.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.407+0000 7f69e77fe640 1 -- 172.21.15.72:0/964178947 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d8009c10 con 0x7f69d4004800 2024-04-15T15:14:59.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.407+0000 7f69eebf4640 1 -- 172.21.15.72:0/964178947 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69d4004800 msgr2=0x7f69d4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:59.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.407+0000 7f69eebf4640 1 --2- 172.21.15.72:0/964178947 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69d4004800 0x7f69d4004c80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f69d8009fc0 tx=0x7f69d4005830 comp rx=0 tx=0).stop 2024-04-15T15:14:59.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.407+0000 7f69ed191640 1 -- 172.21.15.72:0/964178947 reap_dead start 2024-04-15T15:14:59.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.407+0000 7f69eebf4640 1 -- 172.21.15.72:0/964178947 shutdown_connections 2024-04-15T15:14:59.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.407+0000 7f69eebf4640 1 -- 172.21.15.72:0/964178947 >> 172.21.15.72:0/964178947 conn(0x7f69e80b3f90 msgr2=0x7f69e80b43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:59.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.407+0000 7f69eebf4640 1 -- 172.21.15.72:0/964178947 shutdown_connections 2024-04-15T15:14:59.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.407+0000 7f69eebf4640 1 -- 172.21.15.72:0/964178947 wait complete. 2024-04-15T15:14:59.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.409+0000 7f69eebf4640 1 Processor -- start 2024-04-15T15:14:59.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.409+0000 7f69eebf4640 1 -- start start 2024-04-15T15:14:59.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.409+0000 7f69eebf4640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f69d4004800 0x7f69e8157af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:59.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.409+0000 7f69eebf4640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69e81f4800 0x7f69e815c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:59.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.409+0000 7f69eebf4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69e81f4c80 0x7f69e815c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:59.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.409+0000 7f69eebf4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f69e80d2210 con 0x7f69e81f4c80 2024-04-15T15:14:59.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.409+0000 7f69eebf4640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f69e80d2090 con 0x7f69e81f4800 2024-04-15T15:14:59.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.409+0000 7f69eebf4640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f69e80d16d0 con 0x7f69d4004800 2024-04-15T15:14:59.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.410+0000 7f69ed191640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69e81f4c80 0x7f69e815c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:59.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.410+0000 7f69ed191640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69e81f4c80 0x7f69e815c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45226/0 (socket says 172.21.15.72:45226) 2024-04-15T15:14:59.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.410+0000 7f69ed191640 1 -- 172.21.15.72:0/313527343 learned_addr learned my addr 172.21.15.72:0/313527343 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:14:59.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.410+0000 7f69ec990640 1 --2- 172.21.15.72:0/313527343 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f69d4004800 0x7f69e8157af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:59.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.410+0000 7f69e7fff640 1 --2- 172.21.15.72:0/313527343 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69e81f4800 0x7f69e815c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:59.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.410+0000 7f69ed191640 1 -- 172.21.15.72:0/313527343 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f69d4004800 msgr2=0x7f69e8157af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:59.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.410+0000 7f69ed191640 1 --2- 172.21.15.72:0/313527343 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f69d4004800 0x7f69e8157af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.410+0000 7f69ed191640 1 -- 172.21.15.72:0/313527343 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69e81f4800 msgr2=0x7f69e815c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:59.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.410+0000 7f69ed191640 1 --2- 172.21.15.72:0/313527343 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69e81f4800 0x7f69e815c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.410+0000 7f69ed191640 1 -- 172.21.15.72:0/313527343 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69e815ac00 con 0x7f69e81f4c80 2024-04-15T15:14:59.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.410+0000 7f69ec990640 1 --2- 172.21.15.72:0/313527343 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f69d4004800 0x7f69e8157af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:14:59.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.410+0000 7f69e7fff640 1 --2- 172.21.15.72:0/313527343 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69e81f4800 0x7f69e815c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:14:59.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.410+0000 7f69ed191640 1 --2- 172.21.15.72:0/313527343 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69e81f4c80 0x7f69e815c8b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f69e80740f0 tx=0x7f69d000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:59.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.410+0000 7f69e57fa640 1 -- 172.21.15.72:0/313527343 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d000c5b0 con 0x7f69e81f4c80 2024-04-15T15:14:59.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.410+0000 7f69e57fa640 1 -- 172.21.15.72:0/313527343 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69d0018070 con 0x7f69e81f4c80 2024-04-15T15:14:59.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.410+0000 7f69eebf4640 1 -- 172.21.15.72:0/313527343 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69e8158510 con 0x7f69e81f4c80 2024-04-15T15:14:59.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.410+0000 7f69e57fa640 1 -- 172.21.15.72:0/313527343 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69d000c9b0 con 0x7f69e81f4c80 2024-04-15T15:14:59.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.410+0000 7f69eebf4640 1 -- 172.21.15.72:0/313527343 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69e8158ad0 con 0x7f69e81f4c80 2024-04-15T15:14:59.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.411+0000 7f69eebf4640 1 -- 172.21.15.72:0/313527343 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69e80689e0 con 0x7f69e81f4c80 2024-04-15T15:14:59.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.412+0000 7f69e57fa640 1 -- 172.21.15.72:0/313527343 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f69d000cb50 con 0x7f69e81f4c80 2024-04-15T15:14:59.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.412+0000 7f69e57fa640 1 --2- 172.21.15.72:0/313527343 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f69c806c920 0x7f69c806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:59.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.412+0000 7f69ec990640 1 --2- 172.21.15.72:0/313527343 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f69c806c920 0x7f69c806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:59.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.412+0000 7f69e57fa640 1 -- 172.21.15.72:0/313527343 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f69d008b940 con 0x7f69e81f4c80 2024-04-15T15:14:59.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.413+0000 7f69ec990640 1 --2- 172.21.15.72:0/313527343 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f69c806c920 0x7f69c806ede0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f69e0009bd0 tx=0x7f69e0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:59.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.417+0000 7f69e57fa640 1 -- 172.21.15.72:0/313527343 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f69e815ac00 con 0x7f69e81f4c80 2024-04-15T15:14:59.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.632+0000 7f69eebf4640 1 -- 172.21.15.72:0/313527343 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f69e80b2e70 con 0x7f69e81f4c80 2024-04-15T15:14:59.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.632+0000 7f69e57fa640 1 -- 172.21.15.72:0/313527343 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) ==== 53+0+88 (secure 0 0 0) 0x7f69d0059ac0 con 0x7f69e81f4c80 2024-04-15T15:14:59.634 INFO:teuthology.orchestra.run.smithi072.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-15T15:14:59.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.634+0000 7f69eebf4640 1 -- 172.21.15.72:0/313527343 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f69c806c920 msgr2=0x7f69c806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:59.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.634+0000 7f69eebf4640 1 --2- 172.21.15.72:0/313527343 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f69c806c920 0x7f69c806ede0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f69e0009bd0 tx=0x7f69e0008040 comp rx=0 tx=0).stop 2024-04-15T15:14:59.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.634+0000 7f69eebf4640 1 -- 172.21.15.72:0/313527343 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69e81f4c80 msgr2=0x7f69e815c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:59.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.634+0000 7f69eebf4640 1 --2- 172.21.15.72:0/313527343 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69e81f4c80 0x7f69e815c8b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f69e80740f0 tx=0x7f69d000eb20 comp rx=0 tx=0).stop 2024-04-15T15:14:59.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.634+0000 7f69eebf4640 1 -- 172.21.15.72:0/313527343 shutdown_connections 2024-04-15T15:14:59.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.634+0000 7f69eebf4640 1 --2- 172.21.15.72:0/313527343 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f69c806c920 0x7f69c806ede0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.634+0000 7f69eebf4640 1 --2- 172.21.15.72:0/313527343 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69e81f4c80 0x7f69e815c8b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.634+0000 7f69eebf4640 1 --2- 172.21.15.72:0/313527343 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69e81f4800 0x7f69e815c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.634+0000 7f69eebf4640 1 --2- 172.21.15.72:0/313527343 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f69d4004800 0x7f69e8157af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:59.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.634+0000 7f69eebf4640 1 -- 172.21.15.72:0/313527343 >> 172.21.15.72:0/313527343 conn(0x7f69e80b3f90 msgr2=0x7f69e80cbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:59.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.634+0000 7f69eebf4640 1 -- 172.21.15.72:0/313527343 shutdown_connections 2024-04-15T15:14:59.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59.634+0000 7f69eebf4640 1 -- 172.21.15.72:0/313527343 wait complete. 2024-04-15T15:14:59.647 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-15T15:14:59.647 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:14:59.647 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi072.front.sepia.ceph.com 2024-04-15T15:14:59.647 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-15T15:14:59.647 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-15T15:14:59.647 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-15T15:14:59.647 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-15T15:14:59.647 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:14:59.647 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-15T15:14:59.647 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:14:59.648 DEBUG:teuthology.orchestra.run.smithi072:> ip addr 2024-04-15T15:14:59.663 INFO:teuthology.orchestra.run.smithi072.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:14:59.664 INFO:teuthology.orchestra.run.smithi072.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:14:59.664 INFO:teuthology.orchestra.run.smithi072.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:14:59.664 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:14:59.664 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 ::1/128 scope host 2024-04-15T15:14:59.664 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:14:59.664 INFO:teuthology.orchestra.run.smithi072.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:14:59.664 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f4 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:14:59.664 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f0 2024-04-15T15:14:59.664 INFO:teuthology.orchestra.run.smithi072.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:14:59.664 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f5 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:14:59.664 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f1 2024-04-15T15:14:59.664 INFO:teuthology.orchestra.run.smithi072.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:14:59.664 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:64 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:14:59.664 INFO:teuthology.orchestra.run.smithi072.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:14:59.665 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:65 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:14:59.665 INFO:teuthology.orchestra.run.smithi072.stdout: inet 172.21.15.72/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-15T15:14:59.665 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft 42704sec preferred_lft 42704sec 2024-04-15T15:14:59.665 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::ec4:7aff:fe8f:b765/64 scope link 2024-04-15T15:14:59.665 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:14:59.665 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-15T15:14:59.666 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:14:59.666 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-04-15T15:14:59.666 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link add name ceph-brx type bridge 2024-04-15T15:14:59.666 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip addr flush dev ceph-brx 2024-04-15T15:14:59.666 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set ceph-brx up 2024-04-15T15:14:59.666 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-15T15:14:59.666 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-04-15T15:14:59.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:14:59.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:14:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:14:59.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:14:59.875 DEBUG:teuthology.orchestra.run.smithi072:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:14:59.904 INFO:teuthology.orchestra.run.smithi072.stdout:1 2024-04-15T15:14:59.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:14:59.905 DEBUG:teuthology.orchestra.run.smithi072:> route 2024-04-15T15:14:59.974 INFO:teuthology.orchestra.run.smithi072.stdout:Kernel IP routing table 2024-04-15T15:14:59.974 INFO:teuthology.orchestra.run.smithi072.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:14:59.974 INFO:teuthology.orchestra.run.smithi072.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-15T15:14:59.975 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-15T15:14:59.975 INFO:teuthology.orchestra.run.smithi072.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:14:59.975 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:14:59.975 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-04-15T15:14:59.975 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-15T15:14:59.975 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-15T15:14:59.975 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-15T15:14:59.975 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-04-15T15:15:00.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:00.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:00.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:00.282 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-04-15T15:15:00.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:00.298 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list-id 2024-04-15T15:15:00.353 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:00.353 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-04-15T15:15:00.353 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-15T15:15:00.353 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-15T15:15:00.353 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-04-15T15:15:00.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:00.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:00.478 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-15T15:15:00.478 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:00.478 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-04-15T15:15:00.478 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-15T15:15:00.478 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-15T15:15:00.478 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-15T15:15:00.478 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-15T15:15:00.479 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-15T15:15:00.479 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-04-15T15:15:00.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:00.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:00.617 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:00.617 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-04-15T15:15:00.617 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set brx.0 up 2024-04-15T15:15:00.617 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set dev brx.0 master ceph-brx 2024-04-15T15:15:00.617 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-04-15T15:15:00.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:00 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:00.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:00 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:00.701 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-15T15:15:00.701 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:15:00.701 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-15T15:15:00.716 INFO:teuthology.orchestra.run.smithi072.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-15T15:15:00.717 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:15:00.717 DEBUG:teuthology.orchestra.run.smithi072:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-15T15:15:00.773 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-04-15T15:15:00.843 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:15:00.900 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-04-15T15:15:00.900 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-04-15T15:15:00.900 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-04-15T15:15:00.900 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-04-15T15:15:00.900 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-04-15T15:15:00.900 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-04-15T15:15:00.900 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-04-15T15:15:00.900 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-04-15T15:15:00.901 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-04-15T15:15:00.901 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-04-15T15:15:00.901 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-04-15T15:15:00.901 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-04-15T15:15:00.901 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:15:00.901 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-04-15T15:15:00.901 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-04-15T15:15:00.901 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-04-15T15:15:00.901 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-04-15T15:15:00.901 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:15:00.901 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-04-15T15:15:00.901 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-04-15T15:15:00.901 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:15:00.901 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:15:00.901 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:15:00.902 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-04-15T15:15:00.902 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-04-15T15:15:00.902 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:15:00.902 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:15:00.902 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:15:00.902 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:15:00.902 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-04-15T15:15:00.902 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-15T15:15:00.902 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-15T15:15:00.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:00.902 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-04-15T15:15:00.956 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-15T15:15:00.956 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-15T15:15:00.999 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-04-15T15:15:01.028 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:15:01.086 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-04-15T15:15:01.086 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-04-15T15:15:01.086 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-04-15T15:15:01.086 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-04-15T15:15:01.086 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-04-15T15:15:01.086 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-04-15T15:15:01.086 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-04-15T15:15:01.086 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-04-15T15:15:01.086 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-04-15T15:15:01.086 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-04-15T15:15:01.086 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-04-15T15:15:01.087 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-04-15T15:15:01.087 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:15:01.087 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-04-15T15:15:01.087 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-04-15T15:15:01.087 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-04-15T15:15:01.087 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-04-15T15:15:01.087 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:15:01.087 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-04-15T15:15:01.087 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-04-15T15:15:01.087 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:15:01.087 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:15:01.087 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:15:01.087 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-04-15T15:15:01.087 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-04-15T15:15:01.088 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:15:01.088 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:15:01.088 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:15:01.088 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:15:01.088 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-04-15T15:15:01.088 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-15T15:15:01.088 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-15T15:15:01.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:01.088 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-04-15T15:15:01.090 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi072.stderr:2024-04-15T15:15:01.089+0000 7f162923a480 -1 init, newargv = 0x555c1c49f1d0 newargc=15 2024-04-15T15:15:01.090 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi072.stderr:ceph-fuse[44584]: starting ceph client 2024-04-15T15:15:01.096 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi072.stderr:ceph-fuse[44584]: starting fuse 2024-04-15T15:15:01.106 INFO:teuthology.orchestra.run.smithi072.stdout:43 2024-04-15T15:15:01.106 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-04-15T15:15:01.106 DEBUG:teuthology.orchestra.run.smithi072:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:15:01.106 DEBUG:teuthology.orchestra.run.smithi072:> import glob 2024-04-15T15:15:01.106 DEBUG:teuthology.orchestra.run.smithi072:> import re 2024-04-15T15:15:01.106 DEBUG:teuthology.orchestra.run.smithi072:> import os 2024-04-15T15:15:01.106 DEBUG:teuthology.orchestra.run.smithi072:> import subprocess 2024-04-15T15:15:01.106 DEBUG:teuthology.orchestra.run.smithi072:> 2024-04-15T15:15:01.106 DEBUG:teuthology.orchestra.run.smithi072:> def _find_admin_socket(client_name): 2024-04-15T15:15:01.107 DEBUG:teuthology.orchestra.run.smithi072:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-15T15:15:01.107 DEBUG:teuthology.orchestra.run.smithi072:> files = glob.glob(asok_path) 2024-04-15T15:15:01.107 DEBUG:teuthology.orchestra.run.smithi072:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-15T15:15:01.107 DEBUG:teuthology.orchestra.run.smithi072:> 2024-04-15T15:15:01.107 DEBUG:teuthology.orchestra.run.smithi072:> # Given a non-glob path, it better be there 2024-04-15T15:15:01.107 DEBUG:teuthology.orchestra.run.smithi072:> if "*" not in asok_path: 2024-04-15T15:15:01.107 DEBUG:teuthology.orchestra.run.smithi072:> assert(len(files) == 1) 2024-04-15T15:15:01.107 DEBUG:teuthology.orchestra.run.smithi072:> return files[0] 2024-04-15T15:15:01.107 DEBUG:teuthology.orchestra.run.smithi072:> 2024-04-15T15:15:01.107 DEBUG:teuthology.orchestra.run.smithi072:> for f in files: 2024-04-15T15:15:01.107 DEBUG:teuthology.orchestra.run.smithi072:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:15:01.107 DEBUG:teuthology.orchestra.run.smithi072:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:15:01.108 DEBUG:teuthology.orchestra.run.smithi072:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:15:01.108 DEBUG:teuthology.orchestra.run.smithi072:> contents = proc_f.read() 2024-04-15T15:15:01.108 DEBUG:teuthology.orchestra.run.smithi072:> if mountpoint in contents: 2024-04-15T15:15:01.108 DEBUG:teuthology.orchestra.run.smithi072:> return f 2024-04-15T15:15:01.108 DEBUG:teuthology.orchestra.run.smithi072:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:15:01.108 DEBUG:teuthology.orchestra.run.smithi072:> 2024-04-15T15:15:01.108 DEBUG:teuthology.orchestra.run.smithi072:> print(_find_admin_socket("client.admin")) 2024-04-15T15:15:01.108 DEBUG:teuthology.orchestra.run.smithi072:> ' 2024-04-15T15:15:01.229 INFO:teuthology.orchestra.run.smithi072.stdout:/var/run/ceph/ceph-client.admin.44584.asok 2024-04-15T15:15:01.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:01.239 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.44584.asok 2024-04-15T15:15:01.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:01.240 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.44584.asok status 2024-04-15T15:15:01.379 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-04-15T15:15:01.379 INFO:teuthology.orchestra.run.smithi072.stdout: "metadata": { 2024-04-15T15:15:01.379 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:15:01.379 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:15:01.380 INFO:teuthology.orchestra.run.smithi072.stdout: "entity_id": "admin", 2024-04-15T15:15:01.380 INFO:teuthology.orchestra.run.smithi072.stdout: "hostname": "smithi072", 2024-04-15T15:15:01.380 INFO:teuthology.orchestra.run.smithi072.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-15T15:15:01.380 INFO:teuthology.orchestra.run.smithi072.stdout: "pid": "44584", 2024-04-15T15:15:01.380 INFO:teuthology.orchestra.run.smithi072.stdout: "root": "/" 2024-04-15T15:15:01.380 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:15:01.380 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_count": 0, 2024-04-15T15:15:01.380 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_pinned_count": 0, 2024-04-15T15:15:01.380 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 4467, 2024-04-15T15:15:01.380 INFO:teuthology.orchestra.run.smithi072.stdout: "inst": { 2024-04-15T15:15:01.380 INFO:teuthology.orchestra.run.smithi072.stdout: "name": { 2024-04-15T15:15:01.380 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "client", 2024-04-15T15:15:01.380 INFO:teuthology.orchestra.run.smithi072.stdout: "num": 4467 2024-04-15T15:15:01.380 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:15:01.380 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-04-15T15:15:01.381 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-04-15T15:15:01.381 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:15:01.381 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 3471264184 2024-04-15T15:15:01.381 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:15:01.381 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:15:01.381 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-04-15T15:15:01.381 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-04-15T15:15:01.381 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:15:01.381 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 3471264184 2024-04-15T15:15:01.381 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:15:01.381 INFO:teuthology.orchestra.run.smithi072.stdout: "inst_str": "client.4467 192.168.0.1:0/3471264184", 2024-04-15T15:15:01.381 INFO:teuthology.orchestra.run.smithi072.stdout: "addr_str": "192.168.0.1:0/3471264184", 2024-04-15T15:15:01.381 INFO:teuthology.orchestra.run.smithi072.stdout: "inode_count": 1, 2024-04-15T15:15:01.381 INFO:teuthology.orchestra.run.smithi072.stdout: "mds_epoch": 7, 2024-04-15T15:15:01.382 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch": 24, 2024-04-15T15:15:01.382 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:15:01.382 INFO:teuthology.orchestra.run.smithi072.stdout: "blocklisted": false, 2024-04-15T15:15:01.382 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs" 2024-04-15T15:15:01.382 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-04-15T15:15:01.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:01.387 DEBUG:teuthology.orchestra.run.smithi072:> stat --file-system '--printf=%T 2024-04-15T15:15:01.388 DEBUG:teuthology.orchestra.run.smithi072:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-15T15:15:01.406 INFO:teuthology.orchestra.run.smithi072.stdout:fuseblk 2024-04-15T15:15:01.406 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-15T15:15:01.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:01.407 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-15T15:15:01.475 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-15T15:15:01.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-15T15:15:01.579 INFO:teuthology.orchestra.run.smithi072.stdout:# file: . 2024-04-15T15:15:01.579 INFO:teuthology.orchestra.run.smithi072.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-15T15:15:01.579 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:01.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:01.585 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:15:01.585 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:15:01.602 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:01.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.743+0000 7f63a333f640 1 Processor -- start 2024-04-15T15:15:01.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.743+0000 7f63a333f640 1 -- start start 2024-04-15T15:15:01.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.744+0000 7f63a333f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f639c0d1790 0x7f639c0b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:01.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.744+0000 7f63a333f640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f639c0b5790 0x7f639c0cd950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:01.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.744+0000 7f63a333f640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f639c15caf0 con 0x7f639c15c710 2024-04-15T15:15:01.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.744+0000 7f63a333f640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f639c06f510 con 0x7f639c0d1790 2024-04-15T15:15:01.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.744+0000 7f63a333f640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f639c0ccb90 con 0x7f639c0b5790 2024-04-15T15:15:01.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.744+0000 7f63a08da640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f639c0d1790 0x7f639c0b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:01.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.744+0000 7f63a10db640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f639c15c710 0x7f639c0d1080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:44988/0 (socket says 172.21.15.72:44988) 2024-04-15T15:15:01.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.744+0000 7f63a10db640 1 -- 172.21.15.72:0/899231736 learned_addr learned my addr 172.21.15.72:0/899231736 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:01.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.744+0000 7f63a08da640 1 -- 172.21.15.72:0/899231736 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f639c0b5790 msgr2=0x7f639c0cd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:01.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.744+0000 7f63a18dc640 1 --2- 172.21.15.72:0/899231736 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f639c0b5790 0x7f639c0cd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:01.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.744+0000 7f638ffff640 1 -- 172.21.15.72:0/899231736 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1962685687 0 0) 0x7f639c15caf0 con 0x7f639c15c710 2024-04-15T15:15:01.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.744+0000 7f63a08da640 1 --2- 172.21.15.72:0/899231736 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f639c0b5790 0x7f639c0cd950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:01.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.744+0000 7f63a08da640 1 -- 172.21.15.72:0/899231736 >> v1:172.21.15.72:6789/0 conn(0x7f639c15c710 legacy=0x7f639c0d1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:01.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.744+0000 7f63a08da640 1 -- 172.21.15.72:0/899231736 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f639c0cce70 con 0x7f639c0d1790 2024-04-15T15:15:01.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.744+0000 7f63a18dc640 1 --2- 172.21.15.72:0/899231736 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f639c0b5790 0x7f639c0cd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:01.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.745+0000 7f63a08da640 1 --2- 172.21.15.72:0/899231736 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f639c0d1790 0x7f639c0b66b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f63980097a0 tx=0x7f639802b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84f45e367adb732d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:01.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.745+0000 7f638ffff640 1 -- 172.21.15.72:0/899231736 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6398034070 con 0x7f639c0d1790 2024-04-15T15:15:01.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.745+0000 7f638ffff640 1 -- 172.21.15.72:0/899231736 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6398030800 con 0x7f639c0d1790 2024-04-15T15:15:01.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.745+0000 7f638ffff640 1 -- 172.21.15.72:0/899231736 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6398030bd0 con 0x7f639c0d1790 2024-04-15T15:15:01.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.745+0000 7f63a333f640 1 -- 172.21.15.72:0/899231736 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f639c0d1790 msgr2=0x7f639c0b66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:01.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.745+0000 7f63a333f640 1 --2- 172.21.15.72:0/899231736 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f639c0d1790 0x7f639c0b66b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f63980097a0 tx=0x7f639802b940 comp rx=0 tx=0).stop 2024-04-15T15:15:01.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.745+0000 7f63a333f640 1 -- 172.21.15.72:0/899231736 shutdown_connections 2024-04-15T15:15:01.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.745+0000 7f63a333f640 1 --2- 172.21.15.72:0/899231736 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f639c0b5790 0x7f639c0cd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:01.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.745+0000 7f63a333f640 1 --2- 172.21.15.72:0/899231736 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f639c0d1790 0x7f639c0b66b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:01.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.745+0000 7f63a333f640 1 -- 172.21.15.72:0/899231736 >> 172.21.15.72:0/899231736 conn(0x7f639c0b3f90 msgr2=0x7f639c0b43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:01.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.746+0000 7f63a333f640 1 -- 172.21.15.72:0/899231736 shutdown_connections 2024-04-15T15:15:01.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.746+0000 7f63a333f640 1 -- 172.21.15.72:0/899231736 wait complete. 2024-04-15T15:15:01.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.747+0000 7f63a333f640 1 Processor -- start 2024-04-15T15:15:01.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.747+0000 7f63a333f640 1 -- start start 2024-04-15T15:15:01.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.747+0000 7f63a333f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f639c0b5790 0x7f639c158c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:01.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.747+0000 7f63a333f640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f639c0d1790 0x7f639c15dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:01.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.747+0000 7f63a333f640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f639c15c710 0x7f639c15e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:01.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.747+0000 7f63a333f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f639c0d1cd0 con 0x7f639c0b5790 2024-04-15T15:15:01.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.747+0000 7f63a333f640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f639c0cd430 con 0x7f639c15c710 2024-04-15T15:15:01.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.747+0000 7f63a333f640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f639c0b6cd0 con 0x7f639c0d1790 2024-04-15T15:15:01.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.747+0000 7f63a10db640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f639c0b5790 0x7f639c158c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:01.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.747+0000 7f63a10db640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f639c0b5790 0x7f639c158c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44242/0 (socket says 172.21.15.72:44242) 2024-04-15T15:15:01.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.747+0000 7f63a10db640 1 -- 172.21.15.72:0/3834138508 learned_addr learned my addr 172.21.15.72:0/3834138508 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:01.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.748+0000 7f63a08da640 1 --2- 172.21.15.72:0/3834138508 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f639c0d1790 0x7f639c15dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:01.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.748+0000 7f63a18dc640 1 --2- 172.21.15.72:0/3834138508 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f639c15c710 0x7f639c15e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:01.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.748+0000 7f63a10db640 1 -- 172.21.15.72:0/3834138508 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f639c0d1790 msgr2=0x7f639c15dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:01.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.748+0000 7f63a10db640 1 --2- 172.21.15.72:0/3834138508 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f639c0d1790 0x7f639c15dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:01.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.748+0000 7f63a10db640 1 -- 172.21.15.72:0/3834138508 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f639c15c710 msgr2=0x7f639c15e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:01.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.748+0000 7f63a10db640 1 --2- 172.21.15.72:0/3834138508 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f639c15c710 0x7f639c15e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:01.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.748+0000 7f63a10db640 1 -- 172.21.15.72:0/3834138508 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f639c1591d0 con 0x7f639c0b5790 2024-04-15T15:15:01.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.748+0000 7f63a18dc640 1 --2- 172.21.15.72:0/3834138508 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f639c15c710 0x7f639c15e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:01.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.748+0000 7f63a10db640 1 --2- 172.21.15.72:0/3834138508 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f639c0b5790 0x7f639c158c90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f639000c500 tx=0x7f639000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:01.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.748+0000 7f638dffb640 1 -- 172.21.15.72:0/3834138508 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6390018070 con 0x7f639c0b5790 2024-04-15T15:15:01.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.748+0000 7f638dffb640 1 -- 172.21.15.72:0/3834138508 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6390007580 con 0x7f639c0b5790 2024-04-15T15:15:01.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.748+0000 7f63a333f640 1 -- 172.21.15.72:0/3834138508 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f639c0c9780 con 0x7f639c0b5790 2024-04-15T15:15:01.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.748+0000 7f638dffb640 1 -- 172.21.15.72:0/3834138508 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6390004a20 con 0x7f639c0b5790 2024-04-15T15:15:01.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.748+0000 7f63a333f640 1 -- 172.21.15.72:0/3834138508 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f639c0c9d40 con 0x7f639c0b5790 2024-04-15T15:15:01.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.749+0000 7f63a333f640 1 -- 172.21.15.72:0/3834138508 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f639c0689e0 con 0x7f639c0b5790 2024-04-15T15:15:01.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.750+0000 7f638dffb640 1 -- 172.21.15.72:0/3834138508 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f639001b070 con 0x7f639c0b5790 2024-04-15T15:15:01.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.750+0000 7f638dffb640 1 --2- 172.21.15.72:0/3834138508 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f638006c920 0x7f638006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:01.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.750+0000 7f638dffb640 1 -- 172.21.15.72:0/3834138508 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f639008b0b0 con 0x7f639c0b5790 2024-04-15T15:15:01.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.750+0000 7f63a08da640 1 --2- 172.21.15.72:0/3834138508 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f638006c920 0x7f638006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:01.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.751+0000 7f63a08da640 1 --2- 172.21.15.72:0/3834138508 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f638006c920 0x7f638006ede0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6398036000 tx=0x7f639803b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:01.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.755+0000 7f638dffb640 1 -- 172.21.15.72:0/3834138508 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6390059230 con 0x7f639c0b5790 2024-04-15T15:15:01.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.933+0000 7f63a333f640 1 -- 172.21.15.72:0/3834138508 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f639c15f620 con 0x7f639c0b5790 2024-04-15T15:15:01.936 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.935+0000 7f638dffb640 1 -- 172.21.15.72:0/3834138508 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7f6390058b30 con 0x7f639c0b5790 2024-04-15T15:15:01.937 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:01.937 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":24,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:14:56.451113+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:14:49.438366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:14:50.368379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:14:54.081157+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:01.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.937+0000 7f63a333f640 1 -- 172.21.15.72:0/3834138508 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f638006c920 msgr2=0x7f638006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:01.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.937+0000 7f63a333f640 1 --2- 172.21.15.72:0/3834138508 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f638006c920 0x7f638006ede0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6398036000 tx=0x7f639803b040 comp rx=0 tx=0).stop 2024-04-15T15:15:01.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.937+0000 7f63a333f640 1 -- 172.21.15.72:0/3834138508 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f639c0b5790 msgr2=0x7f639c158c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:01.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.937+0000 7f63a333f640 1 --2- 172.21.15.72:0/3834138508 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f639c0b5790 0x7f639c158c90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f639000c500 tx=0x7f639000c9c0 comp rx=0 tx=0).stop 2024-04-15T15:15:01.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.937+0000 7f63a333f640 1 -- 172.21.15.72:0/3834138508 shutdown_connections 2024-04-15T15:15:01.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.937+0000 7f63a333f640 1 --2- 172.21.15.72:0/3834138508 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f638006c920 0x7f638006ede0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:01.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.937+0000 7f63a333f640 1 --2- 172.21.15.72:0/3834138508 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f639c15c710 0x7f639c15e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:01.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.937+0000 7f63a333f640 1 --2- 172.21.15.72:0/3834138508 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f639c0d1790 0x7f639c15dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:01.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.937+0000 7f63a333f640 1 --2- 172.21.15.72:0/3834138508 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f639c0b5790 0x7f639c158c90 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:01.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.937+0000 7f63a333f640 1 -- 172.21.15.72:0/3834138508 >> 172.21.15.72:0/3834138508 conn(0x7f639c0b3f90 msgr2=0x7f639c06f8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:01.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.937+0000 7f63a333f640 1 -- 172.21.15.72:0/3834138508 shutdown_connections 2024-04-15T15:15:01.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:01.937+0000 7f63a333f640 1 -- 172.21.15.72:0/3834138508 wait complete. 2024-04-15T15:15:01.949 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:39.359102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:14:49.438366+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:14:50.368379+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:14:54.081157+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:15:01.950 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:02.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.055+0000 7efee72ba640 1 Processor -- start 2024-04-15T15:15:02.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.055+0000 7efee72ba640 1 -- start start 2024-04-15T15:15:02.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.056+0000 7efee72ba640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efee006c1f0 0x7efee006ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:02.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.056+0000 7efee72ba640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efee006f1a0 0x7efee00cdc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:02.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.056+0000 7efee72ba640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efee006d720 con 0x7efee0145f10 2024-04-15T15:15:02.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.056+0000 7efee72ba640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7efee01462f0 con 0x7efee006f1a0 2024-04-15T15:15:02.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.056+0000 7efee72ba640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7efee006d290 con 0x7efee006c1f0 2024-04-15T15:15:02.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.056+0000 7efee5857640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efee006f1a0 0x7efee00cdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:02.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.056+0000 7efee5056640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7efee0145f10 0x7efee00d0580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:44992/0 (socket says 172.21.15.72:44992) 2024-04-15T15:15:02.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.056+0000 7efee5056640 1 -- 172.21.15.72:0/2718530111 learned_addr learned my addr 172.21.15.72:0/2718530111 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:02.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.056+0000 7efee4855640 1 --2- 172.21.15.72:0/2718530111 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efee006c1f0 0x7efee006ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:02.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.056+0000 7efee5857640 1 -- 172.21.15.72:0/2718530111 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efee006c1f0 msgr2=0x7efee006ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:02.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.056+0000 7efee5857640 1 --2- 172.21.15.72:0/2718530111 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efee006c1f0 0x7efee006ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:02.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.056+0000 7efee5857640 1 -- 172.21.15.72:0/2718530111 >> v1:172.21.15.72:6789/0 conn(0x7efee0145f10 legacy=0x7efee00d0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:02.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.056+0000 7efed3fff640 1 -- 172.21.15.72:0/2718530111 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3816348647 0 0) 0x7efee006d720 con 0x7efee0145f10 2024-04-15T15:15:02.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.056+0000 7efee5857640 1 -- 172.21.15.72:0/2718530111 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efee00d0c90 con 0x7efee006f1a0 2024-04-15T15:15:02.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.057+0000 7efee5857640 1 --2- 172.21.15.72:0/2718530111 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efee006f1a0 0x7efee00cdc10 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7efed8009d90 tx=0x7efed802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3103c91b1b4142 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:02.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.057+0000 7efed3fff640 1 -- 172.21.15.72:0/2718530111 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed8039070 con 0x7efee006f1a0 2024-04-15T15:15:02.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.057+0000 7efed3fff640 1 -- 172.21.15.72:0/2718530111 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efed802cc50 con 0x7efee006f1a0 2024-04-15T15:15:02.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.057+0000 7efed3fff640 1 -- 172.21.15.72:0/2718530111 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed8034440 con 0x7efee006f1a0 2024-04-15T15:15:02.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.057+0000 7efee72ba640 1 -- 172.21.15.72:0/2718530111 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efee006f1a0 msgr2=0x7efee00cdc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:02.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.057+0000 7efee72ba640 1 --2- 172.21.15.72:0/2718530111 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efee006f1a0 0x7efee00cdc10 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7efed8009d90 tx=0x7efed802bea0 comp rx=0 tx=0).stop 2024-04-15T15:15:02.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.057+0000 7efee72ba640 1 -- 172.21.15.72:0/2718530111 shutdown_connections 2024-04-15T15:15:02.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.058+0000 7efee72ba640 1 --2- 172.21.15.72:0/2718530111 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efee006f1a0 0x7efee00cdc10 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:02.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.058+0000 7efee72ba640 1 --2- 172.21.15.72:0/2718530111 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efee006c1f0 0x7efee006ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:02.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.058+0000 7efee72ba640 1 -- 172.21.15.72:0/2718530111 >> 172.21.15.72:0/2718530111 conn(0x7efee00d13a0 msgr2=0x7efee00d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:02.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.058+0000 7efee72ba640 1 -- 172.21.15.72:0/2718530111 shutdown_connections 2024-04-15T15:15:02.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.058+0000 7efee72ba640 1 -- 172.21.15.72:0/2718530111 wait complete. 2024-04-15T15:15:02.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.059+0000 7efee72ba640 1 Processor -- start 2024-04-15T15:15:02.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.059+0000 7efee72ba640 1 -- start start 2024-04-15T15:15:02.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.059+0000 7efee72ba640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efee006f1a0 0x7efee015c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:02.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.059+0000 7efee72ba640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efee0145f10 0x7efee013f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:02.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.059+0000 7efee72ba640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efee013f600 0x7efee013c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:02.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.059+0000 7efee72ba640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7efee006cc40 con 0x7efee013f600 2024-04-15T15:15:02.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.059+0000 7efee72ba640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7efee006c960 con 0x7efee0145f10 2024-04-15T15:15:02.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.059+0000 7efee72ba640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7efee006cf20 con 0x7efee006f1a0 2024-04-15T15:15:02.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.060+0000 7efee5857640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efee013f600 0x7efee013c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:02.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.060+0000 7efee5857640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efee013f600 0x7efee013c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44264/0 (socket says 172.21.15.72:44264) 2024-04-15T15:15:02.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.060+0000 7efee5857640 1 -- 172.21.15.72:0/2854798820 learned_addr learned my addr 172.21.15.72:0/2854798820 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:02.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.060+0000 7efee5056640 1 --2- 172.21.15.72:0/2854798820 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efee006f1a0 0x7efee015c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:02.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.060+0000 7efee4855640 1 --2- 172.21.15.72:0/2854798820 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efee0145f10 0x7efee013f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:02.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.060+0000 7efee5857640 1 -- 172.21.15.72:0/2854798820 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efee006f1a0 msgr2=0x7efee015c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:02.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.060+0000 7efee5857640 1 --2- 172.21.15.72:0/2854798820 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efee006f1a0 0x7efee015c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:02.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.060+0000 7efee5857640 1 -- 172.21.15.72:0/2854798820 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efee0145f10 msgr2=0x7efee013f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:02.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.060+0000 7efee5857640 1 --2- 172.21.15.72:0/2854798820 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efee0145f10 0x7efee013f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:02.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.060+0000 7efee5857640 1 -- 172.21.15.72:0/2854798820 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efee013b570 con 0x7efee013f600 2024-04-15T15:15:02.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.060+0000 7efee5857640 1 --2- 172.21.15.72:0/2854798820 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efee013f600 0x7efee013c8b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7efed8009d90 tx=0x7efed8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:02.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.060+0000 7efed1ffb640 1 -- 172.21.15.72:0/2854798820 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed8039050 con 0x7efee013f600 2024-04-15T15:15:02.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.060+0000 7efed1ffb640 1 -- 172.21.15.72:0/2854798820 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efed8045070 con 0x7efee013f600 2024-04-15T15:15:02.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.060+0000 7efee72ba640 1 -- 172.21.15.72:0/2854798820 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efee013b850 con 0x7efee013f600 2024-04-15T15:15:02.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.060+0000 7efed1ffb640 1 -- 172.21.15.72:0/2854798820 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed803e610 con 0x7efee013f600 2024-04-15T15:15:02.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.060+0000 7efee72ba640 1 -- 172.21.15.72:0/2854798820 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efee013be10 con 0x7efee013f600 2024-04-15T15:15:02.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.061+0000 7efee72ba640 1 -- 172.21.15.72:0/2854798820 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efee0075c50 con 0x7efee013f600 2024-04-15T15:15:02.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.062+0000 7efed1ffb640 1 -- 172.21.15.72:0/2854798820 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efed803e7b0 con 0x7efee013f600 2024-04-15T15:15:02.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.062+0000 7efed1ffb640 1 --2- 172.21.15.72:0/2854798820 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7efec406c8f0 0x7efec406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:02.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.062+0000 7efed1ffb640 1 -- 172.21.15.72:0/2854798820 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7efed80ae950 con 0x7efee013f600 2024-04-15T15:15:02.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.062+0000 7efee5056640 1 --2- 172.21.15.72:0/2854798820 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7efec406c8f0 0x7efec406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:02.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.063+0000 7efee5056640 1 --2- 172.21.15.72:0/2854798820 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7efec406c8f0 0x7efec406edb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efedc000f60 tx=0x7efedc007c00 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:02.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.067+0000 7efed1ffb640 1 -- 172.21.15.72:0/2854798820 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efed8032e20 con 0x7efee013f600 2024-04-15T15:15:02.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.244+0000 7efee72ba640 1 -- 172.21.15.72:0/2854798820 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7efee00d2130 con 0x7efee013f600 2024-04-15T15:15:02.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.244+0000 7efed1ffb640 1 -- 172.21.15.72:0/2854798820 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7efed807ca40 con 0x7efee013f600 2024-04-15T15:15:02.246 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-15T15:15:02.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.246+0000 7efee72ba640 1 -- 172.21.15.72:0/2854798820 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7efec406c8f0 msgr2=0x7efec406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:02.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.246+0000 7efee72ba640 1 --2- 172.21.15.72:0/2854798820 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7efec406c8f0 0x7efec406edb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efedc000f60 tx=0x7efedc007c00 comp rx=0 tx=0).stop 2024-04-15T15:15:02.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.246+0000 7efee72ba640 1 -- 172.21.15.72:0/2854798820 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efee013f600 msgr2=0x7efee013c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:02.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.246+0000 7efee72ba640 1 --2- 172.21.15.72:0/2854798820 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efee013f600 0x7efee013c8b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7efed8009d90 tx=0x7efed8004a40 comp rx=0 tx=0).stop 2024-04-15T15:15:02.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.246+0000 7efee72ba640 1 -- 172.21.15.72:0/2854798820 shutdown_connections 2024-04-15T15:15:02.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.246+0000 7efee72ba640 1 --2- 172.21.15.72:0/2854798820 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7efec406c8f0 0x7efec406edb0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:02.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.246+0000 7efee72ba640 1 --2- 172.21.15.72:0/2854798820 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efee013f600 0x7efee013c8b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:02.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.246+0000 7efee72ba640 1 --2- 172.21.15.72:0/2854798820 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efee0145f10 0x7efee013f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:02.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.246+0000 7efee72ba640 1 --2- 172.21.15.72:0/2854798820 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efee006f1a0 0x7efee015c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:02.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.246+0000 7efee72ba640 1 -- 172.21.15.72:0/2854798820 >> 172.21.15.72:0/2854798820 conn(0x7efee00d13a0 msgr2=0x7efee00b28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:02.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.246+0000 7efee72ba640 1 -- 172.21.15.72:0/2854798820 shutdown_connections 2024-04-15T15:15:02.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.246+0000 7efee72ba640 1 -- 172.21.15.72:0/2854798820 wait complete. 2024-04-15T15:15:02.259 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:15:02.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.363+0000 7f50425e9640 1 Processor -- start 2024-04-15T15:15:02.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.363+0000 7f50425e9640 1 -- start start 2024-04-15T15:15:02.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.364+0000 7f50425e9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f503c15c630 0x7f503c06cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:02.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.364+0000 7f50425e9640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f503c06ed30 0x7f503c0ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:02.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.364+0000 7f50425e9640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f503c06bd50 con 0x7f503c0ccb90 2024-04-15T15:15:02.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.364+0000 7f50425e9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f503c06d2f0 con 0x7f503c15c630 2024-04-15T15:15:02.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.364+0000 7f50425e9640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f503c0cd530 con 0x7f503c06ed30 2024-04-15T15:15:02.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.364+0000 7f503bfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f503c15c630 0x7f503c06cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:02.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.364+0000 7f503bfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f503c15c630 0x7f503c06cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44274/0 (socket says 172.21.15.72:44274) 2024-04-15T15:15:02.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.364+0000 7f503bfff640 1 -- 172.21.15.72:0/4141013345 learned_addr learned my addr 172.21.15.72:0/4141013345 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:02.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.364+0000 7f5040b86640 1 --2- 172.21.15.72:0/4141013345 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f503c06ed30 0x7f503c0ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:02.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.364+0000 7f503bfff640 1 -- 172.21.15.72:0/4141013345 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f503c06ed30 msgr2=0x7f503c0ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:02.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.364+0000 7f503bfff640 1 --2- 172.21.15.72:0/4141013345 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f503c06ed30 0x7f503c0ccff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:02.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.364+0000 7f503bfff640 1 -- 172.21.15.72:0/4141013345 >> v1:172.21.15.175:6789/0 conn(0x7f503c0ccb90 legacy=0x7f503c06e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:02.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.364+0000 7f503bfff640 1 -- 172.21.15.72:0/4141013345 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f503c06bff0 con 0x7f503c15c630 2024-04-15T15:15:02.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.364+0000 7f5040b86640 1 --2- 172.21.15.72:0/4141013345 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f503c06ed30 0x7f503c0ccff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:02.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.365+0000 7f503bfff640 1 --2- 172.21.15.72:0/4141013345 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f503c15c630 0x7f503c06cdb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f502c009870 tx=0x7f502c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18720aaa6c6caf83 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:02.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.365+0000 7f503affd640 1 -- 172.21.15.72:0/4141013345 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f502c035020 con 0x7f503c15c630 2024-04-15T15:15:02.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.365+0000 7f503affd640 1 -- 172.21.15.72:0/4141013345 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f502c031640 con 0x7f503c15c630 2024-04-15T15:15:02.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.365+0000 7f503affd640 1 -- 172.21.15.72:0/4141013345 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f502c02d040 con 0x7f503c15c630 2024-04-15T15:15:02.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.366+0000 7f50425e9640 1 -- 172.21.15.72:0/4141013345 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f503c15c630 msgr2=0x7f503c06cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:02.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.366+0000 7f50425e9640 1 --2- 172.21.15.72:0/4141013345 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f503c15c630 0x7f503c06cdb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f502c009870 tx=0x7f502c02b8f0 comp rx=0 tx=0).stop 2024-04-15T15:15:02.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.366+0000 7f50425e9640 1 -- 172.21.15.72:0/4141013345 shutdown_connections 2024-04-15T15:15:02.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.366+0000 7f50425e9640 1 --2- 172.21.15.72:0/4141013345 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f503c06ed30 0x7f503c0ccff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:02.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.366+0000 7f50425e9640 1 --2- 172.21.15.72:0/4141013345 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f503c15c630 0x7f503c06cdb0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:02.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.366+0000 7f50425e9640 1 -- 172.21.15.72:0/4141013345 >> 172.21.15.72:0/4141013345 conn(0x7f503c0b31d0 msgr2=0x7f503c0b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:02.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.366+0000 7f50425e9640 1 -- 172.21.15.72:0/4141013345 shutdown_connections 2024-04-15T15:15:02.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.366+0000 7f50425e9640 1 -- 172.21.15.72:0/4141013345 wait complete. 2024-04-15T15:15:02.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.367+0000 7f50425e9640 1 Processor -- start 2024-04-15T15:15:02.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.367+0000 7f50425e9640 1 -- start start 2024-04-15T15:15:02.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.368+0000 7f50425e9640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f503c06ed30 0x7f503c1f3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:02.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.368+0000 7f50425e9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f503c0ccb90 0x7f503c14feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:02.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.368+0000 7f50425e9640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f503c15c630 0x7f503c1f3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:02.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.368+0000 7f50425e9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f503c06c5b0 con 0x7f503c0ccb90 2024-04-15T15:15:02.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.368+0000 7f50425e9640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f503c06c920 con 0x7f503c15c630 2024-04-15T15:15:02.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.368+0000 7f50425e9640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f503c15ca10 con 0x7f503c06ed30 2024-04-15T15:15:02.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.368+0000 7f503b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f503c0ccb90 0x7f503c14feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:02.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.368+0000 7f503b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f503c0ccb90 0x7f503c14feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44282/0 (socket says 172.21.15.72:44282) 2024-04-15T15:15:02.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.368+0000 7f503b7fe640 1 -- 172.21.15.72:0/1064811139 learned_addr learned my addr 172.21.15.72:0/1064811139 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:02.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.368+0000 7f503bfff640 1 --2- 172.21.15.72:0/1064811139 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f503c06ed30 0x7f503c1f3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:02.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.368+0000 7f5040b86640 1 --2- 172.21.15.72:0/1064811139 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f503c15c630 0x7f503c1f3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:02.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.368+0000 7f503b7fe640 1 -- 172.21.15.72:0/1064811139 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f503c06ed30 msgr2=0x7f503c1f3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:02.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.368+0000 7f503b7fe640 1 --2- 172.21.15.72:0/1064811139 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f503c06ed30 0x7f503c1f3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:02.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.368+0000 7f503b7fe640 1 -- 172.21.15.72:0/1064811139 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f503c15c630 msgr2=0x7f503c1f3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:02.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.368+0000 7f503b7fe640 1 --2- 172.21.15.72:0/1064811139 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f503c15c630 0x7f503c1f3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:02.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.368+0000 7f503b7fe640 1 -- 172.21.15.72:0/1064811139 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f503c1f3740 con 0x7f503c0ccb90 2024-04-15T15:15:02.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.368+0000 7f503bfff640 1 --2- 172.21.15.72:0/1064811139 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f503c06ed30 0x7f503c1f3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:02.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.368+0000 7f5040b86640 1 --2- 172.21.15.72:0/1064811139 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f503c15c630 0x7f503c1f3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:02.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.368+0000 7f503b7fe640 1 --2- 172.21.15.72:0/1064811139 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f503c0ccb90 0x7f503c14feb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f503000c430 tx=0x7f503000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:02.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.368+0000 7f5038ff9640 1 -- 172.21.15.72:0/1064811139 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5030011790 con 0x7f503c0ccb90 2024-04-15T15:15:02.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.368+0000 7f5038ff9640 1 -- 172.21.15.72:0/1064811139 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5030017070 con 0x7f503c0ccb90 2024-04-15T15:15:02.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.368+0000 7f5038ff9640 1 -- 172.21.15.72:0/1064811139 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5030011b70 con 0x7f503c0ccb90 2024-04-15T15:15:02.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.368+0000 7f50425e9640 1 -- 172.21.15.72:0/1064811139 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f503c1f3f10 con 0x7f503c0ccb90 2024-04-15T15:15:02.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.369+0000 7f50425e9640 1 -- 172.21.15.72:0/1064811139 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f503c1f44d0 con 0x7f503c0ccb90 2024-04-15T15:15:02.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.369+0000 7f50425e9640 1 -- 172.21.15.72:0/1064811139 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f503c075c50 con 0x7f503c0ccb90 2024-04-15T15:15:02.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.370+0000 7f5038ff9640 1 -- 172.21.15.72:0/1064811139 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5030011d10 con 0x7f503c0ccb90 2024-04-15T15:15:02.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.370+0000 7f5038ff9640 1 --2- 172.21.15.72:0/1064811139 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f502806c8f0 0x7f502806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:02.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.370+0000 7f5038ff9640 1 -- 172.21.15.72:0/1064811139 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f503008bdc0 con 0x7f503c0ccb90 2024-04-15T15:15:02.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.370+0000 7f503bfff640 1 --2- 172.21.15.72:0/1064811139 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f502806c8f0 0x7f502806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:02.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.371+0000 7f503bfff640 1 --2- 172.21.15.72:0/1064811139 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f502806c8f0 0x7f502806edb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f502c036000 tx=0x7f502c03c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:02.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.375+0000 7f5038ff9640 1 -- 172.21.15.72:0/1064811139 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5030059e30 con 0x7f503c0ccb90 2024-04-15T15:15:02.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.547+0000 7f50425e9640 1 -- 172.21.15.72:0/1064811139 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f503c15ba20 con 0x7f503c0ccb90 2024-04-15T15:15:02.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.548+0000 7f5038ff9640 1 -- 172.21.15.72:0/1064811139 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7f5030059730 con 0x7f503c0ccb90 2024-04-15T15:15:02.549 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:02.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.550+0000 7f50425e9640 1 -- 172.21.15.72:0/1064811139 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f502806c8f0 msgr2=0x7f502806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:02.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.550+0000 7f50425e9640 1 --2- 172.21.15.72:0/1064811139 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f502806c8f0 0x7f502806edb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f502c036000 tx=0x7f502c03c040 comp rx=0 tx=0).stop 2024-04-15T15:15:02.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.550+0000 7f50425e9640 1 -- 172.21.15.72:0/1064811139 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f503c0ccb90 msgr2=0x7f503c14feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:02.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.550+0000 7f50425e9640 1 --2- 172.21.15.72:0/1064811139 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f503c0ccb90 0x7f503c14feb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f503000c430 tx=0x7f503000c8f0 comp rx=0 tx=0).stop 2024-04-15T15:15:02.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.550+0000 7f50425e9640 1 -- 172.21.15.72:0/1064811139 shutdown_connections 2024-04-15T15:15:02.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.550+0000 7f50425e9640 1 --2- 172.21.15.72:0/1064811139 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f502806c8f0 0x7f502806edb0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:02.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.550+0000 7f50425e9640 1 --2- 172.21.15.72:0/1064811139 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f503c15c630 0x7f503c1f3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:02.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.550+0000 7f50425e9640 1 --2- 172.21.15.72:0/1064811139 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f503c0ccb90 0x7f503c14feb0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:02.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.550+0000 7f50425e9640 1 --2- 172.21.15.72:0/1064811139 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f503c06ed30 0x7f503c1f3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:02.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.550+0000 7f50425e9640 1 -- 172.21.15.72:0/1064811139 >> 172.21.15.72:0/1064811139 conn(0x7f503c0b31d0 msgr2=0x7f503c06e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:02.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.550+0000 7f50425e9640 1 -- 172.21.15.72:0/1064811139 shutdown_connections 2024-04-15T15:15:02.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.550+0000 7f50425e9640 1 -- 172.21.15.72:0/1064811139 wait complete. 2024-04-15T15:15:02.563 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:15:02.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.666+0000 7f04cb474640 1 Processor -- start 2024-04-15T15:15:02.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.667+0000 7f04cb474640 1 -- start start 2024-04-15T15:15:02.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.667+0000 7f04cb474640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f04c4158d60 0x7f04c415ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:02.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.667+0000 7f04cb474640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f04c406d910 0x7f04c4160750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:02.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.667+0000 7f04cb474640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04c415f360 con 0x7f04c406f7f0 2024-04-15T15:15:02.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.667+0000 7f04cb474640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f04c415ffe0 con 0x7f04c4158d60 2024-04-15T15:15:02.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.667+0000 7f04cb474640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f04c415f630 con 0x7f04c406d910 2024-04-15T15:15:02.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.668+0000 7f04c9210640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f04c4158d60 0x7f04c415ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:02.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.668+0000 7f04c9210640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f04c4158d60 0x7f04c415ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44288/0 (socket says 172.21.15.72:44288) 2024-04-15T15:15:02.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.668+0000 7f04c9210640 1 -- 172.21.15.72:0/1240504801 learned_addr learned my addr 172.21.15.72:0/1240504801 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:02.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.668+0000 7f04c9a11640 1 --2- 172.21.15.72:0/1240504801 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f04c406d910 0x7f04c4160750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:02.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.668+0000 7f04c9210640 1 -- 172.21.15.72:0/1240504801 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f04c406d910 msgr2=0x7f04c4160750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:02.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.668+0000 7f04c9210640 1 --2- 172.21.15.72:0/1240504801 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f04c406d910 0x7f04c4160750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:02.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.668+0000 7f04c9210640 1 -- 172.21.15.72:0/1240504801 >> v1:172.21.15.175:6789/0 conn(0x7f04c406f7f0 legacy=0x7f04c406d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:02.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.668+0000 7f04c9210640 1 -- 172.21.15.72:0/1240504801 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04c415f910 con 0x7f04c4158d60 2024-04-15T15:15:02.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.668+0000 7f04c9a11640 1 --2- 172.21.15.72:0/1240504801 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f04c406d910 0x7f04c4160750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:02.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.669+0000 7f04c9210640 1 --2- 172.21.15.72:0/1240504801 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04c4158d60 0x7f04c415ee20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f04bc009e50 tx=0x7f04bc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12e8a31402b2b4d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:02.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.669+0000 7f04bbfff640 1 -- 172.21.15.72:0/1240504801 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04bc02cb00 con 0x7f04c4158d60 2024-04-15T15:15:02.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.669+0000 7f04bbfff640 1 -- 172.21.15.72:0/1240504801 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04bc02cca0 con 0x7f04c4158d60 2024-04-15T15:15:02.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.669+0000 7f04bbfff640 1 -- 172.21.15.72:0/1240504801 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04bc034610 con 0x7f04c4158d60 2024-04-15T15:15:02.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.669+0000 7f04cb474640 1 -- 172.21.15.72:0/1240504801 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04c4158d60 msgr2=0x7f04c415ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:02.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.669+0000 7f04cb474640 1 --2- 172.21.15.72:0/1240504801 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04c4158d60 0x7f04c415ee20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f04bc009e50 tx=0x7f04bc02bf60 comp rx=0 tx=0).stop 2024-04-15T15:15:02.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.669+0000 7f04cb474640 1 -- 172.21.15.72:0/1240504801 shutdown_connections 2024-04-15T15:15:02.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.669+0000 7f04cb474640 1 --2- 172.21.15.72:0/1240504801 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f04c406d910 0x7f04c4160750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:02.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.669+0000 7f04cb474640 1 --2- 172.21.15.72:0/1240504801 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04c4158d60 0x7f04c415ee20 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:02.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.669+0000 7f04cb474640 1 -- 172.21.15.72:0/1240504801 >> 172.21.15.72:0/1240504801 conn(0x7f04c4072400 msgr2=0x7f04c4072810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:02.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.669+0000 7f04cb474640 1 -- 172.21.15.72:0/1240504801 shutdown_connections 2024-04-15T15:15:02.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.669+0000 7f04cb474640 1 -- 172.21.15.72:0/1240504801 wait complete. 2024-04-15T15:15:02.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.671+0000 7f04cb474640 1 Processor -- start 2024-04-15T15:15:02.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.671+0000 7f04cb474640 1 -- start start 2024-04-15T15:15:02.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.671+0000 7f04cb474640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04c406d910 0x7f04c414a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:02.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.671+0000 7f04cb474640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f04c406f7f0 0x7f04c41522b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:02.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.671+0000 7f04cb474640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04c4158d60 0x7f04c4153da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:02.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.671+0000 7f04cb474640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f04c4159140 con 0x7f04c406d910 2024-04-15T15:15:02.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.671+0000 7f04cb474640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f04c4160230 con 0x7f04c4158d60 2024-04-15T15:15:02.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.671+0000 7f04cb474640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f04c41592c0 con 0x7f04c406f7f0 2024-04-15T15:15:02.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.671+0000 7f04c9210640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04c406d910 0x7f04c414a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:02.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.671+0000 7f04c9210640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04c406d910 0x7f04c414a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44294/0 (socket says 172.21.15.72:44294) 2024-04-15T15:15:02.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.671+0000 7f04c9210640 1 -- 172.21.15.72:0/4178917657 learned_addr learned my addr 172.21.15.72:0/4178917657 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:02.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.672+0000 7f04c9a11640 1 --2- 172.21.15.72:0/4178917657 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04c4158d60 0x7f04c4153da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:02.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.672+0000 7f04c8a0f640 1 --2- 172.21.15.72:0/4178917657 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f04c406f7f0 0x7f04c41522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:02.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.672+0000 7f04c9210640 1 -- 172.21.15.72:0/4178917657 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f04c406f7f0 msgr2=0x7f04c41522b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:02.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.672+0000 7f04c9210640 1 --2- 172.21.15.72:0/4178917657 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f04c406f7f0 0x7f04c41522b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:02.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.672+0000 7f04c9210640 1 -- 172.21.15.72:0/4178917657 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04c4158d60 msgr2=0x7f04c4153da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:02.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.672+0000 7f04c9210640 1 --2- 172.21.15.72:0/4178917657 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04c4158d60 0x7f04c4153da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:02.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.672+0000 7f04c9210640 1 -- 172.21.15.72:0/4178917657 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04c4152d50 con 0x7f04c406d910 2024-04-15T15:15:02.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.672+0000 7f04c8a0f640 1 --2- 172.21.15.72:0/4178917657 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f04c406f7f0 0x7f04c41522b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:02.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.672+0000 7f04c9210640 1 --2- 172.21.15.72:0/4178917657 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04c406d910 0x7f04c414a330 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f04bc036040 tx=0x7f04bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:02.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.672+0000 7f04b9ffb640 1 -- 172.21.15.72:0/4178917657 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04bc043020 con 0x7f04c406d910 2024-04-15T15:15:02.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.672+0000 7f04b9ffb640 1 -- 172.21.15.72:0/4178917657 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04bc02ce50 con 0x7f04c406d910 2024-04-15T15:15:02.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.672+0000 7f04cb474640 1 -- 172.21.15.72:0/4178917657 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04c4153030 con 0x7f04c406d910 2024-04-15T15:15:02.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.672+0000 7f04b9ffb640 1 -- 172.21.15.72:0/4178917657 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04bc03e690 con 0x7f04c406d910 2024-04-15T15:15:02.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.672+0000 7f04cb474640 1 -- 172.21.15.72:0/4178917657 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04c41515d0 con 0x7f04c406d910 2024-04-15T15:15:02.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.673+0000 7f04cb474640 1 -- 172.21.15.72:0/4178917657 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04c415e9f0 con 0x7f04c406d910 2024-04-15T15:15:02.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.674+0000 7f04b9ffb640 1 -- 172.21.15.72:0/4178917657 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f04bc046020 con 0x7f04c406d910 2024-04-15T15:15:02.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.674+0000 7f04b9ffb640 1 --2- 172.21.15.72:0/4178917657 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f04a806c8f0 0x7f04a806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:02.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.674+0000 7f04b9ffb640 1 -- 172.21.15.72:0/4178917657 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f04bc0ae5f0 con 0x7f04c406d910 2024-04-15T15:15:02.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.674+0000 7f04c8a0f640 1 --2- 172.21.15.72:0/4178917657 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f04a806c8f0 0x7f04a806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:02.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.675+0000 7f04c8a0f640 1 --2- 172.21.15.72:0/4178917657 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f04a806c8f0 0x7f04a806edb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f04c4154d80 tx=0x7f04b4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:02.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.678+0000 7f04b9ffb640 1 -- 172.21.15.72:0/4178917657 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f04c415e9f0 con 0x7f04c406d910 2024-04-15T15:15:02.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.852+0000 7f04cb474640 1 -- 172.21.15.72:0/4178917657 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f04c4075c50 con 0x7f04c406d910 2024-04-15T15:15:02.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.852+0000 7f04b9ffb640 1 -- 172.21.15.72:0/4178917657 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7f04bc04d030 con 0x7f04c406d910 2024-04-15T15:15:02.854 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-04-15T15:15:02.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.854+0000 7f04cb474640 1 -- 172.21.15.72:0/4178917657 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f04a806c8f0 msgr2=0x7f04a806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:02.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.854+0000 7f04cb474640 1 --2- 172.21.15.72:0/4178917657 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f04a806c8f0 0x7f04a806edb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f04c4154d80 tx=0x7f04b4009040 comp rx=0 tx=0).stop 2024-04-15T15:15:02.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.854+0000 7f04cb474640 1 -- 172.21.15.72:0/4178917657 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04c406d910 msgr2=0x7f04c414a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:02.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.854+0000 7f04cb474640 1 --2- 172.21.15.72:0/4178917657 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04c406d910 0x7f04c414a330 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f04bc036040 tx=0x7f04bc004a40 comp rx=0 tx=0).stop 2024-04-15T15:15:02.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.854+0000 7f04cb474640 1 -- 172.21.15.72:0/4178917657 shutdown_connections 2024-04-15T15:15:02.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.854+0000 7f04cb474640 1 --2- 172.21.15.72:0/4178917657 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f04a806c8f0 0x7f04a806edb0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:02.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.854+0000 7f04cb474640 1 --2- 172.21.15.72:0/4178917657 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04c4158d60 0x7f04c4153da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:02.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.854+0000 7f04cb474640 1 --2- 172.21.15.72:0/4178917657 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f04c406f7f0 0x7f04c41522b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:02.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.854+0000 7f04cb474640 1 --2- 172.21.15.72:0/4178917657 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04c406d910 0x7f04c414a330 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:02.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.854+0000 7f04cb474640 1 -- 172.21.15.72:0/4178917657 >> 172.21.15.72:0/4178917657 conn(0x7f04c4072400 msgr2=0x7f04c40ae480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:02.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.854+0000 7f04cb474640 1 -- 172.21.15.72:0/4178917657 shutdown_connections 2024-04-15T15:15:02.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.855+0000 7f04cb474640 1 -- 172.21.15.72:0/4178917657 wait complete. 2024-04-15T15:15:02.867 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:15:02.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.970+0000 7f23898d3640 1 Processor -- start 2024-04-15T15:15:02.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.970+0000 7f23898d3640 1 -- start start 2024-04-15T15:15:02.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.971+0000 7f23898d3640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f238406f370 0x7f23841675d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:02.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.971+0000 7f23898d3640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f238406d720 con 0x7f238406c1f0 2024-04-15T15:15:02.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.971+0000 7f23898d3640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23840d0c90 con 0x7f2384145f10 2024-04-15T15:15:02.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.971+0000 7f23898d3640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f23841462f0 con 0x7f238406f370 2024-04-15T15:15:02.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.971+0000 7f23827fc640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f238406c1f0 0x7f238406ec60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45000/0 (socket says 172.21.15.72:45000) 2024-04-15T15:15:02.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.971+0000 7f23827fc640 1 -- 172.21.15.72:0/731431316 learned_addr learned my addr 172.21.15.72:0/731431316 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:02.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.971+0000 7f23837fe640 1 --2- 172.21.15.72:0/731431316 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f238406f370 0x7f23841675d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:02.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.971+0000 7f2381ffb640 1 -- 172.21.15.72:0/731431316 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1397438097 0 0) 0x7f238406d720 con 0x7f238406c1f0 2024-04-15T15:15:02.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.971+0000 7f2381ffb640 1 -- 172.21.15.72:0/731431316 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f236c003520 con 0x7f238406c1f0 2024-04-15T15:15:02.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.972+0000 7f2381ffb640 1 -- 172.21.15.72:0/731431316 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 164433358 0 0) 0x7f236c003520 con 0x7f238406c1f0 2024-04-15T15:15:02.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.972+0000 7f2381ffb640 1 -- 172.21.15.72:0/731431316 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f238406f370 msgr2=0x7f23841675d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:02.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.972+0000 7f2381ffb640 1 --2- 172.21.15.72:0/731431316 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f238406f370 0x7f23841675d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:02.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.972+0000 7f2381ffb640 1 -- 172.21.15.72:0/731431316 >> v1:172.21.15.175:6789/0 conn(0x7f2384145f10 legacy=0x7f23840d0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:02.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.972+0000 7f2381ffb640 1 -- 172.21.15.72:0/731431316 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f238406c680 con 0x7f238406c1f0 2024-04-15T15:15:02.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.972+0000 7f2381ffb640 1 -- 172.21.15.72:0/731431316 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f2378001670 con 0x7f238406c1f0 2024-04-15T15:15:02.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.972+0000 7f2381ffb640 1 -- 172.21.15.72:0/731431316 >> v1:172.21.15.72:6789/0 conn(0x7f238406c1f0 legacy=0x7f238406ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:02.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.972+0000 7f2381ffb640 1 --2- 172.21.15.72:0/731431316 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f236c0039e0 0x7f236c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:02.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.972+0000 7f2381ffb640 1 --2- 172.21.15.72:0/731431316 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f236c004800 0x7f236c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:02.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.972+0000 7f2381ffb640 1 --2- 172.21.15.72:0/731431316 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f238406f370 0x7f236c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:02.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.972+0000 7f2381ffb640 1 -- 172.21.15.72:0/731431316 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f23841462f0 con 0x7f236c0039e0 2024-04-15T15:15:02.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.972+0000 7f2381ffb640 1 -- 172.21.15.72:0/731431316 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f236c00bb20 con 0x7f236c004800 2024-04-15T15:15:02.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.972+0000 7f2381ffb640 1 -- 172.21.15.72:0/731431316 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f236c00bd20 con 0x7f238406f370 2024-04-15T15:15:02.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.972+0000 7f2382ffd640 1 --2- 172.21.15.72:0/731431316 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f236c0039e0 0x7f236c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:02.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.972+0000 7f23827fc640 1 --2- 172.21.15.72:0/731431316 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f236c004800 0x7f236c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:02.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.972+0000 7f23837fe640 1 --2- 172.21.15.72:0/731431316 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f238406f370 0x7f236c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:02.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.972+0000 7f2382ffd640 1 -- 172.21.15.72:0/731431316 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f238406f370 msgr2=0x7f236c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:02.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.972+0000 7f2382ffd640 1 --2- 172.21.15.72:0/731431316 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f238406f370 0x7f236c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:02.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.972+0000 7f2382ffd640 1 -- 172.21.15.72:0/731431316 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f236c004800 msgr2=0x7f236c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:02.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.972+0000 7f2382ffd640 1 --2- 172.21.15.72:0/731431316 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f236c004800 0x7f236c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:02.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.972+0000 7f2382ffd640 1 -- 172.21.15.72:0/731431316 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f236c003520 con 0x7f236c0039e0 2024-04-15T15:15:02.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.972+0000 7f23837fe640 1 --2- 172.21.15.72:0/731431316 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f238406f370 0x7f236c005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:02.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.973+0000 7f2382ffd640 1 --2- 172.21.15.72:0/731431316 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f236c0039e0 0x7f236c003e00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2374001750 tx=0x7f236c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e53abaa3d07f5b07 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:02.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.973+0000 7f2381ffb640 1 -- 172.21.15.72:0/731431316 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2374001370 con 0x7f236c0039e0 2024-04-15T15:15:02.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.973+0000 7f2381ffb640 1 -- 172.21.15.72:0/731431316 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23740052d0 con 0x7f236c0039e0 2024-04-15T15:15:02.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.973+0000 7f2381ffb640 1 -- 172.21.15.72:0/731431316 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2374003830 con 0x7f236c0039e0 2024-04-15T15:15:02.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.973+0000 7f23898d3640 1 -- 172.21.15.72:0/731431316 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f236c0039e0 msgr2=0x7f236c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:02.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.973+0000 7f23898d3640 1 --2- 172.21.15.72:0/731431316 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f236c0039e0 0x7f236c003e00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2374001750 tx=0x7f236c005830 comp rx=0 tx=0).stop 2024-04-15T15:15:02.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.973+0000 7f23837fe640 1 -- 172.21.15.72:0/731431316 reap_dead start 2024-04-15T15:15:02.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.973+0000 7f23898d3640 1 -- 172.21.15.72:0/731431316 shutdown_connections 2024-04-15T15:15:02.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.973+0000 7f23898d3640 1 -- 172.21.15.72:0/731431316 >> 172.21.15.72:0/731431316 conn(0x7f23840d13a0 msgr2=0x7f23840d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:02.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.973+0000 7f23898d3640 1 -- 172.21.15.72:0/731431316 shutdown_connections 2024-04-15T15:15:02.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.973+0000 7f23898d3640 1 -- 172.21.15.72:0/731431316 wait complete. 2024-04-15T15:15:02.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.975+0000 7f23898d3640 1 Processor -- start 2024-04-15T15:15:02.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.975+0000 7f23898d3640 1 -- start start 2024-04-15T15:15:02.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.975+0000 7f23898d3640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f236c0039e0 0x7f238413f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:02.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.976+0000 7f23898d3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23841f2e20 0x7f238415b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:02.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.976+0000 7f23898d3640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f238413b7c0 0x7f23840cea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:02.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.976+0000 7f23898d3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f238406cc40 con 0x7f23841f2e20 2024-04-15T15:15:02.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.976+0000 7f23898d3640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f238406d290 con 0x7f236c0039e0 2024-04-15T15:15:02.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.976+0000 7f23898d3640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f238406cf20 con 0x7f238413b7c0 2024-04-15T15:15:02.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.976+0000 7f23827fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23841f2e20 0x7f238415b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:02.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.976+0000 7f23827fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23841f2e20 0x7f238415b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44320/0 (socket says 172.21.15.72:44320) 2024-04-15T15:15:02.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.976+0000 7f23827fc640 1 -- 172.21.15.72:0/1831152246 learned_addr learned my addr 172.21.15.72:0/1831152246 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:02.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.976+0000 7f23837fe640 1 --2- 172.21.15.72:0/1831152246 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f238413b7c0 0x7f23840cea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:02.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.976+0000 7f2382ffd640 1 --2- 172.21.15.72:0/1831152246 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f236c0039e0 0x7f238413f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:02.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.976+0000 7f23827fc640 1 -- 172.21.15.72:0/1831152246 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f238413b7c0 msgr2=0x7f23840cea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:02.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.976+0000 7f23827fc640 1 --2- 172.21.15.72:0/1831152246 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f238413b7c0 0x7f23840cea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:02.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.976+0000 7f23827fc640 1 -- 172.21.15.72:0/1831152246 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f236c0039e0 msgr2=0x7f238413f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:02.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.976+0000 7f23827fc640 1 --2- 172.21.15.72:0/1831152246 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f236c0039e0 0x7f238413f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:02.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.976+0000 7f23827fc640 1 -- 172.21.15.72:0/1831152246 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f238413bfd0 con 0x7f23841f2e20 2024-04-15T15:15:02.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.976+0000 7f23837fe640 1 --2- 172.21.15.72:0/1831152246 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f238413b7c0 0x7f23840cea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:02.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.976+0000 7f23827fc640 1 --2- 172.21.15.72:0/1831152246 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23841f2e20 0x7f238415b920 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f23780072c0 tx=0x7f2378007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:02.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.976+0000 7f236bfff640 1 -- 172.21.15.72:0/1831152246 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f237800f8e0 con 0x7f23841f2e20 2024-04-15T15:15:02.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.976+0000 7f236bfff640 1 -- 172.21.15.72:0/1831152246 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2378018070 con 0x7f23841f2e20 2024-04-15T15:15:02.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.976+0000 7f23898d3640 1 -- 172.21.15.72:0/1831152246 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f238413c2b0 con 0x7f23841f2e20 2024-04-15T15:15:02.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.976+0000 7f236bfff640 1 -- 172.21.15.72:0/1831152246 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f237800fd00 con 0x7f23841f2e20 2024-04-15T15:15:02.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.976+0000 7f23898d3640 1 -- 172.21.15.72:0/1831152246 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f238413bc40 con 0x7f23841f2e20 2024-04-15T15:15:02.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.977+0000 7f23898d3640 1 -- 172.21.15.72:0/1831152246 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f238413c7d0 con 0x7f23841f2e20 2024-04-15T15:15:02.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.978+0000 7f236bfff640 1 -- 172.21.15.72:0/1831152246 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f237801d460 con 0x7f23841f2e20 2024-04-15T15:15:02.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.978+0000 7f236bfff640 1 --2- 172.21.15.72:0/1831152246 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f237006c920 0x7f237006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:02.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.978+0000 7f236bfff640 1 -- 172.21.15.72:0/1831152246 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f237808bdc0 con 0x7f23841f2e20 2024-04-15T15:15:02.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.978+0000 7f2382ffd640 1 --2- 172.21.15.72:0/1831152246 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f237006c920 0x7f237006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:02.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.979+0000 7f2382ffd640 1 --2- 172.21.15.72:0/1831152246 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f237006c920 0x7f237006ede0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2374033000 tx=0x7f2374009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:02.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:02.983+0000 7f236bfff640 1 -- 172.21.15.72:0/1831152246 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2378059f40 con 0x7f23841f2e20 2024-04-15T15:15:03.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.159+0000 7f23898d3640 1 -- 172.21.15.72:0/1831152246 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f2384075c50 con 0x7f23841f2e20 2024-04-15T15:15:03.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.159+0000 7f236bfff640 1 -- 172.21.15.72:0/1831152246 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7f2378059840 con 0x7f23841f2e20 2024-04-15T15:15:03.161 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:03.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.161+0000 7f23898d3640 1 -- 172.21.15.72:0/1831152246 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f237006c920 msgr2=0x7f237006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:03.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.161+0000 7f23898d3640 1 --2- 172.21.15.72:0/1831152246 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f237006c920 0x7f237006ede0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2374033000 tx=0x7f2374009030 comp rx=0 tx=0).stop 2024-04-15T15:15:03.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.161+0000 7f23898d3640 1 -- 172.21.15.72:0/1831152246 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23841f2e20 msgr2=0x7f238415b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:03.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.161+0000 7f23898d3640 1 --2- 172.21.15.72:0/1831152246 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23841f2e20 0x7f238415b920 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f23780072c0 tx=0x7f2378007780 comp rx=0 tx=0).stop 2024-04-15T15:15:03.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.161+0000 7f23898d3640 1 -- 172.21.15.72:0/1831152246 shutdown_connections 2024-04-15T15:15:03.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.161+0000 7f23898d3640 1 --2- 172.21.15.72:0/1831152246 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f237006c920 0x7f237006ede0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.161+0000 7f23898d3640 1 --2- 172.21.15.72:0/1831152246 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f238413b7c0 0x7f23840cea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.161+0000 7f23898d3640 1 --2- 172.21.15.72:0/1831152246 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23841f2e20 0x7f238415b920 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.161+0000 7f23898d3640 1 --2- 172.21.15.72:0/1831152246 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f236c0039e0 0x7f238413f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.161+0000 7f23898d3640 1 -- 172.21.15.72:0/1831152246 >> 172.21.15.72:0/1831152246 conn(0x7f23840d13a0 msgr2=0x7f238406ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:03.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.161+0000 7f23898d3640 1 -- 172.21.15.72:0/1831152246 shutdown_connections 2024-04-15T15:15:03.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.161+0000 7f23898d3640 1 -- 172.21.15.72:0/1831152246 wait complete. 2024-04-15T15:15:03.174 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:15:03.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.276+0000 7f8127950640 1 Processor -- start 2024-04-15T15:15:03.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.277+0000 7f8127950640 1 -- start start 2024-04-15T15:15:03.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.277+0000 7f8127950640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f812015e610 0x7f81200d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.277+0000 7f8127950640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f812006c400 0x7f812006c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.277+0000 7f8127950640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81200d2320 con 0x7f812015e9f0 2024-04-15T15:15:03.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.277+0000 7f8127950640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f81200d31d0 con 0x7f812006c400 2024-04-15T15:15:03.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.277+0000 7f8127950640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f81200d25c0 con 0x7f812015e610 2024-04-15T15:15:03.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.278+0000 7f8125eed640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f812006c400 0x7f812006c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:03.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.278+0000 7f8125eed640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f812006c400 0x7f812006c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44324/0 (socket says 172.21.15.72:44324) 2024-04-15T15:15:03.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.278+0000 7f8125eed640 1 -- 172.21.15.72:0/4077422499 learned_addr learned my addr 172.21.15.72:0/4077422499 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:03.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.278+0000 7f8124eeb640 1 --1- 172.21.15.72:0/4077422499 >> v1:172.21.15.72:6789/0 conn(0x7f812015e9f0 0x7f81200d3660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45002/0 (socket says 172.21.15.72:45002) 2024-04-15T15:15:03.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.278+0000 7f8125eed640 1 -- 172.21.15.72:0/4077422499 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f812015e610 msgr2=0x7f81200d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:03.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.278+0000 7f8125eed640 1 --2- 172.21.15.72:0/4077422499 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f812015e610 0x7f81200d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.278+0000 7f8125eed640 1 -- 172.21.15.72:0/4077422499 >> v1:172.21.15.72:6789/0 conn(0x7f812015e9f0 legacy=0x7f81200d3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:03.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.278+0000 7f8117fff640 1 -- 172.21.15.72:0/4077422499 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3614576284 0 0) 0x7f81200d2320 con 0x7f812015e9f0 2024-04-15T15:15:03.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.278+0000 7f8125eed640 1 -- 172.21.15.72:0/4077422499 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81200d28a0 con 0x7f812006c400 2024-04-15T15:15:03.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.279+0000 7f8125eed640 1 --2- 172.21.15.72:0/4077422499 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f812006c400 0x7f812006c890 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8118009870 tx=0x7f811802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ad82d2e019d4f23 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:03.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.279+0000 7f8117fff640 1 -- 172.21.15.72:0/4077422499 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f811802f840 con 0x7f812006c400 2024-04-15T15:15:03.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.279+0000 7f8117fff640 1 -- 172.21.15.72:0/4077422499 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f811802f9e0 con 0x7f812006c400 2024-04-15T15:15:03.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.279+0000 7f8117fff640 1 -- 172.21.15.72:0/4077422499 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f811802fdb0 con 0x7f812006c400 2024-04-15T15:15:03.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.279+0000 7f8127950640 1 -- 172.21.15.72:0/4077422499 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f812006c400 msgr2=0x7f812006c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:03.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.279+0000 7f8127950640 1 --2- 172.21.15.72:0/4077422499 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f812006c400 0x7f812006c890 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8118009870 tx=0x7f811802ba20 comp rx=0 tx=0).stop 2024-04-15T15:15:03.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.279+0000 7f8127950640 1 -- 172.21.15.72:0/4077422499 shutdown_connections 2024-04-15T15:15:03.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.279+0000 7f8127950640 1 --2- 172.21.15.72:0/4077422499 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f812006c400 0x7f812006c890 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.279+0000 7f8127950640 1 --2- 172.21.15.72:0/4077422499 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f812015e610 0x7f81200d0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.279+0000 7f8127950640 1 -- 172.21.15.72:0/4077422499 >> 172.21.15.72:0/4077422499 conn(0x7f81200d4750 msgr2=0x7f81200d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:03.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.279+0000 7f8127950640 1 -- 172.21.15.72:0/4077422499 shutdown_connections 2024-04-15T15:15:03.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.279+0000 7f8127950640 1 -- 172.21.15.72:0/4077422499 wait complete. 2024-04-15T15:15:03.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.281+0000 7f8127950640 1 Processor -- start 2024-04-15T15:15:03.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.281+0000 7f8127950640 1 -- start start 2024-04-15T15:15:03.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.281+0000 7f8127950640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f812006c400 0x7f81200ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.281+0000 7f8127950640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f812015e610 0x7f81201ea420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.281+0000 7f8127950640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f812015e9f0 0x7f81201ea960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.281+0000 7f8127950640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f812006d130 con 0x7f812015e9f0 2024-04-15T15:15:03.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.281+0000 7f8127950640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f81200d2e60 con 0x7f812015e610 2024-04-15T15:15:03.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.281+0000 7f8127950640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f81200d0bd0 con 0x7f812006c400 2024-04-15T15:15:03.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.281+0000 7f8125eed640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f812015e9f0 0x7f81201ea960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:03.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.281+0000 7f8125eed640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f812015e9f0 0x7f81201ea960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44330/0 (socket says 172.21.15.72:44330) 2024-04-15T15:15:03.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.281+0000 7f8125eed640 1 -- 172.21.15.72:0/3754000520 learned_addr learned my addr 172.21.15.72:0/3754000520 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:03.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.282+0000 7f8124eeb640 1 --2- 172.21.15.72:0/3754000520 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f812015e610 0x7f81201ea420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:03.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.282+0000 7f8125eed640 1 -- 172.21.15.72:0/3754000520 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f812006c400 msgr2=0x7f81200ccc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:03.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.282+0000 7f81256ec640 1 --2- 172.21.15.72:0/3754000520 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f812006c400 0x7f81200ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:03.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.282+0000 7f8125eed640 1 --2- 172.21.15.72:0/3754000520 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f812006c400 0x7f81200ccc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.282+0000 7f8125eed640 1 -- 172.21.15.72:0/3754000520 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f812015e610 msgr2=0x7f81201ea420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:03.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.282+0000 7f8125eed640 1 --2- 172.21.15.72:0/3754000520 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f812015e610 0x7f81201ea420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.282+0000 7f8125eed640 1 -- 172.21.15.72:0/3754000520 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81200cd450 con 0x7f812015e9f0 2024-04-15T15:15:03.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.282+0000 7f8124eeb640 1 --2- 172.21.15.72:0/3754000520 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f812015e610 0x7f81201ea420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:03.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.282+0000 7f81256ec640 1 --2- 172.21.15.72:0/3754000520 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f812006c400 0x7f81200ccc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:03.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.282+0000 7f8125eed640 1 --2- 172.21.15.72:0/3754000520 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f812015e9f0 0x7f81201ea960 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f81180090b0 tx=0x7f8118004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:03.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.282+0000 7f8115ffb640 1 -- 172.21.15.72:0/3754000520 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8118047070 con 0x7f812015e9f0 2024-04-15T15:15:03.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.282+0000 7f8115ffb640 1 -- 172.21.15.72:0/3754000520 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8118037c60 con 0x7f812015e9f0 2024-04-15T15:15:03.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.282+0000 7f8127950640 1 -- 172.21.15.72:0/3754000520 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81200cd730 con 0x7f812015e9f0 2024-04-15T15:15:03.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.282+0000 7f8115ffb640 1 -- 172.21.15.72:0/3754000520 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8118004270 con 0x7f812015e9f0 2024-04-15T15:15:03.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.282+0000 7f8127950640 1 -- 172.21.15.72:0/3754000520 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81200cdcf0 con 0x7f812015e9f0 2024-04-15T15:15:03.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.283+0000 7f8127950640 1 -- 172.21.15.72:0/3754000520 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8120075c50 con 0x7f812015e9f0 2024-04-15T15:15:03.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.284+0000 7f8115ffb640 1 -- 172.21.15.72:0/3754000520 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f811804a070 con 0x7f812015e9f0 2024-04-15T15:15:03.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.284+0000 7f8115ffb640 1 --2- 172.21.15.72:0/3754000520 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f810406c8f0 0x7f810406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.284+0000 7f8115ffb640 1 -- 172.21.15.72:0/3754000520 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f81180ae0b0 con 0x7f812015e9f0 2024-04-15T15:15:03.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.284+0000 7f81256ec640 1 --2- 172.21.15.72:0/3754000520 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f810406c8f0 0x7f810406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:03.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.285+0000 7f81256ec640 1 --2- 172.21.15.72:0/3754000520 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f810406c8f0 0x7f810406edb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f81201eafc0 tx=0x7f811c0073e0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:03.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.288+0000 7f8115ffb640 1 -- 172.21.15.72:0/3754000520 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8118033030 con 0x7f812015e9f0 2024-04-15T15:15:03.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.476+0000 7f8127950640 1 -- 172.21.15.72:0/3754000520 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f812015dfd0 con 0x7f812015e9f0 2024-04-15T15:15:03.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.476+0000 7f8115ffb640 1 -- 172.21.15.72:0/3754000520 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) ==== 80+0+0 (secure 0 0 0) 0x7f811807c1a0 con 0x7f812015e9f0 2024-04-15T15:15:03.478 INFO:teuthology.orchestra.run.smithi072.stderr:listed 0 entries 2024-04-15T15:15:03.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.478+0000 7f8127950640 1 -- 172.21.15.72:0/3754000520 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f810406c8f0 msgr2=0x7f810406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:03.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.478+0000 7f8127950640 1 --2- 172.21.15.72:0/3754000520 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f810406c8f0 0x7f810406edb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f81201eafc0 tx=0x7f811c0073e0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.478+0000 7f8127950640 1 -- 172.21.15.72:0/3754000520 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f812015e9f0 msgr2=0x7f81201ea960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:03.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.478+0000 7f8127950640 1 --2- 172.21.15.72:0/3754000520 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f812015e9f0 0x7f81201ea960 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f81180090b0 tx=0x7f8118004730 comp rx=0 tx=0).stop 2024-04-15T15:15:03.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.478+0000 7f8127950640 1 -- 172.21.15.72:0/3754000520 shutdown_connections 2024-04-15T15:15:03.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.478+0000 7f8127950640 1 --2- 172.21.15.72:0/3754000520 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f810406c8f0 0x7f810406edb0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.478+0000 7f8127950640 1 --2- 172.21.15.72:0/3754000520 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f812015e9f0 0x7f81201ea960 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.478+0000 7f8127950640 1 --2- 172.21.15.72:0/3754000520 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f812015e610 0x7f81201ea420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.478+0000 7f8127950640 1 --2- 172.21.15.72:0/3754000520 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f812006c400 0x7f81200ccc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.478+0000 7f8127950640 1 -- 172.21.15.72:0/3754000520 >> 172.21.15.72:0/3754000520 conn(0x7f81200d4750 msgr2=0x7f81200b6b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:03.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.478+0000 7f8127950640 1 -- 172.21.15.72:0/3754000520 shutdown_connections 2024-04-15T15:15:03.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.478+0000 7f8127950640 1 -- 172.21.15.72:0/3754000520 wait complete. 2024-04-15T15:15:03.491 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:15:03.491 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:15:03.507 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:03.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.649+0000 7f47624c8640 1 Processor -- start 2024-04-15T15:15:03.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.649+0000 7f47624c8640 1 -- start start 2024-04-15T15:15:03.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.649+0000 7f47624c8640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f475c161d60 0x7f475c06f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.649+0000 7f47624c8640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f475c06fb30 0x7f475c070310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.649+0000 7f47624c8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f475c070850 0x7f475c0cf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.649+0000 7f47624c8640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f475c0b5650 con 0x7f475c070850 2024-04-15T15:15:03.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.649+0000 7f47624c8640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f475c070c80 con 0x7f475c06fb30 2024-04-15T15:15:03.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.649+0000 7f47624c8640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f475c162140 con 0x7f475c161d60 2024-04-15T15:15:03.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.650+0000 7f4760a65640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f475c070850 0x7f475c0cf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:03.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.650+0000 7f4760a65640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f475c070850 0x7f475c0cf1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44332/0 (socket says 172.21.15.72:44332) 2024-04-15T15:15:03.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.650+0000 7f4760a65640 1 -- 172.21.15.72:0/2523225653 learned_addr learned my addr 172.21.15.72:0/2523225653 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:03.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.650+0000 7f4753fff640 1 --2- 172.21.15.72:0/2523225653 >> v2:172.21.15.175:3300/0 conn(0x7f475c06fb30 0x7f475c070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:03.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.650+0000 7f475bfff640 1 --2- 172.21.15.72:0/2523225653 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f475c161d60 0x7f475c06f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:03.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.650+0000 7f4760a65640 1 -- 172.21.15.72:0/2523225653 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f475c161d60 msgr2=0x7f475c06f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:03.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.650+0000 7f4760a65640 1 --2- 172.21.15.72:0/2523225653 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f475c161d60 0x7f475c06f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.650+0000 7f4760a65640 1 -- 172.21.15.72:0/2523225653 >> v2:172.21.15.175:3300/0 conn(0x7f475c06fb30 msgr2=0x7f475c070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:03.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.650+0000 7f4760a65640 1 --2- 172.21.15.72:0/2523225653 >> v2:172.21.15.175:3300/0 conn(0x7f475c06fb30 0x7f475c070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.650+0000 7f4760a65640 1 -- 172.21.15.72:0/2523225653 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f475c0b42d0 con 0x7f475c070850 2024-04-15T15:15:03.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.650+0000 7f4753fff640 1 --2- 172.21.15.72:0/2523225653 >> v2:172.21.15.175:3300/0 conn(0x7f475c06fb30 0x7f475c070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:03.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.651+0000 7f4760a65640 1 --2- 172.21.15.72:0/2523225653 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f475c070850 0x7f475c0cf1a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f475c0cea50 tx=0x7f475402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57c70419b6a1de27 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:03.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.651+0000 7f475b7fe640 1 -- 172.21.15.72:0/2523225653 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475402ca70 con 0x7f475c070850 2024-04-15T15:15:03.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.651+0000 7f475b7fe640 1 -- 172.21.15.72:0/2523225653 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f475402cc10 con 0x7f475c070850 2024-04-15T15:15:03.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.651+0000 7f475b7fe640 1 -- 172.21.15.72:0/2523225653 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47540343f0 con 0x7f475c070850 2024-04-15T15:15:03.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.651+0000 7f47624c8640 1 -- 172.21.15.72:0/2523225653 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f475c070850 msgr2=0x7f475c0cf1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:03.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.651+0000 7f47624c8640 1 --2- 172.21.15.72:0/2523225653 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f475c070850 0x7f475c0cf1a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f475c0cea50 tx=0x7f475402bed0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.651+0000 7f47624c8640 1 -- 172.21.15.72:0/2523225653 shutdown_connections 2024-04-15T15:15:03.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.651+0000 7f47624c8640 1 --2- 172.21.15.72:0/2523225653 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f475c070850 0x7f475c0cf1a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.651+0000 7f47624c8640 1 --2- 172.21.15.72:0/2523225653 >> v2:172.21.15.175:3300/0 conn(0x7f475c06fb30 0x7f475c070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.651+0000 7f47624c8640 1 --2- 172.21.15.72:0/2523225653 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f475c161d60 0x7f475c06f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.651+0000 7f47624c8640 1 -- 172.21.15.72:0/2523225653 >> 172.21.15.72:0/2523225653 conn(0x7f475c0cdd20 msgr2=0x7f475c0ce100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:03.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.651+0000 7f47624c8640 1 -- 172.21.15.72:0/2523225653 shutdown_connections 2024-04-15T15:15:03.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.652+0000 7f47624c8640 1 -- 172.21.15.72:0/2523225653 wait complete. 2024-04-15T15:15:03.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.653+0000 7f47624c8640 1 Processor -- start 2024-04-15T15:15:03.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.653+0000 7f47624c8640 1 -- start start 2024-04-15T15:15:03.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.653+0000 7f47624c8640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f475c06fb30 0x7f475c06cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.653+0000 7f47624c8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f475c070850 0x7f475c141aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.653+0000 7f47624c8640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f475c161d60 0x7f475c1f3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.653+0000 7f47624c8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f475c0b4b70 con 0x7f475c070850 2024-04-15T15:15:03.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.653+0000 7f47624c8640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f475c0b4890 con 0x7f475c06fb30 2024-04-15T15:15:03.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.653+0000 7f47624c8640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f475c0b4e50 con 0x7f475c161d60 2024-04-15T15:15:03.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.653+0000 7f4753fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f475c070850 0x7f475c141aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:03.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.654+0000 7f4753fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f475c070850 0x7f475c141aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44334/0 (socket says 172.21.15.72:44334) 2024-04-15T15:15:03.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.654+0000 7f4753fff640 1 -- 172.21.15.72:0/4138981818 learned_addr learned my addr 172.21.15.72:0/4138981818 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:03.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.654+0000 7f475bfff640 1 --2- 172.21.15.72:0/4138981818 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f475c06fb30 0x7f475c06cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:03.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.654+0000 7f4760a65640 1 --2- 172.21.15.72:0/4138981818 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f475c161d60 0x7f475c1f3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:03.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.654+0000 7f4753fff640 1 -- 172.21.15.72:0/4138981818 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f475c161d60 msgr2=0x7f475c1f3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:03.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.654+0000 7f4753fff640 1 --2- 172.21.15.72:0/4138981818 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f475c161d60 0x7f475c1f3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.654+0000 7f4753fff640 1 -- 172.21.15.72:0/4138981818 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f475c06fb30 msgr2=0x7f475c06cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:03.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.654+0000 7f4753fff640 1 --2- 172.21.15.72:0/4138981818 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f475c06fb30 0x7f475c06cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.654+0000 7f4753fff640 1 -- 172.21.15.72:0/4138981818 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f475c06d310 con 0x7f475c070850 2024-04-15T15:15:03.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.654+0000 7f475bfff640 1 --2- 172.21.15.72:0/4138981818 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f475c06fb30 0x7f475c06cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:03.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.654+0000 7f4753fff640 1 --2- 172.21.15.72:0/4138981818 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f475c070850 0x7f475c141aa0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f474400c7c0 tx=0x7f474400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:03.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.654+0000 7f47597fa640 1 -- 172.21.15.72:0/4138981818 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4744015070 con 0x7f475c070850 2024-04-15T15:15:03.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.654+0000 7f47597fa640 1 -- 172.21.15.72:0/4138981818 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4744007a30 con 0x7f475c070850 2024-04-15T15:15:03.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.654+0000 7f47597fa640 1 -- 172.21.15.72:0/4138981818 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4744007e00 con 0x7f475c070850 2024-04-15T15:15:03.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.654+0000 7f47624c8640 1 -- 172.21.15.72:0/4138981818 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f475c06d5f0 con 0x7f475c070850 2024-04-15T15:15:03.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.654+0000 7f47624c8640 1 -- 172.21.15.72:0/4138981818 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f475c142590 con 0x7f475c070850 2024-04-15T15:15:03.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.655+0000 7f47624c8640 1 -- 172.21.15.72:0/4138981818 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f475c0690f0 con 0x7f475c070850 2024-04-15T15:15:03.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.656+0000 7f47597fa640 1 -- 172.21.15.72:0/4138981818 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4744018070 con 0x7f475c070850 2024-04-15T15:15:03.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.656+0000 7f47597fa640 1 --2- 172.21.15.72:0/4138981818 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f473c06c8f0 0x7f473c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.656+0000 7f47597fa640 1 -- 172.21.15.72:0/4138981818 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f474408bd70 con 0x7f475c070850 2024-04-15T15:15:03.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.656+0000 7f475bfff640 1 --2- 172.21.15.72:0/4138981818 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f473c06c8f0 0x7f473c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:03.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.656+0000 7f475bfff640 1 --2- 172.21.15.72:0/4138981818 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f473c06c8f0 0x7f473c06edb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f474c00b820 tx=0x7f474c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:03.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.660+0000 7f47597fa640 1 -- 172.21.15.72:0/4138981818 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4744059e60 con 0x7f475c070850 2024-04-15T15:15:03.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.838+0000 7f47624c8640 1 -- 172.21.15.72:0/4138981818 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f475c0ce2a0 con 0x7f475c070850 2024-04-15T15:15:03.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.839+0000 7f47597fa640 1 -- 172.21.15.72:0/4138981818 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7f4744059760 con 0x7f475c070850 2024-04-15T15:15:03.841 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:03.841 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":24,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:14:56.451113+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:14:49.438366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:14:50.368379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:14:54.081157+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:03.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.841+0000 7f47624c8640 1 -- 172.21.15.72:0/4138981818 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f473c06c8f0 msgr2=0x7f473c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:03.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.841+0000 7f47624c8640 1 --2- 172.21.15.72:0/4138981818 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f473c06c8f0 0x7f473c06edb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f474c00b820 tx=0x7f474c008040 comp rx=0 tx=0).stop 2024-04-15T15:15:03.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.841+0000 7f47624c8640 1 -- 172.21.15.72:0/4138981818 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f475c070850 msgr2=0x7f475c141aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:03.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.841+0000 7f47624c8640 1 --2- 172.21.15.72:0/4138981818 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f475c070850 0x7f475c141aa0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f474400c7c0 tx=0x7f474400cc80 comp rx=0 tx=0).stop 2024-04-15T15:15:03.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.841+0000 7f47624c8640 1 -- 172.21.15.72:0/4138981818 shutdown_connections 2024-04-15T15:15:03.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.841+0000 7f47624c8640 1 --2- 172.21.15.72:0/4138981818 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f473c06c8f0 0x7f473c06edb0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.841+0000 7f47624c8640 1 --2- 172.21.15.72:0/4138981818 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f475c161d60 0x7f475c1f3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.841+0000 7f47624c8640 1 --2- 172.21.15.72:0/4138981818 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f475c070850 0x7f475c141aa0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.841+0000 7f47624c8640 1 --2- 172.21.15.72:0/4138981818 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f475c06fb30 0x7f475c06cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.841+0000 7f47624c8640 1 -- 172.21.15.72:0/4138981818 >> 172.21.15.72:0/4138981818 conn(0x7f475c0cdd20 msgr2=0x7f475c15eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:03.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.841+0000 7f47624c8640 1 -- 172.21.15.72:0/4138981818 shutdown_connections 2024-04-15T15:15:03.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.841+0000 7f47624c8640 1 -- 172.21.15.72:0/4138981818 wait complete. 2024-04-15T15:15:03.854 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:39.359102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:14:49.438366+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:14:50.368379+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:14:54.081157+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:15:03.854 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:03.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.958+0000 7fb8965e9640 1 Processor -- start 2024-04-15T15:15:03.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.958+0000 7fb8965e9640 1 -- start start 2024-04-15T15:15:03.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.959+0000 7fb8965e9640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb89015e960 0x7fb8900d1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.959+0000 7fb8965e9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb8900d5010 0x7fb8900cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.959+0000 7fb8965e9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8900b65b0 con 0x7fb8900b11d0 2024-04-15T15:15:03.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.959+0000 7fb8965e9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fb8900b6890 con 0x7fb8900d5010 2024-04-15T15:15:03.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.959+0000 7fb8965e9640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fb8900b6b70 con 0x7fb89015e960 2024-04-15T15:15:03.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.959+0000 7fb894b86640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb8900d5010 0x7fb8900cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:03.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.959+0000 7fb88f7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fb8900b11d0 0x7fb8900d4900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45014/0 (socket says 172.21.15.72:45014) 2024-04-15T15:15:03.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.959+0000 7fb88f7fe640 1 -- 172.21.15.72:0/669437434 learned_addr learned my addr 172.21.15.72:0/669437434 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:03.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.960+0000 7fb88effd640 1 -- 172.21.15.72:0/669437434 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3694266437 0 0) 0x7fb8900b65b0 con 0x7fb8900b11d0 2024-04-15T15:15:03.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.960+0000 7fb88effd640 1 -- 172.21.15.72:0/669437434 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb878003520 con 0x7fb8900b11d0 2024-04-15T15:15:03.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.960+0000 7fb894b86640 1 -- 172.21.15.72:0/669437434 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb89015e960 msgr2=0x7fb8900d1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:03.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.960+0000 7fb894b86640 1 --2- 172.21.15.72:0/669437434 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb89015e960 0x7fb8900d1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.960+0000 7fb894b86640 1 -- 172.21.15.72:0/669437434 >> v1:172.21.15.72:6789/0 conn(0x7fb8900b11d0 legacy=0x7fb8900d4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:03.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.960+0000 7fb894b86640 1 -- 172.21.15.72:0/669437434 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8900b6e50 con 0x7fb8900d5010 2024-04-15T15:15:03.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.960+0000 7fb894b86640 1 --2- 172.21.15.72:0/669437434 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb8900d5010 0x7fb8900cffe0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb888009d90 tx=0x7fb88802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8fb46774db4f41a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:03.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.961+0000 7fb88effd640 1 -- 172.21.15.72:0/669437434 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb888039070 con 0x7fb8900d5010 2024-04-15T15:15:03.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.961+0000 7fb88effd640 1 -- 172.21.15.72:0/669437434 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb88802cc50 con 0x7fb8900d5010 2024-04-15T15:15:03.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.961+0000 7fb88effd640 1 -- 172.21.15.72:0/669437434 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb888034440 con 0x7fb8900d5010 2024-04-15T15:15:03.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.961+0000 7fb8965e9640 1 -- 172.21.15.72:0/669437434 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb8900d5010 msgr2=0x7fb8900cffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:03.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.961+0000 7fb8965e9640 1 --2- 172.21.15.72:0/669437434 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb8900d5010 0x7fb8900cffe0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb888009d90 tx=0x7fb88802bea0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.961+0000 7fb8965e9640 1 -- 172.21.15.72:0/669437434 shutdown_connections 2024-04-15T15:15:03.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.961+0000 7fb8965e9640 1 --2- 172.21.15.72:0/669437434 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb8900d5010 0x7fb8900cffe0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.961+0000 7fb8965e9640 1 --2- 172.21.15.72:0/669437434 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb89015e960 0x7fb8900d1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.961+0000 7fb8965e9640 1 -- 172.21.15.72:0/669437434 >> 172.21.15.72:0/669437434 conn(0x7fb8900b9580 msgr2=0x7fb8900b9990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:03.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.961+0000 7fb8965e9640 1 -- 172.21.15.72:0/669437434 shutdown_connections 2024-04-15T15:15:03.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.961+0000 7fb8965e9640 1 -- 172.21.15.72:0/669437434 wait complete. 2024-04-15T15:15:03.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.962+0000 7fb8965e9640 1 Processor -- start 2024-04-15T15:15:03.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.963+0000 7fb8965e9640 1 -- start start 2024-04-15T15:15:03.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.963+0000 7fb8965e9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb8900b11d0 0x7fb8901f77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.963+0000 7fb8965e9640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb8900d5010 0x7fb890156150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.963+0000 7fb8965e9640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb89015e960 0x7fb89006e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.963+0000 7fb8965e9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb8900cfb40 con 0x7fb8900b11d0 2024-04-15T15:15:03.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.963+0000 7fb8965e9640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fb8900cf9c0 con 0x7fb89015e960 2024-04-15T15:15:03.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.963+0000 7fb8965e9640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fb8900b6040 con 0x7fb8900d5010 2024-04-15T15:15:03.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.963+0000 7fb88ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb8900b11d0 0x7fb8901f77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:03.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.963+0000 7fb88ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb8900b11d0 0x7fb8901f77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44366/0 (socket says 172.21.15.72:44366) 2024-04-15T15:15:03.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.963+0000 7fb88ffff640 1 -- 172.21.15.72:0/3212153744 learned_addr learned my addr 172.21.15.72:0/3212153744 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:03.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.963+0000 7fb894b86640 1 --2- 172.21.15.72:0/3212153744 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb89015e960 0x7fb89006e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:03.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.964+0000 7fb88f7fe640 1 --2- 172.21.15.72:0/3212153744 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb8900d5010 0x7fb890156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:03.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.964+0000 7fb88ffff640 1 -- 172.21.15.72:0/3212153744 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb8900d5010 msgr2=0x7fb890156150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:03.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.964+0000 7fb88ffff640 1 --2- 172.21.15.72:0/3212153744 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb8900d5010 0x7fb890156150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.964+0000 7fb88ffff640 1 -- 172.21.15.72:0/3212153744 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb89015e960 msgr2=0x7fb89006e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:03.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.964+0000 7fb88ffff640 1 --2- 172.21.15.72:0/3212153744 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb89015e960 0x7fb89006e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.964+0000 7fb88ffff640 1 -- 172.21.15.72:0/3212153744 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8901f86c0 con 0x7fb8900b11d0 2024-04-15T15:15:03.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.964+0000 7fb894b86640 1 --2- 172.21.15.72:0/3212153744 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb89015e960 0x7fb89006e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:03.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.964+0000 7fb88f7fe640 1 --2- 172.21.15.72:0/3212153744 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb8900d5010 0x7fb890156150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:03.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.964+0000 7fb88ffff640 1 --2- 172.21.15.72:0/3212153744 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb8900b11d0 0x7fb8901f77b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb88000b410 tx=0x7fb88000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:03.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.964+0000 7fb88cff9640 1 -- 172.21.15.72:0/3212153744 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb880015020 con 0x7fb8900b11d0 2024-04-15T15:15:03.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.964+0000 7fb88cff9640 1 -- 172.21.15.72:0/3212153744 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb880011850 con 0x7fb8900b11d0 2024-04-15T15:15:03.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.964+0000 7fb8965e9640 1 -- 172.21.15.72:0/3212153744 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8901f7e20 con 0x7fb8900b11d0 2024-04-15T15:15:03.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.964+0000 7fb88cff9640 1 -- 172.21.15.72:0/3212153744 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb880011c40 con 0x7fb8900b11d0 2024-04-15T15:15:03.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.964+0000 7fb8965e9640 1 -- 172.21.15.72:0/3212153744 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8901f83e0 con 0x7fb8900b11d0 2024-04-15T15:15:03.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.965+0000 7fb8965e9640 1 -- 172.21.15.72:0/3212153744 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb890075c50 con 0x7fb8900b11d0 2024-04-15T15:15:03.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.965+0000 7fb88cff9640 1 -- 172.21.15.72:0/3212153744 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb880018020 con 0x7fb8900b11d0 2024-04-15T15:15:03.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.966+0000 7fb88cff9640 1 --2- 172.21.15.72:0/3212153744 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb87c06c8f0 0x7fb87c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.966+0000 7fb88cff9640 1 -- 172.21.15.72:0/3212153744 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb88008bd30 con 0x7fb8900b11d0 2024-04-15T15:15:03.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.966+0000 7fb88f7fe640 1 --2- 172.21.15.72:0/3212153744 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb87c06c8f0 0x7fb87c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:03.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.966+0000 7fb88f7fe640 1 --2- 172.21.15.72:0/3212153744 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb87c06c8f0 0x7fb87c06edb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb8900a23c0 tx=0x7fb884009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:03.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:03.970+0000 7fb88cff9640 1 -- 172.21.15.72:0/3212153744 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb88001f020 con 0x7fb8900b11d0 2024-04-15T15:15:04.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.149+0000 7fb8965e9640 1 -- 172.21.15.72:0/3212153744 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb89015c240 con 0x7fb8900b11d0 2024-04-15T15:15:04.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.149+0000 7fb88cff9640 1 -- 172.21.15.72:0/3212153744 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7fb8800172c0 con 0x7fb8900b11d0 2024-04-15T15:15:04.151 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-15T15:15:04.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.151+0000 7fb8965e9640 1 -- 172.21.15.72:0/3212153744 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb87c06c8f0 msgr2=0x7fb87c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:04.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.151+0000 7fb8965e9640 1 --2- 172.21.15.72:0/3212153744 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb87c06c8f0 0x7fb87c06edb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb8900a23c0 tx=0x7fb884009040 comp rx=0 tx=0).stop 2024-04-15T15:15:04.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.151+0000 7fb8965e9640 1 -- 172.21.15.72:0/3212153744 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb8900b11d0 msgr2=0x7fb8901f77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:04.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.151+0000 7fb8965e9640 1 --2- 172.21.15.72:0/3212153744 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb8900b11d0 0x7fb8901f77b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb88000b410 tx=0x7fb88000b8d0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.151+0000 7fb8965e9640 1 -- 172.21.15.72:0/3212153744 shutdown_connections 2024-04-15T15:15:04.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.151+0000 7fb8965e9640 1 --2- 172.21.15.72:0/3212153744 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb87c06c8f0 0x7fb87c06edb0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.151+0000 7fb8965e9640 1 --2- 172.21.15.72:0/3212153744 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb89015e960 0x7fb89006e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.151+0000 7fb8965e9640 1 --2- 172.21.15.72:0/3212153744 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb8900d5010 0x7fb890156150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.151+0000 7fb8965e9640 1 --2- 172.21.15.72:0/3212153744 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb8900b11d0 0x7fb8901f77b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.151+0000 7fb8965e9640 1 -- 172.21.15.72:0/3212153744 >> 172.21.15.72:0/3212153744 conn(0x7fb8900b9580 msgr2=0x7fb8900b9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:04.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.152+0000 7fb8965e9640 1 -- 172.21.15.72:0/3212153744 shutdown_connections 2024-04-15T15:15:04.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.152+0000 7fb8965e9640 1 -- 172.21.15.72:0/3212153744 wait complete. 2024-04-15T15:15:04.164 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:15:04.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.271+0000 7f8b0f104640 1 Processor -- start 2024-04-15T15:15:04.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.271+0000 7f8b0f104640 1 -- start start 2024-04-15T15:15:04.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.272+0000 7f8b0f104640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8b080d5010 0x7f8b081693e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:04.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.272+0000 7f8b0f104640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b080b65b0 con 0x7f8b080b11d0 2024-04-15T15:15:04.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.272+0000 7f8b0f104640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b080b6890 con 0x7f8b0815e960 2024-04-15T15:15:04.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.272+0000 7f8b0f104640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f8b080b6b70 con 0x7f8b080d5010 2024-04-15T15:15:04.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.272+0000 7f8b07fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f8b080b11d0 0x7f8b080d4900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45018/0 (socket says 172.21.15.72:45018) 2024-04-15T15:15:04.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.272+0000 7f8b07fff640 1 -- 172.21.15.72:0/4268287238 learned_addr learned my addr 172.21.15.72:0/4268287238 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:04.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.272+0000 7f8b0d6a1640 1 --2- 172.21.15.72:0/4268287238 >> v2:172.21.15.175:3300/0 conn(0x7f8b080d5010 0x7f8b081693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:04.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.272+0000 7f8b077fe640 1 -- 172.21.15.72:0/4268287238 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3705343716 0 0) 0x7f8b080b65b0 con 0x7f8b080b11d0 2024-04-15T15:15:04.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.272+0000 7f8b077fe640 1 -- 172.21.15.72:0/4268287238 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8af8003520 con 0x7f8b080b11d0 2024-04-15T15:15:04.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.272+0000 7f8b077fe640 1 -- 172.21.15.72:0/4268287238 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2442396555 0 0) 0x7f8af8003520 con 0x7f8b080b11d0 2024-04-15T15:15:04.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.272+0000 7f8b077fe640 1 -- 172.21.15.72:0/4268287238 >> v2:172.21.15.175:3300/0 conn(0x7f8b080d5010 msgr2=0x7f8b081693e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:04.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.272+0000 7f8b077fe640 1 --2- 172.21.15.72:0/4268287238 >> v2:172.21.15.175:3300/0 conn(0x7f8b080d5010 0x7f8b081693e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.272+0000 7f8b077fe640 1 -- 172.21.15.72:0/4268287238 >> v1:172.21.15.175:6789/0 conn(0x7f8b0815e960 legacy=0x7f8b080d1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:04.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.272+0000 7f8b077fe640 1 -- 172.21.15.72:0/4268287238 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b080b6e50 con 0x7f8b080b11d0 2024-04-15T15:15:04.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.272+0000 7f8b077fe640 1 -- 172.21.15.72:0/4268287238 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f8af4001670 con 0x7f8b080b11d0 2024-04-15T15:15:04.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.272+0000 7f8b077fe640 1 -- 172.21.15.72:0/4268287238 >> v1:172.21.15.72:6789/0 conn(0x7f8b080b11d0 legacy=0x7f8b080d4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:04.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.272+0000 7f8b077fe640 1 --2- 172.21.15.72:0/4268287238 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8af80039b0 0x7f8af8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:04.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.272+0000 7f8b077fe640 1 --2- 172.21.15.72:0/4268287238 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8af8004800 0x7f8af8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:04.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.272+0000 7f8b077fe640 1 --2- 172.21.15.72:0/4268287238 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8af80051c0 0x7f8af800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:04.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.272+0000 7f8b077fe640 1 -- 172.21.15.72:0/4268287238 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8b080b6b70 con 0x7f8af8004800 2024-04-15T15:15:04.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.272+0000 7f8b077fe640 1 -- 172.21.15.72:0/4268287238 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f8af800c060 con 0x7f8af80051c0 2024-04-15T15:15:04.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.272+0000 7f8b077fe640 1 -- 172.21.15.72:0/4268287238 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f8af800c260 con 0x7f8af80039b0 2024-04-15T15:15:04.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.272+0000 7f8b07fff640 1 --2- 172.21.15.72:0/4268287238 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8af8004800 0x7f8af8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:04.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.273+0000 7f8b0d6a1640 1 --2- 172.21.15.72:0/4268287238 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8af80051c0 0x7f8af800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:04.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.273+0000 7f8b0cea0640 1 --2- 172.21.15.72:0/4268287238 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8af80039b0 0x7f8af8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:04.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.273+0000 7f8b07fff640 1 -- 172.21.15.72:0/4268287238 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8af80039b0 msgr2=0x7f8af8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:04.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.273+0000 7f8b07fff640 1 --2- 172.21.15.72:0/4268287238 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8af80039b0 0x7f8af8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.273+0000 7f8b07fff640 1 -- 172.21.15.72:0/4268287238 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8af80051c0 msgr2=0x7f8af800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:04.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.273+0000 7f8b07fff640 1 --2- 172.21.15.72:0/4268287238 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8af80051c0 0x7f8af800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.273+0000 7f8b0d6a1640 1 -- 172.21.15.72:0/4268287238 reap_dead start 2024-04-15T15:15:04.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.273+0000 7f8b07fff640 1 -- 172.21.15.72:0/4268287238 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8af8003520 con 0x7f8af8004800 2024-04-15T15:15:04.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.273+0000 7f8b07fff640 1 --2- 172.21.15.72:0/4268287238 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8af8004800 0x7f8af8004c80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8af400c210 tx=0x7f8af8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a19ad8557378ba86 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:04.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.274+0000 7f8b077fe640 1 -- 172.21.15.72:0/4268287238 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af4007d10 con 0x7f8af8004800 2024-04-15T15:15:04.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.274+0000 7f8b077fe640 1 -- 172.21.15.72:0/4268287238 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8af4030070 con 0x7f8af8004800 2024-04-15T15:15:04.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.274+0000 7f8b077fe640 1 -- 172.21.15.72:0/4268287238 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af4005d70 con 0x7f8af8004800 2024-04-15T15:15:04.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.274+0000 7f8b0f104640 1 -- 172.21.15.72:0/4268287238 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8af8004800 msgr2=0x7f8af8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:04.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.274+0000 7f8b0f104640 1 --2- 172.21.15.72:0/4268287238 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8af8004800 0x7f8af8004c80 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8af400c210 tx=0x7f8af8005830 comp rx=0 tx=0).stop 2024-04-15T15:15:04.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.274+0000 7f8b0f104640 1 -- 172.21.15.72:0/4268287238 shutdown_connections 2024-04-15T15:15:04.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.274+0000 7f8b0f104640 1 --2- 172.21.15.72:0/4268287238 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8af8004800 0x7f8af8004c80 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.274+0000 7f8b0f104640 1 -- 172.21.15.72:0/4268287238 >> 172.21.15.72:0/4268287238 conn(0x7f8b080b9580 msgr2=0x7f8b080b9990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:04.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.274+0000 7f8b0f104640 1 -- 172.21.15.72:0/4268287238 shutdown_connections 2024-04-15T15:15:04.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.274+0000 7f8b0f104640 1 -- 172.21.15.72:0/4268287238 wait complete. 2024-04-15T15:15:04.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.275+0000 7f8b0f104640 1 Processor -- start 2024-04-15T15:15:04.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.275+0000 7f8b0f104640 1 -- start start 2024-04-15T15:15:04.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.276+0000 7f8b0f104640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8af8004800 0x7f8b08152b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:04.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.276+0000 7f8b0f104640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b0806e2a0 0x7f8b0815ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:04.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.276+0000 7f8b0f104640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8b0806e720 0x7f8b080cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:04.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.276+0000 7f8b0f104640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8b080b61c0 con 0x7f8af8004800 2024-04-15T15:15:04.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.276+0000 7f8b0f104640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f8b080b6040 con 0x7f8b0806e2a0 2024-04-15T15:15:04.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.276+0000 7f8b0f104640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f8b080b6340 con 0x7f8b0806e720 2024-04-15T15:15:04.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.276+0000 7f8b0cea0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8af8004800 0x7f8b08152b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:04.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.276+0000 7f8b0cea0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8af8004800 0x7f8b08152b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44392/0 (socket says 172.21.15.72:44392) 2024-04-15T15:15:04.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.276+0000 7f8b0cea0640 1 -- 172.21.15.72:0/1747603393 learned_addr learned my addr 172.21.15.72:0/1747603393 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:04.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.276+0000 7f8b07fff640 1 --2- 172.21.15.72:0/1747603393 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b0806e2a0 0x7f8b0815ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:04.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.276+0000 7f8b0d6a1640 1 --2- 172.21.15.72:0/1747603393 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8b0806e720 0x7f8b080cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:04.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.277+0000 7f8b0cea0640 1 -- 172.21.15.72:0/1747603393 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8b0806e720 msgr2=0x7f8b080cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:04.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.277+0000 7f8b0cea0640 1 --2- 172.21.15.72:0/1747603393 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8b0806e720 0x7f8b080cdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.277+0000 7f8b0cea0640 1 -- 172.21.15.72:0/1747603393 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b0806e2a0 msgr2=0x7f8b0815ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:04.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.277+0000 7f8b0cea0640 1 --2- 172.21.15.72:0/1747603393 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b0806e2a0 0x7f8b0815ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.277+0000 7f8b0cea0640 1 -- 172.21.15.72:0/1747603393 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b080ccba0 con 0x7f8af8004800 2024-04-15T15:15:04.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.277+0000 7f8b0d6a1640 1 --2- 172.21.15.72:0/1747603393 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8b0806e720 0x7f8b080cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:04.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.277+0000 7f8b0cea0640 1 --2- 172.21.15.72:0/1747603393 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8af8004800 0x7f8b08152b40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8afc00f5f0 tx=0x7f8afc00fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:04.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.277+0000 7f8b057fa640 1 -- 172.21.15.72:0/1747603393 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8afc0052e0 con 0x7f8af8004800 2024-04-15T15:15:04.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.277+0000 7f8b057fa640 1 -- 172.21.15.72:0/1747603393 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8afc018070 con 0x7f8af8004800 2024-04-15T15:15:04.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.277+0000 7f8b0f104640 1 -- 172.21.15.72:0/1747603393 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b080cce80 con 0x7f8af8004800 2024-04-15T15:15:04.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.277+0000 7f8b057fa640 1 -- 172.21.15.72:0/1747603393 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8afc003810 con 0x7f8af8004800 2024-04-15T15:15:04.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.277+0000 7f8b0f104640 1 -- 172.21.15.72:0/1747603393 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b080cd440 con 0x7f8af8004800 2024-04-15T15:15:04.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.278+0000 7f8b0f104640 1 -- 172.21.15.72:0/1747603393 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b08075c50 con 0x7f8af8004800 2024-04-15T15:15:04.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.278+0000 7f8b057fa640 1 -- 172.21.15.72:0/1747603393 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8afc0039b0 con 0x7f8af8004800 2024-04-15T15:15:04.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.279+0000 7f8b057fa640 1 --2- 172.21.15.72:0/1747603393 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8aec06c8f0 0x7f8aec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:04.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.279+0000 7f8b057fa640 1 -- 172.21.15.72:0/1747603393 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f8afc08bb40 con 0x7f8af8004800 2024-04-15T15:15:04.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.279+0000 7f8b07fff640 1 --2- 172.21.15.72:0/1747603393 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8aec06c8f0 0x7f8aec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:04.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.279+0000 7f8b07fff640 1 --2- 172.21.15.72:0/1747603393 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8aec06c8f0 0x7f8aec06edb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8af4034000 tx=0x7f8af4009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:04.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.283+0000 7f8b057fa640 1 -- 172.21.15.72:0/1747603393 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8afc059cc0 con 0x7f8af8004800 2024-04-15T15:15:04.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.463+0000 7f8b0f104640 1 -- 172.21.15.72:0/1747603393 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8b080b1620 con 0x7f8af8004800 2024-04-15T15:15:04.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.464+0000 7f8b057fa640 1 -- 172.21.15.72:0/1747603393 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7f8afc0595c0 con 0x7f8af8004800 2024-04-15T15:15:04.465 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:04.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.465+0000 7f8b0f104640 1 -- 172.21.15.72:0/1747603393 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8aec06c8f0 msgr2=0x7f8aec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:04.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.465+0000 7f8b0f104640 1 --2- 172.21.15.72:0/1747603393 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8aec06c8f0 0x7f8aec06edb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8af4034000 tx=0x7f8af4009030 comp rx=0 tx=0).stop 2024-04-15T15:15:04.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.465+0000 7f8b0f104640 1 -- 172.21.15.72:0/1747603393 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8af8004800 msgr2=0x7f8b08152b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:04.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.465+0000 7f8b0f104640 1 --2- 172.21.15.72:0/1747603393 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8af8004800 0x7f8b08152b40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8afc00f5f0 tx=0x7f8afc00fab0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.466+0000 7f8b0f104640 1 -- 172.21.15.72:0/1747603393 shutdown_connections 2024-04-15T15:15:04.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.466+0000 7f8b0f104640 1 --2- 172.21.15.72:0/1747603393 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8aec06c8f0 0x7f8aec06edb0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.466+0000 7f8b0f104640 1 --2- 172.21.15.72:0/1747603393 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8b0806e720 0x7f8b080cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.466+0000 7f8b0f104640 1 --2- 172.21.15.72:0/1747603393 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b0806e2a0 0x7f8b0815ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.466+0000 7f8b0f104640 1 --2- 172.21.15.72:0/1747603393 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8af8004800 0x7f8b08152b40 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.466+0000 7f8b0f104640 1 -- 172.21.15.72:0/1747603393 >> 172.21.15.72:0/1747603393 conn(0x7f8b080b9580 msgr2=0x7f8b080b9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:04.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.466+0000 7f8b0f104640 1 -- 172.21.15.72:0/1747603393 shutdown_connections 2024-04-15T15:15:04.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.466+0000 7f8b0f104640 1 -- 172.21.15.72:0/1747603393 wait complete. 2024-04-15T15:15:04.478 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:15:04.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.585+0000 7f3d441a8640 1 Processor -- start 2024-04-15T15:15:04.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.586+0000 7f3d441a8640 1 -- start start 2024-04-15T15:15:04.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.586+0000 7f3d441a8640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3d3c161ee0 0x7f3d3c0c95f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:04.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.586+0000 7f3d441a8640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f3d3c0b67c0 0x7f3d3c0d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:04.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.586+0000 7f3d441a8640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d3c0ccfe0 con 0x7f3d3c0b6c20 2024-04-15T15:15:04.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.586+0000 7f3d441a8640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f3d3c1622c0 con 0x7f3d3c0b67c0 2024-04-15T15:15:04.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.586+0000 7f3d441a8640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f3d3c0cd2c0 con 0x7f3d3c161ee0 2024-04-15T15:15:04.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.587+0000 7f3d42745640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f3d3c0b6c20 0x7f3d3c0ce080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45022/0 (socket says 172.21.15.72:45022) 2024-04-15T15:15:04.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.587+0000 7f3d42745640 1 -- 172.21.15.72:0/4059057437 learned_addr learned my addr 172.21.15.72:0/4059057437 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:04.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.587+0000 7f3d41743640 1 --2- 172.21.15.72:0/4059057437 >> v2:172.21.15.175:3300/0 conn(0x7f3d3c0b67c0 0x7f3d3c0d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:04.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.587+0000 7f3d41f44640 1 --2- 172.21.15.72:0/4059057437 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3d3c161ee0 0x7f3d3c0c95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:04.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.587+0000 7f3d40f42640 1 -- 172.21.15.72:0/4059057437 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1474806032 0 0) 0x7f3d3c0ccfe0 con 0x7f3d3c0b6c20 2024-04-15T15:15:04.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.587+0000 7f3d40f42640 1 -- 172.21.15.72:0/4059057437 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d28003520 con 0x7f3d3c0b6c20 2024-04-15T15:15:04.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.587+0000 7f3d40f42640 1 -- 172.21.15.72:0/4059057437 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2737523649 0 0) 0x7f3d28003520 con 0x7f3d3c0b6c20 2024-04-15T15:15:04.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.587+0000 7f3d41743640 1 -- 172.21.15.72:0/4059057437 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3d3c161ee0 msgr2=0x7f3d3c0c95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:04.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.587+0000 7f3d41743640 1 --2- 172.21.15.72:0/4059057437 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3d3c161ee0 0x7f3d3c0c95f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.587+0000 7f3d41743640 1 -- 172.21.15.72:0/4059057437 >> v1:172.21.15.72:6789/0 conn(0x7f3d3c0b6c20 legacy=0x7f3d3c0ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:04.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.587+0000 7f3d41743640 1 -- 172.21.15.72:0/4059057437 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d3c0cd5a0 con 0x7f3d3c0b67c0 2024-04-15T15:15:04.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.588+0000 7f3d41743640 1 --2- 172.21.15.72:0/4059057437 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3d3c0b67c0 0x7f3d3c0d1100 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3d300097d0 tx=0x7f3d3002b770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f488f8e9c6fb7e9c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:04.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.588+0000 7f3d40f42640 1 -- 172.21.15.72:0/4059057437 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d30037070 con 0x7f3d3c0b67c0 2024-04-15T15:15:04.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.588+0000 7f3d40f42640 1 -- 172.21.15.72:0/4059057437 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d30030510 con 0x7f3d3c0b67c0 2024-04-15T15:15:04.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.588+0000 7f3d40f42640 1 -- 172.21.15.72:0/4059057437 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d340019f0 con 0x7f3d3c0b67c0 2024-04-15T15:15:04.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.589+0000 7f3d441a8640 1 -- 172.21.15.72:0/4059057437 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3d3c0b67c0 msgr2=0x7f3d3c0d1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:04.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.589+0000 7f3d441a8640 1 --2- 172.21.15.72:0/4059057437 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3d3c0b67c0 0x7f3d3c0d1100 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3d300097d0 tx=0x7f3d3002b770 comp rx=0 tx=0).stop 2024-04-15T15:15:04.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.589+0000 7f3d441a8640 1 -- 172.21.15.72:0/4059057437 shutdown_connections 2024-04-15T15:15:04.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.589+0000 7f3d441a8640 1 --2- 172.21.15.72:0/4059057437 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3d3c0b67c0 0x7f3d3c0d1100 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.589+0000 7f3d441a8640 1 --2- 172.21.15.72:0/4059057437 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3d3c161ee0 0x7f3d3c0c95f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.589+0000 7f3d441a8640 1 -- 172.21.15.72:0/4059057437 >> 172.21.15.72:0/4059057437 conn(0x7f3d3c0b4010 msgr2=0x7f3d3c0b4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:04.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.589+0000 7f3d441a8640 1 -- 172.21.15.72:0/4059057437 shutdown_connections 2024-04-15T15:15:04.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.589+0000 7f3d441a8640 1 -- 172.21.15.72:0/4059057437 wait complete. 2024-04-15T15:15:04.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.590+0000 7f3d441a8640 1 Processor -- start 2024-04-15T15:15:04.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.590+0000 7f3d441a8640 1 -- start start 2024-04-15T15:15:04.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.591+0000 7f3d441a8640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3d3c0b67c0 0x7f3d3c159f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:04.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.591+0000 7f3d441a8640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3d3c0b6c20 0x7f3d3c1f4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:04.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.591+0000 7f3d441a8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d3c161ee0 0x7f3d3c1f4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:04.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.591+0000 7f3d441a8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3c0ccb70 con 0x7f3d3c161ee0 2024-04-15T15:15:04.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.591+0000 7f3d441a8640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3c0c9b30 con 0x7f3d3c0b67c0 2024-04-15T15:15:04.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.591+0000 7f3d441a8640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f3d3c0cccf0 con 0x7f3d3c0b6c20 2024-04-15T15:15:04.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.591+0000 7f3d42745640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d3c161ee0 0x7f3d3c1f4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:04.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.591+0000 7f3d42745640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d3c161ee0 0x7f3d3c1f4e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44406/0 (socket says 172.21.15.72:44406) 2024-04-15T15:15:04.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.591+0000 7f3d42745640 1 -- 172.21.15.72:0/3862743088 learned_addr learned my addr 172.21.15.72:0/3862743088 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:04.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.591+0000 7f3d41f44640 1 --2- 172.21.15.72:0/3862743088 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3d3c0b67c0 0x7f3d3c159f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:04.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.591+0000 7f3d41743640 1 --2- 172.21.15.72:0/3862743088 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3d3c0b6c20 0x7f3d3c1f4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:04.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.591+0000 7f3d42745640 1 -- 172.21.15.72:0/3862743088 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3d3c0b6c20 msgr2=0x7f3d3c1f4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:04.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.591+0000 7f3d42745640 1 --2- 172.21.15.72:0/3862743088 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3d3c0b6c20 0x7f3d3c1f4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.591+0000 7f3d42745640 1 -- 172.21.15.72:0/3862743088 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3d3c0b67c0 msgr2=0x7f3d3c159f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:04.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.591+0000 7f3d42745640 1 --2- 172.21.15.72:0/3862743088 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3d3c0b67c0 0x7f3d3c159f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.591+0000 7f3d42745640 1 -- 172.21.15.72:0/3862743088 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d3c15c5e0 con 0x7f3d3c161ee0 2024-04-15T15:15:04.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.591+0000 7f3d41f44640 1 --2- 172.21.15.72:0/3862743088 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3d3c0b67c0 0x7f3d3c159f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:04.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.591+0000 7f3d42745640 1 --2- 172.21.15.72:0/3862743088 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d3c161ee0 0x7f3d3c1f4e60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3d340018c0 tx=0x7f3d3400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:04.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.591+0000 7f3d2e7fc640 1 -- 172.21.15.72:0/3862743088 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d34012400 con 0x7f3d3c161ee0 2024-04-15T15:15:04.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.591+0000 7f3d2e7fc640 1 -- 172.21.15.72:0/3862743088 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d340125c0 con 0x7f3d3c161ee0 2024-04-15T15:15:04.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.591+0000 7f3d441a8640 1 -- 172.21.15.72:0/3862743088 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d3c15c8c0 con 0x7f3d3c161ee0 2024-04-15T15:15:04.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.591+0000 7f3d2e7fc640 1 -- 172.21.15.72:0/3862743088 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d340129b0 con 0x7f3d3c161ee0 2024-04-15T15:15:04.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.591+0000 7f3d441a8640 1 -- 172.21.15.72:0/3862743088 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d3c15ce80 con 0x7f3d3c161ee0 2024-04-15T15:15:04.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.592+0000 7f3d441a8640 1 -- 172.21.15.72:0/3862743088 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d3c075c50 con 0x7f3d3c161ee0 2024-04-15T15:15:04.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.593+0000 7f3d2e7fc640 1 -- 172.21.15.72:0/3862743088 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3d34012b50 con 0x7f3d3c161ee0 2024-04-15T15:15:04.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.593+0000 7f3d2e7fc640 1 --2- 172.21.15.72:0/3862743088 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3d2006c8f0 0x7f3d2006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:04.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.593+0000 7f3d2e7fc640 1 -- 172.21.15.72:0/3862743088 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f3d3408c6c0 con 0x7f3d3c161ee0 2024-04-15T15:15:04.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.593+0000 7f3d41f44640 1 --2- 172.21.15.72:0/3862743088 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3d2006c8f0 0x7f3d2006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:04.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.594+0000 7f3d41f44640 1 --2- 172.21.15.72:0/3862743088 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3d2006c8f0 0x7f3d2006edb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3d38009a20 tx=0x7f3d38008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:04.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.598+0000 7f3d2e7fc640 1 -- 172.21.15.72:0/3862743088 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3d340182b0 con 0x7f3d3c161ee0 2024-04-15T15:15:04.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.772+0000 7f3d441a8640 1 -- 172.21.15.72:0/3862743088 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3d3c0d1e50 con 0x7f3d3c161ee0 2024-04-15T15:15:04.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.773+0000 7f3d2e7fc640 1 -- 172.21.15.72:0/3862743088 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7f3d3405a900 con 0x7f3d3c161ee0 2024-04-15T15:15:04.774 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-04-15T15:15:04.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.775+0000 7f3d441a8640 1 -- 172.21.15.72:0/3862743088 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3d2006c8f0 msgr2=0x7f3d2006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:04.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.775+0000 7f3d441a8640 1 --2- 172.21.15.72:0/3862743088 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3d2006c8f0 0x7f3d2006edb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3d38009a20 tx=0x7f3d38008040 comp rx=0 tx=0).stop 2024-04-15T15:15:04.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.775+0000 7f3d441a8640 1 -- 172.21.15.72:0/3862743088 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d3c161ee0 msgr2=0x7f3d3c1f4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:04.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.775+0000 7f3d441a8640 1 --2- 172.21.15.72:0/3862743088 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d3c161ee0 0x7f3d3c1f4e60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3d340018c0 tx=0x7f3d3400a430 comp rx=0 tx=0).stop 2024-04-15T15:15:04.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.775+0000 7f3d441a8640 1 -- 172.21.15.72:0/3862743088 shutdown_connections 2024-04-15T15:15:04.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.775+0000 7f3d441a8640 1 --2- 172.21.15.72:0/3862743088 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3d2006c8f0 0x7f3d2006edb0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.775+0000 7f3d441a8640 1 --2- 172.21.15.72:0/3862743088 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d3c161ee0 0x7f3d3c1f4e60 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.775+0000 7f3d441a8640 1 --2- 172.21.15.72:0/3862743088 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3d3c0b6c20 0x7f3d3c1f4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.775+0000 7f3d441a8640 1 --2- 172.21.15.72:0/3862743088 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3d3c0b67c0 0x7f3d3c159f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.775+0000 7f3d441a8640 1 -- 172.21.15.72:0/3862743088 >> 172.21.15.72:0/3862743088 conn(0x7f3d3c0b4010 msgr2=0x7f3d3c160010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:04.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.775+0000 7f3d441a8640 1 -- 172.21.15.72:0/3862743088 shutdown_connections 2024-04-15T15:15:04.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.775+0000 7f3d441a8640 1 -- 172.21.15.72:0/3862743088 wait complete. 2024-04-15T15:15:04.787 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:15:04.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.891+0000 7f690b40d640 1 Processor -- start 2024-04-15T15:15:04.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.891+0000 7f690b40d640 1 -- start start 2024-04-15T15:15:04.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.892+0000 7f690b40d640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f690406c400 0x7f690406c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:04.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.892+0000 7f690b40d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69040d2320 con 0x7f690415e610 2024-04-15T15:15:04.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.892+0000 7f690b40d640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69040d25c0 con 0x7f690415e9f0 2024-04-15T15:15:04.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.892+0000 7f690b40d640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f69040d31d0 con 0x7f690406c400 2024-04-15T15:15:04.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.892+0000 7f69091a9640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f690415e610 0x7f69040d0690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45032/0 (socket says 172.21.15.72:45032) 2024-04-15T15:15:04.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.892+0000 7f69091a9640 1 -- 172.21.15.72:0/3306378584 learned_addr learned my addr 172.21.15.72:0/3306378584 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:04.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.893+0000 7f68fbfff640 1 -- 172.21.15.72:0/3306378584 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 65165785 0 0) 0x7f69040d2320 con 0x7f690415e610 2024-04-15T15:15:04.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.893+0000 7f69099aa640 1 --2- 172.21.15.72:0/3306378584 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f690406c400 0x7f690406c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:04.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.893+0000 7f68fbfff640 1 -- 172.21.15.72:0/3306378584 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68f0003520 con 0x7f690415e610 2024-04-15T15:15:04.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.893+0000 7f68fbfff640 1 -- 172.21.15.72:0/3306378584 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2449578172 0 0) 0x7f68f0003520 con 0x7f690415e610 2024-04-15T15:15:04.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.893+0000 7f68fbfff640 1 -- 172.21.15.72:0/3306378584 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f690406c400 msgr2=0x7f690406c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:04.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.893+0000 7f68fbfff640 1 --2- 172.21.15.72:0/3306378584 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f690406c400 0x7f690406c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.893+0000 7f68fbfff640 1 -- 172.21.15.72:0/3306378584 >> v1:172.21.15.175:6789/0 conn(0x7f690415e9f0 legacy=0x7f69040d3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:04.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.893+0000 7f68fbfff640 1 -- 172.21.15.72:0/3306378584 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69040d28a0 con 0x7f690415e610 2024-04-15T15:15:04.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.893+0000 7f68fbfff640 1 -- 172.21.15.72:0/3306378584 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f6900001670 con 0x7f690415e610 2024-04-15T15:15:04.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.893+0000 7f68fbfff640 1 -- 172.21.15.72:0/3306378584 >> v1:172.21.15.72:6789/0 conn(0x7f690415e610 legacy=0x7f69040d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:04.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.893+0000 7f68fbfff640 1 --2- 172.21.15.72:0/3306378584 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f68f00039e0 0x7f68f0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:04.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.893+0000 7f68fbfff640 1 --2- 172.21.15.72:0/3306378584 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f690406c400 0x7f68f0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:04.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.893+0000 7f68fbfff640 1 --2- 172.21.15.72:0/3306378584 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68f0004d40 0x7f68f00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:04.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.893+0000 7f68fbfff640 1 -- 172.21.15.72:0/3306378584 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f69040d31d0 con 0x7f68f0004d40 2024-04-15T15:15:04.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.893+0000 7f68fbfff640 1 -- 172.21.15.72:0/3306378584 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f68f000ba70 con 0x7f68f00039e0 2024-04-15T15:15:04.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.893+0000 7f68fbfff640 1 -- 172.21.15.72:0/3306378584 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f68f000bc20 con 0x7f690406c400 2024-04-15T15:15:04.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.893+0000 7f69099aa640 1 --2- 172.21.15.72:0/3306378584 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68f0004d40 0x7f68f00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:04.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.893+0000 7f69091a9640 1 --2- 172.21.15.72:0/3306378584 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f68f00039e0 0x7f68f0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:04.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.893+0000 7f69089a8640 1 --2- 172.21.15.72:0/3306378584 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f690406c400 0x7f68f0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:04.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.893+0000 7f69099aa640 1 -- 172.21.15.72:0/3306378584 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f690406c400 msgr2=0x7f68f0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:04.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.893+0000 7f69099aa640 1 --2- 172.21.15.72:0/3306378584 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f690406c400 0x7f68f0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.893+0000 7f69099aa640 1 -- 172.21.15.72:0/3306378584 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f68f00039e0 msgr2=0x7f68f0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:04.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.893+0000 7f69099aa640 1 --2- 172.21.15.72:0/3306378584 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f68f00039e0 0x7f68f0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.893+0000 7f69099aa640 1 -- 172.21.15.72:0/3306378584 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68f0003520 con 0x7f68f0004d40 2024-04-15T15:15:04.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.893+0000 7f69089a8640 1 --2- 172.21.15.72:0/3306378584 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f690406c400 0x7f68f0004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:04.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.894+0000 7f69099aa640 1 --2- 172.21.15.72:0/3306378584 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68f0004d40 0x7f68f00051c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f68fc009fc0 tx=0x7f68f0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4002a8a3dd12fa5 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:04.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.894+0000 7f68fbfff640 1 -- 172.21.15.72:0/3306378584 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68fc039070 con 0x7f68f0004d40 2024-04-15T15:15:04.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.894+0000 7f68fbfff640 1 -- 172.21.15.72:0/3306378584 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68fc02cc20 con 0x7f68f0004d40 2024-04-15T15:15:04.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.894+0000 7f68fbfff640 1 -- 172.21.15.72:0/3306378584 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68fc009c10 con 0x7f68f0004d40 2024-04-15T15:15:04.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.895+0000 7f690b40d640 1 -- 172.21.15.72:0/3306378584 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68f0004d40 msgr2=0x7f68f00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:04.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.895+0000 7f690b40d640 1 --2- 172.21.15.72:0/3306378584 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f68f0004d40 0x7f68f00051c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f68fc009fc0 tx=0x7f68f0005830 comp rx=0 tx=0).stop 2024-04-15T15:15:04.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.895+0000 7f69099aa640 1 -- 172.21.15.72:0/3306378584 reap_dead start 2024-04-15T15:15:04.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.895+0000 7f690b40d640 1 -- 172.21.15.72:0/3306378584 shutdown_connections 2024-04-15T15:15:04.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.895+0000 7f690b40d640 1 -- 172.21.15.72:0/3306378584 >> 172.21.15.72:0/3306378584 conn(0x7f69040d4750 msgr2=0x7f69040d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:04.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.895+0000 7f690b40d640 1 -- 172.21.15.72:0/3306378584 shutdown_connections 2024-04-15T15:15:04.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.895+0000 7f690b40d640 1 -- 172.21.15.72:0/3306378584 wait complete. 2024-04-15T15:15:04.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.896+0000 7f690b40d640 1 Processor -- start 2024-04-15T15:15:04.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.896+0000 7f690b40d640 1 -- start start 2024-04-15T15:15:04.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.897+0000 7f690b40d640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f68f0004d40 0x7f69041f7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:04.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.897+0000 7f690b40d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69040ce040 0x7f69040ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:04.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.897+0000 7f690b40d640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6904160660 0x7f6904160ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:04.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.897+0000 7f690b40d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f690406d130 con 0x7f69040ce040 2024-04-15T15:15:04.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.897+0000 7f690b40d640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f69040d2e60 con 0x7f68f0004d40 2024-04-15T15:15:04.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.897+0000 7f690b40d640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f690406c890 con 0x7f6904160660 2024-04-15T15:15:04.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.897+0000 7f69089a8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69040ce040 0x7f69040ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:04.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.897+0000 7f69089a8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69040ce040 0x7f69040ce4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44412/0 (socket says 172.21.15.72:44412) 2024-04-15T15:15:04.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.897+0000 7f69089a8640 1 -- 172.21.15.72:0/1992860331 learned_addr learned my addr 172.21.15.72:0/1992860331 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:04.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.897+0000 7f69091a9640 1 --2- 172.21.15.72:0/1992860331 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f68f0004d40 0x7f69041f7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:04.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.897+0000 7f69099aa640 1 --2- 172.21.15.72:0/1992860331 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6904160660 0x7f6904160ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:04.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.897+0000 7f69089a8640 1 -- 172.21.15.72:0/1992860331 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6904160660 msgr2=0x7f6904160ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:04.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.897+0000 7f69089a8640 1 --2- 172.21.15.72:0/1992860331 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6904160660 0x7f6904160ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.897+0000 7f69089a8640 1 -- 172.21.15.72:0/1992860331 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f68f0004d40 msgr2=0x7f69041f7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:04.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.897+0000 7f69089a8640 1 --2- 172.21.15.72:0/1992860331 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f68f0004d40 0x7f69041f7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.897+0000 7f69089a8640 1 -- 172.21.15.72:0/1992860331 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69041613e0 con 0x7f69040ce040 2024-04-15T15:15:04.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.897+0000 7f69089a8640 1 --2- 172.21.15.72:0/1992860331 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69040ce040 0x7f69040ce4c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f68f400e880 tx=0x7f68f400ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:04.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.897+0000 7f68f9ffb640 1 -- 172.21.15.72:0/1992860331 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f4001d30 con 0x7f69040ce040 2024-04-15T15:15:04.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.897+0000 7f68f9ffb640 1 -- 172.21.15.72:0/1992860331 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68f401a070 con 0x7f69040ce040 2024-04-15T15:15:04.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.897+0000 7f690b40d640 1 -- 172.21.15.72:0/1992860331 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69041616c0 con 0x7f69040ce040 2024-04-15T15:15:04.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.897+0000 7f68f9ffb640 1 -- 172.21.15.72:0/1992860331 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f400cab0 con 0x7f69040ce040 2024-04-15T15:15:04.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.898+0000 7f690b40d640 1 -- 172.21.15.72:0/1992860331 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6904161c80 con 0x7f69040ce040 2024-04-15T15:15:04.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.899+0000 7f690b40d640 1 -- 172.21.15.72:0/1992860331 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6904075c50 con 0x7f69040ce040 2024-04-15T15:15:04.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.899+0000 7f68f9ffb640 1 -- 172.21.15.72:0/1992860331 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f68f400cc50 con 0x7f69040ce040 2024-04-15T15:15:04.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.899+0000 7f68f9ffb640 1 --2- 172.21.15.72:0/1992860331 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f68e806cac0 0x7f68e806ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:04.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.899+0000 7f68f9ffb640 1 -- 172.21.15.72:0/1992860331 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f68f408c300 con 0x7f69040ce040 2024-04-15T15:15:04.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.900+0000 7f69091a9640 1 --2- 172.21.15.72:0/1992860331 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f68e806cac0 0x7f68e806ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:04.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.900+0000 7f69091a9640 1 --2- 172.21.15.72:0/1992860331 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f68e806cac0 0x7f68e806ef80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6900001470 tx=0x7f6900009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:04.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:04.904+0000 7f68f9ffb640 1 -- 172.21.15.72:0/1992860331 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f68f405a3f0 con 0x7f69040ce040 2024-04-15T15:15:05.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.080+0000 7f690b40d640 1 -- 172.21.15.72:0/1992860331 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f69040b7080 con 0x7f69040ce040 2024-04-15T15:15:05.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.081+0000 7f68f9ffb640 1 -- 172.21.15.72:0/1992860331 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7f68f4059cf0 con 0x7f69040ce040 2024-04-15T15:15:05.082 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:05.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.083+0000 7f690b40d640 1 -- 172.21.15.72:0/1992860331 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f68e806cac0 msgr2=0x7f68e806ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:05.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.083+0000 7f690b40d640 1 --2- 172.21.15.72:0/1992860331 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f68e806cac0 0x7f68e806ef80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6900001470 tx=0x7f6900009040 comp rx=0 tx=0).stop 2024-04-15T15:15:05.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.083+0000 7f690b40d640 1 -- 172.21.15.72:0/1992860331 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69040ce040 msgr2=0x7f69040ce4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:05.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.083+0000 7f690b40d640 1 --2- 172.21.15.72:0/1992860331 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69040ce040 0x7f69040ce4c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f68f400e880 tx=0x7f68f400ed40 comp rx=0 tx=0).stop 2024-04-15T15:15:05.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.083+0000 7f690b40d640 1 -- 172.21.15.72:0/1992860331 shutdown_connections 2024-04-15T15:15:05.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.083+0000 7f690b40d640 1 --2- 172.21.15.72:0/1992860331 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f68e806cac0 0x7f68e806ef80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:05.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.083+0000 7f690b40d640 1 --2- 172.21.15.72:0/1992860331 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6904160660 0x7f6904160ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:05.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.083+0000 7f690b40d640 1 --2- 172.21.15.72:0/1992860331 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69040ce040 0x7f69040ce4c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:05.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.083+0000 7f690b40d640 1 --2- 172.21.15.72:0/1992860331 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f68f0004d40 0x7f69041f7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:05.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.083+0000 7f690b40d640 1 -- 172.21.15.72:0/1992860331 >> 172.21.15.72:0/1992860331 conn(0x7f69040d4750 msgr2=0x7f690406b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:05.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.083+0000 7f690b40d640 1 -- 172.21.15.72:0/1992860331 shutdown_connections 2024-04-15T15:15:05.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.083+0000 7f690b40d640 1 -- 172.21.15.72:0/1992860331 wait complete. 2024-04-15T15:15:05.095 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:15:05.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.198+0000 7f24fa700640 1 Processor -- start 2024-04-15T15:15:05.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.198+0000 7f24fa700640 1 -- start start 2024-04-15T15:15:05.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.199+0000 7f24fa700640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f24f40b0270 0x7f24f40cea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:05.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.199+0000 7f24fa700640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24f415ce00 con 0x7f24f40d1570 2024-04-15T15:15:05.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.199+0000 7f24fa700640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24f415d0e0 con 0x7f24f4156cf0 2024-04-15T15:15:05.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.199+0000 7f24fa700640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f24f415d3c0 con 0x7f24f40b0270 2024-04-15T15:15:05.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.199+0000 7f24f37fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f24f40b0270 0x7f24f40cea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:05.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.199+0000 7f24f8c9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f24f40d1570 0x7f24f40cef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45046/0 (socket says 172.21.15.72:45046) 2024-04-15T15:15:05.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.199+0000 7f24f8c9d640 1 -- 172.21.15.72:0/2765596432 learned_addr learned my addr 172.21.15.72:0/2765596432 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:05.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.199+0000 7f24f37fe640 1 -- 172.21.15.72:0/2765596432 >> v1:172.21.15.175:6789/0 conn(0x7f24f4156cf0 legacy=0x7f24f406ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:05.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.199+0000 7f24f37fe640 1 -- 172.21.15.72:0/2765596432 >> v1:172.21.15.72:6789/0 conn(0x7f24f40d1570 legacy=0x7f24f40cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:05.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.199+0000 7f24f37fe640 1 -- 172.21.15.72:0/2765596432 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24f415d6a0 con 0x7f24f40b0270 2024-04-15T15:15:05.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.199+0000 7f24f2ffd640 1 -- 172.21.15.72:0/2765596432 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 802457492 0 0) 0x7f24f415ce00 con 0x7f24f40d1570 2024-04-15T15:15:05.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.200+0000 7f24f37fe640 1 --2- 172.21.15.72:0/2765596432 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24f40b0270 0x7f24f40cea50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f24e80097d0 tx=0x7f24e802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae7dfec0e9cb26f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:05.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.200+0000 7f24f2ffd640 1 -- 172.21.15.72:0/2765596432 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e80315a0 con 0x7f24f40b0270 2024-04-15T15:15:05.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.200+0000 7f24f2ffd640 1 -- 172.21.15.72:0/2765596432 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24e8037070 con 0x7f24f40b0270 2024-04-15T15:15:05.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.200+0000 7f24f2ffd640 1 -- 172.21.15.72:0/2765596432 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e8031960 con 0x7f24f40b0270 2024-04-15T15:15:05.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.200+0000 7f24fa700640 1 -- 172.21.15.72:0/2765596432 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24f40b0270 msgr2=0x7f24f40cea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:05.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.200+0000 7f24fa700640 1 --2- 172.21.15.72:0/2765596432 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24f40b0270 0x7f24f40cea50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f24e80097d0 tx=0x7f24e802b840 comp rx=0 tx=0).stop 2024-04-15T15:15:05.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.200+0000 7f24fa700640 1 -- 172.21.15.72:0/2765596432 shutdown_connections 2024-04-15T15:15:05.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.200+0000 7f24fa700640 1 --2- 172.21.15.72:0/2765596432 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24f40b0270 0x7f24f40cea50 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:05.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.200+0000 7f24fa700640 1 -- 172.21.15.72:0/2765596432 >> 172.21.15.72:0/2765596432 conn(0x7f24f40d0a50 msgr2=0x7f24f40d0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:05.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.200+0000 7f24fa700640 1 -- 172.21.15.72:0/2765596432 shutdown_connections 2024-04-15T15:15:05.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.200+0000 7f24fa700640 1 -- 172.21.15.72:0/2765596432 wait complete. 2024-04-15T15:15:05.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.202+0000 7f24fa700640 1 Processor -- start 2024-04-15T15:15:05.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.202+0000 7f24fa700640 1 -- start start 2024-04-15T15:15:05.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.202+0000 7f24fa700640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f24f40b0270 0x7f24f41401c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:05.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.202+0000 7f24fa700640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24f40d1570 0x7f24f413e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:05.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.202+0000 7f24fa700640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f24f4156cf0 0x7f24f413e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:05.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.202+0000 7f24fa700640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f24f406dbb0 con 0x7f24f40d1570 2024-04-15T15:15:05.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.202+0000 7f24fa700640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f24f41570d0 con 0x7f24f40b0270 2024-04-15T15:15:05.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.202+0000 7f24fa700640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f24f406d540 con 0x7f24f4156cf0 2024-04-15T15:15:05.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.203+0000 7f24f37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24f40d1570 0x7f24f413e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:05.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.203+0000 7f24f37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24f40d1570 0x7f24f413e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44428/0 (socket says 172.21.15.72:44428) 2024-04-15T15:15:05.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.203+0000 7f24f37fe640 1 -- 172.21.15.72:0/2457196855 learned_addr learned my addr 172.21.15.72:0/2457196855 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:05.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.203+0000 7f24f3fff640 1 --2- 172.21.15.72:0/2457196855 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f24f40b0270 0x7f24f41401c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:05.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.203+0000 7f24f8c9d640 1 --2- 172.21.15.72:0/2457196855 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f24f4156cf0 0x7f24f413e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:05.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.203+0000 7f24f37fe640 1 -- 172.21.15.72:0/2457196855 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f24f4156cf0 msgr2=0x7f24f413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:05.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.203+0000 7f24f37fe640 1 --2- 172.21.15.72:0/2457196855 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f24f4156cf0 0x7f24f413e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:05.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.203+0000 7f24f37fe640 1 -- 172.21.15.72:0/2457196855 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f24f40b0270 msgr2=0x7f24f41401c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:05.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.203+0000 7f24f37fe640 1 --2- 172.21.15.72:0/2457196855 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f24f40b0270 0x7f24f41401c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:05.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.203+0000 7f24f37fe640 1 -- 172.21.15.72:0/2457196855 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24f413e540 con 0x7f24f40d1570 2024-04-15T15:15:05.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.203+0000 7f24f8c9d640 1 --2- 172.21.15.72:0/2457196855 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f24f4156cf0 0x7f24f413e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:05.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.203+0000 7f24f3fff640 1 --2- 172.21.15.72:0/2457196855 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f24f40b0270 0x7f24f41401c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:05.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.203+0000 7f24f37fe640 1 --2- 172.21.15.72:0/2457196855 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24f40d1570 0x7f24f413e000 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f24e80097d0 tx=0x7f24e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:05.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.203+0000 7f24f0ff9640 1 -- 172.21.15.72:0/2457196855 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e8032840 con 0x7f24f40d1570 2024-04-15T15:15:05.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.203+0000 7f24f0ff9640 1 -- 172.21.15.72:0/2457196855 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24e80329e0 con 0x7f24f40d1570 2024-04-15T15:15:05.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.203+0000 7f24f0ff9640 1 -- 172.21.15.72:0/2457196855 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24e8032dd0 con 0x7f24f40d1570 2024-04-15T15:15:05.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.203+0000 7f24fa700640 1 -- 172.21.15.72:0/2457196855 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24f413f120 con 0x7f24f40d1570 2024-04-15T15:15:05.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.203+0000 7f24fa700640 1 -- 172.21.15.72:0/2457196855 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24f413f6e0 con 0x7f24f40d1570 2024-04-15T15:15:05.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.204+0000 7f24fa700640 1 -- 172.21.15.72:0/2457196855 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24f4071ea0 con 0x7f24f40d1570 2024-04-15T15:15:05.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.205+0000 7f24f0ff9640 1 -- 172.21.15.72:0/2457196855 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f24e803b3f0 con 0x7f24f40d1570 2024-04-15T15:15:05.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.205+0000 7f24f0ff9640 1 --2- 172.21.15.72:0/2457196855 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f24e006c920 0x7f24e006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:05.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.205+0000 7f24f3fff640 1 --2- 172.21.15.72:0/2457196855 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f24e006c920 0x7f24e006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:05.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.205+0000 7f24f0ff9640 1 -- 172.21.15.72:0/2457196855 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f24e802f030 con 0x7f24f40d1570 2024-04-15T15:15:05.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.206+0000 7f24f3fff640 1 --2- 172.21.15.72:0/2457196855 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f24e006c920 0x7f24e006ede0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f24e4003eb0 tx=0x7f24e4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:05.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.210+0000 7f24f0ff9640 1 -- 172.21.15.72:0/2457196855 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f24f4071ea0 con 0x7f24f40d1570 2024-04-15T15:15:05.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.396+0000 7f24fa700640 1 -- 172.21.15.72:0/2457196855 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f24f4068d10 con 0x7f24f40d1570 2024-04-15T15:15:05.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.397+0000 7f24f0ff9640 1 -- 172.21.15.72:0/2457196855 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) ==== 80+0+0 (secure 0 0 0) 0x7f24f413e540 con 0x7f24f40d1570 2024-04-15T15:15:05.401 INFO:teuthology.orchestra.run.smithi072.stderr:listed 0 entries 2024-04-15T15:15:05.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.399+0000 7f24fa700640 1 -- 172.21.15.72:0/2457196855 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f24e006c920 msgr2=0x7f24e006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:05.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.399+0000 7f24fa700640 1 --2- 172.21.15.72:0/2457196855 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f24e006c920 0x7f24e006ede0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f24e4003eb0 tx=0x7f24e4009040 comp rx=0 tx=0).stop 2024-04-15T15:15:05.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.399+0000 7f24fa700640 1 -- 172.21.15.72:0/2457196855 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24f40d1570 msgr2=0x7f24f413e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:05.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.399+0000 7f24fa700640 1 --2- 172.21.15.72:0/2457196855 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24f40d1570 0x7f24f413e000 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f24e80097d0 tx=0x7f24e80046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:05.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.399+0000 7f24fa700640 1 -- 172.21.15.72:0/2457196855 shutdown_connections 2024-04-15T15:15:05.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.399+0000 7f24fa700640 1 --2- 172.21.15.72:0/2457196855 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f24e006c920 0x7f24e006ede0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:05.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.399+0000 7f24fa700640 1 --2- 172.21.15.72:0/2457196855 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f24f4156cf0 0x7f24f413e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:05.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.399+0000 7f24fa700640 1 --2- 172.21.15.72:0/2457196855 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24f40d1570 0x7f24f413e000 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:05.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.399+0000 7f24fa700640 1 --2- 172.21.15.72:0/2457196855 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f24f40b0270 0x7f24f41401c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:05.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.399+0000 7f24fa700640 1 -- 172.21.15.72:0/2457196855 >> 172.21.15.72:0/2457196855 conn(0x7f24f40d0a50 msgr2=0x7f24f415f570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:05.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.399+0000 7f24fa700640 1 -- 172.21.15.72:0/2457196855 shutdown_connections 2024-04-15T15:15:05.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.399+0000 7f24fa700640 1 -- 172.21.15.72:0/2457196855 wait complete. 2024-04-15T15:15:05.411 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi072.front.sepia.ceph.com... 2024-04-15T15:15:05.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:05.411 DEBUG:teuthology.orchestra.run.smithi072:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-15T15:15:05.438 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi072.stderr:ceph-fuse[44584]: fuse finished with error 0 and tester_r 0 2024-04-15T15:15:05.440 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:15:05.480 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi072.front.sepia.ceph.com 2024-04-15T15:15:05.481 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:05.481 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-15T15:15:05.543 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:05.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.685+0000 7fcb602d3640 1 Processor -- start 2024-04-15T15:15:05.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.685+0000 7fcb602d3640 1 -- start start 2024-04-15T15:15:05.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.686+0000 7fcb602d3640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcb5815e640 0x7fcb580cf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:05.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.686+0000 7fcb602d3640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcb580cd4d0 0x7fcb580d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:05.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.686+0000 7fcb602d3640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb581623d0 con 0x7fcb580cf7f0 2024-04-15T15:15:05.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.686+0000 7fcb602d3640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fcb58162fa0 con 0x7fcb580cd4d0 2024-04-15T15:15:05.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.686+0000 7fcb602d3640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fcb580cfc30 con 0x7fcb5815e640 2024-04-15T15:15:05.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.686+0000 7fcb5e870640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcb580cd4d0 0x7fcb580d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:05.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.686+0000 7fcb5e870640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcb580cd4d0 0x7fcb580d0040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44430/0 (socket says 172.21.15.72:44430) 2024-04-15T15:15:05.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.686+0000 7fcb5e870640 1 -- 172.21.15.72:0/2382073763 learned_addr learned my addr 172.21.15.72:0/2382073763 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:05.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.686+0000 7fcb5e06f640 1 --2- 172.21.15.72:0/2382073763 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcb5815e640 0x7fcb580cf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:05.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.686+0000 7fcb5e870640 1 -- 172.21.15.72:0/2382073763 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcb5815e640 msgr2=0x7fcb580cf2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:05.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.686+0000 7fcb5e870640 1 --2- 172.21.15.72:0/2382073763 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcb5815e640 0x7fcb580cf2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:05.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.686+0000 7fcb5e870640 1 -- 172.21.15.72:0/2382073763 >> v1:172.21.15.175:6789/0 conn(0x7fcb580cf7f0 legacy=0x7fcb580ccdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:05.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.686+0000 7fcb5e870640 1 -- 172.21.15.72:0/2382073763 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb58162670 con 0x7fcb580cd4d0 2024-04-15T15:15:05.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.687+0000 7fcb5e870640 1 --2- 172.21.15.72:0/2382073763 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb580cd4d0 0x7fcb580d0040 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fcb50009e50 tx=0x7fcb5002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2928d14748e4810a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:05.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.687+0000 7fcb5d06d640 1 -- 172.21.15.72:0/2382073763 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5002ca70 con 0x7fcb580cd4d0 2024-04-15T15:15:05.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.687+0000 7fcb5d06d640 1 -- 172.21.15.72:0/2382073763 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb5002cc10 con 0x7fcb580cd4d0 2024-04-15T15:15:05.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.687+0000 7fcb5d06d640 1 -- 172.21.15.72:0/2382073763 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb500343f0 con 0x7fcb580cd4d0 2024-04-15T15:15:05.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.688+0000 7fcb602d3640 1 -- 172.21.15.72:0/2382073763 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb580cd4d0 msgr2=0x7fcb580d0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:05.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.688+0000 7fcb602d3640 1 --2- 172.21.15.72:0/2382073763 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb580cd4d0 0x7fcb580d0040 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fcb50009e50 tx=0x7fcb5002bed0 comp rx=0 tx=0).stop 2024-04-15T15:15:05.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.688+0000 7fcb602d3640 1 -- 172.21.15.72:0/2382073763 shutdown_connections 2024-04-15T15:15:05.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.688+0000 7fcb602d3640 1 --2- 172.21.15.72:0/2382073763 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb580cd4d0 0x7fcb580d0040 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:05.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.688+0000 7fcb602d3640 1 --2- 172.21.15.72:0/2382073763 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcb5815e640 0x7fcb580cf2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:05.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.688+0000 7fcb602d3640 1 -- 172.21.15.72:0/2382073763 >> 172.21.15.72:0/2382073763 conn(0x7fcb580b43c0 msgr2=0x7fcb580b47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:05.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.688+0000 7fcb602d3640 1 -- 172.21.15.72:0/2382073763 shutdown_connections 2024-04-15T15:15:05.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.688+0000 7fcb602d3640 1 -- 172.21.15.72:0/2382073763 wait complete. 2024-04-15T15:15:05.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.689+0000 7fcb602d3640 1 Processor -- start 2024-04-15T15:15:05.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.689+0000 7fcb602d3640 1 -- start start 2024-04-15T15:15:05.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.689+0000 7fcb602d3640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcb580cd4d0 0x7fcb58155cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:05.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.689+0000 7fcb602d3640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb580cf7f0 0x7fcb580d2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:05.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.689+0000 7fcb602d3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb5815e640 0x7fcb580b1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:05.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.689+0000 7fcb602d3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5815ea20 con 0x7fcb5815e640 2024-04-15T15:15:05.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.690+0000 7fcb602d3640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fcb58162c30 con 0x7fcb580cf7f0 2024-04-15T15:15:05.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.690+0000 7fcb602d3640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fcb580d0580 con 0x7fcb580cd4d0 2024-04-15T15:15:05.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.690+0000 7fcb5e870640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb5815e640 0x7fcb580b1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:05.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.690+0000 7fcb5e870640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb5815e640 0x7fcb580b1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44446/0 (socket says 172.21.15.72:44446) 2024-04-15T15:15:05.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.690+0000 7fcb5e870640 1 -- 172.21.15.72:0/3634968443 learned_addr learned my addr 172.21.15.72:0/3634968443 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:05.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.690+0000 7fcb5e870640 1 -- 172.21.15.72:0/3634968443 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcb580cd4d0 msgr2=0x7fcb58155cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:05.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.690+0000 7fcb5e870640 1 --2- 172.21.15.72:0/3634968443 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcb580cd4d0 0x7fcb58155cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:05.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.690+0000 7fcb5e870640 1 -- 172.21.15.72:0/3634968443 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb580cf7f0 msgr2=0x7fcb580d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:05.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.690+0000 7fcb5e870640 1 --2- 172.21.15.72:0/3634968443 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb580cf7f0 0x7fcb580d2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:05.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.690+0000 7fcb5e870640 1 -- 172.21.15.72:0/3634968443 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb581561f0 con 0x7fcb5815e640 2024-04-15T15:15:05.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.690+0000 7fcb5e870640 1 --2- 172.21.15.72:0/3634968443 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb5815e640 0x7fcb580b1e50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcb50009e50 tx=0x7fcb50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:05.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.690+0000 7fcb4effd640 1 -- 172.21.15.72:0/3634968443 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb50043020 con 0x7fcb5815e640 2024-04-15T15:15:05.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.690+0000 7fcb4effd640 1 -- 172.21.15.72:0/3634968443 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb5003e450 con 0x7fcb5815e640 2024-04-15T15:15:05.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.690+0000 7fcb602d3640 1 -- 172.21.15.72:0/3634968443 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb580d0d40 con 0x7fcb5815e640 2024-04-15T15:15:05.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.690+0000 7fcb4effd640 1 -- 172.21.15.72:0/3634968443 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb50009510 con 0x7fcb5815e640 2024-04-15T15:15:05.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.690+0000 7fcb602d3640 1 -- 172.21.15.72:0/3634968443 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb580d1300 con 0x7fcb5815e640 2024-04-15T15:15:05.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.691+0000 7fcb602d3640 1 -- 172.21.15.72:0/3634968443 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb580689e0 con 0x7fcb5815e640 2024-04-15T15:15:05.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.692+0000 7fcb4effd640 1 -- 172.21.15.72:0/3634968443 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcb50046020 con 0x7fcb5815e640 2024-04-15T15:15:05.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.692+0000 7fcb4effd640 1 --2- 172.21.15.72:0/3634968443 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcb3c06c8f0 0x7fcb3c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:05.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.692+0000 7fcb4effd640 1 -- 172.21.15.72:0/3634968443 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fcb500ae650 con 0x7fcb5815e640 2024-04-15T15:15:05.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.692+0000 7fcb5e06f640 1 --2- 172.21.15.72:0/3634968443 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcb3c06c8f0 0x7fcb3c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:05.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.693+0000 7fcb5e06f640 1 --2- 172.21.15.72:0/3634968443 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcb3c06c8f0 0x7fcb3c06edb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcb54006fd0 tx=0x7fcb54008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:05.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.696+0000 7fcb4effd640 1 -- 172.21.15.72:0/3634968443 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcb5004d030 con 0x7fcb5815e640 2024-04-15T15:15:05.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.871+0000 7fcb602d3640 1 -- 172.21.15.72:0/3634968443 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcb580ca190 con 0x7fcb5815e640 2024-04-15T15:15:05.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.873+0000 7fcb4effd640 1 -- 172.21.15.72:0/3634968443 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7fcb581561f0 con 0x7fcb5815e640 2024-04-15T15:15:05.874 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:05.874 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":24,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:14:56.451113+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:14:49.438366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:14:50.368379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:14:54.081157+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:05.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.874+0000 7fcb602d3640 1 -- 172.21.15.72:0/3634968443 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcb3c06c8f0 msgr2=0x7fcb3c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:05.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.874+0000 7fcb602d3640 1 --2- 172.21.15.72:0/3634968443 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcb3c06c8f0 0x7fcb3c06edb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcb54006fd0 tx=0x7fcb54008040 comp rx=0 tx=0).stop 2024-04-15T15:15:05.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.874+0000 7fcb602d3640 1 -- 172.21.15.72:0/3634968443 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb5815e640 msgr2=0x7fcb580b1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:05.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.874+0000 7fcb602d3640 1 --2- 172.21.15.72:0/3634968443 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb5815e640 0x7fcb580b1e50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcb50009e50 tx=0x7fcb50004a40 comp rx=0 tx=0).stop 2024-04-15T15:15:05.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.875+0000 7fcb602d3640 1 -- 172.21.15.72:0/3634968443 shutdown_connections 2024-04-15T15:15:05.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.875+0000 7fcb602d3640 1 --2- 172.21.15.72:0/3634968443 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcb3c06c8f0 0x7fcb3c06edb0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:05.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.875+0000 7fcb602d3640 1 --2- 172.21.15.72:0/3634968443 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb5815e640 0x7fcb580b1e50 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:05.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.875+0000 7fcb602d3640 1 --2- 172.21.15.72:0/3634968443 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb580cf7f0 0x7fcb580d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:05.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.875+0000 7fcb602d3640 1 --2- 172.21.15.72:0/3634968443 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcb580cd4d0 0x7fcb58155cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:05.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.875+0000 7fcb602d3640 1 -- 172.21.15.72:0/3634968443 >> 172.21.15.72:0/3634968443 conn(0x7fcb580b43c0 msgr2=0x7fcb58159720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:05.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.875+0000 7fcb602d3640 1 -- 172.21.15.72:0/3634968443 shutdown_connections 2024-04-15T15:15:05.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.875+0000 7fcb602d3640 1 -- 172.21.15.72:0/3634968443 wait complete. 2024-04-15T15:15:05.887 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:05.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.988+0000 7f8455582640 1 Processor -- start 2024-04-15T15:15:05.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.988+0000 7f8455582640 1 -- start start 2024-04-15T15:15:05.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.989+0000 7f8455582640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f845015e830 0x7f84500d0450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:05.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.989+0000 7f8455582640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8450070310 0x7f8450070760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:05.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.989+0000 7f8455582640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f84500b4370 con 0x7f84500b6820 2024-04-15T15:15:05.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.989+0000 7f8455582640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f8450070ca0 con 0x7f845015e830 2024-04-15T15:15:05.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.989+0000 7f8455582640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f84500b4c60 con 0x7f8450070310 2024-04-15T15:15:05.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.989+0000 7f844effd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f845015e830 0x7f84500d0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:05.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.989+0000 7f844effd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f845015e830 0x7f84500d0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44460/0 (socket says 172.21.15.72:44460) 2024-04-15T15:15:05.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.989+0000 7f844effd640 1 -- 172.21.15.72:0/3139854510 learned_addr learned my addr 172.21.15.72:0/3139854510 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:05.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.990+0000 7f844f7fe640 1 --2- 172.21.15.72:0/3139854510 >> v2:172.21.15.175:3300/0 conn(0x7f8450070310 0x7f8450070760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:05.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.990+0000 7f844effd640 1 -- 172.21.15.72:0/3139854510 >> v2:172.21.15.175:3300/0 conn(0x7f8450070310 msgr2=0x7f8450070760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:05.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.990+0000 7f844effd640 1 --2- 172.21.15.72:0/3139854510 >> v2:172.21.15.175:3300/0 conn(0x7f8450070310 0x7f8450070760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:05.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.990+0000 7f844effd640 1 -- 172.21.15.72:0/3139854510 >> v1:172.21.15.175:6789/0 conn(0x7f84500b6820 legacy=0x7f84500b50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:05.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.990+0000 7f844effd640 1 -- 172.21.15.72:0/3139854510 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84500b4610 con 0x7f845015e830 2024-04-15T15:15:05.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.990+0000 7f844f7fe640 1 --2- 172.21.15.72:0/3139854510 >> v2:172.21.15.175:3300/0 conn(0x7f8450070310 0x7f8450070760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:05.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.990+0000 7f844effd640 1 --2- 172.21.15.72:0/3139854510 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f845015e830 0x7f84500d0450 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f84500d03e0 tx=0x7f844002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9140e8d433ea29d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:05.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.990+0000 7f844dffb640 1 -- 172.21.15.72:0/3139854510 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f844002ca50 con 0x7f845015e830 2024-04-15T15:15:05.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.991+0000 7f844dffb640 1 -- 172.21.15.72:0/3139854510 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f844002cbf0 con 0x7f845015e830 2024-04-15T15:15:05.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.991+0000 7f844dffb640 1 -- 172.21.15.72:0/3139854510 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84400343f0 con 0x7f845015e830 2024-04-15T15:15:05.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.991+0000 7f8455582640 1 -- 172.21.15.72:0/3139854510 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f845015e830 msgr2=0x7f84500d0450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:05.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.991+0000 7f8455582640 1 --2- 172.21.15.72:0/3139854510 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f845015e830 0x7f84500d0450 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f84500d03e0 tx=0x7f844002beb0 comp rx=0 tx=0).stop 2024-04-15T15:15:05.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.991+0000 7f8455582640 1 -- 172.21.15.72:0/3139854510 shutdown_connections 2024-04-15T15:15:05.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.991+0000 7f8455582640 1 --2- 172.21.15.72:0/3139854510 >> v2:172.21.15.175:3300/0 conn(0x7f8450070310 0x7f8450070760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:05.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.991+0000 7f8455582640 1 --2- 172.21.15.72:0/3139854510 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f845015e830 0x7f84500d0450 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:05.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.991+0000 7f8455582640 1 -- 172.21.15.72:0/3139854510 >> 172.21.15.72:0/3139854510 conn(0x7f84500cdd20 msgr2=0x7f84500ce130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:05.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.991+0000 7f8455582640 1 -- 172.21.15.72:0/3139854510 shutdown_connections 2024-04-15T15:15:05.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.991+0000 7f8455582640 1 -- 172.21.15.72:0/3139854510 wait complete. 2024-04-15T15:15:05.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.992+0000 7f8455582640 1 Processor -- start 2024-04-15T15:15:05.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.992+0000 7f8455582640 1 -- start start 2024-04-15T15:15:05.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.993+0000 7f8455582640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8450070310 0x7f845015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:05.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.993+0000 7f8455582640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84500b6820 0x7f845015f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:05.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.993+0000 7f8455582640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f845015e830 0x7f845015cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:05.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.993+0000 7f8455582640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f84500d0ae0 con 0x7f8450070310 2024-04-15T15:15:05.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.993+0000 7f8455582640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f845015ec10 con 0x7f84500b6820 2024-04-15T15:15:05.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.993+0000 7f8455582640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f84500ae3d0 con 0x7f845015e830 2024-04-15T15:15:05.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.993+0000 7f844effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8450070310 0x7f845015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:05.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.993+0000 7f844effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8450070310 0x7f845015ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44470/0 (socket says 172.21.15.72:44470) 2024-04-15T15:15:05.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.993+0000 7f844effd640 1 -- 172.21.15.72:0/2973923241 learned_addr learned my addr 172.21.15.72:0/2973923241 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:05.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.993+0000 7f844e7fc640 1 --2- 172.21.15.72:0/2973923241 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84500b6820 0x7f845015f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:05.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.993+0000 7f844f7fe640 1 --2- 172.21.15.72:0/2973923241 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f845015e830 0x7f845015cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:05.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.994+0000 7f844effd640 1 -- 172.21.15.72:0/2973923241 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f845015e830 msgr2=0x7f845015cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:05.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.994+0000 7f844effd640 1 --2- 172.21.15.72:0/2973923241 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f845015e830 0x7f845015cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:05.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.994+0000 7f844effd640 1 -- 172.21.15.72:0/2973923241 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84500b6820 msgr2=0x7f845015f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:05.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.994+0000 7f844effd640 1 --2- 172.21.15.72:0/2973923241 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84500b6820 0x7f845015f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:05.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.994+0000 7f844effd640 1 -- 172.21.15.72:0/2973923241 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84501f4060 con 0x7f8450070310 2024-04-15T15:15:05.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.994+0000 7f844f7fe640 1 --2- 172.21.15.72:0/2973923241 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f845015e830 0x7f845015cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:05.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.994+0000 7f844effd640 1 --2- 172.21.15.72:0/2973923241 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8450070310 0x7f845015ed90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8440036040 tx=0x7f8440004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:05.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.994+0000 7f84377fe640 1 -- 172.21.15.72:0/2973923241 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8440043020 con 0x7f8450070310 2024-04-15T15:15:05.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.994+0000 7f84377fe640 1 -- 172.21.15.72:0/2973923241 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f844003e450 con 0x7f8450070310 2024-04-15T15:15:05.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.994+0000 7f8455582640 1 -- 172.21.15.72:0/2973923241 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84501f4320 con 0x7f8450070310 2024-04-15T15:15:05.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.994+0000 7f84377fe640 1 -- 172.21.15.72:0/2973923241 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8440009510 con 0x7f8450070310 2024-04-15T15:15:05.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.994+0000 7f8455582640 1 -- 172.21.15.72:0/2973923241 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84501f48e0 con 0x7f8450070310 2024-04-15T15:15:05.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.995+0000 7f8455582640 1 -- 172.21.15.72:0/2973923241 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84500690f0 con 0x7f8450070310 2024-04-15T15:15:05.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.995+0000 7f84377fe640 1 -- 172.21.15.72:0/2973923241 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8440046020 con 0x7f8450070310 2024-04-15T15:15:05.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.996+0000 7f84377fe640 1 --2- 172.21.15.72:0/2973923241 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f843c06c8f0 0x7f843c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:05.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.996+0000 7f84377fe640 1 -- 172.21.15.72:0/2973923241 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f84400ae850 con 0x7f8450070310 2024-04-15T15:15:05.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.996+0000 7f844e7fc640 1 --2- 172.21.15.72:0/2973923241 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f843c06c8f0 0x7f843c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:05.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:05.996+0000 7f844e7fc640 1 --2- 172.21.15.72:0/2973923241 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f843c06c8f0 0x7f843c06edb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f84500d0490 tx=0x7f8444009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:06.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.000+0000 7f84377fe640 1 -- 172.21.15.72:0/2973923241 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f844004d030 con 0x7f8450070310 2024-04-15T15:15:06.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.179+0000 7f8455582640 1 -- 172.21.15.72:0/2973923241 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f84500b0310 con 0x7f8450070310 2024-04-15T15:15:06.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.180+0000 7f84377fe640 1 -- 172.21.15.72:0/2973923241 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7f844007c940 con 0x7f8450070310 2024-04-15T15:15:06.233 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:06.233 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":24,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:14:56.451113+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:14:49.438366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:14:50.368379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:14:54.081157+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:06.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.182+0000 7f8455582640 1 -- 172.21.15.72:0/2973923241 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f843c06c8f0 msgr2=0x7f843c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:06.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.182+0000 7f8455582640 1 --2- 172.21.15.72:0/2973923241 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f843c06c8f0 0x7f843c06edb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f84500d0490 tx=0x7f8444009040 comp rx=0 tx=0).stop 2024-04-15T15:15:06.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.182+0000 7f8455582640 1 -- 172.21.15.72:0/2973923241 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8450070310 msgr2=0x7f845015ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:06.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.182+0000 7f8455582640 1 --2- 172.21.15.72:0/2973923241 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8450070310 0x7f845015ed90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8440036040 tx=0x7f8440004a40 comp rx=0 tx=0).stop 2024-04-15T15:15:06.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.182+0000 7f8455582640 1 -- 172.21.15.72:0/2973923241 shutdown_connections 2024-04-15T15:15:06.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.182+0000 7f8455582640 1 --2- 172.21.15.72:0/2973923241 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f843c06c8f0 0x7f843c06edb0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:06.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.182+0000 7f8455582640 1 --2- 172.21.15.72:0/2973923241 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f845015e830 0x7f845015cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:06.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.182+0000 7f8455582640 1 --2- 172.21.15.72:0/2973923241 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f84500b6820 0x7f845015f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:06.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.182+0000 7f8455582640 1 --2- 172.21.15.72:0/2973923241 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8450070310 0x7f845015ed90 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:06.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.182+0000 7f8455582640 1 -- 172.21.15.72:0/2973923241 >> 172.21.15.72:0/2973923241 conn(0x7f84500cdd20 msgr2=0x7f84500b0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:06.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.182+0000 7f8455582640 1 -- 172.21.15.72:0/2973923241 shutdown_connections 2024-04-15T15:15:06.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.182+0000 7f8455582640 1 -- 172.21.15.72:0/2973923241 wait complete. 2024-04-15T15:15:06.236 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-15T15:15:06.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.340+0000 7f97d5138640 1 Processor -- start 2024-04-15T15:15:06.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.340+0000 7f97d5138640 1 -- start start 2024-04-15T15:15:06.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.341+0000 7f97d5138640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f97d015f9f0 0x7f97d00cca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:06.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.341+0000 7f97d5138640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97d006df30 con 0x7f97d015c250 2024-04-15T15:15:06.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.341+0000 7f97d5138640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97d006e210 con 0x7f97d00d0010 2024-04-15T15:15:06.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.341+0000 7f97d5138640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f97d006e4f0 con 0x7f97d015f9f0 2024-04-15T15:15:06.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.341+0000 7f97ced9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f97d015c250 0x7f97d006d6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45048/0 (socket says 172.21.15.72:45048) 2024-04-15T15:15:06.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.341+0000 7f97ced9d640 1 -- 172.21.15.72:0/2944343614 learned_addr learned my addr 172.21.15.72:0/2944343614 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:06.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.341+0000 7f97ce59c640 1 --2- 172.21.15.72:0/2944343614 >> v2:172.21.15.72:3300/0 conn(0x7f97d015f9f0 0x7f97d00cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:06.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.341+0000 7f97cdd9b640 1 -- 172.21.15.72:0/2944343614 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3533749655 0 0) 0x7f97d006df30 con 0x7f97d015c250 2024-04-15T15:15:06.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.341+0000 7f97cdd9b640 1 -- 172.21.15.72:0/2944343614 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f97b8003520 con 0x7f97d015c250 2024-04-15T15:15:06.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.341+0000 7f97ce59c640 1 -- 172.21.15.72:0/2944343614 >> v1:172.21.15.175:6789/0 conn(0x7f97d00d0010 legacy=0x7f97d00b3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:06.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.341+0000 7f97ce59c640 1 -- 172.21.15.72:0/2944343614 >> v1:172.21.15.72:6789/0 conn(0x7f97d015c250 legacy=0x7f97d006d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:06.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.341+0000 7f97ce59c640 1 -- 172.21.15.72:0/2944343614 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97d00b3610 con 0x7f97d015f9f0 2024-04-15T15:15:06.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.342+0000 7f97ce59c640 1 --2- 172.21.15.72:0/2944343614 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97d015f9f0 0x7f97d00cca90 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f97c40097a0 tx=0x7f97c402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45e6a7493c4a94f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:06.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.342+0000 7f97cdd9b640 1 -- 172.21.15.72:0/2944343614 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c4034070 con 0x7f97d015f9f0 2024-04-15T15:15:06.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.342+0000 7f97cdd9b640 1 -- 172.21.15.72:0/2944343614 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97c4030610 con 0x7f97d015f9f0 2024-04-15T15:15:06.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.342+0000 7f97cdd9b640 1 -- 172.21.15.72:0/2944343614 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c40309e0 con 0x7f97d015f9f0 2024-04-15T15:15:06.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.343+0000 7f97d5138640 1 -- 172.21.15.72:0/2944343614 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97d015f9f0 msgr2=0x7f97d00cca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:06.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.343+0000 7f97d5138640 1 --2- 172.21.15.72:0/2944343614 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97d015f9f0 0x7f97d00cca90 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f97c40097a0 tx=0x7f97c402b960 comp rx=0 tx=0).stop 2024-04-15T15:15:06.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.343+0000 7f97d5138640 1 -- 172.21.15.72:0/2944343614 shutdown_connections 2024-04-15T15:15:06.345 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.343+0000 7f97d5138640 1 --2- 172.21.15.72:0/2944343614 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97d015f9f0 0x7f97d00cca90 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:06.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.343+0000 7f97d5138640 1 -- 172.21.15.72:0/2944343614 >> 172.21.15.72:0/2944343614 conn(0x7f97d00ceec0 msgr2=0x7f97d00cf2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:06.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.343+0000 7f97d5138640 1 -- 172.21.15.72:0/2944343614 shutdown_connections 2024-04-15T15:15:06.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.343+0000 7f97d5138640 1 -- 172.21.15.72:0/2944343614 wait complete. 2024-04-15T15:15:06.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.344+0000 7f97d5138640 1 Processor -- start 2024-04-15T15:15:06.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.344+0000 7f97d5138640 1 -- start start 2024-04-15T15:15:06.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.345+0000 7f97d5138640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f97d00d0010 0x7f97d00cb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:06.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.345+0000 7f97d5138640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f97d015c250 0x7f97d015c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:06.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.345+0000 7f97d5138640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97d015f9f0 0x7f97d01f3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:06.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.345+0000 7f97d5138640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f97d015c7b0 con 0x7f97d015f9f0 2024-04-15T15:15:06.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.345+0000 7f97d5138640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f97d015c630 con 0x7f97d015c250 2024-04-15T15:15:06.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.345+0000 7f97d5138640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f97d00ccfd0 con 0x7f97d00d0010 2024-04-15T15:15:06.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.345+0000 7f97cf59e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97d015f9f0 0x7f97d01f3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:06.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.345+0000 7f97cf59e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97d015f9f0 0x7f97d01f3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44488/0 (socket says 172.21.15.72:44488) 2024-04-15T15:15:06.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.345+0000 7f97cf59e640 1 -- 172.21.15.72:0/2237881295 learned_addr learned my addr 172.21.15.72:0/2237881295 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:06.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.345+0000 7f97ce59c640 1 --2- 172.21.15.72:0/2237881295 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f97d015c250 0x7f97d015c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:06.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.345+0000 7f97ced9d640 1 --2- 172.21.15.72:0/2237881295 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f97d00d0010 0x7f97d00cb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:06.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.345+0000 7f97cf59e640 1 -- 172.21.15.72:0/2237881295 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f97d00d0010 msgr2=0x7f97d00cb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:06.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.345+0000 7f97cf59e640 1 --2- 172.21.15.72:0/2237881295 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f97d00d0010 0x7f97d00cb1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:06.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.345+0000 7f97cf59e640 1 -- 172.21.15.72:0/2237881295 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f97d015c250 msgr2=0x7f97d015c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:06.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.345+0000 7f97cf59e640 1 --2- 172.21.15.72:0/2237881295 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f97d015c250 0x7f97d015c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:06.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.345+0000 7f97cf59e640 1 -- 172.21.15.72:0/2237881295 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97d015dc80 con 0x7f97d015f9f0 2024-04-15T15:15:06.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.345+0000 7f97cf59e640 1 --2- 172.21.15.72:0/2237881295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97d015f9f0 0x7f97d01f3a10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f97c8001eb0 tx=0x7f97c800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:06.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.346+0000 7f97b77fe640 1 -- 172.21.15.72:0/2237881295 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c8017070 con 0x7f97d015f9f0 2024-04-15T15:15:06.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.346+0000 7f97b77fe640 1 -- 172.21.15.72:0/2237881295 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97c80052d0 con 0x7f97d015f9f0 2024-04-15T15:15:06.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.346+0000 7f97d5138640 1 -- 172.21.15.72:0/2237881295 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97d015d100 con 0x7f97d015f9f0 2024-04-15T15:15:06.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.346+0000 7f97b77fe640 1 -- 172.21.15.72:0/2237881295 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c8003810 con 0x7f97d015f9f0 2024-04-15T15:15:06.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.346+0000 7f97d5138640 1 -- 172.21.15.72:0/2237881295 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97d015d6c0 con 0x7f97d015f9f0 2024-04-15T15:15:06.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.347+0000 7f97d5138640 1 -- 172.21.15.72:0/2237881295 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97d0075c50 con 0x7f97d015f9f0 2024-04-15T15:15:06.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.347+0000 7f97b77fe640 1 -- 172.21.15.72:0/2237881295 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f97c801a070 con 0x7f97d015f9f0 2024-04-15T15:15:06.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.347+0000 7f97b77fe640 1 --2- 172.21.15.72:0/2237881295 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f97bc06c8f0 0x7f97bc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:06.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.348+0000 7f97b77fe640 1 -- 172.21.15.72:0/2237881295 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f97c808b190 con 0x7f97d015f9f0 2024-04-15T15:15:06.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.348+0000 7f97ced9d640 1 --2- 172.21.15.72:0/2237881295 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f97bc06c8f0 0x7f97bc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:06.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.348+0000 7f97ced9d640 1 --2- 172.21.15.72:0/2237881295 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f97bc06c8f0 0x7f97bc06edb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f97c0001560 tx=0x7f97c0001a90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:06.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.352+0000 7f97b77fe640 1 -- 172.21.15.72:0/2237881295 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f97c8058c30 con 0x7f97d015f9f0 2024-04-15T15:15:06.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:06.567+0000 7f97d5138640 1 -- 172.21.15.72:0/2237881295 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f97d00b83c0 con 0x7f97d015f9f0 2024-04-15T15:15:07.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.128+0000 7f97b77fe640 1 -- 172.21.15.72:0/2237881295 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) ==== 118+0+0 (secure 0 0 0) 0x7f97c805cf30 con 0x7f97d015f9f0 2024-04-15T15:15:07.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.131+0000 7f97d5138640 1 -- 172.21.15.72:0/2237881295 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f97bc06c8f0 msgr2=0x7f97bc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:07.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.131+0000 7f97d5138640 1 --2- 172.21.15.72:0/2237881295 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f97bc06c8f0 0x7f97bc06edb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f97c0001560 tx=0x7f97c0001a90 comp rx=0 tx=0).stop 2024-04-15T15:15:07.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.131+0000 7f97d5138640 1 -- 172.21.15.72:0/2237881295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97d015f9f0 msgr2=0x7f97d01f3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:07.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.131+0000 7f97d5138640 1 --2- 172.21.15.72:0/2237881295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97d015f9f0 0x7f97d01f3a10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f97c8001eb0 tx=0x7f97c800c890 comp rx=0 tx=0).stop 2024-04-15T15:15:07.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.131+0000 7f97d5138640 1 -- 172.21.15.72:0/2237881295 shutdown_connections 2024-04-15T15:15:07.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.131+0000 7f97d5138640 1 --2- 172.21.15.72:0/2237881295 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f97bc06c8f0 0x7f97bc06edb0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.131+0000 7f97d5138640 1 --2- 172.21.15.72:0/2237881295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97d015f9f0 0x7f97d01f3a10 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.131+0000 7f97d5138640 1 --2- 172.21.15.72:0/2237881295 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f97d015c250 0x7f97d015c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.131+0000 7f97d5138640 1 --2- 172.21.15.72:0/2237881295 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f97d00d0010 0x7f97d00cb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.131+0000 7f97d5138640 1 -- 172.21.15.72:0/2237881295 >> 172.21.15.72:0/2237881295 conn(0x7f97d00ceec0 msgr2=0x7f97d00cf2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:07.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.131+0000 7f97d5138640 1 -- 172.21.15.72:0/2237881295 shutdown_connections 2024-04-15T15:15:07.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.131+0000 7f97d5138640 1 -- 172.21.15.72:0/2237881295 wait complete. 2024-04-15T15:15:07.146 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2024-04-15T15:15:07.147 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-15T15:15:07.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.256+0000 7f00efbb5640 1 Processor -- start 2024-04-15T15:15:07.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.256+0000 7f00efbb5640 1 -- start start 2024-04-15T15:15:07.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.257+0000 7f00efbb5640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f00e80b67c0 0x7f00e80d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:07.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.257+0000 7f00efbb5640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f00e80b6c20 0x7f00e80d1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:07.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.257+0000 7f00efbb5640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00e80ccfe0 con 0x7f00e8161ee0 2024-04-15T15:15:07.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.257+0000 7f00efbb5640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f00e81622c0 con 0x7f00e80b67c0 2024-04-15T15:15:07.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.257+0000 7f00efbb5640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f00e80cd2c0 con 0x7f00e80b6c20 2024-04-15T15:15:07.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.257+0000 7f00ed951640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f00e8161ee0 0x7f00e80c95f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45058/0 (socket says 172.21.15.72:45058) 2024-04-15T15:15:07.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.257+0000 7f00ed951640 1 -- 172.21.15.72:0/1682777867 learned_addr learned my addr 172.21.15.72:0/1682777867 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:07.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.257+0000 7f00ed150640 1 --2- 172.21.15.72:0/1682777867 >> v2:172.21.15.175:3300/0 conn(0x7f00e80b67c0 0x7f00e80d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:07.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.257+0000 7f00ee152640 1 --2- 172.21.15.72:0/1682777867 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f00e80b6c20 0x7f00e80d1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:07.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.258+0000 7f00ec94f640 1 -- 172.21.15.72:0/1682777867 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 72014455 0 0) 0x7f00e80ccfe0 con 0x7f00e8161ee0 2024-04-15T15:15:07.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.258+0000 7f00ec94f640 1 -- 172.21.15.72:0/1682777867 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f00d4003520 con 0x7f00e8161ee0 2024-04-15T15:15:07.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.258+0000 7f00ec94f640 1 -- 172.21.15.72:0/1682777867 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1129253013 0 0) 0x7f00d4003520 con 0x7f00e8161ee0 2024-04-15T15:15:07.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.258+0000 7f00ec94f640 1 -- 172.21.15.72:0/1682777867 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f00e80b6c20 msgr2=0x7f00e80d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:07.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.258+0000 7f00ec94f640 1 --2- 172.21.15.72:0/1682777867 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f00e80b6c20 0x7f00e80d1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.258+0000 7f00ec94f640 1 -- 172.21.15.72:0/1682777867 >> v2:172.21.15.175:3300/0 conn(0x7f00e80b67c0 msgr2=0x7f00e80d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:07.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.258+0000 7f00ec94f640 1 --2- 172.21.15.72:0/1682777867 >> v2:172.21.15.175:3300/0 conn(0x7f00e80b67c0 0x7f00e80d1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.258+0000 7f00ec94f640 1 -- 172.21.15.72:0/1682777867 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00e80cd5a0 con 0x7f00e8161ee0 2024-04-15T15:15:07.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.258+0000 7f00ec94f640 1 -- 172.21.15.72:0/1682777867 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f00e4001670 con 0x7f00e8161ee0 2024-04-15T15:15:07.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.258+0000 7f00ec94f640 1 -- 172.21.15.72:0/1682777867 >> v1:172.21.15.72:6789/0 conn(0x7f00e8161ee0 legacy=0x7f00e80c95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:07.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.258+0000 7f00ec94f640 1 --2- 172.21.15.72:0/1682777867 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f00d40039b0 0x7f00d4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:07.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.258+0000 7f00ec94f640 1 --2- 172.21.15.72:0/1682777867 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f00e80b6c20 0x7f00d4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:07.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.258+0000 7f00ec94f640 1 --2- 172.21.15.72:0/1682777867 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00d4004d40 0x7f00d40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:07.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.258+0000 7f00ec94f640 1 -- 172.21.15.72:0/1682777867 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f00e80cd2c0 con 0x7f00d40039b0 2024-04-15T15:15:07.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.258+0000 7f00ec94f640 1 -- 172.21.15.72:0/1682777867 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f00d400b9c0 con 0x7f00d4004d40 2024-04-15T15:15:07.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.258+0000 7f00ec94f640 1 -- 172.21.15.72:0/1682777867 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f00d400bb40 con 0x7f00e80b6c20 2024-04-15T15:15:07.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.258+0000 7f00ed951640 1 --2- 172.21.15.72:0/1682777867 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f00d40039b0 0x7f00d4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:07.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.258+0000 7f00ee152640 1 --2- 172.21.15.72:0/1682777867 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00d4004d40 0x7f00d40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:07.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.258+0000 7f00ed150640 1 --2- 172.21.15.72:0/1682777867 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f00e80b6c20 0x7f00d4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:07.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.258+0000 7f00ed951640 1 -- 172.21.15.72:0/1682777867 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f00e80b6c20 msgr2=0x7f00d4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:07.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.258+0000 7f00ed951640 1 --2- 172.21.15.72:0/1682777867 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f00e80b6c20 0x7f00d4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.258+0000 7f00ed951640 1 -- 172.21.15.72:0/1682777867 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00d4004d40 msgr2=0x7f00d40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:07.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.258+0000 7f00ed951640 1 --2- 172.21.15.72:0/1682777867 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00d4004d40 0x7f00d40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.258+0000 7f00ed951640 1 -- 172.21.15.72:0/1682777867 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00d4003520 con 0x7f00d40039b0 2024-04-15T15:15:07.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.259+0000 7f00ed951640 1 --2- 172.21.15.72:0/1682777867 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f00d40039b0 0x7f00d4003dd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f00e400c210 tx=0x7f00d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd5e381273f07c48 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:07.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.259+0000 7f00ec94f640 1 -- 172.21.15.72:0/1682777867 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e4007d50 con 0x7f00d40039b0 2024-04-15T15:15:07.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.259+0000 7f00ec94f640 1 -- 172.21.15.72:0/1682777867 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00e4030070 con 0x7f00d40039b0 2024-04-15T15:15:07.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.259+0000 7f00ec94f640 1 -- 172.21.15.72:0/1682777867 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e402cd20 con 0x7f00d40039b0 2024-04-15T15:15:07.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.260+0000 7f00efbb5640 1 -- 172.21.15.72:0/1682777867 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f00d40039b0 msgr2=0x7f00d4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:07.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.260+0000 7f00efbb5640 1 --2- 172.21.15.72:0/1682777867 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f00d40039b0 0x7f00d4003dd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f00e400c210 tx=0x7f00d4005830 comp rx=0 tx=0).stop 2024-04-15T15:15:07.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.260+0000 7f00ee152640 1 -- 172.21.15.72:0/1682777867 reap_dead start 2024-04-15T15:15:07.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.260+0000 7f00efbb5640 1 -- 172.21.15.72:0/1682777867 shutdown_connections 2024-04-15T15:15:07.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.260+0000 7f00efbb5640 1 -- 172.21.15.72:0/1682777867 >> 172.21.15.72:0/1682777867 conn(0x7f00e80b4010 msgr2=0x7f00e80b4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:07.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.260+0000 7f00efbb5640 1 -- 172.21.15.72:0/1682777867 shutdown_connections 2024-04-15T15:15:07.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.260+0000 7f00efbb5640 1 -- 172.21.15.72:0/1682777867 wait complete. 2024-04-15T15:15:07.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.261+0000 7f00efbb5640 1 Processor -- start 2024-04-15T15:15:07.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.261+0000 7f00efbb5640 1 -- start start 2024-04-15T15:15:07.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.262+0000 7f00efbb5640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f00d40039b0 0x7f00e81f7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:07.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.262+0000 7f00efbb5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f00e8159600 0x7f00e8159a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:07.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.262+0000 7f00efbb5640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00e815cf00 0x7f00e815d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:07.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.262+0000 7f00efbb5640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f00e80cccf0 con 0x7f00e8159600 2024-04-15T15:15:07.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.262+0000 7f00efbb5640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f00e80ccb70 con 0x7f00e815cf00 2024-04-15T15:15:07.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.262+0000 7f00efbb5640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f00e80d1640 con 0x7f00d40039b0 2024-04-15T15:15:07.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.262+0000 7f00ed150640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f00e8159600 0x7f00e8159a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:07.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.262+0000 7f00ed150640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f00e8159600 0x7f00e8159a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44506/0 (socket says 172.21.15.72:44506) 2024-04-15T15:15:07.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.262+0000 7f00ed150640 1 -- 172.21.15.72:0/4007194038 learned_addr learned my addr 172.21.15.72:0/4007194038 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:07.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.262+0000 7f00ee152640 1 --2- 172.21.15.72:0/4007194038 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00e815cf00 0x7f00e815d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:07.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.262+0000 7f00ed951640 1 --2- 172.21.15.72:0/4007194038 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f00d40039b0 0x7f00e81f7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:07.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.262+0000 7f00ed150640 1 -- 172.21.15.72:0/4007194038 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f00d40039b0 msgr2=0x7f00e81f7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:07.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.262+0000 7f00ed150640 1 --2- 172.21.15.72:0/4007194038 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f00d40039b0 0x7f00e81f7520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.262+0000 7f00ed150640 1 -- 172.21.15.72:0/4007194038 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00e815cf00 msgr2=0x7f00e815d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:07.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.262+0000 7f00ed150640 1 --2- 172.21.15.72:0/4007194038 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00e815cf00 0x7f00e815d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.262+0000 7f00ed150640 1 -- 172.21.15.72:0/4007194038 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00e815a0c0 con 0x7f00e8159600 2024-04-15T15:15:07.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.262+0000 7f00ee152640 1 --2- 172.21.15.72:0/4007194038 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00e815cf00 0x7f00e815d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:07.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.262+0000 7f00ed150640 1 --2- 172.21.15.72:0/4007194038 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f00e8159600 0x7f00e8159a80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f00d8011a00 tx=0x7f00d8011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:07.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.262+0000 7f00de7fc640 1 -- 172.21.15.72:0/4007194038 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d800fa90 con 0x7f00e8159600 2024-04-15T15:15:07.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.262+0000 7f00de7fc640 1 -- 172.21.15.72:0/4007194038 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00d800fc30 con 0x7f00e8159600 2024-04-15T15:15:07.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.262+0000 7f00efbb5640 1 -- 172.21.15.72:0/4007194038 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00e815a3a0 con 0x7f00e8159600 2024-04-15T15:15:07.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.262+0000 7f00de7fc640 1 -- 172.21.15.72:0/4007194038 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d8014440 con 0x7f00e8159600 2024-04-15T15:15:07.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.263+0000 7f00efbb5640 1 -- 172.21.15.72:0/4007194038 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00e815a960 con 0x7f00e8159600 2024-04-15T15:15:07.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.264+0000 7f00efbb5640 1 -- 172.21.15.72:0/4007194038 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00e8075c50 con 0x7f00e8159600 2024-04-15T15:15:07.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.264+0000 7f00de7fc640 1 -- 172.21.15.72:0/4007194038 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f00d8012040 con 0x7f00e8159600 2024-04-15T15:15:07.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.265+0000 7f00de7fc640 1 --2- 172.21.15.72:0/4007194038 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f00cc06c8f0 0x7f00cc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:07.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.265+0000 7f00ed951640 1 --2- 172.21.15.72:0/4007194038 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f00cc06c8f0 0x7f00cc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:07.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.265+0000 7f00de7fc640 1 -- 172.21.15.72:0/4007194038 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f00d808c500 con 0x7f00e8159600 2024-04-15T15:15:07.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.265+0000 7f00ed951640 1 --2- 172.21.15.72:0/4007194038 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f00cc06c8f0 0x7f00cc06edb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f00e4034000 tx=0x7f00e4007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:07.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.269+0000 7f00de7fc640 1 -- 172.21.15.72:0/4007194038 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f00d8019030 con 0x7f00e8159600 2024-04-15T15:15:07.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:07.484+0000 7f00efbb5640 1 -- 172.21.15.72:0/4007194038 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f00e80c7be0 con 0x7f00e8159600 2024-04-15T15:15:08.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.137+0000 7f00de7fc640 1 -- 172.21.15.72:0/4007194038 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) ==== 122+0+0 (secure 0 0 0) 0x7f00d805a560 con 0x7f00e8159600 2024-04-15T15:15:08.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.140+0000 7f00efbb5640 1 -- 172.21.15.72:0/4007194038 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f00cc06c8f0 msgr2=0x7f00cc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:08.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.140+0000 7f00efbb5640 1 --2- 172.21.15.72:0/4007194038 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f00cc06c8f0 0x7f00cc06edb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f00e4034000 tx=0x7f00e4007000 comp rx=0 tx=0).stop 2024-04-15T15:15:08.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.140+0000 7f00efbb5640 1 -- 172.21.15.72:0/4007194038 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f00e8159600 msgr2=0x7f00e8159a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:08.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.140+0000 7f00efbb5640 1 --2- 172.21.15.72:0/4007194038 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f00e8159600 0x7f00e8159a80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f00d8011a00 tx=0x7f00d8011ec0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.140+0000 7f00efbb5640 1 -- 172.21.15.72:0/4007194038 shutdown_connections 2024-04-15T15:15:08.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.140+0000 7f00efbb5640 1 --2- 172.21.15.72:0/4007194038 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f00cc06c8f0 0x7f00cc06edb0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.140+0000 7f00efbb5640 1 --2- 172.21.15.72:0/4007194038 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00e815cf00 0x7f00e815d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.140+0000 7f00efbb5640 1 --2- 172.21.15.72:0/4007194038 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f00e8159600 0x7f00e8159a80 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.140+0000 7f00efbb5640 1 --2- 172.21.15.72:0/4007194038 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f00d40039b0 0x7f00e81f7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.140+0000 7f00efbb5640 1 -- 172.21.15.72:0/4007194038 >> 172.21.15.72:0/4007194038 conn(0x7f00e80b4010 msgr2=0x7f00e80b5b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:08.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.140+0000 7f00efbb5640 1 -- 172.21.15.72:0/4007194038 shutdown_connections 2024-04-15T15:15:08.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.140+0000 7f00efbb5640 1 -- 172.21.15.72:0/4007194038 wait complete. 2024-04-15T15:15:08.155 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-15T15:15:08.155 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:15:08.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.261+0000 7f09b8540640 1 Processor -- start 2024-04-15T15:15:08.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.261+0000 7f09b8540640 1 -- start start 2024-04-15T15:15:08.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.262+0000 7f09b8540640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f09b015c7c0 0x7f09b006e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:08.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.262+0000 7f09b8540640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f09b006b960 0x7f09b006bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:08.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.262+0000 7f09b8540640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09b00cffd0 con 0x7f09b00af700 2024-04-15T15:15:08.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.262+0000 7f09b8540640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f09b015cba0 con 0x7f09b015c7c0 2024-04-15T15:15:08.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.262+0000 7f09b8540640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f09b006dc10 con 0x7f09b006b960 2024-04-15T15:15:08.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.263+0000 7f09b6add640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f09b006b960 0x7f09b006bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:08.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.263+0000 7f09b62dc640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f09b015c7c0 0x7f09b006e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:08.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.263+0000 7f09b62dc640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f09b015c7c0 0x7f09b006e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.72:59814/0 (socket says 172.21.15.72:59814) 2024-04-15T15:15:08.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.263+0000 7f09b6add640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f09b006b960 0x7f09b006bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3301/0 says I am v2:172.21.15.72:57548/0 (socket says 172.21.15.72:57548) 2024-04-15T15:15:08.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.263+0000 7f09b62dc640 1 -- 172.21.15.72:0/1135388084 learned_addr learned my addr 172.21.15.72:0/1135388084 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:08.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.263+0000 7f09b5adb640 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f09b00af700 0x7f09b006e960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.72:53322/0 (socket says 172.21.15.72:53322) 2024-04-15T15:15:08.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.263+0000 7f09b62dc640 1 -- 172.21.15.72:0/1135388084 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f09b006b960 msgr2=0x7f09b006bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:08.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.263+0000 7f09b62dc640 1 --2- 172.21.15.72:0/1135388084 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f09b006b960 0x7f09b006bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.263+0000 7f09b62dc640 1 -- 172.21.15.72:0/1135388084 >> v1:172.21.15.175:6789/0 conn(0x7f09b00af700 legacy=0x7f09b006e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:08.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.263+0000 7f09b62dc640 1 -- 172.21.15.72:0/1135388084 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09b006de60 con 0x7f09b015c7c0 2024-04-15T15:15:08.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.264+0000 7f09b62dc640 1 --2- 172.21.15.72:0/1135388084 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09b015c7c0 0x7f09b006e420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f09a8009870 tx=0x7f09a802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=343d49db768364d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:08.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.264+0000 7f09b52da640 1 -- 172.21.15.72:0/1135388084 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a8034070 con 0x7f09b015c7c0 2024-04-15T15:15:08.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.264+0000 7f09b52da640 1 -- 172.21.15.72:0/1135388084 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09a8030840 con 0x7f09b015c7c0 2024-04-15T15:15:08.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.264+0000 7f09b52da640 1 -- 172.21.15.72:0/1135388084 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a8030c10 con 0x7f09b015c7c0 2024-04-15T15:15:08.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.265+0000 7f09b8540640 1 -- 172.21.15.72:0/1135388084 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09b015c7c0 msgr2=0x7f09b006e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:08.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.265+0000 7f09b8540640 1 --2- 172.21.15.72:0/1135388084 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09b015c7c0 0x7f09b006e420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f09a8009870 tx=0x7f09a802b980 comp rx=0 tx=0).stop 2024-04-15T15:15:08.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.265+0000 7f09b8540640 1 -- 172.21.15.72:0/1135388084 shutdown_connections 2024-04-15T15:15:08.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.265+0000 7f09b8540640 1 --2- 172.21.15.72:0/1135388084 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f09b006b960 0x7f09b006bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.265+0000 7f09b8540640 1 --2- 172.21.15.72:0/1135388084 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09b015c7c0 0x7f09b006e420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.265+0000 7f09b8540640 1 -- 172.21.15.72:0/1135388084 >> 172.21.15.72:0/1135388084 conn(0x7f09b00b1dd0 msgr2=0x7f09b00b21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:08.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.265+0000 7f09b8540640 1 -- 172.21.15.72:0/1135388084 shutdown_connections 2024-04-15T15:15:08.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.265+0000 7f09b8540640 1 -- 172.21.15.72:0/1135388084 wait complete. 2024-04-15T15:15:08.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.266+0000 7f09b8540640 1 Processor -- start 2024-04-15T15:15:08.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.266+0000 7f09b8540640 1 -- start start 2024-04-15T15:15:08.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.267+0000 7f09b8540640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f09b006b960 0x7f09b0150a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:08.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.267+0000 7f09b8540640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09b00af700 0x7f09b006e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:08.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.267+0000 7f09b8540640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09b015c7c0 0x7f09b006e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:08.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.267+0000 7f09b8540640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f09b00cfab0 con 0x7f09b015c7c0 2024-04-15T15:15:08.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.267+0000 7f09b8540640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f09b00cf7d0 con 0x7f09b00af700 2024-04-15T15:15:08.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.267+0000 7f09b8540640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f09b00cdc60 con 0x7f09b006b960 2024-04-15T15:15:08.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.267+0000 7f09b6add640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09b015c7c0 0x7f09b006e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:08.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.267+0000 7f09b6add640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09b015c7c0 0x7f09b006e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44510/0 (socket says 172.21.15.72:44510) 2024-04-15T15:15:08.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.267+0000 7f09b6add640 1 -- 172.21.15.72:0/1584809000 learned_addr learned my addr 172.21.15.72:0/1584809000 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:08.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.267+0000 7f09b62dc640 1 --2- 172.21.15.72:0/1584809000 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f09b006b960 0x7f09b0150a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:08.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.267+0000 7f09b5adb640 1 --2- 172.21.15.72:0/1584809000 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09b00af700 0x7f09b006e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:08.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.267+0000 7f09b6add640 1 -- 172.21.15.72:0/1584809000 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f09b006b960 msgr2=0x7f09b0150a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:08.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.267+0000 7f09b6add640 1 --2- 172.21.15.72:0/1584809000 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f09b006b960 0x7f09b0150a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.267+0000 7f09b6add640 1 -- 172.21.15.72:0/1584809000 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09b00af700 msgr2=0x7f09b006e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:08.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.267+0000 7f09b6add640 1 --2- 172.21.15.72:0/1584809000 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09b00af700 0x7f09b006e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.267+0000 7f09b6add640 1 -- 172.21.15.72:0/1584809000 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09b015b0f0 con 0x7f09b015c7c0 2024-04-15T15:15:08.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.267+0000 7f09b6add640 1 --2- 172.21.15.72:0/1584809000 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09b015c7c0 0x7f09b006e8c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f09a000efc0 tx=0x7f09a000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:08.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.268+0000 7f09aeffd640 1 -- 172.21.15.72:0/1584809000 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a0018020 con 0x7f09b015c7c0 2024-04-15T15:15:08.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.268+0000 7f09aeffd640 1 -- 172.21.15.72:0/1584809000 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09a0014480 con 0x7f09b015c7c0 2024-04-15T15:15:08.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.268+0000 7f09b8540640 1 -- 172.21.15.72:0/1584809000 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09b01f3420 con 0x7f09b015c7c0 2024-04-15T15:15:08.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.268+0000 7f09aeffd640 1 -- 172.21.15.72:0/1584809000 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a0014870 con 0x7f09b015c7c0 2024-04-15T15:15:08.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.268+0000 7f09b8540640 1 -- 172.21.15.72:0/1584809000 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09b01f3a30 con 0x7f09b015c7c0 2024-04-15T15:15:08.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.269+0000 7f09b8540640 1 -- 172.21.15.72:0/1584809000 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09b00689e0 con 0x7f09b015c7c0 2024-04-15T15:15:08.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.269+0000 7f09aeffd640 1 -- 172.21.15.72:0/1584809000 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f09a0005d10 con 0x7f09b015c7c0 2024-04-15T15:15:08.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.269+0000 7f09aeffd640 1 --2- 172.21.15.72:0/1584809000 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f099c06c8f0 0x7f099c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:08.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.270+0000 7f09b62dc640 1 --2- 172.21.15.72:0/1584809000 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f099c06c8f0 0x7f099c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:08.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.270+0000 7f09aeffd640 1 -- 172.21.15.72:0/1584809000 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f09a008cb60 con 0x7f09b015c7c0 2024-04-15T15:15:08.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.270+0000 7f09b62dc640 1 --2- 172.21.15.72:0/1584809000 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f099c06c8f0 0x7f099c06edb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f09a8036000 tx=0x7f09a803b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:08.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.274+0000 7f09aeffd640 1 -- 172.21.15.72:0/1584809000 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f09a0022030 con 0x7f09b015c7c0 2024-04-15T15:15:08.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.484+0000 7f09b8540640 1 -- 172.21.15.72:0/1584809000 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f09b00ce670 con 0x7f09b015c7c0 2024-04-15T15:15:08.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.484+0000 7f09aeffd640 1 -- 172.21.15.72:0/1584809000 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5183 (secure 0 0 0) 0x7f09a0010040 con 0x7f09b015c7c0 2024-04-15T15:15:08.486 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:08.486 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4267,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6833/1659343919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1659343919},{"type":"v1","addr":"172.21.15.175:6833","nonce":1659343919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6835/4113588141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6834","nonce":4113588141},{"type":"v1","addr":"172.21.15.175:6835","nonce":4113588141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4296,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/4151548261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4151548261},{"type":"v1","addr":"172.21.15.72:6836","nonce":4151548261}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:14:51.411091+0000","modified":"2024-04-15T15:15:07.485436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/3458067804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3458067804},{"type":"v1","addr":"172.21.15.72:6837","nonce":3458067804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4299,"qdb_cluster":[4299]},"id":1}]} 2024-04-15T15:15:08.487 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 9 2024-04-15T15:15:08.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.486+0000 7f09b8540640 1 -- 172.21.15.72:0/1584809000 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f099c06c8f0 msgr2=0x7f099c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:08.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.486+0000 7f09b8540640 1 --2- 172.21.15.72:0/1584809000 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f099c06c8f0 0x7f099c06edb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f09a8036000 tx=0x7f09a803b040 comp rx=0 tx=0).stop 2024-04-15T15:15:08.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.486+0000 7f09b8540640 1 -- 172.21.15.72:0/1584809000 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09b015c7c0 msgr2=0x7f09b006e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:08.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.486+0000 7f09b8540640 1 --2- 172.21.15.72:0/1584809000 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09b015c7c0 0x7f09b006e8c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f09a000efc0 tx=0x7f09a000c490 comp rx=0 tx=0).stop 2024-04-15T15:15:08.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.486+0000 7f09b8540640 1 -- 172.21.15.72:0/1584809000 shutdown_connections 2024-04-15T15:15:08.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.486+0000 7f09b8540640 1 --2- 172.21.15.72:0/1584809000 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f099c06c8f0 0x7f099c06edb0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.486+0000 7f09b8540640 1 --2- 172.21.15.72:0/1584809000 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09b015c7c0 0x7f09b006e8c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.486+0000 7f09b8540640 1 --2- 172.21.15.72:0/1584809000 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09b00af700 0x7f09b006e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.486+0000 7f09b8540640 1 --2- 172.21.15.72:0/1584809000 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f09b006b960 0x7f09b0150a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.486+0000 7f09b8540640 1 -- 172.21.15.72:0/1584809000 >> 172.21.15.72:0/1584809000 conn(0x7f09b00b1dd0 msgr2=0x7f09b00ada30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:08.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.486+0000 7f09b8540640 1 -- 172.21.15.72:0/1584809000 shutdown_connections 2024-04-15T15:15:08.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.486+0000 7f09b8540640 1 -- 172.21.15.72:0/1584809000 wait complete. 2024-04-15T15:15:08.499 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:08.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.601+0000 7f0d82e2a640 1 Processor -- start 2024-04-15T15:15:08.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.601+0000 7f0d82e2a640 1 -- start start 2024-04-15T15:15:08.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.602+0000 7f0d82e2a640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0d7c0d12b0 0x7f0d7c0d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:08.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.602+0000 7f0d82e2a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f0d7c0d2ea0 0x7f0d7c0d45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:08.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.602+0000 7f0d82e2a640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d7c162220 con 0x7f0d7c161e40 2024-04-15T15:15:08.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.602+0000 7f0d82e2a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f0d7c0b3f00 con 0x7f0d7c0d2ea0 2024-04-15T15:15:08.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.602+0000 7f0d82e2a640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f0d7c0d21f0 con 0x7f0d7c0d12b0 2024-04-15T15:15:08.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.602+0000 7f0d813c7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f0d7c0d2ea0 0x7f0d7c0d45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:08.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.602+0000 7f0d813c7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f0d7c0d2ea0 0x7f0d7c0d45b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44526/0 (socket says 172.21.15.72:44526) 2024-04-15T15:15:08.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.602+0000 7f0d813c7640 1 -- 172.21.15.72:0/1495410795 learned_addr learned my addr 172.21.15.72:0/1495410795 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:08.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.602+0000 7f0d813c7640 1 -- 172.21.15.72:0/1495410795 >> v2:172.21.15.175:3300/0 conn(0x7f0d7c0d12b0 msgr2=0x7f0d7c0d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:08.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.602+0000 7f0d813c7640 1 --2- 172.21.15.72:0/1495410795 >> v2:172.21.15.175:3300/0 conn(0x7f0d7c0d12b0 0x7f0d7c0d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.602+0000 7f0d813c7640 1 -- 172.21.15.72:0/1495410795 >> v1:172.21.15.175:6789/0 conn(0x7f0d7c161e40 legacy=0x7f0d7c0b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:08.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.602+0000 7f0d813c7640 1 -- 172.21.15.72:0/1495410795 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d7c0d1820 con 0x7f0d7c0d2ea0 2024-04-15T15:15:08.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.603+0000 7f0d813c7640 1 --2- 172.21.15.72:0/1495410795 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d7c0d2ea0 0x7f0d7c0d45b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0d64009870 tx=0x7f0d6402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcc36d06114aa632 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:08.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.603+0000 7f0d7b7fe640 1 -- 172.21.15.72:0/1495410795 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d6402e070 con 0x7f0d7c0d2ea0 2024-04-15T15:15:08.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.603+0000 7f0d7b7fe640 1 -- 172.21.15.72:0/1495410795 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d64037070 con 0x7f0d7c0d2ea0 2024-04-15T15:15:08.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.603+0000 7f0d7b7fe640 1 -- 172.21.15.72:0/1495410795 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d64031a00 con 0x7f0d7c0d2ea0 2024-04-15T15:15:08.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.603+0000 7f0d82e2a640 1 -- 172.21.15.72:0/1495410795 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d7c0d2ea0 msgr2=0x7f0d7c0d45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:08.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.603+0000 7f0d82e2a640 1 --2- 172.21.15.72:0/1495410795 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d7c0d2ea0 0x7f0d7c0d45b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f0d64009870 tx=0x7f0d6402b880 comp rx=0 tx=0).stop 2024-04-15T15:15:08.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.603+0000 7f0d82e2a640 1 -- 172.21.15.72:0/1495410795 shutdown_connections 2024-04-15T15:15:08.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.603+0000 7f0d82e2a640 1 --2- 172.21.15.72:0/1495410795 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d7c0d2ea0 0x7f0d7c0d45b0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.603+0000 7f0d82e2a640 1 --2- 172.21.15.72:0/1495410795 >> v2:172.21.15.175:3300/0 conn(0x7f0d7c0d12b0 0x7f0d7c0d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.603+0000 7f0d82e2a640 1 -- 172.21.15.72:0/1495410795 >> 172.21.15.72:0/1495410795 conn(0x7f0d7c09d970 msgr2=0x7f0d7c0b95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:08.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.603+0000 7f0d82e2a640 1 -- 172.21.15.72:0/1495410795 shutdown_connections 2024-04-15T15:15:08.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.603+0000 7f0d82e2a640 1 -- 172.21.15.72:0/1495410795 wait complete. 2024-04-15T15:15:08.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.605+0000 7f0d82e2a640 1 Processor -- start 2024-04-15T15:15:08.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.605+0000 7f0d82e2a640 1 -- start start 2024-04-15T15:15:08.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.605+0000 7f0d82e2a640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0d7c0d12b0 0x7f0d7c0b4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:08.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.605+0000 7f0d82e2a640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0d7c161e40 0x7f0d7c06e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:08.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.605+0000 7f0d82e2a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d7c152a30 0x7f0d7c0c8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:08.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.606+0000 7f0d82e2a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f0d7c0d2440 con 0x7f0d7c152a30 2024-04-15T15:15:08.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.606+0000 7f0d82e2a640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f0d7c0d1de0 con 0x7f0d7c0d12b0 2024-04-15T15:15:08.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.606+0000 7f0d82e2a640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f0d7c0d4af0 con 0x7f0d7c161e40 2024-04-15T15:15:08.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.606+0000 7f0d813c7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d7c152a30 0x7f0d7c0c8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:08.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.606+0000 7f0d813c7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d7c152a30 0x7f0d7c0c8080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44536/0 (socket says 172.21.15.72:44536) 2024-04-15T15:15:08.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.606+0000 7f0d813c7640 1 -- 172.21.15.72:0/2273530505 learned_addr learned my addr 172.21.15.72:0/2273530505 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:08.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.606+0000 7f0d80bc6640 1 --2- 172.21.15.72:0/2273530505 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0d7c0d12b0 0x7f0d7c0b4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:08.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.606+0000 7f0d7bfff640 1 --2- 172.21.15.72:0/2273530505 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0d7c161e40 0x7f0d7c06e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:08.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.606+0000 7f0d813c7640 1 -- 172.21.15.72:0/2273530505 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0d7c161e40 msgr2=0x7f0d7c06e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:08.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.606+0000 7f0d813c7640 1 --2- 172.21.15.72:0/2273530505 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0d7c161e40 0x7f0d7c06e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.606+0000 7f0d813c7640 1 -- 172.21.15.72:0/2273530505 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0d7c0d12b0 msgr2=0x7f0d7c0b4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:08.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.606+0000 7f0d813c7640 1 --2- 172.21.15.72:0/2273530505 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0d7c0d12b0 0x7f0d7c0b4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.606+0000 7f0d813c7640 1 -- 172.21.15.72:0/2273530505 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d7c0b5460 con 0x7f0d7c152a30 2024-04-15T15:15:08.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.606+0000 7f0d813c7640 1 --2- 172.21.15.72:0/2273530505 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d7c152a30 0x7f0d7c0c8080 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0d64009870 tx=0x7f0d640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:08.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.606+0000 7f0d797fa640 1 -- 172.21.15.72:0/2273530505 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d64032880 con 0x7f0d7c152a30 2024-04-15T15:15:08.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.606+0000 7f0d797fa640 1 -- 172.21.15.72:0/2273530505 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d64032a20 con 0x7f0d7c152a30 2024-04-15T15:15:08.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.606+0000 7f0d797fa640 1 -- 172.21.15.72:0/2273530505 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d64032e10 con 0x7f0d7c152a30 2024-04-15T15:15:08.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.606+0000 7f0d82e2a640 1 -- 172.21.15.72:0/2273530505 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d7c0b5740 con 0x7f0d7c152a30 2024-04-15T15:15:08.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.606+0000 7f0d82e2a640 1 -- 172.21.15.72:0/2273530505 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d7c0b5d00 con 0x7f0d7c152a30 2024-04-15T15:15:08.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.607+0000 7f0d82e2a640 1 -- 172.21.15.72:0/2273530505 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d7c0689e0 con 0x7f0d7c152a30 2024-04-15T15:15:08.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.608+0000 7f0d797fa640 1 -- 172.21.15.72:0/2273530505 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0d6404c060 con 0x7f0d7c152a30 2024-04-15T15:15:08.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.608+0000 7f0d797fa640 1 --2- 172.21.15.72:0/2273530505 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0d5c06c8f0 0x7f0d5c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:08.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.608+0000 7f0d797fa640 1 -- 172.21.15.72:0/2273530505 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f0d640ad8a0 con 0x7f0d7c152a30 2024-04-15T15:15:08.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.608+0000 7f0d80bc6640 1 --2- 172.21.15.72:0/2273530505 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0d5c06c8f0 0x7f0d5c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:08.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.609+0000 7f0d80bc6640 1 --2- 172.21.15.72:0/2273530505 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0d5c06c8f0 0x7f0d5c06edb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0d74006fd0 tx=0x7f0d74009060 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:08.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.612+0000 7f0d797fa640 1 -- 172.21.15.72:0/2273530505 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0d64040070 con 0x7f0d7c152a30 2024-04-15T15:15:08.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.792+0000 7f0d82e2a640 1 -- 172.21.15.72:0/2273530505 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0d7c0d4ec0 con 0x7f0d7c152a30 2024-04-15T15:15:08.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.793+0000 7f0d797fa640 1 -- 172.21.15.72:0/2273530505 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7f0d6407b460 con 0x7f0d7c152a30 2024-04-15T15:15:08.795 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:08.795 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":24,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:14:56.451113+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:14:49.438366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:14:50.368379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:14:54.081157+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:08.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.795+0000 7f0d82e2a640 1 -- 172.21.15.72:0/2273530505 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0d5c06c8f0 msgr2=0x7f0d5c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:08.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.795+0000 7f0d82e2a640 1 --2- 172.21.15.72:0/2273530505 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0d5c06c8f0 0x7f0d5c06edb0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0d74006fd0 tx=0x7f0d74009060 comp rx=0 tx=0).stop 2024-04-15T15:15:08.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.795+0000 7f0d82e2a640 1 -- 172.21.15.72:0/2273530505 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d7c152a30 msgr2=0x7f0d7c0c8080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:08.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.795+0000 7f0d82e2a640 1 --2- 172.21.15.72:0/2273530505 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d7c152a30 0x7f0d7c0c8080 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0d64009870 tx=0x7f0d640046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.795+0000 7f0d82e2a640 1 -- 172.21.15.72:0/2273530505 shutdown_connections 2024-04-15T15:15:08.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.795+0000 7f0d82e2a640 1 --2- 172.21.15.72:0/2273530505 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0d5c06c8f0 0x7f0d5c06edb0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.795+0000 7f0d82e2a640 1 --2- 172.21.15.72:0/2273530505 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d7c152a30 0x7f0d7c0c8080 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.795+0000 7f0d82e2a640 1 --2- 172.21.15.72:0/2273530505 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0d7c161e40 0x7f0d7c06e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.795+0000 7f0d82e2a640 1 --2- 172.21.15.72:0/2273530505 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0d7c0d12b0 0x7f0d7c0b4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.795+0000 7f0d82e2a640 1 -- 172.21.15.72:0/2273530505 >> 172.21.15.72:0/2273530505 conn(0x7f0d7c09d970 msgr2=0x7f0d7c0b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:08.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.795+0000 7f0d82e2a640 1 -- 172.21.15.72:0/2273530505 shutdown_connections 2024-04-15T15:15:08.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.795+0000 7f0d82e2a640 1 -- 172.21.15.72:0/2273530505 wait complete. 2024-04-15T15:15:08.808 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-15T15:15:08.808 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:15:08.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.909+0000 7f4af34c8640 1 Processor -- start 2024-04-15T15:15:08.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.910+0000 7f4af34c8640 1 -- start start 2024-04-15T15:15:08.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.910+0000 7f4af34c8640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4aec15c480 0x7f4aec06ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:08.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.910+0000 7f4af34c8640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f4aec0ce670 0x7f4aec0cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:08.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.910+0000 7f4af34c8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4aec0cf300 0x7f4aec0cff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:08.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.910+0000 7f4af34c8640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f4aec0d1b30 con 0x7f4aec0cf300 2024-04-15T15:15:08.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.910+0000 7f4af34c8640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f4aec0cca00 con 0x7f4aec0ce670 2024-04-15T15:15:08.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.910+0000 7f4af34c8640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f4aec0d1d80 con 0x7f4aec15c480 2024-04-15T15:15:08.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.911+0000 7f4af1a65640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4aec0cf300 0x7f4aec0cff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:08.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.911+0000 7f4af1a65640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4aec0cf300 0x7f4aec0cff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44542/0 (socket says 172.21.15.72:44542) 2024-04-15T15:15:08.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.911+0000 7f4af1a65640 1 -- 172.21.15.72:0/3172770978 learned_addr learned my addr 172.21.15.72:0/3172770978 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:08.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.911+0000 7f4af0a63640 1 --2- 172.21.15.72:0/3172770978 >> v2:172.21.15.175:3300/0 conn(0x7f4aec0ce670 0x7f4aec0cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:08.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.911+0000 7f4af1264640 1 --2- 172.21.15.72:0/3172770978 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4aec15c480 0x7f4aec06ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:08.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.911+0000 7f4af1a65640 1 -- 172.21.15.72:0/3172770978 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4aec15c480 msgr2=0x7f4aec06ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:08.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.911+0000 7f4af1a65640 1 --2- 172.21.15.72:0/3172770978 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4aec15c480 0x7f4aec06ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.911+0000 7f4af1a65640 1 -- 172.21.15.72:0/3172770978 >> v2:172.21.15.175:3300/0 conn(0x7f4aec0ce670 msgr2=0x7f4aec0cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:08.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.911+0000 7f4af1a65640 1 --2- 172.21.15.72:0/3172770978 >> v2:172.21.15.175:3300/0 conn(0x7f4aec0ce670 0x7f4aec0cedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.911+0000 7f4af1a65640 1 -- 172.21.15.72:0/3172770978 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aec0d2060 con 0x7f4aec0cf300 2024-04-15T15:15:08.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.911+0000 7f4af0a63640 1 --2- 172.21.15.72:0/3172770978 >> v2:172.21.15.175:3300/0 conn(0x7f4aec0ce670 0x7f4aec0cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:08.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.912+0000 7f4af1a65640 1 --2- 172.21.15.72:0/3172770978 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4aec0cf300 0x7f4aec0cff80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4aec0d3420 tx=0x7f4ae402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93d289d2f765332a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:08.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.912+0000 7f4ae3fff640 1 -- 172.21.15.72:0/3172770978 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae4035020 con 0x7f4aec0cf300 2024-04-15T15:15:08.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.912+0000 7f4ae3fff640 1 -- 172.21.15.72:0/3172770978 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ae4031870 con 0x7f4aec0cf300 2024-04-15T15:15:08.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.912+0000 7f4ae3fff640 1 -- 172.21.15.72:0/3172770978 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ae4031c40 con 0x7f4aec0cf300 2024-04-15T15:15:08.918 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.912+0000 7f4af34c8640 1 -- 172.21.15.72:0/3172770978 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4aec0cf300 msgr2=0x7f4aec0cff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:08.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.912+0000 7f4af34c8640 1 --2- 172.21.15.72:0/3172770978 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4aec0cf300 0x7f4aec0cff80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4aec0d3420 tx=0x7f4ae402b910 comp rx=0 tx=0).stop 2024-04-15T15:15:08.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.912+0000 7f4af34c8640 1 -- 172.21.15.72:0/3172770978 shutdown_connections 2024-04-15T15:15:08.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.912+0000 7f4af34c8640 1 --2- 172.21.15.72:0/3172770978 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4aec0cf300 0x7f4aec0cff80 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.912+0000 7f4af34c8640 1 --2- 172.21.15.72:0/3172770978 >> v2:172.21.15.175:3300/0 conn(0x7f4aec0ce670 0x7f4aec0cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.912+0000 7f4af34c8640 1 --2- 172.21.15.72:0/3172770978 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4aec15c480 0x7f4aec06ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.912+0000 7f4af34c8640 1 -- 172.21.15.72:0/3172770978 >> 172.21.15.72:0/3172770978 conn(0x7f4aec0b3fa0 msgr2=0x7f4aec0b43b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:08.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.912+0000 7f4af34c8640 1 -- 172.21.15.72:0/3172770978 shutdown_connections 2024-04-15T15:15:08.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.912+0000 7f4af34c8640 1 -- 172.21.15.72:0/3172770978 wait complete. 2024-04-15T15:15:08.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.914+0000 7f4af34c8640 1 Processor -- start 2024-04-15T15:15:08.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.914+0000 7f4af34c8640 1 -- start start 2024-04-15T15:15:08.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.914+0000 7f4af34c8640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4aec0ce670 0x7f4aec156a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:08.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.914+0000 7f4af34c8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4aec0cf300 0x7f4aec06d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:08.919 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.914+0000 7f4af34c8640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4aec15c480 0x7f4aec0cb490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:08.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.914+0000 7f4af34c8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f4aec0cd170 con 0x7f4aec0cf300 2024-04-15T15:15:08.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.914+0000 7f4af34c8640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f4aec0cead0 con 0x7f4aec0ce670 2024-04-15T15:15:08.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.914+0000 7f4af34c8640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f4aec15c860 con 0x7f4aec15c480 2024-04-15T15:15:08.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.914+0000 7f4af0a63640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4aec0cf300 0x7f4aec06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:08.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.914+0000 7f4af0a63640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4aec0cf300 0x7f4aec06d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44554/0 (socket says 172.21.15.72:44554) 2024-04-15T15:15:08.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.914+0000 7f4af0a63640 1 -- 172.21.15.72:0/3374835058 learned_addr learned my addr 172.21.15.72:0/3374835058 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:08.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.915+0000 7f4af1264640 1 --2- 172.21.15.72:0/3374835058 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4aec0ce670 0x7f4aec156a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:08.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.915+0000 7f4af1a65640 1 --2- 172.21.15.72:0/3374835058 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4aec15c480 0x7f4aec0cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:08.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.915+0000 7f4af0a63640 1 -- 172.21.15.72:0/3374835058 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4aec15c480 msgr2=0x7f4aec0cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:08.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.915+0000 7f4af0a63640 1 --2- 172.21.15.72:0/3374835058 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4aec15c480 0x7f4aec0cb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.915+0000 7f4af0a63640 1 -- 172.21.15.72:0/3374835058 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4aec0ce670 msgr2=0x7f4aec156a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:08.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.915+0000 7f4af0a63640 1 --2- 172.21.15.72:0/3374835058 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4aec0ce670 0x7f4aec156a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.915+0000 7f4af0a63640 1 -- 172.21.15.72:0/3374835058 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aec06dd50 con 0x7f4aec0cf300 2024-04-15T15:15:08.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.915+0000 7f4af1264640 1 --2- 172.21.15.72:0/3374835058 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4aec0ce670 0x7f4aec156a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:08.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.915+0000 7f4af1a65640 1 --2- 172.21.15.72:0/3374835058 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4aec15c480 0x7f4aec0cb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:08.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.915+0000 7f4af0a63640 1 --2- 172.21.15.72:0/3374835058 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4aec0cf300 0x7f4aec06d300 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4adc00e760 tx=0x7f4adc00ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:08.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.915+0000 7f4ae1ffb640 1 -- 172.21.15.72:0/3374835058 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4adc00f040 con 0x7f4aec0cf300 2024-04-15T15:15:08.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.915+0000 7f4ae1ffb640 1 -- 172.21.15.72:0/3374835058 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4adc004230 con 0x7f4aec0cf300 2024-04-15T15:15:08.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.915+0000 7f4af34c8640 1 -- 172.21.15.72:0/3374835058 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4aec06e030 con 0x7f4aec0cf300 2024-04-15T15:15:08.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.915+0000 7f4ae1ffb640 1 -- 172.21.15.72:0/3374835058 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4adc00ee70 con 0x7f4aec0cf300 2024-04-15T15:15:08.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.915+0000 7f4af34c8640 1 -- 172.21.15.72:0/3374835058 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4aec06e5f0 con 0x7f4aec0cf300 2024-04-15T15:15:08.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.916+0000 7f4af34c8640 1 -- 172.21.15.72:0/3374835058 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4aec0690f0 con 0x7f4aec0cf300 2024-04-15T15:15:08.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.917+0000 7f4ae1ffb640 1 -- 172.21.15.72:0/3374835058 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4adc00c6d0 con 0x7f4aec0cf300 2024-04-15T15:15:08.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.917+0000 7f4ae1ffb640 1 --2- 172.21.15.72:0/3374835058 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4ad006c920 0x7f4ad006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:08.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.917+0000 7f4ae1ffb640 1 -- 172.21.15.72:0/3374835058 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f4adc08bb70 con 0x7f4aec0cf300 2024-04-15T15:15:08.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.917+0000 7f4af1264640 1 --2- 172.21.15.72:0/3374835058 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4ad006c920 0x7f4ad006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:08.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.917+0000 7f4af1264640 1 --2- 172.21.15.72:0/3374835058 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4ad006c920 0x7f4ad006ede0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4ae8009ed0 tx=0x7f4ae800b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:08.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:08.921+0000 7f4ae1ffb640 1 -- 172.21.15.72:0/3374835058 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4adc017020 con 0x7f4aec0cf300 2024-04-15T15:15:09.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.092+0000 7f4af34c8640 1 -- 172.21.15.72:0/3374835058 --> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4aec0b2cf0 con 0x7f4ad006c920 2024-04-15T15:15:09.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.100+0000 7f4ae1ffb640 1 -- 172.21.15.72:0/3374835058 <== mgr.4101 v2:172.21.15.72:6832/3503807805 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332534 (secure 0 0 0) 0x7f4ae806c310 con 0x7f4ad006c920 2024-04-15T15:15:09.106 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:09.108 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-04-15T15:15:09.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.104+0000 7f4af34c8640 1 -- 172.21.15.72:0/3374835058 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4ad006c920 msgr2=0x7f4ad006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:09.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.104+0000 7f4af34c8640 1 --2- 172.21.15.72:0/3374835058 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4ad006c920 0x7f4ad006ede0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4ae8009ed0 tx=0x7f4ae800b040 comp rx=0 tx=0).stop 2024-04-15T15:15:09.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.104+0000 7f4af34c8640 1 -- 172.21.15.72:0/3374835058 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4aec0cf300 msgr2=0x7f4aec06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:09.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.104+0000 7f4af34c8640 1 --2- 172.21.15.72:0/3374835058 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4aec0cf300 0x7f4aec06d300 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4adc00e760 tx=0x7f4adc00ec20 comp rx=0 tx=0).stop 2024-04-15T15:15:09.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.104+0000 7f4af34c8640 1 -- 172.21.15.72:0/3374835058 shutdown_connections 2024-04-15T15:15:09.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.104+0000 7f4af34c8640 1 --2- 172.21.15.72:0/3374835058 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4ad006c920 0x7f4ad006ede0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.104+0000 7f4af34c8640 1 --2- 172.21.15.72:0/3374835058 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4aec15c480 0x7f4aec0cb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.104+0000 7f4af34c8640 1 --2- 172.21.15.72:0/3374835058 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4aec0cf300 0x7f4aec06d300 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.104+0000 7f4af34c8640 1 --2- 172.21.15.72:0/3374835058 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4aec0ce670 0x7f4aec156a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.104+0000 7f4af34c8640 1 -- 172.21.15.72:0/3374835058 >> 172.21.15.72:0/3374835058 conn(0x7f4aec0b3fa0 msgr2=0x7f4aec0b5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:09.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.104+0000 7f4af34c8640 1 -- 172.21.15.72:0/3374835058 shutdown_connections 2024-04-15T15:15:09.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.104+0000 7f4af34c8640 1 -- 172.21.15.72:0/3374835058 wait complete. 2024-04-15T15:15:09.117 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-04-15T15:15:07.347060+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594867,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":206,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219392,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755328,"statfs":{"total":773094113280,"available":772869455872,"internally_reserved":0,"allocated":3284992,"data_stored":1991782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2089,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.240670"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137048+0000","last_change":"2024-04-15T15:14:51.075131+0000","last_active":"2024-04-15T15:14:57.137048+0000","last_peered":"2024-04-15T15:14:57.137048+0000","last_clean":"2024-04-15T15:14:57.137048+0000","last_became_active":"2024-04-15T15:14:51.074841+0000","last_became_peered":"2024-04-15T15:14:51.074841+0000","last_unstale":"2024-04-15T15:14:57.137048+0000","last_undegraded":"2024-04-15T15:14:57.137048+0000","last_fullsized":"2024-04-15T15:14:57.137048+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:19:56.231155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138223+0000","last_change":"2024-04-15T15:14:52.077480+0000","last_active":"2024-04-15T15:14:57.138223+0000","last_peered":"2024-04-15T15:14:57.138223+0000","last_clean":"2024-04-15T15:14:57.138223+0000","last_became_active":"2024-04-15T15:14:52.077166+0000","last_became_peered":"2024-04-15T15:14:52.077166+0000","last_unstale":"2024-04-15T15:14:57.138223+0000","last_undegraded":"2024-04-15T15:14:57.138223+0000","last_fullsized":"2024-04-15T15:14:57.138223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:13:01.642838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136181+0000","last_change":"2024-04-15T15:14:56.255424+0000","last_active":"2024-04-15T15:14:57.136181+0000","last_peered":"2024-04-15T15:14:57.136181+0000","last_clean":"2024-04-15T15:14:57.136181+0000","last_became_active":"2024-04-15T15:14:56.254987+0000","last_became_peered":"2024-04-15T15:14:56.254987+0000","last_unstale":"2024-04-15T15:14:57.136181+0000","last_undegraded":"2024-04-15T15:14:57.136181+0000","last_fullsized":"2024-04-15T15:14:57.136181+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:11:24.196989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138454+0000","last_change":"2024-04-15T15:14:51.071376+0000","last_active":"2024-04-15T15:14:57.138454+0000","last_peered":"2024-04-15T15:14:57.138454+0000","last_clean":"2024-04-15T15:14:57.138454+0000","last_became_active":"2024-04-15T15:14:51.071108+0000","last_became_peered":"2024-04-15T15:14:51.071108+0000","last_unstale":"2024-04-15T15:14:57.138454+0000","last_undegraded":"2024-04-15T15:14:57.138454+0000","last_fullsized":"2024-04-15T15:14:57.138454+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:50:01.422527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136732+0000","last_change":"2024-04-15T15:14:52.083839+0000","last_active":"2024-04-15T15:14:57.136732+0000","last_peered":"2024-04-15T15:14:57.136732+0000","last_clean":"2024-04-15T15:14:57.136732+0000","last_became_active":"2024-04-15T15:14:52.083533+0000","last_became_peered":"2024-04-15T15:14:52.083533+0000","last_unstale":"2024-04-15T15:14:57.136732+0000","last_undegraded":"2024-04-15T15:14:57.136732+0000","last_fullsized":"2024-04-15T15:14:57.136732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:34:51.775979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135872+0000","last_change":"2024-04-15T15:14:56.098375+0000","last_active":"2024-04-15T15:14:57.135872+0000","last_peered":"2024-04-15T15:14:57.135872+0000","last_clean":"2024-04-15T15:14:57.135872+0000","last_became_active":"2024-04-15T15:14:56.097969+0000","last_became_peered":"2024-04-15T15:14:56.097969+0000","last_unstale":"2024-04-15T15:14:57.135872+0000","last_undegraded":"2024-04-15T15:14:57.135872+0000","last_fullsized":"2024-04-15T15:14:57.135872+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:12:44.445524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136739+0000","last_change":"2024-04-15T15:14:51.079869+0000","last_active":"2024-04-15T15:14:57.136739+0000","last_peered":"2024-04-15T15:14:57.136739+0000","last_clean":"2024-04-15T15:14:57.136739+0000","last_became_active":"2024-04-15T15:14:51.079576+0000","last_became_peered":"2024-04-15T15:14:51.079576+0000","last_unstale":"2024-04-15T15:14:57.136739+0000","last_undegraded":"2024-04-15T15:14:57.136739+0000","last_fullsized":"2024-04-15T15:14:57.136739+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:58:40.440083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109631+0000","last_change":"2024-04-15T15:14:52.075307+0000","last_active":"2024-04-15T15:14:57.109631+0000","last_peered":"2024-04-15T15:14:57.109631+0000","last_clean":"2024-04-15T15:14:57.109631+0000","last_became_active":"2024-04-15T15:14:52.075001+0000","last_became_peered":"2024-04-15T15:14:52.075001+0000","last_unstale":"2024-04-15T15:14:57.109631+0000","last_undegraded":"2024-04-15T15:14:57.109631+0000","last_fullsized":"2024-04-15T15:14:57.109631+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:01:24.679360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136399+0000","last_change":"2024-04-15T15:14:56.098292+0000","last_active":"2024-04-15T15:14:57.136399+0000","last_peered":"2024-04-15T15:14:57.136399+0000","last_clean":"2024-04-15T15:14:57.136399+0000","last_became_active":"2024-04-15T15:14:56.097848+0000","last_became_peered":"2024-04-15T15:14:56.097848+0000","last_unstale":"2024-04-15T15:14:57.136399+0000","last_undegraded":"2024-04-15T15:14:57.136399+0000","last_fullsized":"2024-04-15T15:14:57.136399+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:26:49.486090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136078+0000","last_change":"2024-04-15T15:14:51.079754+0000","last_active":"2024-04-15T15:14:57.136078+0000","last_peered":"2024-04-15T15:14:57.136078+0000","last_clean":"2024-04-15T15:14:57.136078+0000","last_became_active":"2024-04-15T15:14:51.079452+0000","last_became_peered":"2024-04-15T15:14:51.079452+0000","last_unstale":"2024-04-15T15:14:57.136078+0000","last_undegraded":"2024-04-15T15:14:57.136078+0000","last_fullsized":"2024-04-15T15:14:57.136078+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:19:37.987965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.110221+0000","last_change":"2024-04-15T15:14:52.076654+0000","last_active":"2024-04-15T15:14:57.110221+0000","last_peered":"2024-04-15T15:14:57.110221+0000","last_clean":"2024-04-15T15:14:57.110221+0000","last_became_active":"2024-04-15T15:14:52.076283+0000","last_became_peered":"2024-04-15T15:14:52.076283+0000","last_unstale":"2024-04-15T15:14:57.110221+0000","last_undegraded":"2024-04-15T15:14:57.110221+0000","last_fullsized":"2024-04-15T15:14:57.110221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:57:57.665646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136259+0000","last_change":"2024-04-15T15:14:56.105257+0000","last_active":"2024-04-15T15:14:57.136259+0000","last_peered":"2024-04-15T15:14:57.136259+0000","last_clean":"2024-04-15T15:14:57.136259+0000","last_became_active":"2024-04-15T15:14:56.104856+0000","last_became_peered":"2024-04-15T15:14:56.104856+0000","last_unstale":"2024-04-15T15:14:57.136259+0000","last_undegraded":"2024-04-15T15:14:57.136259+0000","last_fullsized":"2024-04-15T15:14:57.136259+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:10:23.721558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108699+0000","last_change":"2024-04-15T15:14:52.086412+0000","last_active":"2024-04-15T15:14:57.108699+0000","last_peered":"2024-04-15T15:14:57.108699+0000","last_clean":"2024-04-15T15:14:57.108699+0000","last_became_active":"2024-04-15T15:14:52.085479+0000","last_became_peered":"2024-04-15T15:14:52.085479+0000","last_unstale":"2024-04-15T15:14:57.108699+0000","last_undegraded":"2024-04-15T15:14:57.108699+0000","last_fullsized":"2024-04-15T15:14:57.108699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:13:01.642838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136325+0000","last_change":"2024-04-15T15:14:51.080109+0000","last_active":"2024-04-15T15:14:57.136325+0000","last_peered":"2024-04-15T15:14:57.136325+0000","last_clean":"2024-04-15T15:14:57.136325+0000","last_became_active":"2024-04-15T15:14:51.079768+0000","last_became_peered":"2024-04-15T15:14:51.079768+0000","last_unstale":"2024-04-15T15:14:57.136325+0000","last_undegraded":"2024-04-15T15:14:57.136325+0000","last_fullsized":"2024-04-15T15:14:57.136325+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:07:48.416478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136732+0000","last_change":"2024-04-15T15:14:56.102424+0000","last_active":"2024-04-15T15:14:57.136732+0000","last_peered":"2024-04-15T15:14:57.136732+0000","last_clean":"2024-04-15T15:14:57.136732+0000","last_became_active":"2024-04-15T15:14:56.101985+0000","last_became_peered":"2024-04-15T15:14:56.101985+0000","last_unstale":"2024-04-15T15:14:57.136732+0000","last_undegraded":"2024-04-15T15:14:57.136732+0000","last_fullsized":"2024-04-15T15:14:57.136732+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:01:28.704663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136649+0000","last_change":"2024-04-15T15:14:52.084991+0000","last_active":"2024-04-15T15:14:57.136649+0000","last_peered":"2024-04-15T15:14:57.136649+0000","last_clean":"2024-04-15T15:14:57.136649+0000","last_became_active":"2024-04-15T15:14:52.082250+0000","last_became_peered":"2024-04-15T15:14:52.082250+0000","last_unstale":"2024-04-15T15:14:57.136649+0000","last_undegraded":"2024-04-15T15:14:57.136649+0000","last_fullsized":"2024-04-15T15:14:57.136649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:26:45.460787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137232+0000","last_change":"2024-04-15T15:14:51.071863+0000","last_active":"2024-04-15T15:14:57.137232+0000","last_peered":"2024-04-15T15:14:57.137232+0000","last_clean":"2024-04-15T15:14:57.137232+0000","last_became_active":"2024-04-15T15:14:51.071595+0000","last_became_peered":"2024-04-15T15:14:51.071595+0000","last_unstale":"2024-04-15T15:14:57.137232+0000","last_undegraded":"2024-04-15T15:14:57.137232+0000","last_fullsized":"2024-04-15T15:14:57.137232+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:19:37.987965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136625+0000","last_change":"2024-04-15T15:14:56.099259+0000","last_active":"2024-04-15T15:14:57.136625+0000","last_peered":"2024-04-15T15:14:57.136625+0000","last_clean":"2024-04-15T15:14:57.136625+0000","last_became_active":"2024-04-15T15:14:56.098862+0000","last_became_peered":"2024-04-15T15:14:56.098862+0000","last_unstale":"2024-04-15T15:14:57.136625+0000","last_undegraded":"2024-04-15T15:14:57.136625+0000","last_fullsized":"2024-04-15T15:14:57.136625+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:38:39.210922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136403+0000","last_change":"2024-04-15T15:14:52.080320+0000","last_active":"2024-04-15T15:14:57.136403+0000","last_peered":"2024-04-15T15:14:57.136403+0000","last_clean":"2024-04-15T15:14:57.136403+0000","last_became_active":"2024-04-15T15:14:52.080064+0000","last_became_peered":"2024-04-15T15:14:52.080064+0000","last_unstale":"2024-04-15T15:14:57.136403+0000","last_undegraded":"2024-04-15T15:14:57.136403+0000","last_fullsized":"2024-04-15T15:14:57.136403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:10:19.696254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136107+0000","last_change":"2024-04-15T15:14:51.075197+0000","last_active":"2024-04-15T15:14:57.136107+0000","last_peered":"2024-04-15T15:14:57.136107+0000","last_clean":"2024-04-15T15:14:57.136107+0000","last_became_active":"2024-04-15T15:14:51.074955+0000","last_became_peered":"2024-04-15T15:14:51.074955+0000","last_unstale":"2024-04-15T15:14:57.136107+0000","last_undegraded":"2024-04-15T15:14:57.136107+0000","last_fullsized":"2024-04-15T15:14:57.136107+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:18:41.399059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137490+0000","last_change":"2024-04-15T15:14:56.253169+0000","last_active":"2024-04-15T15:14:57.137490+0000","last_peered":"2024-04-15T15:14:57.137490+0000","last_clean":"2024-04-15T15:14:57.137490+0000","last_became_active":"2024-04-15T15:14:56.252823+0000","last_became_peered":"2024-04-15T15:14:56.252823+0000","last_unstale":"2024-04-15T15:14:57.137490+0000","last_undegraded":"2024-04-15T15:14:57.137490+0000","last_fullsized":"2024-04-15T15:14:57.137490+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:03:07.876012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136337+0000","last_change":"2024-04-15T15:14:52.075278+0000","last_active":"2024-04-15T15:14:57.136337+0000","last_peered":"2024-04-15T15:14:57.136337+0000","last_clean":"2024-04-15T15:14:57.136337+0000","last_became_active":"2024-04-15T15:14:52.075003+0000","last_became_peered":"2024-04-15T15:14:52.075003+0000","last_unstale":"2024-04-15T15:14:57.136337+0000","last_undegraded":"2024-04-15T15:14:57.136337+0000","last_fullsized":"2024-04-15T15:14:57.136337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:47:55.994716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135597+0000","last_change":"2024-04-15T15:14:51.076530+0000","last_active":"2024-04-15T15:14:57.135597+0000","last_peered":"2024-04-15T15:14:57.135597+0000","last_clean":"2024-04-15T15:14:57.135597+0000","last_became_active":"2024-04-15T15:14:51.076247+0000","last_became_peered":"2024-04-15T15:14:51.076247+0000","last_unstale":"2024-04-15T15:14:57.135597+0000","last_undegraded":"2024-04-15T15:14:57.135597+0000","last_fullsized":"2024-04-15T15:14:57.135597+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:47:54.990677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136888+0000","last_change":"2024-04-15T15:14:56.098477+0000","last_active":"2024-04-15T15:14:57.136888+0000","last_peered":"2024-04-15T15:14:57.136888+0000","last_clean":"2024-04-15T15:14:57.136888+0000","last_became_active":"2024-04-15T15:14:56.097839+0000","last_became_peered":"2024-04-15T15:14:56.097839+0000","last_unstale":"2024-04-15T15:14:57.136888+0000","last_undegraded":"2024-04-15T15:14:57.136888+0000","last_fullsized":"2024-04-15T15:14:57.136888+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:12.500763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138519+0000","last_change":"2024-04-15T15:14:52.079014+0000","last_active":"2024-04-15T15:14:57.138519+0000","last_peered":"2024-04-15T15:14:57.138519+0000","last_clean":"2024-04-15T15:14:57.138519+0000","last_became_active":"2024-04-15T15:14:52.078708+0000","last_became_peered":"2024-04-15T15:14:52.078708+0000","last_unstale":"2024-04-15T15:14:57.138519+0000","last_undegraded":"2024-04-15T15:14:57.138519+0000","last_fullsized":"2024-04-15T15:14:57.138519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:37:31.656434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109619+0000","last_change":"2024-04-15T15:14:51.075067+0000","last_active":"2024-04-15T15:14:57.109619+0000","last_peered":"2024-04-15T15:14:57.109619+0000","last_clean":"2024-04-15T15:14:57.109619+0000","last_became_active":"2024-04-15T15:14:51.074780+0000","last_became_peered":"2024-04-15T15:14:51.074780+0000","last_unstale":"2024-04-15T15:14:57.109619+0000","last_undegraded":"2024-04-15T15:14:57.109619+0000","last_fullsized":"2024-04-15T15:14:57.109619+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:47:54.990677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136173+0000","last_change":"2024-04-15T15:14:56.252842+0000","last_active":"2024-04-15T15:14:57.136173+0000","last_peered":"2024-04-15T15:14:57.136173+0000","last_clean":"2024-04-15T15:14:57.136173+0000","last_became_active":"2024-04-15T15:14:56.252432+0000","last_became_peered":"2024-04-15T15:14:56.252432+0000","last_unstale":"2024-04-15T15:14:57.136173+0000","last_undegraded":"2024-04-15T15:14:57.136173+0000","last_fullsized":"2024-04-15T15:14:57.136173+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:51:26.840184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138191+0000","last_change":"2024-04-15T15:14:52.082404+0000","last_active":"2024-04-15T15:14:57.138191+0000","last_peered":"2024-04-15T15:14:57.138191+0000","last_clean":"2024-04-15T15:14:57.138191+0000","last_became_active":"2024-04-15T15:14:52.082136+0000","last_became_peered":"2024-04-15T15:14:52.082136+0000","last_unstale":"2024-04-15T15:14:57.138191+0000","last_undegraded":"2024-04-15T15:14:57.138191+0000","last_fullsized":"2024-04-15T15:14:57.138191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:38:35.185619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137543+0000","last_change":"2024-04-15T15:14:51.073915+0000","last_active":"2024-04-15T15:14:57.137543+0000","last_peered":"2024-04-15T15:14:57.137543+0000","last_clean":"2024-04-15T15:14:57.137543+0000","last_became_active":"2024-04-15T15:14:51.073639+0000","last_became_peered":"2024-04-15T15:14:51.073639+0000","last_unstale":"2024-04-15T15:14:57.137543+0000","last_undegraded":"2024-04-15T15:14:57.137543+0000","last_fullsized":"2024-04-15T15:14:57.137543+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:41:20.990702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135980+0000","last_change":"2024-04-15T15:14:56.251221+0000","last_active":"2024-04-15T15:14:57.135980+0000","last_peered":"2024-04-15T15:14:57.135980+0000","last_clean":"2024-04-15T15:14:57.135980+0000","last_became_active":"2024-04-15T15:14:56.250888+0000","last_became_peered":"2024-04-15T15:14:56.250888+0000","last_unstale":"2024-04-15T15:14:57.135980+0000","last_undegraded":"2024-04-15T15:14:57.135980+0000","last_fullsized":"2024-04-15T15:14:57.135980+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:42:46.671056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137233+0000","last_change":"2024-04-15T15:14:52.077347+0000","last_active":"2024-04-15T15:14:57.137233+0000","last_peered":"2024-04-15T15:14:57.137233+0000","last_clean":"2024-04-15T15:14:57.137233+0000","last_became_active":"2024-04-15T15:14:52.077053+0000","last_became_peered":"2024-04-15T15:14:52.077053+0000","last_unstale":"2024-04-15T15:14:57.137233+0000","last_undegraded":"2024-04-15T15:14:57.137233+0000","last_fullsized":"2024-04-15T15:14:57.137233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:13:01.642838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136936+0000","last_change":"2024-04-15T15:14:51.076396+0000","last_active":"2024-04-15T15:14:57.136936+0000","last_peered":"2024-04-15T15:14:57.136936+0000","last_clean":"2024-04-15T15:14:57.136936+0000","last_became_active":"2024-04-15T15:14:51.076124+0000","last_became_peered":"2024-04-15T15:14:51.076124+0000","last_unstale":"2024-04-15T15:14:57.136936+0000","last_undegraded":"2024-04-15T15:14:57.136936+0000","last_fullsized":"2024-04-15T15:14:57.136936+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:31:57.289767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137379+0000","last_change":"2024-04-15T15:14:56.251375+0000","last_active":"2024-04-15T15:14:57.137379+0000","last_peered":"2024-04-15T15:14:57.137379+0000","last_clean":"2024-04-15T15:14:57.137379+0000","last_became_active":"2024-04-15T15:14:56.250973+0000","last_became_peered":"2024-04-15T15:14:56.250973+0000","last_unstale":"2024-04-15T15:14:57.137379+0000","last_undegraded":"2024-04-15T15:14:57.137379+0000","last_fullsized":"2024-04-15T15:14:57.137379+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:58:01.690949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.110314+0000","last_change":"2024-04-15T15:14:52.082863+0000","last_active":"2024-04-15T15:14:57.110314+0000","last_peered":"2024-04-15T15:14:57.110314+0000","last_clean":"2024-04-15T15:14:57.110314+0000","last_became_active":"2024-04-15T15:14:52.082370+0000","last_became_peered":"2024-04-15T15:14:52.082370+0000","last_unstale":"2024-04-15T15:14:57.110314+0000","last_undegraded":"2024-04-15T15:14:57.110314+0000","last_fullsized":"2024-04-15T15:14:57.110314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:08.475460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137335+0000","last_change":"2024-04-15T15:14:51.073016+0000","last_active":"2024-04-15T15:14:57.137335+0000","last_peered":"2024-04-15T15:14:57.137335+0000","last_clean":"2024-04-15T15:14:57.137335+0000","last_became_active":"2024-04-15T15:14:51.072736+0000","last_became_peered":"2024-04-15T15:14:51.072736+0000","last_unstale":"2024-04-15T15:14:57.137335+0000","last_undegraded":"2024-04-15T15:14:57.137335+0000","last_fullsized":"2024-04-15T15:14:57.137335+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:44:33.685298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136866+0000","last_change":"2024-04-15T15:14:56.105315+0000","last_active":"2024-04-15T15:14:57.136866+0000","last_peered":"2024-04-15T15:14:57.136866+0000","last_clean":"2024-04-15T15:14:57.136866+0000","last_became_active":"2024-04-15T15:14:56.104923+0000","last_became_peered":"2024-04-15T15:14:56.104923+0000","last_unstale":"2024-04-15T15:14:57.136866+0000","last_undegraded":"2024-04-15T15:14:57.136866+0000","last_fullsized":"2024-04-15T15:14:57.136866+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:01:28.704663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136866+0000","last_change":"2024-04-15T15:14:52.078353+0000","last_active":"2024-04-15T15:14:57.136866+0000","last_peered":"2024-04-15T15:14:57.136866+0000","last_clean":"2024-04-15T15:14:57.136866+0000","last_became_active":"2024-04-15T15:14:52.076556+0000","last_became_peered":"2024-04-15T15:14:52.076556+0000","last_unstale":"2024-04-15T15:14:57.136866+0000","last_undegraded":"2024-04-15T15:14:57.136866+0000","last_fullsized":"2024-04-15T15:14:57.136866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:19:38.992004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109062+0000","last_change":"2024-04-15T15:14:51.072875+0000","last_active":"2024-04-15T15:14:57.109062+0000","last_peered":"2024-04-15T15:14:57.109062+0000","last_clean":"2024-04-15T15:14:57.109062+0000","last_became_active":"2024-04-15T15:14:51.072114+0000","last_became_peered":"2024-04-15T15:14:51.072114+0000","last_unstale":"2024-04-15T15:14:57.109062+0000","last_undegraded":"2024-04-15T15:14:57.109062+0000","last_fullsized":"2024-04-15T15:14:57.109062+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:18:41.399059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137344+0000","last_change":"2024-04-15T15:14:56.098426+0000","last_active":"2024-04-15T15:14:57.137344+0000","last_peered":"2024-04-15T15:14:57.137344+0000","last_clean":"2024-04-15T15:14:57.137344+0000","last_became_active":"2024-04-15T15:14:56.098066+0000","last_became_peered":"2024-04-15T15:14:56.098066+0000","last_unstale":"2024-04-15T15:14:57.137344+0000","last_undegraded":"2024-04-15T15:14:57.137344+0000","last_fullsized":"2024-04-15T15:14:57.137344+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:12.500763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136427+0000","last_change":"2024-04-15T15:14:52.077474+0000","last_active":"2024-04-15T15:14:57.136427+0000","last_peered":"2024-04-15T15:14:57.136427+0000","last_clean":"2024-04-15T15:14:57.136427+0000","last_became_active":"2024-04-15T15:14:52.077206+0000","last_became_peered":"2024-04-15T15:14:52.077206+0000","last_unstale":"2024-04-15T15:14:57.136427+0000","last_undegraded":"2024-04-15T15:14:57.136427+0000","last_fullsized":"2024-04-15T15:14:57.136427+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:34:51.775979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.107727+0000","last_change":"2024-04-15T15:14:51.067544+0000","last_active":"2024-04-15T15:14:57.107727+0000","last_peered":"2024-04-15T15:14:57.107727+0000","last_clean":"2024-04-15T15:14:57.107727+0000","last_became_active":"2024-04-15T15:14:51.067233+0000","last_became_peered":"2024-04-15T15:14:51.067233+0000","last_unstale":"2024-04-15T15:14:57.107727+0000","last_undegraded":"2024-04-15T15:14:57.107727+0000","last_fullsized":"2024-04-15T15:14:57.107727+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:44:33.685298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137463+0000","last_change":"2024-04-15T15:14:56.103911+0000","last_active":"2024-04-15T15:14:57.137463+0000","last_peered":"2024-04-15T15:14:57.137463+0000","last_clean":"2024-04-15T15:14:57.137463+0000","last_became_active":"2024-04-15T15:14:56.103035+0000","last_became_peered":"2024-04-15T15:14:56.103035+0000","last_unstale":"2024-04-15T15:14:57.137463+0000","last_undegraded":"2024-04-15T15:14:57.137463+0000","last_fullsized":"2024-04-15T15:14:57.137463+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:21:52.495367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136227+0000","last_change":"2024-04-15T15:14:52.078870+0000","last_active":"2024-04-15T15:14:57.136227+0000","last_peered":"2024-04-15T15:14:57.136227+0000","last_clean":"2024-04-15T15:14:57.136227+0000","last_became_active":"2024-04-15T15:14:52.078573+0000","last_became_peered":"2024-04-15T15:14:52.078573+0000","last_unstale":"2024-04-15T15:14:57.136227+0000","last_undegraded":"2024-04-15T15:14:57.136227+0000","last_fullsized":"2024-04-15T15:14:57.136227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:51:36.740221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135636+0000","last_change":"2024-04-15T15:14:51.077104+0000","last_active":"2024-04-15T15:14:57.135636+0000","last_peered":"2024-04-15T15:14:57.135636+0000","last_clean":"2024-04-15T15:14:57.135636+0000","last_became_active":"2024-04-15T15:14:51.076857+0000","last_became_peered":"2024-04-15T15:14:51.076857+0000","last_unstale":"2024-04-15T15:14:57.135636+0000","last_undegraded":"2024-04-15T15:14:57.135636+0000","last_fullsized":"2024-04-15T15:14:57.135636+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:18:41.399059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138372+0000","last_change":"2024-04-15T15:14:56.251372+0000","last_active":"2024-04-15T15:14:57.138372+0000","last_peered":"2024-04-15T15:14:57.138372+0000","last_clean":"2024-04-15T15:14:57.138372+0000","last_became_active":"2024-04-15T15:14:56.250961+0000","last_became_peered":"2024-04-15T15:14:56.250961+0000","last_unstale":"2024-04-15T15:14:57.138372+0000","last_undegraded":"2024-04-15T15:14:57.138372+0000","last_fullsized":"2024-04-15T15:14:57.138372+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:13:03.031872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136619+0000","last_change":"2024-04-15T15:14:52.089029+0000","last_active":"2024-04-15T15:14:57.136619+0000","last_peered":"2024-04-15T15:14:57.136619+0000","last_clean":"2024-04-15T15:14:57.136619+0000","last_became_active":"2024-04-15T15:14:52.088646+0000","last_became_peered":"2024-04-15T15:14:52.088646+0000","last_unstale":"2024-04-15T15:14:57.136619+0000","last_undegraded":"2024-04-15T15:14:57.136619+0000","last_fullsized":"2024-04-15T15:14:57.136619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:37:31.656434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108973+0000","last_change":"2024-04-15T15:14:51.067071+0000","last_active":"2024-04-15T15:14:57.108973+0000","last_peered":"2024-04-15T15:14:57.108973+0000","last_clean":"2024-04-15T15:14:57.108973+0000","last_became_active":"2024-04-15T15:14:51.066382+0000","last_became_peered":"2024-04-15T15:14:51.066382+0000","last_unstale":"2024-04-15T15:14:57.108973+0000","last_undegraded":"2024-04-15T15:14:57.108973+0000","last_fullsized":"2024-04-15T15:14:57.108973+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:50:01.422527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135699+0000","last_change":"2024-04-15T15:14:40.494279+0000","last_active":"2024-04-15T15:14:57.135699+0000","last_peered":"2024-04-15T15:14:57.135699+0000","last_clean":"2024-04-15T15:14:57.135699+0000","last_became_active":"2024-04-15T15:14:40.493404+0000","last_became_peered":"2024-04-15T15:14:40.493404+0000","last_unstale":"2024-04-15T15:14:57.135699+0000","last_undegraded":"2024-04-15T15:14:57.135699+0000","last_fullsized":"2024-04-15T15:14:57.135699+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:39.418508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:39.418508+0000","last_clean_scrub_stamp":"2024-04-15T15:14:39.418508+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:01:51.789246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109541+0000","last_change":"2024-04-15T15:14:56.253853+0000","last_active":"2024-04-15T15:14:57.109541+0000","last_peered":"2024-04-15T15:14:57.109541+0000","last_clean":"2024-04-15T15:14:57.109541+0000","last_became_active":"2024-04-15T15:14:56.253323+0000","last_became_peered":"2024-04-15T15:14:56.253323+0000","last_unstale":"2024-04-15T15:14:57.109541+0000","last_undegraded":"2024-04-15T15:14:57.109541+0000","last_fullsized":"2024-04-15T15:14:57.109541+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:03:07.876012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136598+0000","last_change":"2024-04-15T15:14:52.093103+0000","last_active":"2024-04-15T15:14:57.136598+0000","last_peered":"2024-04-15T15:14:57.136598+0000","last_clean":"2024-04-15T15:14:57.136598+0000","last_became_active":"2024-04-15T15:14:52.092786+0000","last_became_peered":"2024-04-15T15:14:52.092786+0000","last_unstale":"2024-04-15T15:14:57.136598+0000","last_undegraded":"2024-04-15T15:14:57.136598+0000","last_fullsized":"2024-04-15T15:14:57.136598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:57:57.665646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108044+0000","last_change":"2024-04-15T15:14:51.067545+0000","last_active":"2024-04-15T15:14:57.108044+0000","last_peered":"2024-04-15T15:14:57.108044+0000","last_clean":"2024-04-15T15:14:57.108044+0000","last_became_active":"2024-04-15T15:14:51.067229+0000","last_became_peered":"2024-04-15T15:14:51.067229+0000","last_unstale":"2024-04-15T15:14:57.108044+0000","last_undegraded":"2024-04-15T15:14:57.108044+0000","last_fullsized":"2024-04-15T15:14:57.108044+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:41:20.990702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108641+0000","last_change":"2024-04-15T15:14:56.251780+0000","last_active":"2024-04-15T15:14:57.108641+0000","last_peered":"2024-04-15T15:14:57.108641+0000","last_clean":"2024-04-15T15:14:57.108641+0000","last_became_active":"2024-04-15T15:14:56.251079+0000","last_became_peered":"2024-04-15T15:14:56.251079+0000","last_unstale":"2024-04-15T15:14:57.108641+0000","last_undegraded":"2024-04-15T15:14:57.108641+0000","last_fullsized":"2024-04-15T15:14:57.108641+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:12.500763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109714+0000","last_change":"2024-04-15T15:14:52.074261+0000","last_active":"2024-04-15T15:14:57.109714+0000","last_peered":"2024-04-15T15:14:57.109714+0000","last_clean":"2024-04-15T15:14:57.109714+0000","last_became_active":"2024-04-15T15:14:52.073979+0000","last_became_peered":"2024-04-15T15:14:52.073979+0000","last_unstale":"2024-04-15T15:14:57.109714+0000","last_undegraded":"2024-04-15T15:14:57.109714+0000","last_fullsized":"2024-04-15T15:14:57.109714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:03:03.850709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137650+0000","last_change":"2024-04-15T15:14:51.074612+0000","last_active":"2024-04-15T15:14:57.137650+0000","last_peered":"2024-04-15T15:14:57.137650+0000","last_clean":"2024-04-15T15:14:57.137650+0000","last_became_active":"2024-04-15T15:14:51.074176+0000","last_became_peered":"2024-04-15T15:14:51.074176+0000","last_unstale":"2024-04-15T15:14:57.137650+0000","last_undegraded":"2024-04-15T15:14:57.137650+0000","last_fullsized":"2024-04-15T15:14:57.137650+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:50:01.422527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137604+0000","last_change":"2024-04-15T15:14:56.103129+0000","last_active":"2024-04-15T15:14:57.137604+0000","last_peered":"2024-04-15T15:14:57.137604+0000","last_clean":"2024-04-15T15:14:57.137604+0000","last_became_active":"2024-04-15T15:14:56.102757+0000","last_became_peered":"2024-04-15T15:14:56.102757+0000","last_unstale":"2024-04-15T15:14:57.137604+0000","last_undegraded":"2024-04-15T15:14:57.137604+0000","last_fullsized":"2024-04-15T15:14:57.137604+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:12.500763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136942+0000","last_change":"2024-04-15T15:14:52.087082+0000","last_active":"2024-04-15T15:14:57.136942+0000","last_peered":"2024-04-15T15:14:57.136942+0000","last_clean":"2024-04-15T15:14:57.136942+0000","last_became_active":"2024-04-15T15:14:52.086517+0000","last_became_peered":"2024-04-15T15:14:52.086517+0000","last_unstale":"2024-04-15T15:14:57.136942+0000","last_undegraded":"2024-04-15T15:14:57.136942+0000","last_fullsized":"2024-04-15T15:14:57.136942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:08:15.196752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.107969+0000","last_change":"2024-04-15T15:14:51.067941+0000","last_active":"2024-04-15T15:14:57.107969+0000","last_peered":"2024-04-15T15:14:57.107969+0000","last_clean":"2024-04-15T15:14:57.107969+0000","last_became_active":"2024-04-15T15:14:51.067669+0000","last_became_peered":"2024-04-15T15:14:51.067669+0000","last_unstale":"2024-04-15T15:14:57.107969+0000","last_undegraded":"2024-04-15T15:14:57.107969+0000","last_fullsized":"2024-04-15T15:14:57.107969+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:50:01.422527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108366+0000","last_change":"2024-04-15T15:14:56.252233+0000","last_active":"2024-04-15T15:14:57.108366+0000","last_peered":"2024-04-15T15:14:57.108366+0000","last_clean":"2024-04-15T15:14:57.108366+0000","last_became_active":"2024-04-15T15:14:56.251084+0000","last_became_peered":"2024-04-15T15:14:56.251084+0000","last_unstale":"2024-04-15T15:14:57.108366+0000","last_undegraded":"2024-04-15T15:14:57.108366+0000","last_fullsized":"2024-04-15T15:14:57.108366+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:42:46.671056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136568+0000","last_change":"2024-04-15T15:14:52.085987+0000","last_active":"2024-04-15T15:14:57.136568+0000","last_peered":"2024-04-15T15:14:57.136568+0000","last_clean":"2024-04-15T15:14:57.136568+0000","last_became_active":"2024-04-15T15:14:52.085579+0000","last_became_peered":"2024-04-15T15:14:52.085579+0000","last_unstale":"2024-04-15T15:14:57.136568+0000","last_undegraded":"2024-04-15T15:14:57.136568+0000","last_fullsized":"2024-04-15T15:14:57.136568+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:08.475460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136963+0000","last_change":"2024-04-15T15:14:51.066803+0000","last_active":"2024-04-15T15:14:57.136963+0000","last_peered":"2024-04-15T15:14:57.136963+0000","last_clean":"2024-04-15T15:14:57.136963+0000","last_became_active":"2024-04-15T15:14:51.066103+0000","last_became_peered":"2024-04-15T15:14:51.066103+0000","last_unstale":"2024-04-15T15:14:57.136963+0000","last_undegraded":"2024-04-15T15:14:57.136963+0000","last_fullsized":"2024-04-15T15:14:57.136963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:35:13.836817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135900+0000","last_change":"2024-04-15T15:14:56.098601+0000","last_active":"2024-04-15T15:14:57.135900+0000","last_peered":"2024-04-15T15:14:57.135900+0000","last_clean":"2024-04-15T15:14:57.135900+0000","last_became_active":"2024-04-15T15:14:56.098191+0000","last_became_peered":"2024-04-15T15:14:56.098191+0000","last_unstale":"2024-04-15T15:14:57.135900+0000","last_undegraded":"2024-04-15T15:14:57.135900+0000","last_fullsized":"2024-04-15T15:14:57.135900+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:12:11.797852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135812+0000","last_change":"2024-04-15T15:14:52.086933+0000","last_active":"2024-04-15T15:14:57.135812+0000","last_peered":"2024-04-15T15:14:57.135812+0000","last_clean":"2024-04-15T15:14:57.135812+0000","last_became_active":"2024-04-15T15:14:52.086546+0000","last_became_peered":"2024-04-15T15:14:52.086546+0000","last_unstale":"2024-04-15T15:14:57.135812+0000","last_undegraded":"2024-04-15T15:14:57.135812+0000","last_fullsized":"2024-04-15T15:14:57.135812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:10:19.696254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.107592+0000","last_change":"2024-04-15T15:14:51.071716+0000","last_active":"2024-04-15T15:14:57.107592+0000","last_peered":"2024-04-15T15:14:57.107592+0000","last_clean":"2024-04-15T15:14:57.107592+0000","last_became_active":"2024-04-15T15:14:51.071084+0000","last_became_peered":"2024-04-15T15:14:51.071084+0000","last_unstale":"2024-04-15T15:14:57.107592+0000","last_undegraded":"2024-04-15T15:14:57.107592+0000","last_fullsized":"2024-04-15T15:14:57.107592+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:01:16.772802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138211+0000","last_change":"2024-04-15T15:14:56.102060+0000","last_active":"2024-04-15T15:14:57.138211+0000","last_peered":"2024-04-15T15:14:57.138211+0000","last_clean":"2024-04-15T15:14:57.138211+0000","last_became_active":"2024-04-15T15:14:56.101644+0000","last_became_peered":"2024-04-15T15:14:56.101644+0000","last_unstale":"2024-04-15T15:14:57.138211+0000","last_undegraded":"2024-04-15T15:14:57.138211+0000","last_fullsized":"2024-04-15T15:14:57.138211+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:03:07.876012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109139+0000","last_change":"2024-04-15T15:14:52.088688+0000","last_active":"2024-04-15T15:14:57.109139+0000","last_peered":"2024-04-15T15:14:57.109139+0000","last_clean":"2024-04-15T15:14:57.109139+0000","last_became_active":"2024-04-15T15:14:52.088433+0000","last_became_peered":"2024-04-15T15:14:52.088433+0000","last_unstale":"2024-04-15T15:14:57.109139+0000","last_undegraded":"2024-04-15T15:14:57.109139+0000","last_fullsized":"2024-04-15T15:14:57.109139+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:24:30.714153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.110238+0000","last_change":"2024-04-15T15:14:51.073931+0000","last_active":"2024-04-15T15:14:57.110238+0000","last_peered":"2024-04-15T15:14:57.110238+0000","last_clean":"2024-04-15T15:14:57.110238+0000","last_became_active":"2024-04-15T15:14:51.073571+0000","last_became_peered":"2024-04-15T15:14:51.073571+0000","last_unstale":"2024-04-15T15:14:57.110238+0000","last_undegraded":"2024-04-15T15:14:57.110238+0000","last_fullsized":"2024-04-15T15:14:57.110238+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:44:33.685298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138190+0000","last_change":"2024-04-15T15:14:56.251177+0000","last_active":"2024-04-15T15:14:57.138190+0000","last_peered":"2024-04-15T15:14:57.138190+0000","last_clean":"2024-04-15T15:14:57.138190+0000","last_became_active":"2024-04-15T15:14:56.250803+0000","last_became_peered":"2024-04-15T15:14:56.250803+0000","last_unstale":"2024-04-15T15:14:57.138190+0000","last_undegraded":"2024-04-15T15:14:57.138190+0000","last_fullsized":"2024-04-15T15:14:57.138190+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:39:36.394347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136175+0000","last_change":"2024-04-15T15:14:52.086912+0000","last_active":"2024-04-15T15:14:57.136175+0000","last_peered":"2024-04-15T15:14:57.136175+0000","last_clean":"2024-04-15T15:14:57.136175+0000","last_became_active":"2024-04-15T15:14:52.085621+0000","last_became_peered":"2024-04-15T15:14:52.085621+0000","last_unstale":"2024-04-15T15:14:57.136175+0000","last_undegraded":"2024-04-15T15:14:57.136175+0000","last_fullsized":"2024-04-15T15:14:57.136175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:38:35.185619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108968+0000","last_change":"2024-04-15T15:14:51.068349+0000","last_active":"2024-04-15T15:14:57.108968+0000","last_peered":"2024-04-15T15:14:57.108968+0000","last_clean":"2024-04-15T15:14:57.108968+0000","last_became_active":"2024-04-15T15:14:51.068078+0000","last_became_peered":"2024-04-15T15:14:51.068078+0000","last_unstale":"2024-04-15T15:14:57.108968+0000","last_undegraded":"2024-04-15T15:14:57.108968+0000","last_fullsized":"2024-04-15T15:14:57.108968+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:47:54.990677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136871+0000","last_change":"2024-04-15T15:14:56.251089+0000","last_active":"2024-04-15T15:14:57.136871+0000","last_peered":"2024-04-15T15:14:57.136871+0000","last_clean":"2024-04-15T15:14:57.136871+0000","last_became_active":"2024-04-15T15:14:56.250735+0000","last_became_peered":"2024-04-15T15:14:56.250735+0000","last_unstale":"2024-04-15T15:14:57.136871+0000","last_undegraded":"2024-04-15T15:14:57.136871+0000","last_fullsized":"2024-04-15T15:14:57.136871+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:39:38.108475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138515+0000","last_change":"2024-04-15T15:14:52.076410+0000","last_active":"2024-04-15T15:14:57.138515+0000","last_peered":"2024-04-15T15:14:57.138515+0000","last_clean":"2024-04-15T15:14:57.138515+0000","last_became_active":"2024-04-15T15:14:52.076115+0000","last_became_peered":"2024-04-15T15:14:52.076115+0000","last_unstale":"2024-04-15T15:14:57.138515+0000","last_undegraded":"2024-04-15T15:14:57.138515+0000","last_fullsized":"2024-04-15T15:14:57.138515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:48:45.790927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137841+0000","last_change":"2024-04-15T15:14:51.072889+0000","last_active":"2024-04-15T15:14:57.137841+0000","last_peered":"2024-04-15T15:14:57.137841+0000","last_clean":"2024-04-15T15:14:57.137841+0000","last_became_active":"2024-04-15T15:14:51.072611+0000","last_became_peered":"2024-04-15T15:14:51.072611+0000","last_unstale":"2024-04-15T15:14:57.137841+0000","last_undegraded":"2024-04-15T15:14:57.137841+0000","last_fullsized":"2024-04-15T15:14:57.137841+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:20:40.577242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.110168+0000","last_change":"2024-04-15T15:14:56.253070+0000","last_active":"2024-04-15T15:14:57.110168+0000","last_peered":"2024-04-15T15:14:57.110168+0000","last_clean":"2024-04-15T15:14:57.110168+0000","last_became_active":"2024-04-15T15:14:56.252652+0000","last_became_peered":"2024-04-15T15:14:56.252652+0000","last_unstale":"2024-04-15T15:14:57.110168+0000","last_undegraded":"2024-04-15T15:14:57.110168+0000","last_fullsized":"2024-04-15T15:14:57.110168+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:34:50.166643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136518+0000","last_change":"2024-04-15T15:14:52.092734+0000","last_active":"2024-04-15T15:14:57.136518+0000","last_peered":"2024-04-15T15:14:57.136518+0000","last_clean":"2024-04-15T15:14:57.136518+0000","last_became_active":"2024-04-15T15:14:52.092105+0000","last_became_peered":"2024-04-15T15:14:52.092105+0000","last_unstale":"2024-04-15T15:14:57.136518+0000","last_undegraded":"2024-04-15T15:14:57.136518+0000","last_fullsized":"2024-04-15T15:14:57.136518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:01:24.679360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:15:01.094899+0000","last_change":"2024-04-15T15:14:51.072503+0000","last_active":"2024-04-15T15:15:01.094899+0000","last_peered":"2024-04-15T15:15:01.094899+0000","last_clean":"2024-04-15T15:15:01.094899+0000","last_became_active":"2024-04-15T15:14:51.072212+0000","last_became_peered":"2024-04-15T15:14:51.072212+0000","last_unstale":"2024-04-15T15:15:01.094899+0000","last_undegraded":"2024-04-15T15:15:01.094899+0000","last_fullsized":"2024-04-15T15:15:01.094899+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:18:41.399059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3701,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137069+0000","last_change":"2024-04-15T15:14:56.105406+0000","last_active":"2024-04-15T15:14:57.137069+0000","last_peered":"2024-04-15T15:14:57.137069+0000","last_clean":"2024-04-15T15:14:57.137069+0000","last_became_active":"2024-04-15T15:14:56.104951+0000","last_became_peered":"2024-04-15T15:14:56.104951+0000","last_unstale":"2024-04-15T15:14:57.137069+0000","last_undegraded":"2024-04-15T15:14:57.137069+0000","last_fullsized":"2024-04-15T15:14:57.137069+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:58:01.690949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136751+0000","last_change":"2024-04-15T15:14:52.083396+0000","last_active":"2024-04-15T15:14:57.136751+0000","last_peered":"2024-04-15T15:14:57.136751+0000","last_clean":"2024-04-15T15:14:57.136751+0000","last_became_active":"2024-04-15T15:14:52.082661+0000","last_became_peered":"2024-04-15T15:14:52.082661+0000","last_unstale":"2024-04-15T15:14:57.136751+0000","last_undegraded":"2024-04-15T15:14:57.136751+0000","last_fullsized":"2024-04-15T15:14:57.136751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:07:49.420517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137624+0000","last_change":"2024-04-15T15:14:51.074372+0000","last_active":"2024-04-15T15:14:57.137624+0000","last_peered":"2024-04-15T15:14:57.137624+0000","last_clean":"2024-04-15T15:14:57.137624+0000","last_became_active":"2024-04-15T15:14:51.074030+0000","last_became_peered":"2024-04-15T15:14:51.074030+0000","last_unstale":"2024-04-15T15:14:57.137624+0000","last_undegraded":"2024-04-15T15:14:57.137624+0000","last_fullsized":"2024-04-15T15:14:57.137624+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:47:54.990677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136760+0000","last_change":"2024-04-15T15:14:56.099051+0000","last_active":"2024-04-15T15:14:57.136760+0000","last_peered":"2024-04-15T15:14:57.136760+0000","last_clean":"2024-04-15T15:14:57.136760+0000","last_became_active":"2024-04-15T15:14:56.098034+0000","last_became_peered":"2024-04-15T15:14:56.098034+0000","last_unstale":"2024-04-15T15:14:57.136760+0000","last_undegraded":"2024-04-15T15:14:57.136760+0000","last_fullsized":"2024-04-15T15:14:57.136760+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:58:01.690949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136894+0000","last_change":"2024-04-15T15:14:52.084508+0000","last_active":"2024-04-15T15:14:57.136894+0000","last_peered":"2024-04-15T15:14:57.136894+0000","last_clean":"2024-04-15T15:14:57.136894+0000","last_became_active":"2024-04-15T15:14:52.083550+0000","last_became_peered":"2024-04-15T15:14:52.083550+0000","last_unstale":"2024-04-15T15:14:57.136894+0000","last_undegraded":"2024-04-15T15:14:57.136894+0000","last_fullsized":"2024-04-15T15:14:57.136894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:12:40.420221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108919+0000","last_change":"2024-04-15T15:14:51.071093+0000","last_active":"2024-04-15T15:14:57.108919+0000","last_peered":"2024-04-15T15:14:57.108919+0000","last_clean":"2024-04-15T15:14:57.108919+0000","last_became_active":"2024-04-15T15:14:51.070810+0000","last_became_peered":"2024-04-15T15:14:51.070810+0000","last_unstale":"2024-04-15T15:14:57.108919+0000","last_undegraded":"2024-04-15T15:14:57.108919+0000","last_fullsized":"2024-04-15T15:14:57.108919+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:41:20.990702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136272+0000","last_change":"2024-04-15T15:14:56.092919+0000","last_active":"2024-04-15T15:14:57.136272+0000","last_peered":"2024-04-15T15:14:57.136272+0000","last_clean":"2024-04-15T15:14:57.136272+0000","last_became_active":"2024-04-15T15:14:56.092459+0000","last_became_peered":"2024-04-15T15:14:56.092459+0000","last_unstale":"2024-04-15T15:14:57.136272+0000","last_undegraded":"2024-04-15T15:14:57.136272+0000","last_fullsized":"2024-04-15T15:14:57.136272+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:46:38.024003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108815+0000","last_change":"2024-04-15T15:14:52.079880+0000","last_active":"2024-04-15T15:14:57.108815+0000","last_peered":"2024-04-15T15:14:57.108815+0000","last_clean":"2024-04-15T15:14:57.108815+0000","last_became_active":"2024-04-15T15:14:52.079619+0000","last_became_peered":"2024-04-15T15:14:52.079619+0000","last_unstale":"2024-04-15T15:14:57.108815+0000","last_undegraded":"2024-04-15T15:14:57.108815+0000","last_fullsized":"2024-04-15T15:14:57.108815+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:21:48.470063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109438+0000","last_change":"2024-04-15T15:14:51.075415+0000","last_active":"2024-04-15T15:14:57.109438+0000","last_peered":"2024-04-15T15:14:57.109438+0000","last_clean":"2024-04-15T15:14:57.109438+0000","last_became_active":"2024-04-15T15:14:51.075014+0000","last_became_peered":"2024-04-15T15:14:51.075014+0000","last_unstale":"2024-04-15T15:14:57.109438+0000","last_undegraded":"2024-04-15T15:14:57.109438+0000","last_fullsized":"2024-04-15T15:14:57.109438+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:41:20.990702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135982+0000","last_change":"2024-04-15T15:14:56.253630+0000","last_active":"2024-04-15T15:14:57.135982+0000","last_peered":"2024-04-15T15:14:57.135982+0000","last_clean":"2024-04-15T15:14:57.135982+0000","last_became_active":"2024-04-15T15:14:56.253253+0000","last_became_peered":"2024-04-15T15:14:56.253253+0000","last_unstale":"2024-04-15T15:14:57.135982+0000","last_undegraded":"2024-04-15T15:14:57.135982+0000","last_fullsized":"2024-04-15T15:14:57.135982+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:34:50.166643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138051+0000","last_change":"2024-04-15T15:14:56.253163+0000","last_active":"2024-04-15T15:14:57.138051+0000","last_peered":"2024-04-15T15:14:57.138051+0000","last_clean":"2024-04-15T15:14:57.138051+0000","last_became_active":"2024-04-15T15:14:56.252769+0000","last_became_peered":"2024-04-15T15:14:56.252769+0000","last_unstale":"2024-04-15T15:14:57.138051+0000","last_undegraded":"2024-04-15T15:14:57.138051+0000","last_fullsized":"2024-04-15T15:14:57.138051+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:21:52.495367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135905+0000","last_change":"2024-04-15T15:14:52.086725+0000","last_active":"2024-04-15T15:14:57.135905+0000","last_peered":"2024-04-15T15:14:57.135905+0000","last_clean":"2024-04-15T15:14:57.135905+0000","last_became_active":"2024-04-15T15:14:52.086220+0000","last_became_peered":"2024-04-15T15:14:52.086220+0000","last_unstale":"2024-04-15T15:14:57.135905+0000","last_undegraded":"2024-04-15T15:14:57.135905+0000","last_fullsized":"2024-04-15T15:14:57.135905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:21:50.111549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108433+0000","last_change":"2024-04-15T15:14:51.069916+0000","last_active":"2024-04-15T15:14:57.108433+0000","last_peered":"2024-04-15T15:14:57.108433+0000","last_clean":"2024-04-15T15:14:57.108433+0000","last_became_active":"2024-04-15T15:14:51.069631+0000","last_became_peered":"2024-04-15T15:14:51.069631+0000","last_unstale":"2024-04-15T15:14:57.108433+0000","last_undegraded":"2024-04-15T15:14:57.108433+0000","last_fullsized":"2024-04-15T15:14:57.108433+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:44:33.685298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109306+0000","last_change":"2024-04-15T15:14:56.102205+0000","last_active":"2024-04-15T15:14:57.109306+0000","last_peered":"2024-04-15T15:14:57.109306+0000","last_clean":"2024-04-15T15:14:57.109306+0000","last_became_active":"2024-04-15T15:14:56.101827+0000","last_became_peered":"2024-04-15T15:14:56.101827+0000","last_unstale":"2024-04-15T15:14:57.109306+0000","last_undegraded":"2024-04-15T15:14:57.109306+0000","last_fullsized":"2024-04-15T15:14:57.109306+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:21:52.495367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136314+0000","last_change":"2024-04-15T15:14:52.079898+0000","last_active":"2024-04-15T15:14:57.136314+0000","last_peered":"2024-04-15T15:14:57.136314+0000","last_clean":"2024-04-15T15:14:57.136314+0000","last_became_active":"2024-04-15T15:14:52.079430+0000","last_became_peered":"2024-04-15T15:14:52.079430+0000","last_unstale":"2024-04-15T15:14:57.136314+0000","last_undegraded":"2024-04-15T15:14:57.136314+0000","last_fullsized":"2024-04-15T15:14:57.136314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:12:07.772549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136279+0000","last_change":"2024-04-15T15:14:51.069720+0000","last_active":"2024-04-15T15:14:57.136279+0000","last_peered":"2024-04-15T15:14:57.136279+0000","last_clean":"2024-04-15T15:14:57.136279+0000","last_became_active":"2024-04-15T15:14:51.069416+0000","last_became_peered":"2024-04-15T15:14:51.069416+0000","last_unstale":"2024-04-15T15:14:57.136279+0000","last_undegraded":"2024-04-15T15:14:57.136279+0000","last_fullsized":"2024-04-15T15:14:57.136279+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:20:40.577242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108868+0000","last_change":"2024-04-15T15:14:51.070913+0000","last_active":"2024-04-15T15:14:57.108868+0000","last_peered":"2024-04-15T15:14:57.108868+0000","last_clean":"2024-04-15T15:14:57.108868+0000","last_became_active":"2024-04-15T15:14:51.070640+0000","last_became_peered":"2024-04-15T15:14:51.070640+0000","last_unstale":"2024-04-15T15:14:57.108868+0000","last_undegraded":"2024-04-15T15:14:57.108868+0000","last_fullsized":"2024-04-15T15:14:57.108868+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:19:37.987965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137741+0000","last_change":"2024-04-15T15:14:52.078627+0000","last_active":"2024-04-15T15:14:57.137741+0000","last_peered":"2024-04-15T15:14:57.137741+0000","last_clean":"2024-04-15T15:14:57.137741+0000","last_became_active":"2024-04-15T15:14:52.078333+0000","last_became_peered":"2024-04-15T15:14:52.078333+0000","last_unstale":"2024-04-15T15:14:57.137741+0000","last_undegraded":"2024-04-15T15:14:57.137741+0000","last_fullsized":"2024-04-15T15:14:57.137741+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:50:41.263447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108003+0000","last_change":"2024-04-15T15:14:56.093858+0000","last_active":"2024-04-15T15:14:57.108003+0000","last_peered":"2024-04-15T15:14:57.108003+0000","last_clean":"2024-04-15T15:14:57.108003+0000","last_became_active":"2024-04-15T15:14:56.093481+0000","last_became_peered":"2024-04-15T15:14:56.093481+0000","last_unstale":"2024-04-15T15:14:57.108003+0000","last_undegraded":"2024-04-15T15:14:57.108003+0000","last_fullsized":"2024-04-15T15:14:57.108003+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:48:49.816230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137031+0000","last_change":"2024-04-15T15:14:51.073707+0000","last_active":"2024-04-15T15:14:57.137031+0000","last_peered":"2024-04-15T15:14:57.137031+0000","last_clean":"2024-04-15T15:14:57.137031+0000","last_became_active":"2024-04-15T15:14:51.073451+0000","last_became_peered":"2024-04-15T15:14:51.073451+0000","last_unstale":"2024-04-15T15:14:57.137031+0000","last_undegraded":"2024-04-15T15:14:57.137031+0000","last_fullsized":"2024-04-15T15:14:57.137031+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:01:16.772802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136421+0000","last_change":"2024-04-15T15:14:52.078644+0000","last_active":"2024-04-15T15:14:57.136421+0000","last_peered":"2024-04-15T15:14:57.136421+0000","last_clean":"2024-04-15T15:14:57.136421+0000","last_became_active":"2024-04-15T15:14:52.078177+0000","last_became_peered":"2024-04-15T15:14:52.078177+0000","last_unstale":"2024-04-15T15:14:57.136421+0000","last_undegraded":"2024-04-15T15:14:57.136421+0000","last_fullsized":"2024-04-15T15:14:57.136421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:18:42.403098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136932+0000","last_change":"2024-04-15T15:14:56.092926+0000","last_active":"2024-04-15T15:14:57.136932+0000","last_peered":"2024-04-15T15:14:57.136932+0000","last_clean":"2024-04-15T15:14:57.136932+0000","last_became_active":"2024-04-15T15:14:56.092506+0000","last_became_peered":"2024-04-15T15:14:56.092506+0000","last_unstale":"2024-04-15T15:14:57.136932+0000","last_undegraded":"2024-04-15T15:14:57.136932+0000","last_fullsized":"2024-04-15T15:14:57.136932+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:55:14.590939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108967+0000","last_change":"2024-04-15T15:14:52.082533+0000","last_active":"2024-04-15T15:14:57.108967+0000","last_peered":"2024-04-15T15:14:57.108967+0000","last_clean":"2024-04-15T15:14:57.108967+0000","last_became_active":"2024-04-15T15:14:52.082257+0000","last_became_peered":"2024-04-15T15:14:52.082257+0000","last_unstale":"2024-04-15T15:14:57.108967+0000","last_undegraded":"2024-04-15T15:14:57.108967+0000","last_fullsized":"2024-04-15T15:14:57.108967+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:34:51.775979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.107671+0000","last_change":"2024-04-15T15:14:52.084944+0000","last_active":"2024-04-15T15:14:57.107671+0000","last_peered":"2024-04-15T15:14:57.107671+0000","last_clean":"2024-04-15T15:14:57.107671+0000","last_became_active":"2024-04-15T15:14:52.084665+0000","last_became_peered":"2024-04-15T15:14:52.084665+0000","last_unstale":"2024-04-15T15:14:57.107671+0000","last_undegraded":"2024-04-15T15:14:57.107671+0000","last_fullsized":"2024-04-15T15:14:57.107671+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:58:41.444122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108873+0000","last_change":"2024-04-15T15:14:52.082741+0000","last_active":"2024-04-15T15:14:57.108873+0000","last_peered":"2024-04-15T15:14:57.108873+0000","last_clean":"2024-04-15T15:14:57.108873+0000","last_became_active":"2024-04-15T15:14:52.082494+0000","last_became_peered":"2024-04-15T15:14:52.082494+0000","last_unstale":"2024-04-15T15:14:57.108873+0000","last_undegraded":"2024-04-15T15:14:57.108873+0000","last_fullsized":"2024-04-15T15:14:57.108873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:38:35.185619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137649+0000","last_change":"2024-04-15T15:14:52.076826+0000","last_active":"2024-04-15T15:14:57.137649+0000","last_peered":"2024-04-15T15:14:57.137649+0000","last_clean":"2024-04-15T15:14:57.137649+0000","last_became_active":"2024-04-15T15:14:52.076541+0000","last_became_peered":"2024-04-15T15:14:52.076541+0000","last_unstale":"2024-04-15T15:14:57.137649+0000","last_undegraded":"2024-04-15T15:14:57.137649+0000","last_fullsized":"2024-04-15T15:14:57.137649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:13:01.642838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137024+0000","last_change":"2024-04-15T15:14:52.075468+0000","last_active":"2024-04-15T15:14:57.137024+0000","last_peered":"2024-04-15T15:14:57.137024+0000","last_clean":"2024-04-15T15:14:57.137024+0000","last_became_active":"2024-04-15T15:14:52.075173+0000","last_became_peered":"2024-04-15T15:14:52.075173+0000","last_unstale":"2024-04-15T15:14:57.137024+0000","last_undegraded":"2024-04-15T15:14:57.137024+0000","last_fullsized":"2024-04-15T15:14:57.137024+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:50:41.263447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137555+0000","last_change":"2024-04-15T15:14:52.086866+0000","last_active":"2024-04-15T15:14:57.137555+0000","last_peered":"2024-04-15T15:14:57.137555+0000","last_clean":"2024-04-15T15:14:57.137555+0000","last_became_active":"2024-04-15T15:14:52.086547+0000","last_became_peered":"2024-04-15T15:14:52.086547+0000","last_unstale":"2024-04-15T15:14:57.137555+0000","last_undegraded":"2024-04-15T15:14:57.137555+0000","last_fullsized":"2024-04-15T15:14:57.137555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:37:31.656434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.110000+0000","last_change":"2024-04-15T15:14:52.074917+0000","last_active":"2024-04-15T15:14:57.110000+0000","last_peered":"2024-04-15T15:14:57.110000+0000","last_clean":"2024-04-15T15:14:57.110000+0000","last_became_active":"2024-04-15T15:14:52.074643+0000","last_became_peered":"2024-04-15T15:14:52.074643+0000","last_unstale":"2024-04-15T15:14:57.110000+0000","last_undegraded":"2024-04-15T15:14:57.110000+0000","last_fullsized":"2024-04-15T15:14:57.110000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:12:07.772549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138133+0000","last_change":"2024-04-15T15:14:52.076263+0000","last_active":"2024-04-15T15:14:57.138133+0000","last_peered":"2024-04-15T15:14:57.138133+0000","last_clean":"2024-04-15T15:14:57.138133+0000","last_became_active":"2024-04-15T15:14:52.075961+0000","last_became_peered":"2024-04-15T15:14:52.075961+0000","last_unstale":"2024-04-15T15:14:57.138133+0000","last_undegraded":"2024-04-15T15:14:57.138133+0000","last_fullsized":"2024-04-15T15:14:57.138133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:12:40.420221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108421+0000","last_change":"2024-04-15T15:14:52.074353+0000","last_active":"2024-04-15T15:14:57.108421+0000","last_peered":"2024-04-15T15:14:57.108421+0000","last_clean":"2024-04-15T15:14:57.108421+0000","last_became_active":"2024-04-15T15:14:52.074067+0000","last_became_peered":"2024-04-15T15:14:52.074067+0000","last_unstale":"2024-04-15T15:14:57.108421+0000","last_undegraded":"2024-04-15T15:14:57.108421+0000","last_fullsized":"2024-04-15T15:14:57.108421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:26:45.460787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136083+0000","last_change":"2024-04-15T15:14:52.075541+0000","last_active":"2024-04-15T15:14:57.136083+0000","last_peered":"2024-04-15T15:14:57.136083+0000","last_clean":"2024-04-15T15:14:57.136083+0000","last_became_active":"2024-04-15T15:14:52.075227+0000","last_became_peered":"2024-04-15T15:14:52.075227+0000","last_unstale":"2024-04-15T15:14:57.136083+0000","last_undegraded":"2024-04-15T15:14:57.136083+0000","last_fullsized":"2024-04-15T15:14:57.136083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:24:30.714153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108791+0000","last_change":"2024-04-15T15:14:52.080306+0000","last_active":"2024-04-15T15:14:57.108791+0000","last_peered":"2024-04-15T15:14:57.108791+0000","last_clean":"2024-04-15T15:14:57.108791+0000","last_became_active":"2024-04-15T15:14:52.079784+0000","last_became_peered":"2024-04-15T15:14:52.079784+0000","last_unstale":"2024-04-15T15:14:57.108791+0000","last_undegraded":"2024-04-15T15:14:57.108791+0000","last_fullsized":"2024-04-15T15:14:57.108791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:10:19.696254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137426+0000","last_change":"2024-04-15T15:14:52.081224+0000","last_active":"2024-04-15T15:14:57.137426+0000","last_peered":"2024-04-15T15:14:57.137426+0000","last_clean":"2024-04-15T15:14:57.137426+0000","last_became_active":"2024-04-15T15:14:52.080976+0000","last_became_peered":"2024-04-15T15:14:52.080976+0000","last_unstale":"2024-04-15T15:14:57.137426+0000","last_undegraded":"2024-04-15T15:14:57.137426+0000","last_fullsized":"2024-04-15T15:14:57.137426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:21:50.111549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109464+0000","last_change":"2024-04-15T15:14:52.085463+0000","last_active":"2024-04-15T15:14:57.109464+0000","last_peered":"2024-04-15T15:14:57.109464+0000","last_clean":"2024-04-15T15:14:57.109464+0000","last_became_active":"2024-04-15T15:14:52.085175+0000","last_became_peered":"2024-04-15T15:14:52.085175+0000","last_unstale":"2024-04-15T15:14:57.109464+0000","last_undegraded":"2024-04-15T15:14:57.109464+0000","last_fullsized":"2024-04-15T15:14:57.109464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:50:41.263447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136838+0000","last_change":"2024-04-15T15:14:52.092434+0000","last_active":"2024-04-15T15:14:57.136838+0000","last_peered":"2024-04-15T15:14:57.136838+0000","last_clean":"2024-04-15T15:14:57.136838+0000","last_became_active":"2024-04-15T15:14:52.088973+0000","last_became_peered":"2024-04-15T15:14:52.088973+0000","last_unstale":"2024-04-15T15:14:57.136838+0000","last_undegraded":"2024-04-15T15:14:57.136838+0000","last_fullsized":"2024-04-15T15:14:57.136838+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:03:03.850709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.107875+0000","last_change":"2024-04-15T15:14:52.076802+0000","last_active":"2024-04-15T15:14:57.107875+0000","last_peered":"2024-04-15T15:14:57.107875+0000","last_clean":"2024-04-15T15:14:57.107875+0000","last_became_active":"2024-04-15T15:14:52.076523+0000","last_became_peered":"2024-04-15T15:14:52.076523+0000","last_unstale":"2024-04-15T15:14:57.107875+0000","last_undegraded":"2024-04-15T15:14:57.107875+0000","last_fullsized":"2024-04-15T15:14:57.107875+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:07:49.420517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109362+0000","last_change":"2024-04-15T15:14:52.084418+0000","last_active":"2024-04-15T15:14:57.109362+0000","last_peered":"2024-04-15T15:14:57.109362+0000","last_clean":"2024-04-15T15:14:57.109362+0000","last_became_active":"2024-04-15T15:14:52.083428+0000","last_became_peered":"2024-04-15T15:14:52.083428+0000","last_unstale":"2024-04-15T15:14:57.109362+0000","last_undegraded":"2024-04-15T15:14:57.109362+0000","last_fullsized":"2024-04-15T15:14:57.109362+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:12:59.006568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108279+0000","last_change":"2024-04-15T15:14:52.083894+0000","last_active":"2024-04-15T15:14:57.108279+0000","last_peered":"2024-04-15T15:14:57.108279+0000","last_clean":"2024-04-15T15:14:57.108279+0000","last_became_active":"2024-04-15T15:14:52.083636+0000","last_became_peered":"2024-04-15T15:14:52.083636+0000","last_unstale":"2024-04-15T15:14:57.108279+0000","last_undegraded":"2024-04-15T15:14:57.108279+0000","last_fullsized":"2024-04-15T15:14:57.108279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:12:07.772549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136007+0000","last_change":"2024-04-15T15:14:52.087934+0000","last_active":"2024-04-15T15:14:57.136007+0000","last_peered":"2024-04-15T15:14:57.136007+0000","last_clean":"2024-04-15T15:14:57.136007+0000","last_became_active":"2024-04-15T15:14:52.087040+0000","last_became_peered":"2024-04-15T15:14:52.087040+0000","last_unstale":"2024-04-15T15:14:57.136007+0000","last_undegraded":"2024-04-15T15:14:57.136007+0000","last_fullsized":"2024-04-15T15:14:57.136007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:24:30.714153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138511+0000","last_change":"2024-04-15T15:14:52.079696+0000","last_active":"2024-04-15T15:14:57.138511+0000","last_peered":"2024-04-15T15:14:57.138511+0000","last_clean":"2024-04-15T15:14:57.138511+0000","last_became_active":"2024-04-15T15:14:52.079438+0000","last_became_peered":"2024-04-15T15:14:52.079438+0000","last_unstale":"2024-04-15T15:14:57.138511+0000","last_undegraded":"2024-04-15T15:14:57.138511+0000","last_fullsized":"2024-04-15T15:14:57.138511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:27:02.854265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136589+0000","last_change":"2024-04-15T15:14:52.076841+0000","last_active":"2024-04-15T15:14:57.136589+0000","last_peered":"2024-04-15T15:14:57.136589+0000","last_clean":"2024-04-15T15:14:57.136589+0000","last_became_active":"2024-04-15T15:14:52.076567+0000","last_became_peered":"2024-04-15T15:14:52.076567+0000","last_unstale":"2024-04-15T15:14:57.136589+0000","last_undegraded":"2024-04-15T15:14:57.136589+0000","last_fullsized":"2024-04-15T15:14:57.136589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:51:22.814881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136685+0000","last_change":"2024-04-15T15:14:52.078599+0000","last_active":"2024-04-15T15:14:57.136685+0000","last_peered":"2024-04-15T15:14:57.136685+0000","last_clean":"2024-04-15T15:14:57.136685+0000","last_became_active":"2024-04-15T15:14:52.078231+0000","last_became_peered":"2024-04-15T15:14:52.078231+0000","last_unstale":"2024-04-15T15:14:57.136685+0000","last_undegraded":"2024-04-15T15:14:57.136685+0000","last_fullsized":"2024-04-15T15:14:57.136685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:58:41.444122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109712+0000","last_change":"2024-04-15T15:14:52.080747+0000","last_active":"2024-04-15T15:14:57.109712+0000","last_peered":"2024-04-15T15:14:57.109712+0000","last_clean":"2024-04-15T15:14:57.109712+0000","last_became_active":"2024-04-15T15:14:52.080475+0000","last_became_peered":"2024-04-15T15:14:52.080475+0000","last_unstale":"2024-04-15T15:14:57.109712+0000","last_undegraded":"2024-04-15T15:14:57.109712+0000","last_fullsized":"2024-04-15T15:14:57.109712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:37:31.656434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109996+0000","last_change":"2024-04-15T15:14:52.081405+0000","last_active":"2024-04-15T15:14:57.109996+0000","last_peered":"2024-04-15T15:14:57.109996+0000","last_clean":"2024-04-15T15:14:57.109996+0000","last_became_active":"2024-04-15T15:14:52.081120+0000","last_became_peered":"2024-04-15T15:14:52.081120+0000","last_unstale":"2024-04-15T15:14:57.109996+0000","last_undegraded":"2024-04-15T15:14:57.109996+0000","last_fullsized":"2024-04-15T15:14:57.109996+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:12:40.420221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136834+0000","last_change":"2024-04-15T15:14:52.092337+0000","last_active":"2024-04-15T15:14:57.136834+0000","last_peered":"2024-04-15T15:14:57.136834+0000","last_clean":"2024-04-15T15:14:57.136834+0000","last_became_active":"2024-04-15T15:14:52.091800+0000","last_became_peered":"2024-04-15T15:14:52.091800+0000","last_unstale":"2024-04-15T15:14:57.136834+0000","last_undegraded":"2024-04-15T15:14:57.136834+0000","last_fullsized":"2024-04-15T15:14:57.136834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:12:59.006568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136083+0000","last_change":"2024-04-15T15:14:52.079078+0000","last_active":"2024-04-15T15:14:57.136083+0000","last_peered":"2024-04-15T15:14:57.136083+0000","last_clean":"2024-04-15T15:14:57.136083+0000","last_became_active":"2024-04-15T15:14:52.078796+0000","last_became_peered":"2024-04-15T15:14:52.078796+0000","last_unstale":"2024-04-15T15:14:57.136083+0000","last_undegraded":"2024-04-15T15:14:57.136083+0000","last_fullsized":"2024-04-15T15:14:57.136083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:34:51.775979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135766+0000","last_change":"2024-04-15T15:14:52.091742+0000","last_active":"2024-04-15T15:14:57.135766+0000","last_peered":"2024-04-15T15:14:57.135766+0000","last_clean":"2024-04-15T15:14:57.135766+0000","last_became_active":"2024-04-15T15:14:52.087579+0000","last_became_peered":"2024-04-15T15:14:52.087579+0000","last_unstale":"2024-04-15T15:14:57.135766+0000","last_undegraded":"2024-04-15T15:14:57.135766+0000","last_fullsized":"2024-04-15T15:14:57.135766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:51:22.814881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108707+0000","last_change":"2024-04-15T15:14:52.084878+0000","last_active":"2024-04-15T15:14:57.108707+0000","last_peered":"2024-04-15T15:14:57.108707+0000","last_clean":"2024-04-15T15:14:57.108707+0000","last_became_active":"2024-04-15T15:14:52.084602+0000","last_became_peered":"2024-04-15T15:14:52.084602+0000","last_unstale":"2024-04-15T15:14:57.108707+0000","last_undegraded":"2024-04-15T15:14:57.108707+0000","last_fullsized":"2024-04-15T15:14:57.108707+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:51:22.814881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136011+0000","last_change":"2024-04-15T15:14:52.085909+0000","last_active":"2024-04-15T15:14:57.136011+0000","last_peered":"2024-04-15T15:14:57.136011+0000","last_clean":"2024-04-15T15:14:57.136011+0000","last_became_active":"2024-04-15T15:14:52.085266+0000","last_became_peered":"2024-04-15T15:14:52.085266+0000","last_unstale":"2024-04-15T15:14:57.136011+0000","last_undegraded":"2024-04-15T15:14:57.136011+0000","last_fullsized":"2024-04-15T15:14:57.136011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:26:45.460787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109544+0000","last_change":"2024-04-15T15:14:52.084086+0000","last_active":"2024-04-15T15:14:57.109544+0000","last_peered":"2024-04-15T15:14:57.109544+0000","last_clean":"2024-04-15T15:14:57.109544+0000","last_became_active":"2024-04-15T15:14:52.083193+0000","last_became_peered":"2024-04-15T15:14:52.083193+0000","last_unstale":"2024-04-15T15:14:57.109544+0000","last_undegraded":"2024-04-15T15:14:57.109544+0000","last_fullsized":"2024-04-15T15:14:57.109544+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:12:40.420221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136555+0000","last_change":"2024-04-15T15:14:52.072448+0000","last_active":"2024-04-15T15:14:57.136555+0000","last_peered":"2024-04-15T15:14:57.136555+0000","last_clean":"2024-04-15T15:14:57.136555+0000","last_became_active":"2024-04-15T15:14:52.072162+0000","last_became_peered":"2024-04-15T15:14:52.072162+0000","last_unstale":"2024-04-15T15:14:57.136555+0000","last_undegraded":"2024-04-15T15:14:57.136555+0000","last_fullsized":"2024-04-15T15:14:57.136555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:24:30.714153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137467+0000","last_change":"2024-04-15T15:14:52.087484+0000","last_active":"2024-04-15T15:14:57.137467+0000","last_peered":"2024-04-15T15:14:57.137467+0000","last_clean":"2024-04-15T15:14:57.137467+0000","last_became_active":"2024-04-15T15:14:52.087195+0000","last_became_peered":"2024-04-15T15:14:52.087195+0000","last_unstale":"2024-04-15T15:14:57.137467+0000","last_undegraded":"2024-04-15T15:14:57.137467+0000","last_fullsized":"2024-04-15T15:14:57.137467+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:50:41.263447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136721+0000","last_change":"2024-04-15T15:14:52.083998+0000","last_active":"2024-04-15T15:14:57.136721+0000","last_peered":"2024-04-15T15:14:57.136721+0000","last_clean":"2024-04-15T15:14:57.136721+0000","last_became_active":"2024-04-15T15:14:52.083702+0000","last_became_peered":"2024-04-15T15:14:52.083702+0000","last_unstale":"2024-04-15T15:14:57.136721+0000","last_undegraded":"2024-04-15T15:14:57.136721+0000","last_fullsized":"2024-04-15T15:14:57.136721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:21:48.470063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4499,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":103957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":103957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:15:09.122 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:15:09.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.259+0000 7fdc977e9640 1 Processor -- start 2024-04-15T15:15:09.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.259+0000 7fdc977e9640 1 -- start start 2024-04-15T15:15:09.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.260+0000 7fdc977e9640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdc900b5790 0x7fdc900b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:09.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.260+0000 7fdc977e9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc9015caf0 con 0x7fdc9015c710 2024-04-15T15:15:09.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.260+0000 7fdc977e9640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc9006f510 con 0x7fdc900d1790 2024-04-15T15:15:09.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.260+0000 7fdc977e9640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fdc900ccb90 con 0x7fdc900b5790 2024-04-15T15:15:09.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.260+0000 7fdc95585640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fdc9015c710 0x7fdc900d1080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45068/0 (socket says 172.21.15.72:45068) 2024-04-15T15:15:09.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.260+0000 7fdc95585640 1 -- 172.21.15.72:0/758406577 learned_addr learned my addr 172.21.15.72:0/758406577 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:09.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.260+0000 7fdc95d86640 1 --2- 172.21.15.72:0/758406577 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdc900b5790 0x7fdc900b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:09.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.260+0000 7fdc83fff640 1 -- 172.21.15.72:0/758406577 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3423333165 0 0) 0x7fdc9015caf0 con 0x7fdc9015c710 2024-04-15T15:15:09.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.260+0000 7fdc83fff640 1 -- 172.21.15.72:0/758406577 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc7c003520 con 0x7fdc9015c710 2024-04-15T15:15:09.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.260+0000 7fdc83fff640 1 -- 172.21.15.72:0/758406577 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3148953465 0 0) 0x7fdc7c003520 con 0x7fdc9015c710 2024-04-15T15:15:09.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.260+0000 7fdc83fff640 1 -- 172.21.15.72:0/758406577 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdc900b5790 msgr2=0x7fdc900b66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:09.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.260+0000 7fdc83fff640 1 --2- 172.21.15.72:0/758406577 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdc900b5790 0x7fdc900b66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.260+0000 7fdc83fff640 1 -- 172.21.15.72:0/758406577 >> v1:172.21.15.175:6789/0 conn(0x7fdc900d1790 legacy=0x7fdc900cd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:09.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.260+0000 7fdc83fff640 1 -- 172.21.15.72:0/758406577 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc900cce70 con 0x7fdc9015c710 2024-04-15T15:15:09.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.260+0000 7fdc83fff640 1 -- 172.21.15.72:0/758406577 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7fdc8c001670 con 0x7fdc9015c710 2024-04-15T15:15:09.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.260+0000 7fdc83fff640 1 -- 172.21.15.72:0/758406577 >> v1:172.21.15.72:6789/0 conn(0x7fdc9015c710 legacy=0x7fdc900d1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:09.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.260+0000 7fdc83fff640 1 --2- 172.21.15.72:0/758406577 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdc900b5790 0x7fdc7c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:09.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.260+0000 7fdc83fff640 1 --2- 172.21.15.72:0/758406577 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc7c003f20 0x7fdc7c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:09.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.260+0000 7fdc83fff640 1 --2- 172.21.15.72:0/758406577 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc7c004d40 0x7fdc7c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:09.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.260+0000 7fdc83fff640 1 -- 172.21.15.72:0/758406577 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fdc900ccb90 con 0x7fdc7c004d40 2024-04-15T15:15:09.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.260+0000 7fdc83fff640 1 -- 172.21.15.72:0/758406577 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fdc7c00ba20 con 0x7fdc7c003f20 2024-04-15T15:15:09.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.260+0000 7fdc83fff640 1 -- 172.21.15.72:0/758406577 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fdc7c00bc20 con 0x7fdc900b5790 2024-04-15T15:15:09.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.260+0000 7fdc95d86640 1 --2- 172.21.15.72:0/758406577 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc7c004d40 0x7fdc7c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:09.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.260+0000 7fdc95585640 1 --2- 172.21.15.72:0/758406577 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdc900b5790 0x7fdc7c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:09.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.260+0000 7fdc94d84640 1 --2- 172.21.15.72:0/758406577 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc7c003f20 0x7fdc7c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:09.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.261+0000 7fdc95d86640 1 -- 172.21.15.72:0/758406577 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdc900b5790 msgr2=0x7fdc7c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:09.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.261+0000 7fdc95d86640 1 --2- 172.21.15.72:0/758406577 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdc900b5790 0x7fdc7c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.265 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.261+0000 7fdc95d86640 1 -- 172.21.15.72:0/758406577 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc7c003f20 msgr2=0x7fdc7c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:09.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.261+0000 7fdc95d86640 1 --2- 172.21.15.72:0/758406577 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc7c003f20 0x7fdc7c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.261+0000 7fdc95d86640 1 -- 172.21.15.72:0/758406577 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc7c003520 con 0x7fdc7c004d40 2024-04-15T15:15:09.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.261+0000 7fdc94d84640 1 --2- 172.21.15.72:0/758406577 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc7c003f20 0x7fdc7c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:09.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.261+0000 7fdc95d86640 1 --2- 172.21.15.72:0/758406577 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc7c004d40 0x7fdc7c0051c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fdc88009fc0 tx=0x7fdc7c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd73d5b609dbbeb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:09.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.261+0000 7fdc83fff640 1 -- 172.21.15.72:0/758406577 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc88039070 con 0x7fdc7c004d40 2024-04-15T15:15:09.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.262+0000 7fdc83fff640 1 -- 172.21.15.72:0/758406577 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc8802cc20 con 0x7fdc7c004d40 2024-04-15T15:15:09.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.262+0000 7fdc83fff640 1 -- 172.21.15.72:0/758406577 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc88009c10 con 0x7fdc7c004d40 2024-04-15T15:15:09.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.262+0000 7fdc977e9640 1 -- 172.21.15.72:0/758406577 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc7c004d40 msgr2=0x7fdc7c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:09.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.262+0000 7fdc977e9640 1 --2- 172.21.15.72:0/758406577 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc7c004d40 0x7fdc7c0051c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fdc88009fc0 tx=0x7fdc7c005830 comp rx=0 tx=0).stop 2024-04-15T15:15:09.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.262+0000 7fdc95d86640 1 -- 172.21.15.72:0/758406577 reap_dead start 2024-04-15T15:15:09.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.262+0000 7fdc977e9640 1 -- 172.21.15.72:0/758406577 shutdown_connections 2024-04-15T15:15:09.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.262+0000 7fdc977e9640 1 -- 172.21.15.72:0/758406577 >> 172.21.15.72:0/758406577 conn(0x7fdc900b3f90 msgr2=0x7fdc900b43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:09.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.262+0000 7fdc977e9640 1 -- 172.21.15.72:0/758406577 shutdown_connections 2024-04-15T15:15:09.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.262+0000 7fdc977e9640 1 -- 172.21.15.72:0/758406577 wait complete. 2024-04-15T15:15:09.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.263+0000 7fdc977e9640 1 Processor -- start 2024-04-15T15:15:09.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.263+0000 7fdc977e9640 1 -- start start 2024-04-15T15:15:09.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.264+0000 7fdc977e9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc7c004d40 0x7fdc9014eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:09.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.264+0000 7fdc977e9640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc900cc200 0x7fdc900cfdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:09.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.264+0000 7fdc977e9640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdc900d0300 0x7fdc901f1de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:09.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.264+0000 7fdc977e9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fdc900d1cd0 con 0x7fdc7c004d40 2024-04-15T15:15:09.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.264+0000 7fdc977e9640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fdc900cd430 con 0x7fdc900cc200 2024-04-15T15:15:09.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.264+0000 7fdc977e9640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fdc900b6cd0 con 0x7fdc900d0300 2024-04-15T15:15:09.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.264+0000 7fdc95585640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc7c004d40 0x7fdc9014eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:09.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.264+0000 7fdc95585640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc7c004d40 0x7fdc9014eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44576/0 (socket says 172.21.15.72:44576) 2024-04-15T15:15:09.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.264+0000 7fdc95585640 1 -- 172.21.15.72:0/2509268893 learned_addr learned my addr 172.21.15.72:0/2509268893 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:09.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.264+0000 7fdc94d84640 1 --2- 172.21.15.72:0/2509268893 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc900cc200 0x7fdc900cfdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:09.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.264+0000 7fdc95d86640 1 --2- 172.21.15.72:0/2509268893 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdc900d0300 0x7fdc901f1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:09.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.264+0000 7fdc95585640 1 -- 172.21.15.72:0/2509268893 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdc900d0300 msgr2=0x7fdc901f1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:09.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.264+0000 7fdc95585640 1 --2- 172.21.15.72:0/2509268893 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdc900d0300 0x7fdc901f1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.264+0000 7fdc95585640 1 -- 172.21.15.72:0/2509268893 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc900cc200 msgr2=0x7fdc900cfdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:09.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.264+0000 7fdc95585640 1 --2- 172.21.15.72:0/2509268893 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc900cc200 0x7fdc900cfdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.264+0000 7fdc95585640 1 -- 172.21.15.72:0/2509268893 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc9015be40 con 0x7fdc7c004d40 2024-04-15T15:15:09.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.264+0000 7fdc95d86640 1 --2- 172.21.15.72:0/2509268893 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdc900d0300 0x7fdc901f1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:09.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.265+0000 7fdc95585640 1 --2- 172.21.15.72:0/2509268893 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc7c004d40 0x7fdc9014eb80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c007310 tx=0x7fdc8c0077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:09.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.265+0000 7fdc81ffb640 1 -- 172.21.15.72:0/2509268893 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc8c00f940 con 0x7fdc7c004d40 2024-04-15T15:15:09.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.265+0000 7fdc81ffb640 1 -- 172.21.15.72:0/2509268893 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc8c018070 con 0x7fdc7c004d40 2024-04-15T15:15:09.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.265+0000 7fdc977e9640 1 -- 172.21.15.72:0/2509268893 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc900cb160 con 0x7fdc7c004d40 2024-04-15T15:15:09.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.265+0000 7fdc81ffb640 1 -- 172.21.15.72:0/2509268893 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc8c00fd20 con 0x7fdc7c004d40 2024-04-15T15:15:09.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.265+0000 7fdc977e9640 1 -- 172.21.15.72:0/2509268893 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc900cb720 con 0x7fdc7c004d40 2024-04-15T15:15:09.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.266+0000 7fdc977e9640 1 -- 172.21.15.72:0/2509268893 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc900689e0 con 0x7fdc7c004d40 2024-04-15T15:15:09.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.266+0000 7fdc81ffb640 1 -- 172.21.15.72:0/2509268893 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdc8c01d460 con 0x7fdc7c004d40 2024-04-15T15:15:09.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.267+0000 7fdc81ffb640 1 --2- 172.21.15.72:0/2509268893 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fdc7406c8f0 0x7fdc7406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:09.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.267+0000 7fdc81ffb640 1 -- 172.21.15.72:0/2509268893 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fdc8c08bf20 con 0x7fdc7c004d40 2024-04-15T15:15:09.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.267+0000 7fdc94d84640 1 --2- 172.21.15.72:0/2509268893 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fdc7406c8f0 0x7fdc7406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:09.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.267+0000 7fdc94d84640 1 --2- 172.21.15.72:0/2509268893 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fdc7406c8f0 0x7fdc7406edb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fdc901f5310 tx=0x7fdc84009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:09.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.271+0000 7fdc81ffb640 1 -- 172.21.15.72:0/2509268893 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdc9015be40 con 0x7fdc7c004d40 2024-04-15T15:15:09.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.443+0000 7fdc977e9640 1 -- 172.21.15.72:0/2509268893 --> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdc900b3c00 con 0x7fdc7406c8f0 2024-04-15T15:15:09.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.451+0000 7fdc81ffb640 1 -- 172.21.15.72:0/2509268893 <== mgr.4101 v2:172.21.15.72:6832/3503807805 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332534 (secure 0 0 0) 0x7fdc900b3c00 con 0x7fdc7406c8f0 2024-04-15T15:15:09.457 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:09.459 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-04-15T15:15:09.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.454+0000 7fdc977e9640 1 -- 172.21.15.72:0/2509268893 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fdc7406c8f0 msgr2=0x7fdc7406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:09.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.454+0000 7fdc977e9640 1 --2- 172.21.15.72:0/2509268893 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fdc7406c8f0 0x7fdc7406edb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fdc901f5310 tx=0x7fdc84009040 comp rx=0 tx=0).stop 2024-04-15T15:15:09.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.454+0000 7fdc977e9640 1 -- 172.21.15.72:0/2509268893 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc7c004d40 msgr2=0x7fdc9014eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:09.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.454+0000 7fdc977e9640 1 --2- 172.21.15.72:0/2509268893 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc7c004d40 0x7fdc9014eb80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c007310 tx=0x7fdc8c0077d0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.454+0000 7fdc977e9640 1 -- 172.21.15.72:0/2509268893 shutdown_connections 2024-04-15T15:15:09.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.454+0000 7fdc977e9640 1 --2- 172.21.15.72:0/2509268893 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fdc7406c8f0 0x7fdc7406edb0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.454+0000 7fdc977e9640 1 --2- 172.21.15.72:0/2509268893 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdc900d0300 0x7fdc901f1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.454+0000 7fdc977e9640 1 --2- 172.21.15.72:0/2509268893 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc900cc200 0x7fdc900cfdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.454+0000 7fdc977e9640 1 --2- 172.21.15.72:0/2509268893 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc7c004d40 0x7fdc9014eb80 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.454+0000 7fdc977e9640 1 -- 172.21.15.72:0/2509268893 >> 172.21.15.72:0/2509268893 conn(0x7fdc900b3f90 msgr2=0x7fdc900b7260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:09.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.454+0000 7fdc977e9640 1 -- 172.21.15.72:0/2509268893 shutdown_connections 2024-04-15T15:15:09.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.454+0000 7fdc977e9640 1 -- 172.21.15.72:0/2509268893 wait complete. 2024-04-15T15:15:09.467 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2024-04-15T15:15:09.347333+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594867,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":206,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219392,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755328,"statfs":{"total":773094113280,"available":772869455872,"internally_reserved":0,"allocated":3284992,"data_stored":1991782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2089,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002472"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137048+0000","last_change":"2024-04-15T15:14:51.075131+0000","last_active":"2024-04-15T15:14:57.137048+0000","last_peered":"2024-04-15T15:14:57.137048+0000","last_clean":"2024-04-15T15:14:57.137048+0000","last_became_active":"2024-04-15T15:14:51.074841+0000","last_became_peered":"2024-04-15T15:14:51.074841+0000","last_unstale":"2024-04-15T15:14:57.137048+0000","last_undegraded":"2024-04-15T15:14:57.137048+0000","last_fullsized":"2024-04-15T15:14:57.137048+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:19:56.231155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138223+0000","last_change":"2024-04-15T15:14:52.077480+0000","last_active":"2024-04-15T15:14:57.138223+0000","last_peered":"2024-04-15T15:14:57.138223+0000","last_clean":"2024-04-15T15:14:57.138223+0000","last_became_active":"2024-04-15T15:14:52.077166+0000","last_became_peered":"2024-04-15T15:14:52.077166+0000","last_unstale":"2024-04-15T15:14:57.138223+0000","last_undegraded":"2024-04-15T15:14:57.138223+0000","last_fullsized":"2024-04-15T15:14:57.138223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:13:01.642838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136181+0000","last_change":"2024-04-15T15:14:56.255424+0000","last_active":"2024-04-15T15:14:57.136181+0000","last_peered":"2024-04-15T15:14:57.136181+0000","last_clean":"2024-04-15T15:14:57.136181+0000","last_became_active":"2024-04-15T15:14:56.254987+0000","last_became_peered":"2024-04-15T15:14:56.254987+0000","last_unstale":"2024-04-15T15:14:57.136181+0000","last_undegraded":"2024-04-15T15:14:57.136181+0000","last_fullsized":"2024-04-15T15:14:57.136181+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:11:24.196989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138454+0000","last_change":"2024-04-15T15:14:51.071376+0000","last_active":"2024-04-15T15:14:57.138454+0000","last_peered":"2024-04-15T15:14:57.138454+0000","last_clean":"2024-04-15T15:14:57.138454+0000","last_became_active":"2024-04-15T15:14:51.071108+0000","last_became_peered":"2024-04-15T15:14:51.071108+0000","last_unstale":"2024-04-15T15:14:57.138454+0000","last_undegraded":"2024-04-15T15:14:57.138454+0000","last_fullsized":"2024-04-15T15:14:57.138454+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:50:01.422527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136732+0000","last_change":"2024-04-15T15:14:52.083839+0000","last_active":"2024-04-15T15:14:57.136732+0000","last_peered":"2024-04-15T15:14:57.136732+0000","last_clean":"2024-04-15T15:14:57.136732+0000","last_became_active":"2024-04-15T15:14:52.083533+0000","last_became_peered":"2024-04-15T15:14:52.083533+0000","last_unstale":"2024-04-15T15:14:57.136732+0000","last_undegraded":"2024-04-15T15:14:57.136732+0000","last_fullsized":"2024-04-15T15:14:57.136732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:34:51.775979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135872+0000","last_change":"2024-04-15T15:14:56.098375+0000","last_active":"2024-04-15T15:14:57.135872+0000","last_peered":"2024-04-15T15:14:57.135872+0000","last_clean":"2024-04-15T15:14:57.135872+0000","last_became_active":"2024-04-15T15:14:56.097969+0000","last_became_peered":"2024-04-15T15:14:56.097969+0000","last_unstale":"2024-04-15T15:14:57.135872+0000","last_undegraded":"2024-04-15T15:14:57.135872+0000","last_fullsized":"2024-04-15T15:14:57.135872+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:12:44.445524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136739+0000","last_change":"2024-04-15T15:14:51.079869+0000","last_active":"2024-04-15T15:14:57.136739+0000","last_peered":"2024-04-15T15:14:57.136739+0000","last_clean":"2024-04-15T15:14:57.136739+0000","last_became_active":"2024-04-15T15:14:51.079576+0000","last_became_peered":"2024-04-15T15:14:51.079576+0000","last_unstale":"2024-04-15T15:14:57.136739+0000","last_undegraded":"2024-04-15T15:14:57.136739+0000","last_fullsized":"2024-04-15T15:14:57.136739+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:58:40.440083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109631+0000","last_change":"2024-04-15T15:14:52.075307+0000","last_active":"2024-04-15T15:14:57.109631+0000","last_peered":"2024-04-15T15:14:57.109631+0000","last_clean":"2024-04-15T15:14:57.109631+0000","last_became_active":"2024-04-15T15:14:52.075001+0000","last_became_peered":"2024-04-15T15:14:52.075001+0000","last_unstale":"2024-04-15T15:14:57.109631+0000","last_undegraded":"2024-04-15T15:14:57.109631+0000","last_fullsized":"2024-04-15T15:14:57.109631+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:01:24.679360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136399+0000","last_change":"2024-04-15T15:14:56.098292+0000","last_active":"2024-04-15T15:14:57.136399+0000","last_peered":"2024-04-15T15:14:57.136399+0000","last_clean":"2024-04-15T15:14:57.136399+0000","last_became_active":"2024-04-15T15:14:56.097848+0000","last_became_peered":"2024-04-15T15:14:56.097848+0000","last_unstale":"2024-04-15T15:14:57.136399+0000","last_undegraded":"2024-04-15T15:14:57.136399+0000","last_fullsized":"2024-04-15T15:14:57.136399+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:26:49.486090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136078+0000","last_change":"2024-04-15T15:14:51.079754+0000","last_active":"2024-04-15T15:14:57.136078+0000","last_peered":"2024-04-15T15:14:57.136078+0000","last_clean":"2024-04-15T15:14:57.136078+0000","last_became_active":"2024-04-15T15:14:51.079452+0000","last_became_peered":"2024-04-15T15:14:51.079452+0000","last_unstale":"2024-04-15T15:14:57.136078+0000","last_undegraded":"2024-04-15T15:14:57.136078+0000","last_fullsized":"2024-04-15T15:14:57.136078+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:19:37.987965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.110221+0000","last_change":"2024-04-15T15:14:52.076654+0000","last_active":"2024-04-15T15:14:57.110221+0000","last_peered":"2024-04-15T15:14:57.110221+0000","last_clean":"2024-04-15T15:14:57.110221+0000","last_became_active":"2024-04-15T15:14:52.076283+0000","last_became_peered":"2024-04-15T15:14:52.076283+0000","last_unstale":"2024-04-15T15:14:57.110221+0000","last_undegraded":"2024-04-15T15:14:57.110221+0000","last_fullsized":"2024-04-15T15:14:57.110221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:57:57.665646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136259+0000","last_change":"2024-04-15T15:14:56.105257+0000","last_active":"2024-04-15T15:14:57.136259+0000","last_peered":"2024-04-15T15:14:57.136259+0000","last_clean":"2024-04-15T15:14:57.136259+0000","last_became_active":"2024-04-15T15:14:56.104856+0000","last_became_peered":"2024-04-15T15:14:56.104856+0000","last_unstale":"2024-04-15T15:14:57.136259+0000","last_undegraded":"2024-04-15T15:14:57.136259+0000","last_fullsized":"2024-04-15T15:14:57.136259+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:10:23.721558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108699+0000","last_change":"2024-04-15T15:14:52.086412+0000","last_active":"2024-04-15T15:14:57.108699+0000","last_peered":"2024-04-15T15:14:57.108699+0000","last_clean":"2024-04-15T15:14:57.108699+0000","last_became_active":"2024-04-15T15:14:52.085479+0000","last_became_peered":"2024-04-15T15:14:52.085479+0000","last_unstale":"2024-04-15T15:14:57.108699+0000","last_undegraded":"2024-04-15T15:14:57.108699+0000","last_fullsized":"2024-04-15T15:14:57.108699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:13:01.642838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136325+0000","last_change":"2024-04-15T15:14:51.080109+0000","last_active":"2024-04-15T15:14:57.136325+0000","last_peered":"2024-04-15T15:14:57.136325+0000","last_clean":"2024-04-15T15:14:57.136325+0000","last_became_active":"2024-04-15T15:14:51.079768+0000","last_became_peered":"2024-04-15T15:14:51.079768+0000","last_unstale":"2024-04-15T15:14:57.136325+0000","last_undegraded":"2024-04-15T15:14:57.136325+0000","last_fullsized":"2024-04-15T15:14:57.136325+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:07:48.416478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136732+0000","last_change":"2024-04-15T15:14:56.102424+0000","last_active":"2024-04-15T15:14:57.136732+0000","last_peered":"2024-04-15T15:14:57.136732+0000","last_clean":"2024-04-15T15:14:57.136732+0000","last_became_active":"2024-04-15T15:14:56.101985+0000","last_became_peered":"2024-04-15T15:14:56.101985+0000","last_unstale":"2024-04-15T15:14:57.136732+0000","last_undegraded":"2024-04-15T15:14:57.136732+0000","last_fullsized":"2024-04-15T15:14:57.136732+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:01:28.704663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136649+0000","last_change":"2024-04-15T15:14:52.084991+0000","last_active":"2024-04-15T15:14:57.136649+0000","last_peered":"2024-04-15T15:14:57.136649+0000","last_clean":"2024-04-15T15:14:57.136649+0000","last_became_active":"2024-04-15T15:14:52.082250+0000","last_became_peered":"2024-04-15T15:14:52.082250+0000","last_unstale":"2024-04-15T15:14:57.136649+0000","last_undegraded":"2024-04-15T15:14:57.136649+0000","last_fullsized":"2024-04-15T15:14:57.136649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:26:45.460787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137232+0000","last_change":"2024-04-15T15:14:51.071863+0000","last_active":"2024-04-15T15:14:57.137232+0000","last_peered":"2024-04-15T15:14:57.137232+0000","last_clean":"2024-04-15T15:14:57.137232+0000","last_became_active":"2024-04-15T15:14:51.071595+0000","last_became_peered":"2024-04-15T15:14:51.071595+0000","last_unstale":"2024-04-15T15:14:57.137232+0000","last_undegraded":"2024-04-15T15:14:57.137232+0000","last_fullsized":"2024-04-15T15:14:57.137232+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:19:37.987965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136625+0000","last_change":"2024-04-15T15:14:56.099259+0000","last_active":"2024-04-15T15:14:57.136625+0000","last_peered":"2024-04-15T15:14:57.136625+0000","last_clean":"2024-04-15T15:14:57.136625+0000","last_became_active":"2024-04-15T15:14:56.098862+0000","last_became_peered":"2024-04-15T15:14:56.098862+0000","last_unstale":"2024-04-15T15:14:57.136625+0000","last_undegraded":"2024-04-15T15:14:57.136625+0000","last_fullsized":"2024-04-15T15:14:57.136625+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:38:39.210922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136403+0000","last_change":"2024-04-15T15:14:52.080320+0000","last_active":"2024-04-15T15:14:57.136403+0000","last_peered":"2024-04-15T15:14:57.136403+0000","last_clean":"2024-04-15T15:14:57.136403+0000","last_became_active":"2024-04-15T15:14:52.080064+0000","last_became_peered":"2024-04-15T15:14:52.080064+0000","last_unstale":"2024-04-15T15:14:57.136403+0000","last_undegraded":"2024-04-15T15:14:57.136403+0000","last_fullsized":"2024-04-15T15:14:57.136403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:10:19.696254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136107+0000","last_change":"2024-04-15T15:14:51.075197+0000","last_active":"2024-04-15T15:14:57.136107+0000","last_peered":"2024-04-15T15:14:57.136107+0000","last_clean":"2024-04-15T15:14:57.136107+0000","last_became_active":"2024-04-15T15:14:51.074955+0000","last_became_peered":"2024-04-15T15:14:51.074955+0000","last_unstale":"2024-04-15T15:14:57.136107+0000","last_undegraded":"2024-04-15T15:14:57.136107+0000","last_fullsized":"2024-04-15T15:14:57.136107+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:18:41.399059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137490+0000","last_change":"2024-04-15T15:14:56.253169+0000","last_active":"2024-04-15T15:14:57.137490+0000","last_peered":"2024-04-15T15:14:57.137490+0000","last_clean":"2024-04-15T15:14:57.137490+0000","last_became_active":"2024-04-15T15:14:56.252823+0000","last_became_peered":"2024-04-15T15:14:56.252823+0000","last_unstale":"2024-04-15T15:14:57.137490+0000","last_undegraded":"2024-04-15T15:14:57.137490+0000","last_fullsized":"2024-04-15T15:14:57.137490+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:03:07.876012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136337+0000","last_change":"2024-04-15T15:14:52.075278+0000","last_active":"2024-04-15T15:14:57.136337+0000","last_peered":"2024-04-15T15:14:57.136337+0000","last_clean":"2024-04-15T15:14:57.136337+0000","last_became_active":"2024-04-15T15:14:52.075003+0000","last_became_peered":"2024-04-15T15:14:52.075003+0000","last_unstale":"2024-04-15T15:14:57.136337+0000","last_undegraded":"2024-04-15T15:14:57.136337+0000","last_fullsized":"2024-04-15T15:14:57.136337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:47:55.994716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135597+0000","last_change":"2024-04-15T15:14:51.076530+0000","last_active":"2024-04-15T15:14:57.135597+0000","last_peered":"2024-04-15T15:14:57.135597+0000","last_clean":"2024-04-15T15:14:57.135597+0000","last_became_active":"2024-04-15T15:14:51.076247+0000","last_became_peered":"2024-04-15T15:14:51.076247+0000","last_unstale":"2024-04-15T15:14:57.135597+0000","last_undegraded":"2024-04-15T15:14:57.135597+0000","last_fullsized":"2024-04-15T15:14:57.135597+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:47:54.990677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136888+0000","last_change":"2024-04-15T15:14:56.098477+0000","last_active":"2024-04-15T15:14:57.136888+0000","last_peered":"2024-04-15T15:14:57.136888+0000","last_clean":"2024-04-15T15:14:57.136888+0000","last_became_active":"2024-04-15T15:14:56.097839+0000","last_became_peered":"2024-04-15T15:14:56.097839+0000","last_unstale":"2024-04-15T15:14:57.136888+0000","last_undegraded":"2024-04-15T15:14:57.136888+0000","last_fullsized":"2024-04-15T15:14:57.136888+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:12.500763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138519+0000","last_change":"2024-04-15T15:14:52.079014+0000","last_active":"2024-04-15T15:14:57.138519+0000","last_peered":"2024-04-15T15:14:57.138519+0000","last_clean":"2024-04-15T15:14:57.138519+0000","last_became_active":"2024-04-15T15:14:52.078708+0000","last_became_peered":"2024-04-15T15:14:52.078708+0000","last_unstale":"2024-04-15T15:14:57.138519+0000","last_undegraded":"2024-04-15T15:14:57.138519+0000","last_fullsized":"2024-04-15T15:14:57.138519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:37:31.656434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109619+0000","last_change":"2024-04-15T15:14:51.075067+0000","last_active":"2024-04-15T15:14:57.109619+0000","last_peered":"2024-04-15T15:14:57.109619+0000","last_clean":"2024-04-15T15:14:57.109619+0000","last_became_active":"2024-04-15T15:14:51.074780+0000","last_became_peered":"2024-04-15T15:14:51.074780+0000","last_unstale":"2024-04-15T15:14:57.109619+0000","last_undegraded":"2024-04-15T15:14:57.109619+0000","last_fullsized":"2024-04-15T15:14:57.109619+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:47:54.990677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136173+0000","last_change":"2024-04-15T15:14:56.252842+0000","last_active":"2024-04-15T15:14:57.136173+0000","last_peered":"2024-04-15T15:14:57.136173+0000","last_clean":"2024-04-15T15:14:57.136173+0000","last_became_active":"2024-04-15T15:14:56.252432+0000","last_became_peered":"2024-04-15T15:14:56.252432+0000","last_unstale":"2024-04-15T15:14:57.136173+0000","last_undegraded":"2024-04-15T15:14:57.136173+0000","last_fullsized":"2024-04-15T15:14:57.136173+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:51:26.840184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138191+0000","last_change":"2024-04-15T15:14:52.082404+0000","last_active":"2024-04-15T15:14:57.138191+0000","last_peered":"2024-04-15T15:14:57.138191+0000","last_clean":"2024-04-15T15:14:57.138191+0000","last_became_active":"2024-04-15T15:14:52.082136+0000","last_became_peered":"2024-04-15T15:14:52.082136+0000","last_unstale":"2024-04-15T15:14:57.138191+0000","last_undegraded":"2024-04-15T15:14:57.138191+0000","last_fullsized":"2024-04-15T15:14:57.138191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:38:35.185619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137543+0000","last_change":"2024-04-15T15:14:51.073915+0000","last_active":"2024-04-15T15:14:57.137543+0000","last_peered":"2024-04-15T15:14:57.137543+0000","last_clean":"2024-04-15T15:14:57.137543+0000","last_became_active":"2024-04-15T15:14:51.073639+0000","last_became_peered":"2024-04-15T15:14:51.073639+0000","last_unstale":"2024-04-15T15:14:57.137543+0000","last_undegraded":"2024-04-15T15:14:57.137543+0000","last_fullsized":"2024-04-15T15:14:57.137543+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:41:20.990702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135980+0000","last_change":"2024-04-15T15:14:56.251221+0000","last_active":"2024-04-15T15:14:57.135980+0000","last_peered":"2024-04-15T15:14:57.135980+0000","last_clean":"2024-04-15T15:14:57.135980+0000","last_became_active":"2024-04-15T15:14:56.250888+0000","last_became_peered":"2024-04-15T15:14:56.250888+0000","last_unstale":"2024-04-15T15:14:57.135980+0000","last_undegraded":"2024-04-15T15:14:57.135980+0000","last_fullsized":"2024-04-15T15:14:57.135980+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:42:46.671056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137233+0000","last_change":"2024-04-15T15:14:52.077347+0000","last_active":"2024-04-15T15:14:57.137233+0000","last_peered":"2024-04-15T15:14:57.137233+0000","last_clean":"2024-04-15T15:14:57.137233+0000","last_became_active":"2024-04-15T15:14:52.077053+0000","last_became_peered":"2024-04-15T15:14:52.077053+0000","last_unstale":"2024-04-15T15:14:57.137233+0000","last_undegraded":"2024-04-15T15:14:57.137233+0000","last_fullsized":"2024-04-15T15:14:57.137233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:13:01.642838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136936+0000","last_change":"2024-04-15T15:14:51.076396+0000","last_active":"2024-04-15T15:14:57.136936+0000","last_peered":"2024-04-15T15:14:57.136936+0000","last_clean":"2024-04-15T15:14:57.136936+0000","last_became_active":"2024-04-15T15:14:51.076124+0000","last_became_peered":"2024-04-15T15:14:51.076124+0000","last_unstale":"2024-04-15T15:14:57.136936+0000","last_undegraded":"2024-04-15T15:14:57.136936+0000","last_fullsized":"2024-04-15T15:14:57.136936+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:31:57.289767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137379+0000","last_change":"2024-04-15T15:14:56.251375+0000","last_active":"2024-04-15T15:14:57.137379+0000","last_peered":"2024-04-15T15:14:57.137379+0000","last_clean":"2024-04-15T15:14:57.137379+0000","last_became_active":"2024-04-15T15:14:56.250973+0000","last_became_peered":"2024-04-15T15:14:56.250973+0000","last_unstale":"2024-04-15T15:14:57.137379+0000","last_undegraded":"2024-04-15T15:14:57.137379+0000","last_fullsized":"2024-04-15T15:14:57.137379+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:58:01.690949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.110314+0000","last_change":"2024-04-15T15:14:52.082863+0000","last_active":"2024-04-15T15:14:57.110314+0000","last_peered":"2024-04-15T15:14:57.110314+0000","last_clean":"2024-04-15T15:14:57.110314+0000","last_became_active":"2024-04-15T15:14:52.082370+0000","last_became_peered":"2024-04-15T15:14:52.082370+0000","last_unstale":"2024-04-15T15:14:57.110314+0000","last_undegraded":"2024-04-15T15:14:57.110314+0000","last_fullsized":"2024-04-15T15:14:57.110314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:08.475460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137335+0000","last_change":"2024-04-15T15:14:51.073016+0000","last_active":"2024-04-15T15:14:57.137335+0000","last_peered":"2024-04-15T15:14:57.137335+0000","last_clean":"2024-04-15T15:14:57.137335+0000","last_became_active":"2024-04-15T15:14:51.072736+0000","last_became_peered":"2024-04-15T15:14:51.072736+0000","last_unstale":"2024-04-15T15:14:57.137335+0000","last_undegraded":"2024-04-15T15:14:57.137335+0000","last_fullsized":"2024-04-15T15:14:57.137335+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:44:33.685298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136866+0000","last_change":"2024-04-15T15:14:56.105315+0000","last_active":"2024-04-15T15:14:57.136866+0000","last_peered":"2024-04-15T15:14:57.136866+0000","last_clean":"2024-04-15T15:14:57.136866+0000","last_became_active":"2024-04-15T15:14:56.104923+0000","last_became_peered":"2024-04-15T15:14:56.104923+0000","last_unstale":"2024-04-15T15:14:57.136866+0000","last_undegraded":"2024-04-15T15:14:57.136866+0000","last_fullsized":"2024-04-15T15:14:57.136866+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:01:28.704663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136866+0000","last_change":"2024-04-15T15:14:52.078353+0000","last_active":"2024-04-15T15:14:57.136866+0000","last_peered":"2024-04-15T15:14:57.136866+0000","last_clean":"2024-04-15T15:14:57.136866+0000","last_became_active":"2024-04-15T15:14:52.076556+0000","last_became_peered":"2024-04-15T15:14:52.076556+0000","last_unstale":"2024-04-15T15:14:57.136866+0000","last_undegraded":"2024-04-15T15:14:57.136866+0000","last_fullsized":"2024-04-15T15:14:57.136866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:19:38.992004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109062+0000","last_change":"2024-04-15T15:14:51.072875+0000","last_active":"2024-04-15T15:14:57.109062+0000","last_peered":"2024-04-15T15:14:57.109062+0000","last_clean":"2024-04-15T15:14:57.109062+0000","last_became_active":"2024-04-15T15:14:51.072114+0000","last_became_peered":"2024-04-15T15:14:51.072114+0000","last_unstale":"2024-04-15T15:14:57.109062+0000","last_undegraded":"2024-04-15T15:14:57.109062+0000","last_fullsized":"2024-04-15T15:14:57.109062+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:18:41.399059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137344+0000","last_change":"2024-04-15T15:14:56.098426+0000","last_active":"2024-04-15T15:14:57.137344+0000","last_peered":"2024-04-15T15:14:57.137344+0000","last_clean":"2024-04-15T15:14:57.137344+0000","last_became_active":"2024-04-15T15:14:56.098066+0000","last_became_peered":"2024-04-15T15:14:56.098066+0000","last_unstale":"2024-04-15T15:14:57.137344+0000","last_undegraded":"2024-04-15T15:14:57.137344+0000","last_fullsized":"2024-04-15T15:14:57.137344+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:12.500763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136427+0000","last_change":"2024-04-15T15:14:52.077474+0000","last_active":"2024-04-15T15:14:57.136427+0000","last_peered":"2024-04-15T15:14:57.136427+0000","last_clean":"2024-04-15T15:14:57.136427+0000","last_became_active":"2024-04-15T15:14:52.077206+0000","last_became_peered":"2024-04-15T15:14:52.077206+0000","last_unstale":"2024-04-15T15:14:57.136427+0000","last_undegraded":"2024-04-15T15:14:57.136427+0000","last_fullsized":"2024-04-15T15:14:57.136427+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:34:51.775979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.107727+0000","last_change":"2024-04-15T15:14:51.067544+0000","last_active":"2024-04-15T15:14:57.107727+0000","last_peered":"2024-04-15T15:14:57.107727+0000","last_clean":"2024-04-15T15:14:57.107727+0000","last_became_active":"2024-04-15T15:14:51.067233+0000","last_became_peered":"2024-04-15T15:14:51.067233+0000","last_unstale":"2024-04-15T15:14:57.107727+0000","last_undegraded":"2024-04-15T15:14:57.107727+0000","last_fullsized":"2024-04-15T15:14:57.107727+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:44:33.685298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137463+0000","last_change":"2024-04-15T15:14:56.103911+0000","last_active":"2024-04-15T15:14:57.137463+0000","last_peered":"2024-04-15T15:14:57.137463+0000","last_clean":"2024-04-15T15:14:57.137463+0000","last_became_active":"2024-04-15T15:14:56.103035+0000","last_became_peered":"2024-04-15T15:14:56.103035+0000","last_unstale":"2024-04-15T15:14:57.137463+0000","last_undegraded":"2024-04-15T15:14:57.137463+0000","last_fullsized":"2024-04-15T15:14:57.137463+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:21:52.495367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136227+0000","last_change":"2024-04-15T15:14:52.078870+0000","last_active":"2024-04-15T15:14:57.136227+0000","last_peered":"2024-04-15T15:14:57.136227+0000","last_clean":"2024-04-15T15:14:57.136227+0000","last_became_active":"2024-04-15T15:14:52.078573+0000","last_became_peered":"2024-04-15T15:14:52.078573+0000","last_unstale":"2024-04-15T15:14:57.136227+0000","last_undegraded":"2024-04-15T15:14:57.136227+0000","last_fullsized":"2024-04-15T15:14:57.136227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:51:36.740221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135636+0000","last_change":"2024-04-15T15:14:51.077104+0000","last_active":"2024-04-15T15:14:57.135636+0000","last_peered":"2024-04-15T15:14:57.135636+0000","last_clean":"2024-04-15T15:14:57.135636+0000","last_became_active":"2024-04-15T15:14:51.076857+0000","last_became_peered":"2024-04-15T15:14:51.076857+0000","last_unstale":"2024-04-15T15:14:57.135636+0000","last_undegraded":"2024-04-15T15:14:57.135636+0000","last_fullsized":"2024-04-15T15:14:57.135636+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:18:41.399059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138372+0000","last_change":"2024-04-15T15:14:56.251372+0000","last_active":"2024-04-15T15:14:57.138372+0000","last_peered":"2024-04-15T15:14:57.138372+0000","last_clean":"2024-04-15T15:14:57.138372+0000","last_became_active":"2024-04-15T15:14:56.250961+0000","last_became_peered":"2024-04-15T15:14:56.250961+0000","last_unstale":"2024-04-15T15:14:57.138372+0000","last_undegraded":"2024-04-15T15:14:57.138372+0000","last_fullsized":"2024-04-15T15:14:57.138372+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:13:03.031872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136619+0000","last_change":"2024-04-15T15:14:52.089029+0000","last_active":"2024-04-15T15:14:57.136619+0000","last_peered":"2024-04-15T15:14:57.136619+0000","last_clean":"2024-04-15T15:14:57.136619+0000","last_became_active":"2024-04-15T15:14:52.088646+0000","last_became_peered":"2024-04-15T15:14:52.088646+0000","last_unstale":"2024-04-15T15:14:57.136619+0000","last_undegraded":"2024-04-15T15:14:57.136619+0000","last_fullsized":"2024-04-15T15:14:57.136619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:37:31.656434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108973+0000","last_change":"2024-04-15T15:14:51.067071+0000","last_active":"2024-04-15T15:14:57.108973+0000","last_peered":"2024-04-15T15:14:57.108973+0000","last_clean":"2024-04-15T15:14:57.108973+0000","last_became_active":"2024-04-15T15:14:51.066382+0000","last_became_peered":"2024-04-15T15:14:51.066382+0000","last_unstale":"2024-04-15T15:14:57.108973+0000","last_undegraded":"2024-04-15T15:14:57.108973+0000","last_fullsized":"2024-04-15T15:14:57.108973+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:50:01.422527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135699+0000","last_change":"2024-04-15T15:14:40.494279+0000","last_active":"2024-04-15T15:14:57.135699+0000","last_peered":"2024-04-15T15:14:57.135699+0000","last_clean":"2024-04-15T15:14:57.135699+0000","last_became_active":"2024-04-15T15:14:40.493404+0000","last_became_peered":"2024-04-15T15:14:40.493404+0000","last_unstale":"2024-04-15T15:14:57.135699+0000","last_undegraded":"2024-04-15T15:14:57.135699+0000","last_fullsized":"2024-04-15T15:14:57.135699+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:39.418508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:39.418508+0000","last_clean_scrub_stamp":"2024-04-15T15:14:39.418508+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:01:51.789246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109541+0000","last_change":"2024-04-15T15:14:56.253853+0000","last_active":"2024-04-15T15:14:57.109541+0000","last_peered":"2024-04-15T15:14:57.109541+0000","last_clean":"2024-04-15T15:14:57.109541+0000","last_became_active":"2024-04-15T15:14:56.253323+0000","last_became_peered":"2024-04-15T15:14:56.253323+0000","last_unstale":"2024-04-15T15:14:57.109541+0000","last_undegraded":"2024-04-15T15:14:57.109541+0000","last_fullsized":"2024-04-15T15:14:57.109541+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:03:07.876012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136598+0000","last_change":"2024-04-15T15:14:52.093103+0000","last_active":"2024-04-15T15:14:57.136598+0000","last_peered":"2024-04-15T15:14:57.136598+0000","last_clean":"2024-04-15T15:14:57.136598+0000","last_became_active":"2024-04-15T15:14:52.092786+0000","last_became_peered":"2024-04-15T15:14:52.092786+0000","last_unstale":"2024-04-15T15:14:57.136598+0000","last_undegraded":"2024-04-15T15:14:57.136598+0000","last_fullsized":"2024-04-15T15:14:57.136598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:57:57.665646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108044+0000","last_change":"2024-04-15T15:14:51.067545+0000","last_active":"2024-04-15T15:14:57.108044+0000","last_peered":"2024-04-15T15:14:57.108044+0000","last_clean":"2024-04-15T15:14:57.108044+0000","last_became_active":"2024-04-15T15:14:51.067229+0000","last_became_peered":"2024-04-15T15:14:51.067229+0000","last_unstale":"2024-04-15T15:14:57.108044+0000","last_undegraded":"2024-04-15T15:14:57.108044+0000","last_fullsized":"2024-04-15T15:14:57.108044+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:41:20.990702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108641+0000","last_change":"2024-04-15T15:14:56.251780+0000","last_active":"2024-04-15T15:14:57.108641+0000","last_peered":"2024-04-15T15:14:57.108641+0000","last_clean":"2024-04-15T15:14:57.108641+0000","last_became_active":"2024-04-15T15:14:56.251079+0000","last_became_peered":"2024-04-15T15:14:56.251079+0000","last_unstale":"2024-04-15T15:14:57.108641+0000","last_undegraded":"2024-04-15T15:14:57.108641+0000","last_fullsized":"2024-04-15T15:14:57.108641+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:12.500763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109714+0000","last_change":"2024-04-15T15:14:52.074261+0000","last_active":"2024-04-15T15:14:57.109714+0000","last_peered":"2024-04-15T15:14:57.109714+0000","last_clean":"2024-04-15T15:14:57.109714+0000","last_became_active":"2024-04-15T15:14:52.073979+0000","last_became_peered":"2024-04-15T15:14:52.073979+0000","last_unstale":"2024-04-15T15:14:57.109714+0000","last_undegraded":"2024-04-15T15:14:57.109714+0000","last_fullsized":"2024-04-15T15:14:57.109714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:03:03.850709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137650+0000","last_change":"2024-04-15T15:14:51.074612+0000","last_active":"2024-04-15T15:14:57.137650+0000","last_peered":"2024-04-15T15:14:57.137650+0000","last_clean":"2024-04-15T15:14:57.137650+0000","last_became_active":"2024-04-15T15:14:51.074176+0000","last_became_peered":"2024-04-15T15:14:51.074176+0000","last_unstale":"2024-04-15T15:14:57.137650+0000","last_undegraded":"2024-04-15T15:14:57.137650+0000","last_fullsized":"2024-04-15T15:14:57.137650+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:50:01.422527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137604+0000","last_change":"2024-04-15T15:14:56.103129+0000","last_active":"2024-04-15T15:14:57.137604+0000","last_peered":"2024-04-15T15:14:57.137604+0000","last_clean":"2024-04-15T15:14:57.137604+0000","last_became_active":"2024-04-15T15:14:56.102757+0000","last_became_peered":"2024-04-15T15:14:56.102757+0000","last_unstale":"2024-04-15T15:14:57.137604+0000","last_undegraded":"2024-04-15T15:14:57.137604+0000","last_fullsized":"2024-04-15T15:14:57.137604+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:12.500763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136942+0000","last_change":"2024-04-15T15:14:52.087082+0000","last_active":"2024-04-15T15:14:57.136942+0000","last_peered":"2024-04-15T15:14:57.136942+0000","last_clean":"2024-04-15T15:14:57.136942+0000","last_became_active":"2024-04-15T15:14:52.086517+0000","last_became_peered":"2024-04-15T15:14:52.086517+0000","last_unstale":"2024-04-15T15:14:57.136942+0000","last_undegraded":"2024-04-15T15:14:57.136942+0000","last_fullsized":"2024-04-15T15:14:57.136942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:08:15.196752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.107969+0000","last_change":"2024-04-15T15:14:51.067941+0000","last_active":"2024-04-15T15:14:57.107969+0000","last_peered":"2024-04-15T15:14:57.107969+0000","last_clean":"2024-04-15T15:14:57.107969+0000","last_became_active":"2024-04-15T15:14:51.067669+0000","last_became_peered":"2024-04-15T15:14:51.067669+0000","last_unstale":"2024-04-15T15:14:57.107969+0000","last_undegraded":"2024-04-15T15:14:57.107969+0000","last_fullsized":"2024-04-15T15:14:57.107969+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:50:01.422527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108366+0000","last_change":"2024-04-15T15:14:56.252233+0000","last_active":"2024-04-15T15:14:57.108366+0000","last_peered":"2024-04-15T15:14:57.108366+0000","last_clean":"2024-04-15T15:14:57.108366+0000","last_became_active":"2024-04-15T15:14:56.251084+0000","last_became_peered":"2024-04-15T15:14:56.251084+0000","last_unstale":"2024-04-15T15:14:57.108366+0000","last_undegraded":"2024-04-15T15:14:57.108366+0000","last_fullsized":"2024-04-15T15:14:57.108366+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:42:46.671056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136568+0000","last_change":"2024-04-15T15:14:52.085987+0000","last_active":"2024-04-15T15:14:57.136568+0000","last_peered":"2024-04-15T15:14:57.136568+0000","last_clean":"2024-04-15T15:14:57.136568+0000","last_became_active":"2024-04-15T15:14:52.085579+0000","last_became_peered":"2024-04-15T15:14:52.085579+0000","last_unstale":"2024-04-15T15:14:57.136568+0000","last_undegraded":"2024-04-15T15:14:57.136568+0000","last_fullsized":"2024-04-15T15:14:57.136568+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:08.475460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136963+0000","last_change":"2024-04-15T15:14:51.066803+0000","last_active":"2024-04-15T15:14:57.136963+0000","last_peered":"2024-04-15T15:14:57.136963+0000","last_clean":"2024-04-15T15:14:57.136963+0000","last_became_active":"2024-04-15T15:14:51.066103+0000","last_became_peered":"2024-04-15T15:14:51.066103+0000","last_unstale":"2024-04-15T15:14:57.136963+0000","last_undegraded":"2024-04-15T15:14:57.136963+0000","last_fullsized":"2024-04-15T15:14:57.136963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:35:13.836817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135900+0000","last_change":"2024-04-15T15:14:56.098601+0000","last_active":"2024-04-15T15:14:57.135900+0000","last_peered":"2024-04-15T15:14:57.135900+0000","last_clean":"2024-04-15T15:14:57.135900+0000","last_became_active":"2024-04-15T15:14:56.098191+0000","last_became_peered":"2024-04-15T15:14:56.098191+0000","last_unstale":"2024-04-15T15:14:57.135900+0000","last_undegraded":"2024-04-15T15:14:57.135900+0000","last_fullsized":"2024-04-15T15:14:57.135900+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:12:11.797852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135812+0000","last_change":"2024-04-15T15:14:52.086933+0000","last_active":"2024-04-15T15:14:57.135812+0000","last_peered":"2024-04-15T15:14:57.135812+0000","last_clean":"2024-04-15T15:14:57.135812+0000","last_became_active":"2024-04-15T15:14:52.086546+0000","last_became_peered":"2024-04-15T15:14:52.086546+0000","last_unstale":"2024-04-15T15:14:57.135812+0000","last_undegraded":"2024-04-15T15:14:57.135812+0000","last_fullsized":"2024-04-15T15:14:57.135812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:10:19.696254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.107592+0000","last_change":"2024-04-15T15:14:51.071716+0000","last_active":"2024-04-15T15:14:57.107592+0000","last_peered":"2024-04-15T15:14:57.107592+0000","last_clean":"2024-04-15T15:14:57.107592+0000","last_became_active":"2024-04-15T15:14:51.071084+0000","last_became_peered":"2024-04-15T15:14:51.071084+0000","last_unstale":"2024-04-15T15:14:57.107592+0000","last_undegraded":"2024-04-15T15:14:57.107592+0000","last_fullsized":"2024-04-15T15:14:57.107592+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:01:16.772802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138211+0000","last_change":"2024-04-15T15:14:56.102060+0000","last_active":"2024-04-15T15:14:57.138211+0000","last_peered":"2024-04-15T15:14:57.138211+0000","last_clean":"2024-04-15T15:14:57.138211+0000","last_became_active":"2024-04-15T15:14:56.101644+0000","last_became_peered":"2024-04-15T15:14:56.101644+0000","last_unstale":"2024-04-15T15:14:57.138211+0000","last_undegraded":"2024-04-15T15:14:57.138211+0000","last_fullsized":"2024-04-15T15:14:57.138211+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:03:07.876012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109139+0000","last_change":"2024-04-15T15:14:52.088688+0000","last_active":"2024-04-15T15:14:57.109139+0000","last_peered":"2024-04-15T15:14:57.109139+0000","last_clean":"2024-04-15T15:14:57.109139+0000","last_became_active":"2024-04-15T15:14:52.088433+0000","last_became_peered":"2024-04-15T15:14:52.088433+0000","last_unstale":"2024-04-15T15:14:57.109139+0000","last_undegraded":"2024-04-15T15:14:57.109139+0000","last_fullsized":"2024-04-15T15:14:57.109139+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:24:30.714153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.110238+0000","last_change":"2024-04-15T15:14:51.073931+0000","last_active":"2024-04-15T15:14:57.110238+0000","last_peered":"2024-04-15T15:14:57.110238+0000","last_clean":"2024-04-15T15:14:57.110238+0000","last_became_active":"2024-04-15T15:14:51.073571+0000","last_became_peered":"2024-04-15T15:14:51.073571+0000","last_unstale":"2024-04-15T15:14:57.110238+0000","last_undegraded":"2024-04-15T15:14:57.110238+0000","last_fullsized":"2024-04-15T15:14:57.110238+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:44:33.685298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138190+0000","last_change":"2024-04-15T15:14:56.251177+0000","last_active":"2024-04-15T15:14:57.138190+0000","last_peered":"2024-04-15T15:14:57.138190+0000","last_clean":"2024-04-15T15:14:57.138190+0000","last_became_active":"2024-04-15T15:14:56.250803+0000","last_became_peered":"2024-04-15T15:14:56.250803+0000","last_unstale":"2024-04-15T15:14:57.138190+0000","last_undegraded":"2024-04-15T15:14:57.138190+0000","last_fullsized":"2024-04-15T15:14:57.138190+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:39:36.394347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136175+0000","last_change":"2024-04-15T15:14:52.086912+0000","last_active":"2024-04-15T15:14:57.136175+0000","last_peered":"2024-04-15T15:14:57.136175+0000","last_clean":"2024-04-15T15:14:57.136175+0000","last_became_active":"2024-04-15T15:14:52.085621+0000","last_became_peered":"2024-04-15T15:14:52.085621+0000","last_unstale":"2024-04-15T15:14:57.136175+0000","last_undegraded":"2024-04-15T15:14:57.136175+0000","last_fullsized":"2024-04-15T15:14:57.136175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:38:35.185619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108968+0000","last_change":"2024-04-15T15:14:51.068349+0000","last_active":"2024-04-15T15:14:57.108968+0000","last_peered":"2024-04-15T15:14:57.108968+0000","last_clean":"2024-04-15T15:14:57.108968+0000","last_became_active":"2024-04-15T15:14:51.068078+0000","last_became_peered":"2024-04-15T15:14:51.068078+0000","last_unstale":"2024-04-15T15:14:57.108968+0000","last_undegraded":"2024-04-15T15:14:57.108968+0000","last_fullsized":"2024-04-15T15:14:57.108968+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:47:54.990677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136871+0000","last_change":"2024-04-15T15:14:56.251089+0000","last_active":"2024-04-15T15:14:57.136871+0000","last_peered":"2024-04-15T15:14:57.136871+0000","last_clean":"2024-04-15T15:14:57.136871+0000","last_became_active":"2024-04-15T15:14:56.250735+0000","last_became_peered":"2024-04-15T15:14:56.250735+0000","last_unstale":"2024-04-15T15:14:57.136871+0000","last_undegraded":"2024-04-15T15:14:57.136871+0000","last_fullsized":"2024-04-15T15:14:57.136871+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:39:38.108475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138515+0000","last_change":"2024-04-15T15:14:52.076410+0000","last_active":"2024-04-15T15:14:57.138515+0000","last_peered":"2024-04-15T15:14:57.138515+0000","last_clean":"2024-04-15T15:14:57.138515+0000","last_became_active":"2024-04-15T15:14:52.076115+0000","last_became_peered":"2024-04-15T15:14:52.076115+0000","last_unstale":"2024-04-15T15:14:57.138515+0000","last_undegraded":"2024-04-15T15:14:57.138515+0000","last_fullsized":"2024-04-15T15:14:57.138515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:48:45.790927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137841+0000","last_change":"2024-04-15T15:14:51.072889+0000","last_active":"2024-04-15T15:14:57.137841+0000","last_peered":"2024-04-15T15:14:57.137841+0000","last_clean":"2024-04-15T15:14:57.137841+0000","last_became_active":"2024-04-15T15:14:51.072611+0000","last_became_peered":"2024-04-15T15:14:51.072611+0000","last_unstale":"2024-04-15T15:14:57.137841+0000","last_undegraded":"2024-04-15T15:14:57.137841+0000","last_fullsized":"2024-04-15T15:14:57.137841+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:20:40.577242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.110168+0000","last_change":"2024-04-15T15:14:56.253070+0000","last_active":"2024-04-15T15:14:57.110168+0000","last_peered":"2024-04-15T15:14:57.110168+0000","last_clean":"2024-04-15T15:14:57.110168+0000","last_became_active":"2024-04-15T15:14:56.252652+0000","last_became_peered":"2024-04-15T15:14:56.252652+0000","last_unstale":"2024-04-15T15:14:57.110168+0000","last_undegraded":"2024-04-15T15:14:57.110168+0000","last_fullsized":"2024-04-15T15:14:57.110168+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:34:50.166643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136518+0000","last_change":"2024-04-15T15:14:52.092734+0000","last_active":"2024-04-15T15:14:57.136518+0000","last_peered":"2024-04-15T15:14:57.136518+0000","last_clean":"2024-04-15T15:14:57.136518+0000","last_became_active":"2024-04-15T15:14:52.092105+0000","last_became_peered":"2024-04-15T15:14:52.092105+0000","last_unstale":"2024-04-15T15:14:57.136518+0000","last_undegraded":"2024-04-15T15:14:57.136518+0000","last_fullsized":"2024-04-15T15:14:57.136518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:01:24.679360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:15:01.094899+0000","last_change":"2024-04-15T15:14:51.072503+0000","last_active":"2024-04-15T15:15:01.094899+0000","last_peered":"2024-04-15T15:15:01.094899+0000","last_clean":"2024-04-15T15:15:01.094899+0000","last_became_active":"2024-04-15T15:14:51.072212+0000","last_became_peered":"2024-04-15T15:14:51.072212+0000","last_unstale":"2024-04-15T15:15:01.094899+0000","last_undegraded":"2024-04-15T15:15:01.094899+0000","last_fullsized":"2024-04-15T15:15:01.094899+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:18:41.399059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3701,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137069+0000","last_change":"2024-04-15T15:14:56.105406+0000","last_active":"2024-04-15T15:14:57.137069+0000","last_peered":"2024-04-15T15:14:57.137069+0000","last_clean":"2024-04-15T15:14:57.137069+0000","last_became_active":"2024-04-15T15:14:56.104951+0000","last_became_peered":"2024-04-15T15:14:56.104951+0000","last_unstale":"2024-04-15T15:14:57.137069+0000","last_undegraded":"2024-04-15T15:14:57.137069+0000","last_fullsized":"2024-04-15T15:14:57.137069+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:58:01.690949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136751+0000","last_change":"2024-04-15T15:14:52.083396+0000","last_active":"2024-04-15T15:14:57.136751+0000","last_peered":"2024-04-15T15:14:57.136751+0000","last_clean":"2024-04-15T15:14:57.136751+0000","last_became_active":"2024-04-15T15:14:52.082661+0000","last_became_peered":"2024-04-15T15:14:52.082661+0000","last_unstale":"2024-04-15T15:14:57.136751+0000","last_undegraded":"2024-04-15T15:14:57.136751+0000","last_fullsized":"2024-04-15T15:14:57.136751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:07:49.420517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137624+0000","last_change":"2024-04-15T15:14:51.074372+0000","last_active":"2024-04-15T15:14:57.137624+0000","last_peered":"2024-04-15T15:14:57.137624+0000","last_clean":"2024-04-15T15:14:57.137624+0000","last_became_active":"2024-04-15T15:14:51.074030+0000","last_became_peered":"2024-04-15T15:14:51.074030+0000","last_unstale":"2024-04-15T15:14:57.137624+0000","last_undegraded":"2024-04-15T15:14:57.137624+0000","last_fullsized":"2024-04-15T15:14:57.137624+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:47:54.990677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136760+0000","last_change":"2024-04-15T15:14:56.099051+0000","last_active":"2024-04-15T15:14:57.136760+0000","last_peered":"2024-04-15T15:14:57.136760+0000","last_clean":"2024-04-15T15:14:57.136760+0000","last_became_active":"2024-04-15T15:14:56.098034+0000","last_became_peered":"2024-04-15T15:14:56.098034+0000","last_unstale":"2024-04-15T15:14:57.136760+0000","last_undegraded":"2024-04-15T15:14:57.136760+0000","last_fullsized":"2024-04-15T15:14:57.136760+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:58:01.690949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136894+0000","last_change":"2024-04-15T15:14:52.084508+0000","last_active":"2024-04-15T15:14:57.136894+0000","last_peered":"2024-04-15T15:14:57.136894+0000","last_clean":"2024-04-15T15:14:57.136894+0000","last_became_active":"2024-04-15T15:14:52.083550+0000","last_became_peered":"2024-04-15T15:14:52.083550+0000","last_unstale":"2024-04-15T15:14:57.136894+0000","last_undegraded":"2024-04-15T15:14:57.136894+0000","last_fullsized":"2024-04-15T15:14:57.136894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:12:40.420221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108919+0000","last_change":"2024-04-15T15:14:51.071093+0000","last_active":"2024-04-15T15:14:57.108919+0000","last_peered":"2024-04-15T15:14:57.108919+0000","last_clean":"2024-04-15T15:14:57.108919+0000","last_became_active":"2024-04-15T15:14:51.070810+0000","last_became_peered":"2024-04-15T15:14:51.070810+0000","last_unstale":"2024-04-15T15:14:57.108919+0000","last_undegraded":"2024-04-15T15:14:57.108919+0000","last_fullsized":"2024-04-15T15:14:57.108919+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:41:20.990702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136272+0000","last_change":"2024-04-15T15:14:56.092919+0000","last_active":"2024-04-15T15:14:57.136272+0000","last_peered":"2024-04-15T15:14:57.136272+0000","last_clean":"2024-04-15T15:14:57.136272+0000","last_became_active":"2024-04-15T15:14:56.092459+0000","last_became_peered":"2024-04-15T15:14:56.092459+0000","last_unstale":"2024-04-15T15:14:57.136272+0000","last_undegraded":"2024-04-15T15:14:57.136272+0000","last_fullsized":"2024-04-15T15:14:57.136272+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:46:38.024003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108815+0000","last_change":"2024-04-15T15:14:52.079880+0000","last_active":"2024-04-15T15:14:57.108815+0000","last_peered":"2024-04-15T15:14:57.108815+0000","last_clean":"2024-04-15T15:14:57.108815+0000","last_became_active":"2024-04-15T15:14:52.079619+0000","last_became_peered":"2024-04-15T15:14:52.079619+0000","last_unstale":"2024-04-15T15:14:57.108815+0000","last_undegraded":"2024-04-15T15:14:57.108815+0000","last_fullsized":"2024-04-15T15:14:57.108815+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:21:48.470063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109438+0000","last_change":"2024-04-15T15:14:51.075415+0000","last_active":"2024-04-15T15:14:57.109438+0000","last_peered":"2024-04-15T15:14:57.109438+0000","last_clean":"2024-04-15T15:14:57.109438+0000","last_became_active":"2024-04-15T15:14:51.075014+0000","last_became_peered":"2024-04-15T15:14:51.075014+0000","last_unstale":"2024-04-15T15:14:57.109438+0000","last_undegraded":"2024-04-15T15:14:57.109438+0000","last_fullsized":"2024-04-15T15:14:57.109438+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:41:20.990702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135982+0000","last_change":"2024-04-15T15:14:56.253630+0000","last_active":"2024-04-15T15:14:57.135982+0000","last_peered":"2024-04-15T15:14:57.135982+0000","last_clean":"2024-04-15T15:14:57.135982+0000","last_became_active":"2024-04-15T15:14:56.253253+0000","last_became_peered":"2024-04-15T15:14:56.253253+0000","last_unstale":"2024-04-15T15:14:57.135982+0000","last_undegraded":"2024-04-15T15:14:57.135982+0000","last_fullsized":"2024-04-15T15:14:57.135982+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:34:50.166643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138051+0000","last_change":"2024-04-15T15:14:56.253163+0000","last_active":"2024-04-15T15:14:57.138051+0000","last_peered":"2024-04-15T15:14:57.138051+0000","last_clean":"2024-04-15T15:14:57.138051+0000","last_became_active":"2024-04-15T15:14:56.252769+0000","last_became_peered":"2024-04-15T15:14:56.252769+0000","last_unstale":"2024-04-15T15:14:57.138051+0000","last_undegraded":"2024-04-15T15:14:57.138051+0000","last_fullsized":"2024-04-15T15:14:57.138051+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:21:52.495367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135905+0000","last_change":"2024-04-15T15:14:52.086725+0000","last_active":"2024-04-15T15:14:57.135905+0000","last_peered":"2024-04-15T15:14:57.135905+0000","last_clean":"2024-04-15T15:14:57.135905+0000","last_became_active":"2024-04-15T15:14:52.086220+0000","last_became_peered":"2024-04-15T15:14:52.086220+0000","last_unstale":"2024-04-15T15:14:57.135905+0000","last_undegraded":"2024-04-15T15:14:57.135905+0000","last_fullsized":"2024-04-15T15:14:57.135905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:21:50.111549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108433+0000","last_change":"2024-04-15T15:14:51.069916+0000","last_active":"2024-04-15T15:14:57.108433+0000","last_peered":"2024-04-15T15:14:57.108433+0000","last_clean":"2024-04-15T15:14:57.108433+0000","last_became_active":"2024-04-15T15:14:51.069631+0000","last_became_peered":"2024-04-15T15:14:51.069631+0000","last_unstale":"2024-04-15T15:14:57.108433+0000","last_undegraded":"2024-04-15T15:14:57.108433+0000","last_fullsized":"2024-04-15T15:14:57.108433+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:44:33.685298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109306+0000","last_change":"2024-04-15T15:14:56.102205+0000","last_active":"2024-04-15T15:14:57.109306+0000","last_peered":"2024-04-15T15:14:57.109306+0000","last_clean":"2024-04-15T15:14:57.109306+0000","last_became_active":"2024-04-15T15:14:56.101827+0000","last_became_peered":"2024-04-15T15:14:56.101827+0000","last_unstale":"2024-04-15T15:14:57.109306+0000","last_undegraded":"2024-04-15T15:14:57.109306+0000","last_fullsized":"2024-04-15T15:14:57.109306+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:21:52.495367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136314+0000","last_change":"2024-04-15T15:14:52.079898+0000","last_active":"2024-04-15T15:14:57.136314+0000","last_peered":"2024-04-15T15:14:57.136314+0000","last_clean":"2024-04-15T15:14:57.136314+0000","last_became_active":"2024-04-15T15:14:52.079430+0000","last_became_peered":"2024-04-15T15:14:52.079430+0000","last_unstale":"2024-04-15T15:14:57.136314+0000","last_undegraded":"2024-04-15T15:14:57.136314+0000","last_fullsized":"2024-04-15T15:14:57.136314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:12:07.772549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136279+0000","last_change":"2024-04-15T15:14:51.069720+0000","last_active":"2024-04-15T15:14:57.136279+0000","last_peered":"2024-04-15T15:14:57.136279+0000","last_clean":"2024-04-15T15:14:57.136279+0000","last_became_active":"2024-04-15T15:14:51.069416+0000","last_became_peered":"2024-04-15T15:14:51.069416+0000","last_unstale":"2024-04-15T15:14:57.136279+0000","last_undegraded":"2024-04-15T15:14:57.136279+0000","last_fullsized":"2024-04-15T15:14:57.136279+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:20:40.577242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108868+0000","last_change":"2024-04-15T15:14:51.070913+0000","last_active":"2024-04-15T15:14:57.108868+0000","last_peered":"2024-04-15T15:14:57.108868+0000","last_clean":"2024-04-15T15:14:57.108868+0000","last_became_active":"2024-04-15T15:14:51.070640+0000","last_became_peered":"2024-04-15T15:14:51.070640+0000","last_unstale":"2024-04-15T15:14:57.108868+0000","last_undegraded":"2024-04-15T15:14:57.108868+0000","last_fullsized":"2024-04-15T15:14:57.108868+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:19:37.987965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137741+0000","last_change":"2024-04-15T15:14:52.078627+0000","last_active":"2024-04-15T15:14:57.137741+0000","last_peered":"2024-04-15T15:14:57.137741+0000","last_clean":"2024-04-15T15:14:57.137741+0000","last_became_active":"2024-04-15T15:14:52.078333+0000","last_became_peered":"2024-04-15T15:14:52.078333+0000","last_unstale":"2024-04-15T15:14:57.137741+0000","last_undegraded":"2024-04-15T15:14:57.137741+0000","last_fullsized":"2024-04-15T15:14:57.137741+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:50:41.263447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108003+0000","last_change":"2024-04-15T15:14:56.093858+0000","last_active":"2024-04-15T15:14:57.108003+0000","last_peered":"2024-04-15T15:14:57.108003+0000","last_clean":"2024-04-15T15:14:57.108003+0000","last_became_active":"2024-04-15T15:14:56.093481+0000","last_became_peered":"2024-04-15T15:14:56.093481+0000","last_unstale":"2024-04-15T15:14:57.108003+0000","last_undegraded":"2024-04-15T15:14:57.108003+0000","last_fullsized":"2024-04-15T15:14:57.108003+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:48:49.816230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137031+0000","last_change":"2024-04-15T15:14:51.073707+0000","last_active":"2024-04-15T15:14:57.137031+0000","last_peered":"2024-04-15T15:14:57.137031+0000","last_clean":"2024-04-15T15:14:57.137031+0000","last_became_active":"2024-04-15T15:14:51.073451+0000","last_became_peered":"2024-04-15T15:14:51.073451+0000","last_unstale":"2024-04-15T15:14:57.137031+0000","last_undegraded":"2024-04-15T15:14:57.137031+0000","last_fullsized":"2024-04-15T15:14:57.137031+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:01:16.772802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136421+0000","last_change":"2024-04-15T15:14:52.078644+0000","last_active":"2024-04-15T15:14:57.136421+0000","last_peered":"2024-04-15T15:14:57.136421+0000","last_clean":"2024-04-15T15:14:57.136421+0000","last_became_active":"2024-04-15T15:14:52.078177+0000","last_became_peered":"2024-04-15T15:14:52.078177+0000","last_unstale":"2024-04-15T15:14:57.136421+0000","last_undegraded":"2024-04-15T15:14:57.136421+0000","last_fullsized":"2024-04-15T15:14:57.136421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:18:42.403098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136932+0000","last_change":"2024-04-15T15:14:56.092926+0000","last_active":"2024-04-15T15:14:57.136932+0000","last_peered":"2024-04-15T15:14:57.136932+0000","last_clean":"2024-04-15T15:14:57.136932+0000","last_became_active":"2024-04-15T15:14:56.092506+0000","last_became_peered":"2024-04-15T15:14:56.092506+0000","last_unstale":"2024-04-15T15:14:57.136932+0000","last_undegraded":"2024-04-15T15:14:57.136932+0000","last_fullsized":"2024-04-15T15:14:57.136932+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:55:14.590939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108967+0000","last_change":"2024-04-15T15:14:52.082533+0000","last_active":"2024-04-15T15:14:57.108967+0000","last_peered":"2024-04-15T15:14:57.108967+0000","last_clean":"2024-04-15T15:14:57.108967+0000","last_became_active":"2024-04-15T15:14:52.082257+0000","last_became_peered":"2024-04-15T15:14:52.082257+0000","last_unstale":"2024-04-15T15:14:57.108967+0000","last_undegraded":"2024-04-15T15:14:57.108967+0000","last_fullsized":"2024-04-15T15:14:57.108967+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:34:51.775979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.107671+0000","last_change":"2024-04-15T15:14:52.084944+0000","last_active":"2024-04-15T15:14:57.107671+0000","last_peered":"2024-04-15T15:14:57.107671+0000","last_clean":"2024-04-15T15:14:57.107671+0000","last_became_active":"2024-04-15T15:14:52.084665+0000","last_became_peered":"2024-04-15T15:14:52.084665+0000","last_unstale":"2024-04-15T15:14:57.107671+0000","last_undegraded":"2024-04-15T15:14:57.107671+0000","last_fullsized":"2024-04-15T15:14:57.107671+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:58:41.444122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108873+0000","last_change":"2024-04-15T15:14:52.082741+0000","last_active":"2024-04-15T15:14:57.108873+0000","last_peered":"2024-04-15T15:14:57.108873+0000","last_clean":"2024-04-15T15:14:57.108873+0000","last_became_active":"2024-04-15T15:14:52.082494+0000","last_became_peered":"2024-04-15T15:14:52.082494+0000","last_unstale":"2024-04-15T15:14:57.108873+0000","last_undegraded":"2024-04-15T15:14:57.108873+0000","last_fullsized":"2024-04-15T15:14:57.108873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:38:35.185619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137649+0000","last_change":"2024-04-15T15:14:52.076826+0000","last_active":"2024-04-15T15:14:57.137649+0000","last_peered":"2024-04-15T15:14:57.137649+0000","last_clean":"2024-04-15T15:14:57.137649+0000","last_became_active":"2024-04-15T15:14:52.076541+0000","last_became_peered":"2024-04-15T15:14:52.076541+0000","last_unstale":"2024-04-15T15:14:57.137649+0000","last_undegraded":"2024-04-15T15:14:57.137649+0000","last_fullsized":"2024-04-15T15:14:57.137649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:13:01.642838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137024+0000","last_change":"2024-04-15T15:14:52.075468+0000","last_active":"2024-04-15T15:14:57.137024+0000","last_peered":"2024-04-15T15:14:57.137024+0000","last_clean":"2024-04-15T15:14:57.137024+0000","last_became_active":"2024-04-15T15:14:52.075173+0000","last_became_peered":"2024-04-15T15:14:52.075173+0000","last_unstale":"2024-04-15T15:14:57.137024+0000","last_undegraded":"2024-04-15T15:14:57.137024+0000","last_fullsized":"2024-04-15T15:14:57.137024+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:50:41.263447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137555+0000","last_change":"2024-04-15T15:14:52.086866+0000","last_active":"2024-04-15T15:14:57.137555+0000","last_peered":"2024-04-15T15:14:57.137555+0000","last_clean":"2024-04-15T15:14:57.137555+0000","last_became_active":"2024-04-15T15:14:52.086547+0000","last_became_peered":"2024-04-15T15:14:52.086547+0000","last_unstale":"2024-04-15T15:14:57.137555+0000","last_undegraded":"2024-04-15T15:14:57.137555+0000","last_fullsized":"2024-04-15T15:14:57.137555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:37:31.656434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.110000+0000","last_change":"2024-04-15T15:14:52.074917+0000","last_active":"2024-04-15T15:14:57.110000+0000","last_peered":"2024-04-15T15:14:57.110000+0000","last_clean":"2024-04-15T15:14:57.110000+0000","last_became_active":"2024-04-15T15:14:52.074643+0000","last_became_peered":"2024-04-15T15:14:52.074643+0000","last_unstale":"2024-04-15T15:14:57.110000+0000","last_undegraded":"2024-04-15T15:14:57.110000+0000","last_fullsized":"2024-04-15T15:14:57.110000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:12:07.772549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138133+0000","last_change":"2024-04-15T15:14:52.076263+0000","last_active":"2024-04-15T15:14:57.138133+0000","last_peered":"2024-04-15T15:14:57.138133+0000","last_clean":"2024-04-15T15:14:57.138133+0000","last_became_active":"2024-04-15T15:14:52.075961+0000","last_became_peered":"2024-04-15T15:14:52.075961+0000","last_unstale":"2024-04-15T15:14:57.138133+0000","last_undegraded":"2024-04-15T15:14:57.138133+0000","last_fullsized":"2024-04-15T15:14:57.138133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:12:40.420221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108421+0000","last_change":"2024-04-15T15:14:52.074353+0000","last_active":"2024-04-15T15:14:57.108421+0000","last_peered":"2024-04-15T15:14:57.108421+0000","last_clean":"2024-04-15T15:14:57.108421+0000","last_became_active":"2024-04-15T15:14:52.074067+0000","last_became_peered":"2024-04-15T15:14:52.074067+0000","last_unstale":"2024-04-15T15:14:57.108421+0000","last_undegraded":"2024-04-15T15:14:57.108421+0000","last_fullsized":"2024-04-15T15:14:57.108421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:26:45.460787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136083+0000","last_change":"2024-04-15T15:14:52.075541+0000","last_active":"2024-04-15T15:14:57.136083+0000","last_peered":"2024-04-15T15:14:57.136083+0000","last_clean":"2024-04-15T15:14:57.136083+0000","last_became_active":"2024-04-15T15:14:52.075227+0000","last_became_peered":"2024-04-15T15:14:52.075227+0000","last_unstale":"2024-04-15T15:14:57.136083+0000","last_undegraded":"2024-04-15T15:14:57.136083+0000","last_fullsized":"2024-04-15T15:14:57.136083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:24:30.714153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108791+0000","last_change":"2024-04-15T15:14:52.080306+0000","last_active":"2024-04-15T15:14:57.108791+0000","last_peered":"2024-04-15T15:14:57.108791+0000","last_clean":"2024-04-15T15:14:57.108791+0000","last_became_active":"2024-04-15T15:14:52.079784+0000","last_became_peered":"2024-04-15T15:14:52.079784+0000","last_unstale":"2024-04-15T15:14:57.108791+0000","last_undegraded":"2024-04-15T15:14:57.108791+0000","last_fullsized":"2024-04-15T15:14:57.108791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:10:19.696254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137426+0000","last_change":"2024-04-15T15:14:52.081224+0000","last_active":"2024-04-15T15:14:57.137426+0000","last_peered":"2024-04-15T15:14:57.137426+0000","last_clean":"2024-04-15T15:14:57.137426+0000","last_became_active":"2024-04-15T15:14:52.080976+0000","last_became_peered":"2024-04-15T15:14:52.080976+0000","last_unstale":"2024-04-15T15:14:57.137426+0000","last_undegraded":"2024-04-15T15:14:57.137426+0000","last_fullsized":"2024-04-15T15:14:57.137426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:21:50.111549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109464+0000","last_change":"2024-04-15T15:14:52.085463+0000","last_active":"2024-04-15T15:14:57.109464+0000","last_peered":"2024-04-15T15:14:57.109464+0000","last_clean":"2024-04-15T15:14:57.109464+0000","last_became_active":"2024-04-15T15:14:52.085175+0000","last_became_peered":"2024-04-15T15:14:52.085175+0000","last_unstale":"2024-04-15T15:14:57.109464+0000","last_undegraded":"2024-04-15T15:14:57.109464+0000","last_fullsized":"2024-04-15T15:14:57.109464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:50:41.263447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136838+0000","last_change":"2024-04-15T15:14:52.092434+0000","last_active":"2024-04-15T15:14:57.136838+0000","last_peered":"2024-04-15T15:14:57.136838+0000","last_clean":"2024-04-15T15:14:57.136838+0000","last_became_active":"2024-04-15T15:14:52.088973+0000","last_became_peered":"2024-04-15T15:14:52.088973+0000","last_unstale":"2024-04-15T15:14:57.136838+0000","last_undegraded":"2024-04-15T15:14:57.136838+0000","last_fullsized":"2024-04-15T15:14:57.136838+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:03:03.850709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.107875+0000","last_change":"2024-04-15T15:14:52.076802+0000","last_active":"2024-04-15T15:14:57.107875+0000","last_peered":"2024-04-15T15:14:57.107875+0000","last_clean":"2024-04-15T15:14:57.107875+0000","last_became_active":"2024-04-15T15:14:52.076523+0000","last_became_peered":"2024-04-15T15:14:52.076523+0000","last_unstale":"2024-04-15T15:14:57.107875+0000","last_undegraded":"2024-04-15T15:14:57.107875+0000","last_fullsized":"2024-04-15T15:14:57.107875+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:07:49.420517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109362+0000","last_change":"2024-04-15T15:14:52.084418+0000","last_active":"2024-04-15T15:14:57.109362+0000","last_peered":"2024-04-15T15:14:57.109362+0000","last_clean":"2024-04-15T15:14:57.109362+0000","last_became_active":"2024-04-15T15:14:52.083428+0000","last_became_peered":"2024-04-15T15:14:52.083428+0000","last_unstale":"2024-04-15T15:14:57.109362+0000","last_undegraded":"2024-04-15T15:14:57.109362+0000","last_fullsized":"2024-04-15T15:14:57.109362+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:12:59.006568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108279+0000","last_change":"2024-04-15T15:14:52.083894+0000","last_active":"2024-04-15T15:14:57.108279+0000","last_peered":"2024-04-15T15:14:57.108279+0000","last_clean":"2024-04-15T15:14:57.108279+0000","last_became_active":"2024-04-15T15:14:52.083636+0000","last_became_peered":"2024-04-15T15:14:52.083636+0000","last_unstale":"2024-04-15T15:14:57.108279+0000","last_undegraded":"2024-04-15T15:14:57.108279+0000","last_fullsized":"2024-04-15T15:14:57.108279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:12:07.772549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136007+0000","last_change":"2024-04-15T15:14:52.087934+0000","last_active":"2024-04-15T15:14:57.136007+0000","last_peered":"2024-04-15T15:14:57.136007+0000","last_clean":"2024-04-15T15:14:57.136007+0000","last_became_active":"2024-04-15T15:14:52.087040+0000","last_became_peered":"2024-04-15T15:14:52.087040+0000","last_unstale":"2024-04-15T15:14:57.136007+0000","last_undegraded":"2024-04-15T15:14:57.136007+0000","last_fullsized":"2024-04-15T15:14:57.136007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:24:30.714153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138511+0000","last_change":"2024-04-15T15:14:52.079696+0000","last_active":"2024-04-15T15:14:57.138511+0000","last_peered":"2024-04-15T15:14:57.138511+0000","last_clean":"2024-04-15T15:14:57.138511+0000","last_became_active":"2024-04-15T15:14:52.079438+0000","last_became_peered":"2024-04-15T15:14:52.079438+0000","last_unstale":"2024-04-15T15:14:57.138511+0000","last_undegraded":"2024-04-15T15:14:57.138511+0000","last_fullsized":"2024-04-15T15:14:57.138511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:27:02.854265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136589+0000","last_change":"2024-04-15T15:14:52.076841+0000","last_active":"2024-04-15T15:14:57.136589+0000","last_peered":"2024-04-15T15:14:57.136589+0000","last_clean":"2024-04-15T15:14:57.136589+0000","last_became_active":"2024-04-15T15:14:52.076567+0000","last_became_peered":"2024-04-15T15:14:52.076567+0000","last_unstale":"2024-04-15T15:14:57.136589+0000","last_undegraded":"2024-04-15T15:14:57.136589+0000","last_fullsized":"2024-04-15T15:14:57.136589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:51:22.814881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136685+0000","last_change":"2024-04-15T15:14:52.078599+0000","last_active":"2024-04-15T15:14:57.136685+0000","last_peered":"2024-04-15T15:14:57.136685+0000","last_clean":"2024-04-15T15:14:57.136685+0000","last_became_active":"2024-04-15T15:14:52.078231+0000","last_became_peered":"2024-04-15T15:14:52.078231+0000","last_unstale":"2024-04-15T15:14:57.136685+0000","last_undegraded":"2024-04-15T15:14:57.136685+0000","last_fullsized":"2024-04-15T15:14:57.136685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:58:41.444122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109712+0000","last_change":"2024-04-15T15:14:52.080747+0000","last_active":"2024-04-15T15:14:57.109712+0000","last_peered":"2024-04-15T15:14:57.109712+0000","last_clean":"2024-04-15T15:14:57.109712+0000","last_became_active":"2024-04-15T15:14:52.080475+0000","last_became_peered":"2024-04-15T15:14:52.080475+0000","last_unstale":"2024-04-15T15:14:57.109712+0000","last_undegraded":"2024-04-15T15:14:57.109712+0000","last_fullsized":"2024-04-15T15:14:57.109712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:37:31.656434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109996+0000","last_change":"2024-04-15T15:14:52.081405+0000","last_active":"2024-04-15T15:14:57.109996+0000","last_peered":"2024-04-15T15:14:57.109996+0000","last_clean":"2024-04-15T15:14:57.109996+0000","last_became_active":"2024-04-15T15:14:52.081120+0000","last_became_peered":"2024-04-15T15:14:52.081120+0000","last_unstale":"2024-04-15T15:14:57.109996+0000","last_undegraded":"2024-04-15T15:14:57.109996+0000","last_fullsized":"2024-04-15T15:14:57.109996+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:12:40.420221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136834+0000","last_change":"2024-04-15T15:14:52.092337+0000","last_active":"2024-04-15T15:14:57.136834+0000","last_peered":"2024-04-15T15:14:57.136834+0000","last_clean":"2024-04-15T15:14:57.136834+0000","last_became_active":"2024-04-15T15:14:52.091800+0000","last_became_peered":"2024-04-15T15:14:52.091800+0000","last_unstale":"2024-04-15T15:14:57.136834+0000","last_undegraded":"2024-04-15T15:14:57.136834+0000","last_fullsized":"2024-04-15T15:14:57.136834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:12:59.006568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136083+0000","last_change":"2024-04-15T15:14:52.079078+0000","last_active":"2024-04-15T15:14:57.136083+0000","last_peered":"2024-04-15T15:14:57.136083+0000","last_clean":"2024-04-15T15:14:57.136083+0000","last_became_active":"2024-04-15T15:14:52.078796+0000","last_became_peered":"2024-04-15T15:14:52.078796+0000","last_unstale":"2024-04-15T15:14:57.136083+0000","last_undegraded":"2024-04-15T15:14:57.136083+0000","last_fullsized":"2024-04-15T15:14:57.136083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:34:51.775979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135766+0000","last_change":"2024-04-15T15:14:52.091742+0000","last_active":"2024-04-15T15:14:57.135766+0000","last_peered":"2024-04-15T15:14:57.135766+0000","last_clean":"2024-04-15T15:14:57.135766+0000","last_became_active":"2024-04-15T15:14:52.087579+0000","last_became_peered":"2024-04-15T15:14:52.087579+0000","last_unstale":"2024-04-15T15:14:57.135766+0000","last_undegraded":"2024-04-15T15:14:57.135766+0000","last_fullsized":"2024-04-15T15:14:57.135766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:51:22.814881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108707+0000","last_change":"2024-04-15T15:14:52.084878+0000","last_active":"2024-04-15T15:14:57.108707+0000","last_peered":"2024-04-15T15:14:57.108707+0000","last_clean":"2024-04-15T15:14:57.108707+0000","last_became_active":"2024-04-15T15:14:52.084602+0000","last_became_peered":"2024-04-15T15:14:52.084602+0000","last_unstale":"2024-04-15T15:14:57.108707+0000","last_undegraded":"2024-04-15T15:14:57.108707+0000","last_fullsized":"2024-04-15T15:14:57.108707+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:51:22.814881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136011+0000","last_change":"2024-04-15T15:14:52.085909+0000","last_active":"2024-04-15T15:14:57.136011+0000","last_peered":"2024-04-15T15:14:57.136011+0000","last_clean":"2024-04-15T15:14:57.136011+0000","last_became_active":"2024-04-15T15:14:52.085266+0000","last_became_peered":"2024-04-15T15:14:52.085266+0000","last_unstale":"2024-04-15T15:14:57.136011+0000","last_undegraded":"2024-04-15T15:14:57.136011+0000","last_fullsized":"2024-04-15T15:14:57.136011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:26:45.460787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109544+0000","last_change":"2024-04-15T15:14:52.084086+0000","last_active":"2024-04-15T15:14:57.109544+0000","last_peered":"2024-04-15T15:14:57.109544+0000","last_clean":"2024-04-15T15:14:57.109544+0000","last_became_active":"2024-04-15T15:14:52.083193+0000","last_became_peered":"2024-04-15T15:14:52.083193+0000","last_unstale":"2024-04-15T15:14:57.109544+0000","last_undegraded":"2024-04-15T15:14:57.109544+0000","last_fullsized":"2024-04-15T15:14:57.109544+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:12:40.420221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136555+0000","last_change":"2024-04-15T15:14:52.072448+0000","last_active":"2024-04-15T15:14:57.136555+0000","last_peered":"2024-04-15T15:14:57.136555+0000","last_clean":"2024-04-15T15:14:57.136555+0000","last_became_active":"2024-04-15T15:14:52.072162+0000","last_became_peered":"2024-04-15T15:14:52.072162+0000","last_unstale":"2024-04-15T15:14:57.136555+0000","last_undegraded":"2024-04-15T15:14:57.136555+0000","last_fullsized":"2024-04-15T15:14:57.136555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:24:30.714153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137467+0000","last_change":"2024-04-15T15:14:52.087484+0000","last_active":"2024-04-15T15:14:57.137467+0000","last_peered":"2024-04-15T15:14:57.137467+0000","last_clean":"2024-04-15T15:14:57.137467+0000","last_became_active":"2024-04-15T15:14:52.087195+0000","last_became_peered":"2024-04-15T15:14:52.087195+0000","last_unstale":"2024-04-15T15:14:57.137467+0000","last_undegraded":"2024-04-15T15:14:57.137467+0000","last_fullsized":"2024-04-15T15:14:57.137467+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:50:41.263447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136721+0000","last_change":"2024-04-15T15:14:52.083998+0000","last_active":"2024-04-15T15:14:57.136721+0000","last_peered":"2024-04-15T15:14:57.136721+0000","last_clean":"2024-04-15T15:14:57.136721+0000","last_became_active":"2024-04-15T15:14:52.083702+0000","last_became_peered":"2024-04-15T15:14:52.083702+0000","last_unstale":"2024-04-15T15:14:57.136721+0000","last_undegraded":"2024-04-15T15:14:57.136721+0000","last_fullsized":"2024-04-15T15:14:57.136721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:21:48.470063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4499,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":103957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":103957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:15:09.472 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-15T15:15:09.473 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-15T15:15:09.473 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-15T15:15:09.473 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-15T15:15:09.473 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-15T15:15:09.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.609+0000 7f4122bd6640 1 Processor -- start 2024-04-15T15:15:09.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.609+0000 7f4122bd6640 1 -- start start 2024-04-15T15:15:09.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.610+0000 7f4122bd6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f411c161fb0 0x7f411c162450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:09.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.610+0000 7f4122bd6640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f411c0c9570 0x7f411c162990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:09.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.610+0000 7f4122bd6640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f411c0ccd40 con 0x7f411c0d12f0 2024-04-15T15:15:09.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.610+0000 7f4122bd6640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f411c0ccaf0 con 0x7f411c161fb0 2024-04-15T15:15:09.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.610+0000 7f4122bd6640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f411c0cd020 con 0x7f411c0c9570 2024-04-15T15:15:09.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.610+0000 7f411bfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f411c161fb0 0x7f411c162450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:09.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.610+0000 7f411bfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f411c161fb0 0x7f411c162450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44584/0 (socket says 172.21.15.72:44584) 2024-04-15T15:15:09.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.610+0000 7f411bfff640 1 -- 172.21.15.72:0/3468457017 learned_addr learned my addr 172.21.15.72:0/3468457017 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:09.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.610+0000 7f4121173640 1 --2- 172.21.15.72:0/3468457017 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f411c0c9570 0x7f411c162990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:09.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.610+0000 7f411bfff640 1 -- 172.21.15.72:0/3468457017 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f411c0c9570 msgr2=0x7f411c162990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:09.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.610+0000 7f411bfff640 1 --2- 172.21.15.72:0/3468457017 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f411c0c9570 0x7f411c162990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.610+0000 7f411bfff640 1 -- 172.21.15.72:0/3468457017 >> v1:172.21.15.175:6789/0 conn(0x7f411c0d12f0 legacy=0x7f411c1618a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:09.613 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.610+0000 7f411bfff640 1 -- 172.21.15.72:0/3468457017 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f411c0cd300 con 0x7f411c161fb0 2024-04-15T15:15:09.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.611+0000 7f411bfff640 1 --2- 172.21.15.72:0/3468457017 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f411c161fb0 0x7f411c162450 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f41100098a0 tx=0x7f411002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b234fc9ea0bd911 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:09.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.611+0000 7f411b7fe640 1 -- 172.21.15.72:0/3468457017 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4110034070 con 0x7f411c161fb0 2024-04-15T15:15:09.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.611+0000 7f411b7fe640 1 -- 172.21.15.72:0/3468457017 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4110030840 con 0x7f411c161fb0 2024-04-15T15:15:09.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.611+0000 7f411b7fe640 1 -- 172.21.15.72:0/3468457017 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4110030c10 con 0x7f411c161fb0 2024-04-15T15:15:09.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.612+0000 7f4122bd6640 1 -- 172.21.15.72:0/3468457017 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f411c161fb0 msgr2=0x7f411c162450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:09.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.612+0000 7f4122bd6640 1 --2- 172.21.15.72:0/3468457017 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f411c161fb0 0x7f411c162450 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f41100098a0 tx=0x7f411002b980 comp rx=0 tx=0).stop 2024-04-15T15:15:09.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.612+0000 7f4122bd6640 1 -- 172.21.15.72:0/3468457017 shutdown_connections 2024-04-15T15:15:09.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.612+0000 7f4122bd6640 1 --2- 172.21.15.72:0/3468457017 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f411c0c9570 0x7f411c162990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.612+0000 7f4122bd6640 1 --2- 172.21.15.72:0/3468457017 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f411c161fb0 0x7f411c162450 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.612+0000 7f4122bd6640 1 -- 172.21.15.72:0/3468457017 >> 172.21.15.72:0/3468457017 conn(0x7f411c0b3f90 msgr2=0x7f411c0b43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:09.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.612+0000 7f4122bd6640 1 -- 172.21.15.72:0/3468457017 shutdown_connections 2024-04-15T15:15:09.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.612+0000 7f4122bd6640 1 -- 172.21.15.72:0/3468457017 wait complete. 2024-04-15T15:15:09.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.613+0000 7f4122bd6640 1 Processor -- start 2024-04-15T15:15:09.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.613+0000 7f4122bd6640 1 -- start start 2024-04-15T15:15:09.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.614+0000 7f4122bd6640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f411c0c9570 0x7f411c0c7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:09.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.614+0000 7f4122bd6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f411c0d12f0 0x7f411c15a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:09.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.614+0000 7f4122bd6640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f411c161fb0 0x7f411c15df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:09.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.614+0000 7f4122bd6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f411c0d2210 con 0x7f411c0d12f0 2024-04-15T15:15:09.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.614+0000 7f4122bd6640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f411c0d2090 con 0x7f411c161fb0 2024-04-15T15:15:09.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.614+0000 7f4122bd6640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f411c0b68e0 con 0x7f411c0c9570 2024-04-15T15:15:09.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.614+0000 7f411bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f411c0d12f0 0x7f411c15a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:09.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.614+0000 7f411bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f411c0d12f0 0x7f411c15a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44598/0 (socket says 172.21.15.72:44598) 2024-04-15T15:15:09.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.614+0000 7f411bfff640 1 -- 172.21.15.72:0/1632567940 learned_addr learned my addr 172.21.15.72:0/1632567940 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:09.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.614+0000 7f4121173640 1 --2- 172.21.15.72:0/1632567940 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f411c161fb0 0x7f411c15df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:09.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.614+0000 7f4120972640 1 --2- 172.21.15.72:0/1632567940 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f411c0c9570 0x7f411c0c7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:09.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.614+0000 7f411bfff640 1 -- 172.21.15.72:0/1632567940 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f411c0c9570 msgr2=0x7f411c0c7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:09.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.614+0000 7f411bfff640 1 --2- 172.21.15.72:0/1632567940 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f411c0c9570 0x7f411c0c7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.614+0000 7f411bfff640 1 -- 172.21.15.72:0/1632567940 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f411c161fb0 msgr2=0x7f411c15df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:09.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.614+0000 7f411bfff640 1 --2- 172.21.15.72:0/1632567940 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f411c161fb0 0x7f411c15df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.614+0000 7f411bfff640 1 -- 172.21.15.72:0/1632567940 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f411c1f8320 con 0x7f411c0d12f0 2024-04-15T15:15:09.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.615+0000 7f411bfff640 1 --2- 172.21.15.72:0/1632567940 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f411c0d12f0 0x7f411c15a510 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f41100099c0 tx=0x7f41100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:09.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.615+0000 7f41197fa640 1 -- 172.21.15.72:0/1632567940 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4110034040 con 0x7f411c0d12f0 2024-04-15T15:15:09.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.615+0000 7f41197fa640 1 -- 172.21.15.72:0/1632567940 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f411003ada0 con 0x7f411c0d12f0 2024-04-15T15:15:09.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.615+0000 7f41197fa640 1 -- 172.21.15.72:0/1632567940 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4110044630 con 0x7f411c0d12f0 2024-04-15T15:15:09.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.615+0000 7f4122bd6640 1 -- 172.21.15.72:0/1632567940 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f411c1f90a0 con 0x7f411c0d12f0 2024-04-15T15:15:09.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.615+0000 7f4122bd6640 1 -- 172.21.15.72:0/1632567940 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f411c1f85c0 con 0x7f411c0d12f0 2024-04-15T15:15:09.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.616+0000 7f4122bd6640 1 -- 172.21.15.72:0/1632567940 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f411c0b1d80 con 0x7f411c0d12f0 2024-04-15T15:15:09.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.616+0000 7f41197fa640 1 -- 172.21.15.72:0/1632567940 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4110037070 con 0x7f411c0d12f0 2024-04-15T15:15:09.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.617+0000 7f41197fa640 1 --2- 172.21.15.72:0/1632567940 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f410806c8f0 0x7f410806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:09.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.617+0000 7f41197fa640 1 -- 172.21.15.72:0/1632567940 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f41100addd0 con 0x7f411c0d12f0 2024-04-15T15:15:09.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.617+0000 7f4120972640 1 --2- 172.21.15.72:0/1632567940 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f410806c8f0 0x7f410806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:09.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.617+0000 7f4120972640 1 --2- 172.21.15.72:0/1632567940 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f410806c8f0 0x7f410806edb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f410c001780 tx=0x7f410c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:09.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.621+0000 7f41197fa640 1 -- 172.21.15.72:0/1632567940 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f411003f070 con 0x7f411c0d12f0 2024-04-15T15:15:09.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.843+0000 7f4122bd6640 1 -- 172.21.15.72:0/1632567940 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f411c0689e0 con 0x7f411c0d12f0 2024-04-15T15:15:09.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.846+0000 7f41197fa640 1 -- 172.21.15.72:0/1632567940 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+174137 (secure 0 0 0) 0x7f41100af090 con 0x7f411c0d12f0 2024-04-15T15:15:09.849 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:09.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.848+0000 7f4122bd6640 1 -- 172.21.15.72:0/1632567940 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f410806c8f0 msgr2=0x7f410806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:09.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.848+0000 7f4122bd6640 1 --2- 172.21.15.72:0/1632567940 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f410806c8f0 0x7f410806edb0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f410c001780 tx=0x7f410c009040 comp rx=0 tx=0).stop 2024-04-15T15:15:09.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.848+0000 7f4122bd6640 1 -- 172.21.15.72:0/1632567940 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f411c0d12f0 msgr2=0x7f411c15a510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:09.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.848+0000 7f4122bd6640 1 --2- 172.21.15.72:0/1632567940 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f411c0d12f0 0x7f411c15a510 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f41100099c0 tx=0x7f41100046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.849+0000 7f4122bd6640 1 -- 172.21.15.72:0/1632567940 shutdown_connections 2024-04-15T15:15:09.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.849+0000 7f4122bd6640 1 --2- 172.21.15.72:0/1632567940 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f410806c8f0 0x7f410806edb0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.849+0000 7f4122bd6640 1 --2- 172.21.15.72:0/1632567940 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f411c161fb0 0x7f411c15df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.849+0000 7f4122bd6640 1 --2- 172.21.15.72:0/1632567940 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f411c0d12f0 0x7f411c15a510 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.849+0000 7f4122bd6640 1 --2- 172.21.15.72:0/1632567940 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f411c0c9570 0x7f411c0c7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.849+0000 7f4122bd6640 1 -- 172.21.15.72:0/1632567940 >> 172.21.15.72:0/1632567940 conn(0x7f411c0b3f90 msgr2=0x7f411c0c9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:09.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.849+0000 7f4122bd6640 1 -- 172.21.15.72:0/1632567940 shutdown_connections 2024-04-15T15:15:09.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.849+0000 7f4122bd6640 1 -- 172.21.15.72:0/1632567940 wait complete. 2024-04-15T15:15:09.861 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6832","nonce":3503807805},{"type":"v1","addr":"172.21.15.72:6833","nonce":3503807805}]},"active_addr":"172.21.15.72:6833/3503807805","active_change":"2024-04-15T15:14:31.335588+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4109,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.72:0","nonce":3967569694}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.72:0","nonce":3431639099}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.72:0","nonce":4014454964}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.72:0","nonce":3952858116}]}]} 2024-04-15T15:15:09.864 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-15T15:15:09.864 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-15T15:15:09.864 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:09.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.967+0000 7f27367d4640 1 Processor -- start 2024-04-15T15:15:09.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.967+0000 7f27367d4640 1 -- start start 2024-04-15T15:15:09.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.968+0000 7f27367d4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f273015e7e0 0x7f27300b67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:09.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.968+0000 7f27367d4640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f27300ae360 0x7f2730072b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:09.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.968+0000 7f27367d4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2730162800 con 0x7f27300730c0 2024-04-15T15:15:09.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.968+0000 7f27367d4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f2730073530 con 0x7f273015e7e0 2024-04-15T15:15:09.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.968+0000 7f27367d4640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f2730162ae0 con 0x7f27300ae360 2024-04-15T15:15:09.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.968+0000 7f272ffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f273015e7e0 0x7f27300b67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:09.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.968+0000 7f272ffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f273015e7e0 0x7f27300b67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44608/0 (socket says 172.21.15.72:44608) 2024-04-15T15:15:09.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.968+0000 7f2734d71640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f27300730c0 0x7f27300cf180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45080/0 (socket says 172.21.15.72:45080) 2024-04-15T15:15:09.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.968+0000 7f272ffff640 1 -- 172.21.15.72:0/2827969893 learned_addr learned my addr 172.21.15.72:0/2827969893 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:09.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.968+0000 7f2727fff640 1 --2- 172.21.15.72:0/2827969893 >> v2:172.21.15.175:3300/0 conn(0x7f27300ae360 0x7f2730072b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:09.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.969+0000 7f272ffff640 1 -- 172.21.15.72:0/2827969893 >> v2:172.21.15.175:3300/0 conn(0x7f27300ae360 msgr2=0x7f2730072b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:09.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.969+0000 7f272ffff640 1 --2- 172.21.15.72:0/2827969893 >> v2:172.21.15.175:3300/0 conn(0x7f27300ae360 0x7f2730072b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.969+0000 7f272ffff640 1 -- 172.21.15.72:0/2827969893 >> v1:172.21.15.72:6789/0 conn(0x7f27300730c0 legacy=0x7f27300cf180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:09.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.969+0000 7f272f7fe640 1 -- 172.21.15.72:0/2827969893 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2468503651 0 0) 0x7f2730162800 con 0x7f27300730c0 2024-04-15T15:15:09.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.969+0000 7f272ffff640 1 -- 172.21.15.72:0/2827969893 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2730162dc0 con 0x7f273015e7e0 2024-04-15T15:15:09.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.969+0000 7f272ffff640 1 --2- 172.21.15.72:0/2827969893 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f273015e7e0 0x7f27300b67b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f27300addf0 tx=0x7f272002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bafbbfd7dff3c96d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:09.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.969+0000 7f272f7fe640 1 -- 172.21.15.72:0/2827969893 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2720034070 con 0x7f273015e7e0 2024-04-15T15:15:09.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.969+0000 7f272f7fe640 1 -- 172.21.15.72:0/2827969893 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2720030860 con 0x7f273015e7e0 2024-04-15T15:15:09.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.969+0000 7f272f7fe640 1 -- 172.21.15.72:0/2827969893 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2720030c30 con 0x7f273015e7e0 2024-04-15T15:15:09.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.970+0000 7f27367d4640 1 -- 172.21.15.72:0/2827969893 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f273015e7e0 msgr2=0x7f27300b67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:09.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.970+0000 7f27367d4640 1 --2- 172.21.15.72:0/2827969893 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f273015e7e0 0x7f27300b67b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f27300addf0 tx=0x7f272002b9a0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.970+0000 7f27367d4640 1 -- 172.21.15.72:0/2827969893 shutdown_connections 2024-04-15T15:15:09.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.970+0000 7f27367d4640 1 --2- 172.21.15.72:0/2827969893 >> v2:172.21.15.175:3300/0 conn(0x7f27300ae360 0x7f2730072b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.970+0000 7f27367d4640 1 --2- 172.21.15.72:0/2827969893 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f273015e7e0 0x7f27300b67b0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.970+0000 7f27367d4640 1 -- 172.21.15.72:0/2827969893 >> 172.21.15.72:0/2827969893 conn(0x7f27300b31b0 msgr2=0x7f27300b3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:09.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.970+0000 7f27367d4640 1 -- 172.21.15.72:0/2827969893 shutdown_connections 2024-04-15T15:15:09.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.970+0000 7f27367d4640 1 -- 172.21.15.72:0/2827969893 wait complete. 2024-04-15T15:15:09.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.971+0000 7f27367d4640 1 Processor -- start 2024-04-15T15:15:09.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.971+0000 7f27367d4640 1 -- start start 2024-04-15T15:15:09.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.972+0000 7f27367d4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27300730c0 0x7f273015d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:09.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.972+0000 7f27367d4640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27300ae360 0x7f27301e90e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:09.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.972+0000 7f27367d4640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f273015e7e0 0x7f27301e9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:09.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.972+0000 7f27367d4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f27301623d0 con 0x7f27300730c0 2024-04-15T15:15:09.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.972+0000 7f27367d4640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f273015ebc0 con 0x7f27300ae360 2024-04-15T15:15:09.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.972+0000 7f27367d4640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f2730162550 con 0x7f273015e7e0 2024-04-15T15:15:09.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.972+0000 7f272ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27300730c0 0x7f273015d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:09.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.972+0000 7f272ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27300730c0 0x7f273015d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44614/0 (socket says 172.21.15.72:44614) 2024-04-15T15:15:09.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.972+0000 7f272ffff640 1 -- 172.21.15.72:0/195109148 learned_addr learned my addr 172.21.15.72:0/195109148 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:09.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.972+0000 7f2734d71640 1 --2- 172.21.15.72:0/195109148 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f273015e7e0 0x7f27301e9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:09.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.972+0000 7f2727fff640 1 --2- 172.21.15.72:0/195109148 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27300ae360 0x7f27301e90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:09.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.972+0000 7f272ffff640 1 -- 172.21.15.72:0/195109148 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f273015e7e0 msgr2=0x7f27301e9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:09.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.972+0000 7f272ffff640 1 --2- 172.21.15.72:0/195109148 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f273015e7e0 0x7f27301e9620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.972+0000 7f272ffff640 1 -- 172.21.15.72:0/195109148 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27300ae360 msgr2=0x7f27301e90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:09.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.972+0000 7f272ffff640 1 --2- 172.21.15.72:0/195109148 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27300ae360 0x7f27301e90e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.972+0000 7f272ffff640 1 -- 172.21.15.72:0/195109148 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f273015def0 con 0x7f27300730c0 2024-04-15T15:15:09.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.972+0000 7f2734d71640 1 --2- 172.21.15.72:0/195109148 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f273015e7e0 0x7f27301e9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:09.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.972+0000 7f2727fff640 1 --2- 172.21.15.72:0/195109148 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27300ae360 0x7f27301e90e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:09.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.973+0000 7f272ffff640 1 --2- 172.21.15.72:0/195109148 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27300730c0 0x7f273015d8a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f272002b9e0 tx=0x7f27200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:09.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.973+0000 7f272d7fa640 1 -- 172.21.15.72:0/195109148 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2720034040 con 0x7f27300730c0 2024-04-15T15:15:09.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.973+0000 7f272d7fa640 1 -- 172.21.15.72:0/195109148 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2720039d90 con 0x7f27300730c0 2024-04-15T15:15:09.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.973+0000 7f272d7fa640 1 -- 172.21.15.72:0/195109148 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2720043610 con 0x7f27300730c0 2024-04-15T15:15:09.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.973+0000 7f27367d4640 1 -- 172.21.15.72:0/195109148 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f273015af40 con 0x7f27300730c0 2024-04-15T15:15:09.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.973+0000 7f27367d4640 1 -- 172.21.15.72:0/195109148 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f273015b500 con 0x7f27300730c0 2024-04-15T15:15:09.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.974+0000 7f27367d4640 1 -- 172.21.15.72:0/195109148 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27300689e0 con 0x7f27300730c0 2024-04-15T15:15:09.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.974+0000 7f272d7fa640 1 -- 172.21.15.72:0/195109148 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2720037070 con 0x7f27300730c0 2024-04-15T15:15:09.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.975+0000 7f272d7fa640 1 --2- 172.21.15.72:0/195109148 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f271006c920 0x7f271006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:09.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.975+0000 7f272d7fa640 1 -- 172.21.15.72:0/195109148 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f27200adfd0 con 0x7f27300730c0 2024-04-15T15:15:09.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.975+0000 7f2727fff640 1 --2- 172.21.15.72:0/195109148 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f271006c920 0x7f271006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:09.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.975+0000 7f2727fff640 1 --2- 172.21.15.72:0/195109148 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f271006c920 0x7f271006ede0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2730161600 tx=0x7f2718008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:09.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:09.979+0000 7f272d7fa640 1 -- 172.21.15.72:0/195109148 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f272003e070 con 0x7f27300730c0 2024-04-15T15:15:10.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.156+0000 7f27367d4640 1 -- 172.21.15.72:0/195109148 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f27300d0690 con 0x7f27300730c0 2024-04-15T15:15:10.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.157+0000 7f272d7fa640 1 -- 172.21.15.72:0/195109148 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7f272007c0c0 con 0x7f27300730c0 2024-04-15T15:15:10.159 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:10.159 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":24,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:14:56.451113+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:14:49.438366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:14:50.368379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:14:54.081157+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:10.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.159+0000 7f27367d4640 1 -- 172.21.15.72:0/195109148 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f271006c920 msgr2=0x7f271006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.159+0000 7f27367d4640 1 --2- 172.21.15.72:0/195109148 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f271006c920 0x7f271006ede0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2730161600 tx=0x7f2718008040 comp rx=0 tx=0).stop 2024-04-15T15:15:10.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.159+0000 7f27367d4640 1 -- 172.21.15.72:0/195109148 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27300730c0 msgr2=0x7f273015d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.159+0000 7f27367d4640 1 --2- 172.21.15.72:0/195109148 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27300730c0 0x7f273015d8a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f272002b9e0 tx=0x7f27200046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.159+0000 7f27367d4640 1 -- 172.21.15.72:0/195109148 shutdown_connections 2024-04-15T15:15:10.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.159+0000 7f27367d4640 1 --2- 172.21.15.72:0/195109148 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f271006c920 0x7f271006ede0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.159+0000 7f27367d4640 1 --2- 172.21.15.72:0/195109148 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f273015e7e0 0x7f27301e9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.159+0000 7f27367d4640 1 --2- 172.21.15.72:0/195109148 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f27300ae360 0x7f27301e90e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.159+0000 7f27367d4640 1 --2- 172.21.15.72:0/195109148 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f27300730c0 0x7f273015d8a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.159+0000 7f27367d4640 1 -- 172.21.15.72:0/195109148 >> 172.21.15.72:0/195109148 conn(0x7f27300b31b0 msgr2=0x7f273006dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:10.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.159+0000 7f27367d4640 1 -- 172.21.15.72:0/195109148 shutdown_connections 2024-04-15T15:15:10.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.159+0000 7f27367d4640 1 -- 172.21.15.72:0/195109148 wait complete. 2024-04-15T15:15:10.172 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-15T15:15:10.172 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:10.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.273+0000 7fa98733d640 1 Processor -- start 2024-04-15T15:15:10.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.273+0000 7fa98733d640 1 -- start start 2024-04-15T15:15:10.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.274+0000 7fa98733d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa98015e640 0x7fa9800cf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.274+0000 7fa98733d640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa9800cf7f0 0x7fa9800ccdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.274+0000 7fa98733d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9801623d0 con 0x7fa9800cd300 2024-04-15T15:15:10.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.274+0000 7fa98733d640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fa980162fa0 con 0x7fa98015e640 2024-04-15T15:15:10.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.274+0000 7fa98733d640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fa9800cfc30 con 0x7fa9800cf7f0 2024-04-15T15:15:10.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.274+0000 7fa9850d9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa98015e640 0x7fa9800cf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.274+0000 7fa9850d9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa98015e640 0x7fa9800cf2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44620/0 (socket says 172.21.15.72:44620) 2024-04-15T15:15:10.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.274+0000 7fa9850d9640 1 -- 172.21.15.72:0/1306990757 learned_addr learned my addr 172.21.15.72:0/1306990757 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:10.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.274+0000 7fa9858da640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fa9800cd300 0x7fa9800c7210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45088/0 (socket says 172.21.15.72:45088) 2024-04-15T15:15:10.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.274+0000 7fa9850d9640 1 -- 172.21.15.72:0/1306990757 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa9800cf7f0 msgr2=0x7fa9800ccdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:10.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.274+0000 7fa9850d9640 1 --2- 172.21.15.72:0/1306990757 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa9800cf7f0 0x7fa9800ccdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.275+0000 7fa9850d9640 1 -- 172.21.15.72:0/1306990757 >> v1:172.21.15.72:6789/0 conn(0x7fa9800cd300 legacy=0x7fa9800c7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.275+0000 7fa9850d9640 1 -- 172.21.15.72:0/1306990757 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa980162670 con 0x7fa98015e640 2024-04-15T15:15:10.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.275+0000 7fa977fff640 1 -- 172.21.15.72:0/1306990757 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4079150889 0 0) 0x7fa9801623d0 con 0x7fa9800cd300 2024-04-15T15:15:10.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.275+0000 7fa9850d9640 1 --2- 172.21.15.72:0/1306990757 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa98015e640 0x7fa9800cf2b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa97c009e50 tx=0x7fa97c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0404a28588bf1c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.275+0000 7fa977fff640 1 -- 172.21.15.72:0/1306990757 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa97c02cb00 con 0x7fa98015e640 2024-04-15T15:15:10.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.275+0000 7fa977fff640 1 -- 172.21.15.72:0/1306990757 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa97c02cca0 con 0x7fa98015e640 2024-04-15T15:15:10.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.275+0000 7fa977fff640 1 -- 172.21.15.72:0/1306990757 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa97c034610 con 0x7fa98015e640 2024-04-15T15:15:10.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.276+0000 7fa98733d640 1 -- 172.21.15.72:0/1306990757 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa98015e640 msgr2=0x7fa9800cf2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.276+0000 7fa98733d640 1 --2- 172.21.15.72:0/1306990757 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa98015e640 0x7fa9800cf2b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa97c009e50 tx=0x7fa97c02bf60 comp rx=0 tx=0).stop 2024-04-15T15:15:10.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.276+0000 7fa98733d640 1 -- 172.21.15.72:0/1306990757 shutdown_connections 2024-04-15T15:15:10.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.276+0000 7fa98733d640 1 --2- 172.21.15.72:0/1306990757 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa9800cf7f0 0x7fa9800ccdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.276+0000 7fa98733d640 1 --2- 172.21.15.72:0/1306990757 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa98015e640 0x7fa9800cf2b0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.276+0000 7fa98733d640 1 -- 172.21.15.72:0/1306990757 >> 172.21.15.72:0/1306990757 conn(0x7fa9800b43c0 msgr2=0x7fa9800b47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:10.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.276+0000 7fa98733d640 1 -- 172.21.15.72:0/1306990757 shutdown_connections 2024-04-15T15:15:10.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.276+0000 7fa98733d640 1 -- 172.21.15.72:0/1306990757 wait complete. 2024-04-15T15:15:10.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.277+0000 7fa98733d640 1 Processor -- start 2024-04-15T15:15:10.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.277+0000 7fa98733d640 1 -- start start 2024-04-15T15:15:10.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.278+0000 7fa98733d640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa9800cd300 0x7fa9800ca680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.278+0000 7fa98733d640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa9800cf7f0 0x7fa98014e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.278+0000 7fa98733d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa98015e640 0x7fa9800cbd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.278+0000 7fa98733d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fa98015ea20 con 0x7fa98015e640 2024-04-15T15:15:10.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.278+0000 7fa98733d640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fa980162c30 con 0x7fa9800cd300 2024-04-15T15:15:10.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.278+0000 7fa98733d640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fa9800cd790 con 0x7fa9800cf7f0 2024-04-15T15:15:10.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.278+0000 7fa9858da640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa98015e640 0x7fa9800cbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.278+0000 7fa9858da640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa98015e640 0x7fa9800cbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44624/0 (socket says 172.21.15.72:44624) 2024-04-15T15:15:10.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.278+0000 7fa9858da640 1 -- 172.21.15.72:0/1154712134 learned_addr learned my addr 172.21.15.72:0/1154712134 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:10.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.278+0000 7fa9850d9640 1 --2- 172.21.15.72:0/1154712134 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa9800cd300 0x7fa9800ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.278+0000 7fa9848d8640 1 --2- 172.21.15.72:0/1154712134 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa9800cf7f0 0x7fa98014e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.278+0000 7fa9858da640 1 -- 172.21.15.72:0/1154712134 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa9800cf7f0 msgr2=0x7fa98014e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.278+0000 7fa9858da640 1 --2- 172.21.15.72:0/1154712134 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa9800cf7f0 0x7fa98014e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.278+0000 7fa9858da640 1 -- 172.21.15.72:0/1154712134 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa9800cd300 msgr2=0x7fa9800ca680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.278+0000 7fa9858da640 1 --2- 172.21.15.72:0/1154712134 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa9800cd300 0x7fa9800ca680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.278+0000 7fa9858da640 1 -- 172.21.15.72:0/1154712134 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9800cc290 con 0x7fa98015e640 2024-04-15T15:15:10.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.279+0000 7fa9858da640 1 --2- 172.21.15.72:0/1154712134 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa98015e640 0x7fa9800cbd50 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa97800c480 tx=0x7fa97800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.279+0000 7fa975ffb640 1 -- 172.21.15.72:0/1154712134 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9780074e0 con 0x7fa98015e640 2024-04-15T15:15:10.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.279+0000 7fa975ffb640 1 -- 172.21.15.72:0/1154712134 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9780047e0 con 0x7fa98015e640 2024-04-15T15:15:10.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.279+0000 7fa98733d640 1 -- 172.21.15.72:0/1154712134 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9800cacb0 con 0x7fa98015e640 2024-04-15T15:15:10.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.279+0000 7fa975ffb640 1 -- 172.21.15.72:0/1154712134 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa978004bd0 con 0x7fa98015e640 2024-04-15T15:15:10.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.279+0000 7fa98733d640 1 -- 172.21.15.72:0/1154712134 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9800cb270 con 0x7fa98015e640 2024-04-15T15:15:10.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.280+0000 7fa98733d640 1 -- 172.21.15.72:0/1154712134 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9800689e0 con 0x7fa98015e640 2024-04-15T15:15:10.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.280+0000 7fa975ffb640 1 -- 172.21.15.72:0/1154712134 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa978011070 con 0x7fa98015e640 2024-04-15T15:15:10.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.280+0000 7fa975ffb640 1 --2- 172.21.15.72:0/1154712134 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa96406c8f0 0x7fa96406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.281+0000 7fa975ffb640 1 -- 172.21.15.72:0/1154712134 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa97808b9d0 con 0x7fa98015e640 2024-04-15T15:15:10.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.281+0000 7fa9850d9640 1 --2- 172.21.15.72:0/1154712134 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa96406c8f0 0x7fa96406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.281+0000 7fa9850d9640 1 --2- 172.21.15.72:0/1154712134 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa96406c8f0 0x7fa96406edb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa97c035000 tx=0x7fa97c030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.285+0000 7fa975ffb640 1 -- 172.21.15.72:0/1154712134 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa978059b80 con 0x7fa98015e640 2024-04-15T15:15:10.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.460+0000 7fa98733d640 1 -- 172.21.15.72:0/1154712134 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa9800b35f0 con 0x7fa98015e640 2024-04-15T15:15:10.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.461+0000 7fa975ffb640 1 -- 172.21.15.72:0/1154712134 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7fa978059480 con 0x7fa98015e640 2024-04-15T15:15:10.463 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:10.463 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":24,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:14:56.451113+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:14:49.438366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:14:50.368379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:14:54.081157+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:10.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.463+0000 7fa98733d640 1 -- 172.21.15.72:0/1154712134 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa96406c8f0 msgr2=0x7fa96406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.463+0000 7fa98733d640 1 --2- 172.21.15.72:0/1154712134 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa96406c8f0 0x7fa96406edb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa97c035000 tx=0x7fa97c030000 comp rx=0 tx=0).stop 2024-04-15T15:15:10.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.463+0000 7fa98733d640 1 -- 172.21.15.72:0/1154712134 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa98015e640 msgr2=0x7fa9800cbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.463+0000 7fa98733d640 1 --2- 172.21.15.72:0/1154712134 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa98015e640 0x7fa9800cbd50 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa97800c480 tx=0x7fa97800c880 comp rx=0 tx=0).stop 2024-04-15T15:15:10.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.463+0000 7fa98733d640 1 -- 172.21.15.72:0/1154712134 shutdown_connections 2024-04-15T15:15:10.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.463+0000 7fa98733d640 1 --2- 172.21.15.72:0/1154712134 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa96406c8f0 0x7fa96406edb0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.463+0000 7fa98733d640 1 --2- 172.21.15.72:0/1154712134 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa98015e640 0x7fa9800cbd50 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.463+0000 7fa98733d640 1 --2- 172.21.15.72:0/1154712134 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa9800cf7f0 0x7fa98014e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.463+0000 7fa98733d640 1 --2- 172.21.15.72:0/1154712134 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa9800cd300 0x7fa9800ca680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.463+0000 7fa98733d640 1 -- 172.21.15.72:0/1154712134 >> 172.21.15.72:0/1154712134 conn(0x7fa9800b43c0 msgr2=0x7fa9800cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:10.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.463+0000 7fa98733d640 1 -- 172.21.15.72:0/1154712134 shutdown_connections 2024-04-15T15:15:10.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.463+0000 7fa98733d640 1 -- 172.21.15.72:0/1154712134 wait complete. 2024-04-15T15:15:10.477 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-15T15:15:10.477 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-15T15:15:10.478 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-15T15:15:10.478 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-15T15:15:10.478 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-15T15:15:10.479 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-15T15:15:10.479 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-15T15:15:10.480 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-15T15:15:10.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.651+0000 7f4053af1640 1 Processor -- start 2024-04-15T15:15:10.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.651+0000 7f4053af1640 1 -- start start 2024-04-15T15:15:10.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.652+0000 7f4053af1640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f40440580d0 0x7f4044053f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.652+0000 7f4053af1640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4044048bb0 con 0x7f40440eb9b0 2024-04-15T15:15:10.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.652+0000 7f4053af1640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40440488d0 con 0x7f40440e74e0 2024-04-15T15:15:10.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.652+0000 7f4053af1640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f40440ebe70 con 0x7f40440580d0 2024-04-15T15:15:10.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.653+0000 7f405108c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f40440eb9b0 0x7f4044053810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45092/0 (socket says 172.21.15.72:45092) 2024-04-15T15:15:10.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.653+0000 7f405108c640 1 -- 172.21.15.72:0/962889735 learned_addr learned my addr 172.21.15.72:0/962889735 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:10.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.653+0000 7f405208e640 1 --2- 172.21.15.72:0/962889735 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f40440580d0 0x7f4044053f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.653+0000 7f405088b640 1 -- 172.21.15.72:0/962889735 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1837650780 0 0) 0x7f4044048bb0 con 0x7f40440eb9b0 2024-04-15T15:15:10.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.653+0000 7f405088b640 1 -- 172.21.15.72:0/962889735 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4038003520 con 0x7f40440eb9b0 2024-04-15T15:15:10.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.653+0000 7f405208e640 1 -- 172.21.15.72:0/962889735 >> v1:172.21.15.175:6789/0 conn(0x7f40440e74e0 legacy=0x7f40440eb2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:10.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.653+0000 7f405208e640 1 -- 172.21.15.72:0/962889735 >> v1:172.21.15.72:6789/0 conn(0x7f40440eb9b0 legacy=0x7f4044053810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.653+0000 7f405208e640 1 -- 172.21.15.72:0/962889735 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40440ec150 con 0x7f40440580d0 2024-04-15T15:15:10.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.654+0000 7f0f5b8d3640 1 Processor -- start 2024-04-15T15:15:10.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.655+0000 7f405208e640 1 --2- 172.21.15.72:0/962889735 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f40440580d0 0x7f4044053f20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f404c000b60 tx=0x7f404c069760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dff35486409ab896 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.655+0000 7f0f5b8d3640 1 -- start start 2024-04-15T15:15:10.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.655+0000 7f405088b640 1 -- 172.21.15.72:0/962889735 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f404c0b4070 con 0x7f40440580d0 2024-04-15T15:15:10.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.655+0000 7f405088b640 1 -- 172.21.15.72:0/962889735 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f404c0af440 con 0x7f40440580d0 2024-04-15T15:15:10.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.656+0000 7f40aeb52640 1 Processor -- start 2024-04-15T15:15:10.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.656+0000 7f40aeb52640 1 -- start start 2024-04-15T15:15:10.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.656+0000 7f405088b640 1 -- 172.21.15.72:0/962889735 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f404c0af8b0 con 0x7f40440580d0 2024-04-15T15:15:10.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.656+0000 7f4053af1640 1 -- 172.21.15.72:0/962889735 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f40440580d0 msgr2=0x7f4044053f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.656+0000 7f4053af1640 1 --2- 172.21.15.72:0/962889735 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f40440580d0 0x7f4044053f20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f404c000b60 tx=0x7f404c069760 comp rx=0 tx=0).stop 2024-04-15T15:15:10.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.656+0000 7ffa5ced3640 1 Processor -- start 2024-04-15T15:15:10.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.656+0000 7f0f5b8d3640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f0f5415aaf0 0x7f0f540cd540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.656+0000 7f0f5b8d3640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0f5406c970 0x7f0f540733f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.656+0000 7f0f5b8d3640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f540b0db0 con 0x7f0f54072fa0 2024-04-15T15:15:10.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.656+0000 7f0f5b8d3640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f0f54073930 con 0x7f0f5415aaf0 2024-04-15T15:15:10.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.656+0000 7f0f5b8d3640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f0f540b1920 con 0x7f0f5406c970 2024-04-15T15:15:10.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.656+0000 7f7b49ddb640 1 Processor -- start 2024-04-15T15:15:10.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.656+0000 7f4053af1640 1 -- 172.21.15.72:0/962889735 shutdown_connections 2024-04-15T15:15:10.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.656+0000 7f4053af1640 1 --2- 172.21.15.72:0/962889735 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f40440580d0 0x7f4044053f20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.656+0000 7f4053af1640 1 -- 172.21.15.72:0/962889735 >> 172.21.15.72:0/962889735 conn(0x7f4044055c40 msgr2=0x7f4044056050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:10.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.656+0000 7f0f58e6e640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f0f54072fa0 0x7f0f5415e4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45106/0 (socket says 172.21.15.72:45106) 2024-04-15T15:15:10.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.656+0000 7f0f58e6e640 1 -- 172.21.15.72:0/22716071 learned_addr learned my addr 172.21.15.72:0/22716071 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:10.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.656+0000 7f7b49ddb640 1 -- start start 2024-04-15T15:15:10.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.656+0000 7ffa5ced3640 1 -- start start 2024-04-15T15:15:10.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.657+0000 7f4053af1640 1 -- 172.21.15.72:0/962889735 shutdown_connections 2024-04-15T15:15:10.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.657+0000 7fdd4e5a8640 1 Processor -- start 2024-04-15T15:15:10.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.657+0000 7f0f5966f640 1 --2- 172.21.15.72:0/22716071 >> v2:172.21.15.72:3300/0 conn(0x7f0f5415aaf0 0x7f0f540cd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.657+0000 7f0f59e70640 1 --2- 172.21.15.72:0/22716071 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0f5406c970 0x7f0f540733f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.657+0000 7f0f4bfff640 1 -- 172.21.15.72:0/22716071 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1566833106 0 0) 0x7f0f540b0db0 con 0x7f0f54072fa0 2024-04-15T15:15:10.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.657+0000 7f0f4bfff640 1 -- 172.21.15.72:0/22716071 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f40003440 con 0x7f0f54072fa0 2024-04-15T15:15:10.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.657+0000 7f0f59e70640 1 -- 172.21.15.72:0/22716071 >> v2:172.21.15.72:3300/0 conn(0x7f0f5415aaf0 msgr2=0x7f0f540cd540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:10.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.657+0000 7ffa5ced3640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffa580c3460 0x7ffa580c3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.657+0000 7f4053af1640 1 -- 172.21.15.72:0/962889735 wait complete. 2024-04-15T15:15:10.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.657+0000 7f0f59e70640 1 --2- 172.21.15.72:0/22716071 >> v2:172.21.15.72:3300/0 conn(0x7f0f5415aaf0 0x7f0f540cd540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.657+0000 7f0f59e70640 1 -- 172.21.15.72:0/22716071 >> v1:172.21.15.72:6789/0 conn(0x7f0f54072fa0 legacy=0x7f0f5415e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.657+0000 7f0f59e70640 1 -- 172.21.15.72:0/22716071 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f540b1050 con 0x7f0f5406c970 2024-04-15T15:15:10.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.657+0000 7f0f5966f640 1 --2- 172.21.15.72:0/22716071 >> v2:172.21.15.72:3300/0 conn(0x7f0f5415aaf0 0x7f0f540cd540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:10.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.657+0000 7ffa5ced3640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ffa580b1630 0x7ffa5806b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.657+0000 7ffa5ced3640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa580c3d80 con 0x7ffa58163670 2024-04-15T15:15:10.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.658+0000 7ffa5ced3640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7ffa580d07a0 con 0x7ffa580b1630 2024-04-15T15:15:10.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.658+0000 7ffa5ced3640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7ffa58163be0 con 0x7ffa580c3460 2024-04-15T15:15:10.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.658+0000 7f40aeb52640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f40a8072fa0 0x7f40a80733f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.658+0000 7ffa5659c640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffa580c3460 0x7ffa580c3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.658+0000 7f7b49ddb640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7b3c0df830 0x7f7b3c055d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.658+0000 7f7b49ddb640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7b3c03a0d0 0x7f7b3c03a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.658+0000 7f7b49ddb640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b3c050180 con 0x7f7b3c0366d0 2024-04-15T15:15:10.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.658+0000 7f7b49ddb640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f7b3c03aa40 con 0x7f7b3c03a0d0 2024-04-15T15:15:10.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.658+0000 7f7b49ddb640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f7b3c006c60 con 0x7f7b3c0df830 2024-04-15T15:15:10.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.658+0000 7f40ac8ee640 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f40a815aaf0 0x7f40a80cd540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.72:53376/0 (socket says 172.21.15.72:53376) 2024-04-15T15:15:10.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.658+0000 7f40ac8ee640 1 -- 172.21.15.72:0/3755665291 learned_addr learned my addr 172.21.15.72:0/3755665291 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:10.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.658+0000 7f40a7fff640 1 --2- 172.21.15.72:0/3755665291 >> v2:172.21.15.175:3300/0 conn(0x7f40a8072fa0 0x7f40a80733f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.658+0000 7ffa56d9d640 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7ffa58163670 0x7ffa580d0c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.72:53368/0 (socket says 172.21.15.72:53368) 2024-04-15T15:15:10.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.658+0000 7ffa56d9d640 1 -- 172.21.15.72:0/1504474686 learned_addr learned my addr 172.21.15.72:0/1504474686 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:10.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.658+0000 7ffa55d9b640 1 --2- 172.21.15.72:0/1504474686 >> v2:172.21.15.175:3300/0 conn(0x7ffa580b1630 0x7ffa5806b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.658+0000 7f7b42ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f7b3c0366d0 0x7f7b3c04fa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45110/0 (socket says 172.21.15.72:45110) 2024-04-15T15:15:10.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.658+0000 7f7b42ffd640 1 -- 172.21.15.72:0/4159180729 learned_addr learned my addr 172.21.15.72:0/4159180729 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:10.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.658+0000 7ffa5559a640 1 -- 172.21.15.72:0/1504474686 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2002070700 0 0) 0x7ffa580c3d80 con 0x7ffa58163670 2024-04-15T15:15:10.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.658+0000 7f40aeb52640 1 -- 172.21.15.72:0/3755665291 --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40a80b0db0 con 0x7f40a806b960 2024-04-15T15:15:10.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.658+0000 7fdd4e5a8640 1 -- start start 2024-04-15T15:15:10.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.658+0000 7f40aeb52640 1 -- 172.21.15.72:0/3755665291 --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40a80b1050 con 0x7f40a815aaf0 2024-04-15T15:15:10.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.658+0000 7f7b43fff640 1 --2- 172.21.15.72:0/4159180729 >> v2:172.21.15.72:3300/0 conn(0x7f7b3c03a0d0 0x7f7b3c03a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.658+0000 7f7b437fe640 1 --2- 172.21.15.72:0/4159180729 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7b3c0df830 0x7f7b3c055d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.658+0000 7ffa5559a640 1 -- 172.21.15.72:0/1504474686 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa40003520 con 0x7ffa58163670 2024-04-15T15:15:10.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.659+0000 7f40aeb52640 1 -- 172.21.15.72:0/3755665291 --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f40a8073930 con 0x7f40a8072fa0 2024-04-15T15:15:10.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.658+0000 7ffa5559a640 1 -- 172.21.15.72:0/1504474686 <== mon.1 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2255473060 0 0) 0x7ffa40003520 con 0x7ffa58163670 2024-04-15T15:15:10.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.659+0000 7ffa5559a640 1 -- 172.21.15.72:0/1504474686 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffa580c3460 msgr2=0x7ffa580c3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:10.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.659+0000 7ffa5559a640 1 --2- 172.21.15.72:0/1504474686 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffa580c3460 0x7ffa580c3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.659+0000 7ffa5559a640 1 -- 172.21.15.72:0/1504474686 >> v2:172.21.15.175:3300/0 conn(0x7ffa580b1630 msgr2=0x7ffa5806b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:10.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.659+0000 7ffa5559a640 1 --2- 172.21.15.72:0/1504474686 >> v2:172.21.15.175:3300/0 conn(0x7ffa580b1630 0x7ffa5806b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.659+0000 7ffa5559a640 1 -- 172.21.15.72:0/1504474686 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa58163ec0 con 0x7ffa58163670 2024-04-15T15:15:10.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.659+0000 7ffa5559a640 1 -- 172.21.15.72:0/1504474686 <== mon.1 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7ffa48001670 con 0x7ffa58163670 2024-04-15T15:15:10.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.659+0000 7ffa5559a640 1 -- 172.21.15.72:0/1504474686 >> v1:172.21.15.175:6789/0 conn(0x7ffa58163670 legacy=0x7ffa580d0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.659+0000 7ffa5559a640 1 --2- 172.21.15.72:0/1504474686 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffa400039b0 0x7ffa40003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.659+0000 7f40a77fe640 1 -- 172.21.15.72:0/3755665291 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1965661485 0 0) 0x7f40a80b1050 con 0x7f40a815aaf0 2024-04-15T15:15:10.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.659+0000 7f7b427fc640 1 -- 172.21.15.72:0/4159180729 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2016413023 0 0) 0x7f7b3c050180 con 0x7f7b3c0366d0 2024-04-15T15:15:10.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.659+0000 7f40a77fe640 1 -- 172.21.15.72:0/3755665291 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4090003520 con 0x7f40a815aaf0 2024-04-15T15:15:10.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.659+0000 7f0f59e70640 1 --2- 172.21.15.72:0/22716071 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0f5406c970 0x7f0f540733f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0f500097d0 tx=0x7f0f5002b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2efc21c69a2b0a5f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.659+0000 7f4053af1640 1 Processor -- start 2024-04-15T15:15:10.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.659+0000 7ffa5559a640 1 --2- 172.21.15.72:0/1504474686 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffa580c3460 0x7ffa40004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.659+0000 7ffa5559a640 1 --2- 172.21.15.72:0/1504474686 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffa40004d40 0x7ffa400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.659+0000 7ffa5559a640 1 -- 172.21.15.72:0/1504474686 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ffa580d07a0 con 0x7ffa400039b0 2024-04-15T15:15:10.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.659+0000 7ffa5559a640 1 -- 172.21.15.72:0/1504474686 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7ffa58163be0 con 0x7ffa40004d40 2024-04-15T15:15:10.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.659+0000 7ffa5559a640 1 -- 172.21.15.72:0/1504474686 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7ffa4000b9c0 con 0x7ffa580c3460 2024-04-15T15:15:10.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.659+0000 7f40a77fe640 1 -- 172.21.15.72:0/3755665291 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3380046445 0 0) 0x7f40a80b0db0 con 0x7f40a806b960 2024-04-15T15:15:10.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.659+0000 7f7b437fe640 1 -- 172.21.15.72:0/4159180729 >> v2:172.21.15.72:3300/0 conn(0x7f7b3c03a0d0 msgr2=0x7f7b3c03a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:10.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.659+0000 7f40a77fe640 1 -- 172.21.15.72:0/3755665291 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4090003880 con 0x7f40a806b960 2024-04-15T15:15:10.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.659+0000 7f7b437fe640 1 --2- 172.21.15.72:0/4159180729 >> v2:172.21.15.72:3300/0 conn(0x7f7b3c03a0d0 0x7f7b3c03a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.659+0000 7f7b437fe640 1 -- 172.21.15.72:0/4159180729 >> v1:172.21.15.72:6789/0 conn(0x7f7b3c0366d0 legacy=0x7f7b3c04fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.659+0000 7f7b437fe640 1 -- 172.21.15.72:0/4159180729 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b3c006f40 con 0x7f7b3c0df830 2024-04-15T15:15:10.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.659+0000 7f0f4bfff640 1 -- 172.21.15.72:0/22716071 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f50031770 con 0x7f0f5406c970 2024-04-15T15:15:10.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.659+0000 7f0f4bfff640 1 -- 172.21.15.72:0/22716071 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f50037070 con 0x7f0f5406c970 2024-04-15T15:15:10.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.659+0000 7f0f4bfff640 1 -- 172.21.15.72:0/22716071 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f50031b30 con 0x7f0f5406c970 2024-04-15T15:15:10.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.659+0000 7ffa5659c640 1 --2- 172.21.15.72:0/1504474686 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffa400039b0 0x7ffa40003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.659+0000 7f40a7fff640 1 -- 172.21.15.72:0/3755665291 >> v1:172.21.15.175:6789/0 conn(0x7f40a815aaf0 legacy=0x7f40a80cd540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.659+0000 7f40a7fff640 1 -- 172.21.15.72:0/3755665291 >> v1:172.21.15.72:6789/0 conn(0x7f40a806b960 legacy=0x7f40a806bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.659+0000 7f40a7fff640 1 -- 172.21.15.72:0/3755665291 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40a80b12f0 con 0x7f40a8072fa0 2024-04-15T15:15:10.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.660+0000 7fdd4e5a8640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fdd400e7850 0x7fdd400382b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.660+0000 7fdd4e5a8640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdd400eb330 0x7fdd400eb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.660+0000 7fdd4e5a8640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd400e7c30 con 0x7fdd400ebcf0 2024-04-15T15:15:10.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.660+0000 7fdd47fff640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fdd400e7850 0x7fdd400382b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.660+0000 7fdd477fe640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdd400eb330 0x7fdd400eb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.660+0000 7fdd4e5a8640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fdd40057f50 con 0x7fdd400e7850 2024-04-15T15:15:10.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.660+0000 7fdd4cb45640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fdd400ebcf0 0x7fdd40053550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45140/0 (socket says 172.21.15.72:45140) 2024-04-15T15:15:10.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.660+0000 7fdd477fe640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdd400eb330 0x7fdd400eb7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3301/0 says I am v2:172.21.15.72:57720/0 (socket says 172.21.15.72:57720) 2024-04-15T15:15:10.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.660+0000 7fdd477fe640 1 -- 172.21.15.72:0/769761281 learned_addr learned my addr 172.21.15.72:0/769761281 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:10.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.660+0000 7fdd4e5a8640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fdd40058230 con 0x7fdd400eb330 2024-04-15T15:15:10.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.659+0000 7f4053af1640 1 -- start start 2024-04-15T15:15:10.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.660+0000 7f4053af1640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40440580d0 0x7f40440d2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.660+0000 7f4053af1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f40440e74e0 0x7f404404f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.660+0000 7f7b43fff640 1 --2- 172.21.15.72:0/4159180729 >> v2:172.21.15.72:3300/0 conn(0x7f7b3c03a0d0 0x7f7b3c03a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:10.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.660+0000 7ffa5659c640 1 -- 172.21.15.72:0/1504474686 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffa580c3460 msgr2=0x7ffa40004800 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-15T15:15:10.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.660+0000 7ffa5659c640 1 --2- 172.21.15.72:0/1504474686 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffa580c3460 0x7ffa40004800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.660+0000 7ffa5659c640 1 -- 172.21.15.72:0/1504474686 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffa40004d40 msgr2=0x7ffa400051c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-15T15:15:10.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.660+0000 7ffa5659c640 1 --2- 172.21.15.72:0/1504474686 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffa40004d40 0x7ffa400051c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.660+0000 7ffa5659c640 1 -- 172.21.15.72:0/1504474686 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa40003520 con 0x7ffa400039b0 2024-04-15T15:15:10.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7f40a7fff640 1 --2- 172.21.15.72:0/3755665291 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40a8072fa0 0x7f40a80733f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4098007200 tx=0x7f409802db60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9aaa89df79dba34c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.660+0000 7f4053af1640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f40440eb9b0 0x7f40440e50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.660+0000 7f4053af1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f40440e7a40 con 0x7f40440e74e0 2024-04-15T15:15:10.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.660+0000 7f4053af1640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f40440e78c0 con 0x7f40440580d0 2024-04-15T15:15:10.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.660+0000 7f4053af1640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f40440e7bc0 con 0x7f40440eb9b0 2024-04-15T15:15:10.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.660+0000 7f405108c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f40440e74e0 0x7f404404f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.660+0000 7f405208e640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f40440eb9b0 0x7f40440e50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7f405208e640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f40440eb9b0 0x7f40440e50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3301/0 says I am v2:172.21.15.72:57730/0 (socket says 172.21.15.72:57730) 2024-04-15T15:15:10.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7f405208e640 1 -- 172.21.15.72:0/4025438024 learned_addr learned my addr 172.21.15.72:0/4025438024 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:10.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7fdd46ffd640 1 -- 172.21.15.72:0/769761281 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3283846891 0 0) 0x7fdd400e7c30 con 0x7fdd400ebcf0 2024-04-15T15:15:10.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7fdd46ffd640 1 -- 172.21.15.72:0/769761281 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd38003520 con 0x7fdd400ebcf0 2024-04-15T15:15:10.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7f40a77fe640 1 -- 172.21.15.72:0/3755665291 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4098008df0 con 0x7f40a8072fa0 2024-04-15T15:15:10.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7f40a77fe640 1 -- 172.21.15.72:0/3755665291 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f409800a0d0 con 0x7f40a8072fa0 2024-04-15T15:15:10.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7f405208e640 1 -- 172.21.15.72:0/4025438024 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40440580d0 msgr2=0x7f40440d2450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-15T15:15:10.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7f40a77fe640 1 -- 172.21.15.72:0/3755665291 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f409800a4a0 con 0x7f40a8072fa0 2024-04-15T15:15:10.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7f405188d640 1 --2- 172.21.15.72:0/4025438024 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40440580d0 0x7f40440d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7f405208e640 1 --2- 172.21.15.72:0/4025438024 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40440580d0 0x7f40440d2450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7f405208e640 1 -- 172.21.15.72:0/4025438024 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f40440e74e0 msgr2=0x7f404404f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7f405208e640 1 --2- 172.21.15.72:0/4025438024 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f40440e74e0 0x7f404404f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7f405208e640 1 -- 172.21.15.72:0/4025438024 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f404403a9f0 con 0x7f40440eb9b0 2024-04-15T15:15:10.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7f0f5b8d3640 1 -- 172.21.15.72:0/22716071 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0f5406c970 msgr2=0x7f0f540733f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7f0f5b8d3640 1 --2- 172.21.15.72:0/22716071 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0f5406c970 0x7f0f540733f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0f500097d0 tx=0x7f0f5002b810 comp rx=0 tx=0).stop 2024-04-15T15:15:10.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7fdd46ffd640 1 -- 172.21.15.72:0/769761281 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3059754558 0 0) 0x7fdd38003520 con 0x7fdd400ebcf0 2024-04-15T15:15:10.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7fdd46ffd640 1 -- 172.21.15.72:0/769761281 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdd400eb330 msgr2=0x7fdd400eb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:10.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7fdd46ffd640 1 --2- 172.21.15.72:0/769761281 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdd400eb330 0x7fdd400eb7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7fdd46ffd640 1 -- 172.21.15.72:0/769761281 >> v2:172.21.15.175:3300/0 conn(0x7fdd400e7850 msgr2=0x7fdd400382b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:10.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7fdd46ffd640 1 --2- 172.21.15.72:0/769761281 >> v2:172.21.15.175:3300/0 conn(0x7fdd400e7850 0x7fdd400382b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7fdd46ffd640 1 -- 172.21.15.72:0/769761281 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd40058510 con 0x7fdd400ebcf0 2024-04-15T15:15:10.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7f405108c640 1 --2- 172.21.15.72:0/4025438024 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f40440e74e0 0x7f404404f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:10.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.660+0000 7f7b437fe640 1 --2- 172.21.15.72:0/4159180729 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7b3c0df830 0x7f7b3c055d40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7b440654e0 tx=0x7f7b44065780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7e7c738a3f53fd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7f7b427fc640 1 -- 172.21.15.72:0/4159180729 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b440b4070 con 0x7f7b3c0df830 2024-04-15T15:15:10.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7f7b427fc640 1 -- 172.21.15.72:0/4159180729 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b440af640 con 0x7f7b3c0df830 2024-04-15T15:15:10.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.662+0000 7f7b427fc640 1 -- 172.21.15.72:0/4159180729 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b440afad0 con 0x7f7b3c0df830 2024-04-15T15:15:10.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7f405208e640 1 --2- 172.21.15.72:0/4025438024 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f40440eb9b0 0x7f40440e50a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f404c06a3d0 tx=0x7f404c06b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7f0f5b8d3640 1 -- 172.21.15.72:0/22716071 shutdown_connections 2024-04-15T15:15:10.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7f0f5b8d3640 1 --2- 172.21.15.72:0/22716071 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0f5406c970 0x7f0f540733f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7f0f5b8d3640 1 --2- 172.21.15.72:0/22716071 >> v2:172.21.15.72:3300/0 conn(0x7f0f5415aaf0 0x7f0f540cd540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7f0f5b8d3640 1 -- 172.21.15.72:0/22716071 >> 172.21.15.72:0/22716071 conn(0x7f0f540b5af0 msgr2=0x7f0f540b31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:10.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7ffa5659c640 1 --2- 172.21.15.72:0/1504474686 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffa400039b0 0x7ffa40003dd0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c009ac0 tx=0x7ffa40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6652830bccea7506 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7fdd46ffd640 1 -- 172.21.15.72:0/769761281 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7fdd480505a0 con 0x7fdd400ebcf0 2024-04-15T15:15:10.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7fdd46ffd640 1 -- 172.21.15.72:0/769761281 >> v1:172.21.15.72:6789/0 conn(0x7fdd400ebcf0 legacy=0x7fdd40053550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7fdd46ffd640 1 --2- 172.21.15.72:0/769761281 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdd380039d0 0x7fdd38003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7fdd46ffd640 1 --2- 172.21.15.72:0/769761281 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd38004800 0x7fdd38004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7fdd46ffd640 1 --2- 172.21.15.72:0/769761281 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdd400eb330 0x7fdd380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7fdd46ffd640 1 -- 172.21.15.72:0/769761281 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fdd40057f50 con 0x7fdd38004800 2024-04-15T15:15:10.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7fdd46ffd640 1 -- 172.21.15.72:0/769761281 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fdd40058230 con 0x7fdd380039d0 2024-04-15T15:15:10.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.661+0000 7fdd46ffd640 1 -- 172.21.15.72:0/769761281 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fdd3800b9c0 con 0x7fdd400eb330 2024-04-15T15:15:10.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.662+0000 7f403e7fc640 1 -- 172.21.15.72:0/4025438024 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f404c0b4050 con 0x7f40440eb9b0 2024-04-15T15:15:10.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.662+0000 7ffa5559a640 1 -- 172.21.15.72:0/1504474686 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c039070 con 0x7ffa400039b0 2024-04-15T15:15:10.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.662+0000 7f40aeb52640 1 -- 172.21.15.72:0/3755665291 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40a8072fa0 msgr2=0x7f40a80733f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.662+0000 7f40aeb52640 1 --2- 172.21.15.72:0/3755665291 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40a8072fa0 0x7f40a80733f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4098007200 tx=0x7f409802db60 comp rx=0 tx=0).stop 2024-04-15T15:15:10.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.662+0000 7f0f5b8d3640 1 -- 172.21.15.72:0/22716071 shutdown_connections 2024-04-15T15:15:10.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.662+0000 7fdd47fff640 1 --2- 172.21.15.72:0/769761281 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdd380039d0 0x7fdd38003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.662+0000 7f40aeb52640 1 -- 172.21.15.72:0/3755665291 shutdown_connections 2024-04-15T15:15:10.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.662+0000 7fdd477fe640 1 --2- 172.21.15.72:0/769761281 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd38004800 0x7fdd38004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.662+0000 7f40aeb52640 1 --2- 172.21.15.72:0/3755665291 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40a8072fa0 0x7f40a80733f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.662+0000 7f40aeb52640 1 -- 172.21.15.72:0/3755665291 >> 172.21.15.72:0/3755665291 conn(0x7f40a80b5af0 msgr2=0x7f40a80b31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:10.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.662+0000 7f7b49ddb640 1 -- 172.21.15.72:0/4159180729 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7b3c0df830 msgr2=0x7f7b3c055d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.662+0000 7f7b49ddb640 1 --2- 172.21.15.72:0/4159180729 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7b3c0df830 0x7f7b3c055d40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7b440654e0 tx=0x7f7b44065780 comp rx=0 tx=0).stop 2024-04-15T15:15:10.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.662+0000 7f7b49ddb640 1 -- 172.21.15.72:0/4159180729 shutdown_connections 2024-04-15T15:15:10.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.662+0000 7f7b49ddb640 1 --2- 172.21.15.72:0/4159180729 >> v2:172.21.15.72:3300/0 conn(0x7f7b3c03a0d0 0x7f7b3c03a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.662+0000 7f7b49ddb640 1 --2- 172.21.15.72:0/4159180729 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7b3c0df830 0x7f7b3c055d40 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.662+0000 7f7b49ddb640 1 -- 172.21.15.72:0/4159180729 >> 172.21.15.72:0/4159180729 conn(0x7f7b3c01e6f0 msgr2=0x7f7b3c055420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:10.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.662+0000 7f40aeb52640 1 -- 172.21.15.72:0/3755665291 shutdown_connections 2024-04-15T15:15:10.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.662+0000 7f40aeb52640 1 -- 172.21.15.72:0/3755665291 wait complete. 2024-04-15T15:15:10.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.662+0000 7efcb58c8640 1 Processor -- start 2024-04-15T15:15:10.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.662+0000 7fdd47fff640 1 -- 172.21.15.72:0/769761281 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdd400eb330 msgr2=0x7fdd380051c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-15T15:15:10.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.662+0000 7fdd47fff640 1 --2- 172.21.15.72:0/769761281 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdd400eb330 0x7fdd380051c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.662+0000 7fdd47fff640 1 -- 172.21.15.72:0/769761281 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd38004800 msgr2=0x7fdd38004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:10.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.662+0000 7fdd47fff640 1 --2- 172.21.15.72:0/769761281 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd38004800 0x7fdd38004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.662+0000 7fdd47fff640 1 -- 172.21.15.72:0/769761281 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd40058510 con 0x7fdd380039d0 2024-04-15T15:15:10.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.663+0000 7f7b49ddb640 1 -- 172.21.15.72:0/4159180729 shutdown_connections 2024-04-15T15:15:10.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.663+0000 7efcb58c8640 1 -- start start 2024-04-15T15:15:10.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.662+0000 7ffa5559a640 1 -- 172.21.15.72:0/1504474686 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa4c02c6d0 con 0x7ffa400039b0 2024-04-15T15:15:10.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.663+0000 7ffa5ced3640 1 -- 172.21.15.72:0/1504474686 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffa400039b0 msgr2=0x7ffa40003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.663+0000 7ffa5559a640 1 -- 172.21.15.72:0/1504474686 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c039070 con 0x7ffa400039b0 2024-04-15T15:15:10.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.662+0000 7f4053af1640 1 -- 172.21.15.72:0/4025438024 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f404403acd0 con 0x7f40440eb9b0 2024-04-15T15:15:10.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.662+0000 7f4053af1640 1 -- 172.21.15.72:0/4025438024 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f404403b290 con 0x7f40440eb9b0 2024-04-15T15:15:10.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.663+0000 7f403e7fc640 1 -- 172.21.15.72:0/4025438024 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f404c0b5040 con 0x7f40440eb9b0 2024-04-15T15:15:10.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.663+0000 7f7b49ddb640 1 -- 172.21.15.72:0/4159180729 wait complete. 2024-04-15T15:15:10.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.663+0000 7f403e7fc640 1 -- 172.21.15.72:0/4025438024 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f404c0af7f0 con 0x7f40440eb9b0 2024-04-15T15:15:10.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.663+0000 7f4053af1640 1 -- 172.21.15.72:0/4025438024 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f404403bbc0 con 0x7f40440eb9b0 2024-04-15T15:15:10.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.664+0000 7efcb58c8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efcb00d1480 0x7efcb00d2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.663+0000 7f0f5b8d3640 1 -- 172.21.15.72:0/22716071 wait complete. 2024-04-15T15:15:10.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.664+0000 7f403e7fc640 1 -- 172.21.15.72:0/4025438024 <== mon.2 v2:172.21.15.175:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f404c0c2020 con 0x7f40440eb9b0 2024-04-15T15:15:10.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.664+0000 7f7b49ddb640 1 Processor -- start 2024-04-15T15:15:10.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.664+0000 7f40aeb52640 1 Processor -- start 2024-04-15T15:15:10.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.664+0000 7fdd47fff640 1 --2- 172.21.15.72:0/769761281 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdd380039d0 0x7fdd38003e10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdd3400ba30 tx=0x7fdd38005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bb0bdf2124a6e11 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.664+0000 7efcaf7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efcb00d1480 0x7efcb00d2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.665+0000 7efcaf7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efcb00d1480 0x7efcb00d2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44684/0 (socket says 172.21.15.72:44684) 2024-04-15T15:15:10.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.664+0000 7ffa5ced3640 1 --2- 172.21.15.72:0/1504474686 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffa400039b0 0x7ffa40003dd0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c009ac0 tx=0x7ffa40005830 comp rx=0 tx=0).stop 2024-04-15T15:15:10.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.664+0000 7ffa56d9d640 1 -- 172.21.15.72:0/1504474686 reap_dead start 2024-04-15T15:15:10.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.665+0000 7ffa5ced3640 1 -- 172.21.15.72:0/1504474686 shutdown_connections 2024-04-15T15:15:10.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.665+0000 7f40aeb52640 1 -- start start 2024-04-15T15:15:10.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.665+0000 7efcaf7fe640 1 -- 172.21.15.72:0/3779405512 learned_addr learned my addr 172.21.15.72:0/3779405512 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:10.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.665+0000 7f54c757c640 1 Processor -- start 2024-04-15T15:15:10.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.665+0000 7ffa5ced3640 1 -- 172.21.15.72:0/1504474686 >> 172.21.15.72:0/1504474686 conn(0x7ffa58073080 msgr2=0x7ffa58073490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:10.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.665+0000 7f7b49ddb640 1 -- start start 2024-04-15T15:15:10.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.665+0000 7f403e7fc640 1 --2- 172.21.15.72:0/4025438024 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f403006c940 0x7f403006ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.665+0000 7f54c757c640 1 -- start start 2024-04-15T15:15:10.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.665+0000 7f0f5b8d3640 1 Processor -- start 2024-04-15T15:15:10.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.665+0000 7efcb58c8640 1 -- 172.21.15.72:0/3779405512 --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcb00d1910 con 0x7efcb00d4900 2024-04-15T15:15:10.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.665+0000 7efcb58c8640 1 -- 172.21.15.72:0/3779405512 --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcb00d1ef0 con 0x7efcb00b3590 2024-04-15T15:15:10.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.665+0000 7efcb58c8640 1 -- 172.21.15.72:0/3779405512 --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7efcb00d0840 con 0x7efcb00d1480 2024-04-15T15:15:10.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.666+0000 7efcaf7fe640 1 -- 172.21.15.72:0/3779405512 >> v1:172.21.15.175:6789/0 conn(0x7efcb00b3590 legacy=0x7efcb00d2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.666+0000 7f7b49ddb640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7b3c0366d0 0x7f7b3c04d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.666+0000 7efcae7fc640 1 -- 172.21.15.72:0/3779405512 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 544125082 0 0) 0x7efcb00d1ef0 con 0x7efcb00b3590 2024-04-15T15:15:10.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.666+0000 7efcaf7fe640 1 -- 172.21.15.72:0/3779405512 >> v1:172.21.15.72:6789/0 conn(0x7efcb00d4900 legacy=0x7efcb00d4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.666+0000 7efcaf7fe640 1 -- 172.21.15.72:0/3779405512 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcb00d0b20 con 0x7efcb00d1480 2024-04-15T15:15:10.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.666+0000 7f7b49ddb640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b3c03a0d0 0x7f7b3c0dd360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.666+0000 7f7b49ddb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b3c0df830 0x7f7b3c007920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.666+0000 7f7b49ddb640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f7b3c0dfc10 con 0x7f7b3c0df830 2024-04-15T15:15:10.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.666+0000 7f7b49ddb640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f7b3c007500 con 0x7f7b3c03a0d0 2024-04-15T15:15:10.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.666+0000 7f7b49ddb640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f7b3c036b10 con 0x7f7b3c0366d0 2024-04-15T15:15:10.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.665+0000 7f0f5b8d3640 1 -- start start 2024-04-15T15:15:10.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.666+0000 7f0f5b8d3640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0f5406c970 0x7f0f540b3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.666+0000 7f0f5b8d3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f54072fa0 0x7f0f54152ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.666+0000 7f0f5b8d3640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0f5415aaf0 0x7f0f541515e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.666+0000 7f0f5b8d3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f0f540cda80 con 0x7f0f54072fa0 2024-04-15T15:15:10.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.666+0000 7f0f5b8d3640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f0f540b15b0 con 0x7f0f5415aaf0 2024-04-15T15:15:10.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.666+0000 7f0f5b8d3640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f0f540b0a80 con 0x7f0f5406c970 2024-04-15T15:15:10.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.666+0000 7f7b42ffd640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b3c03a0d0 0x7f7b3c0dd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.666+0000 7f7b42ffd640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b3c03a0d0 0x7f7b3c0dd360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.72:59952/0 (socket says 172.21.15.72:59952) 2024-04-15T15:15:10.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.666+0000 7f7b42ffd640 1 -- 172.21.15.72:0/1532730000 learned_addr learned my addr 172.21.15.72:0/1532730000 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:10.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.666+0000 7f54c757c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f54c015c7c0 0x7f54c006e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.666+0000 7f54c757c640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f54c006b960 0x7f54c006bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.666+0000 7f54c757c640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54c00cffd0 con 0x7f54c00af700 2024-04-15T15:15:10.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.666+0000 7f54c757c640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f54c015cba0 con 0x7f54c015c7c0 2024-04-15T15:15:10.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.666+0000 7f54c757c640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f54c006dc10 con 0x7f54c006b960 2024-04-15T15:15:10.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.666+0000 7f0f58e6e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f54072fa0 0x7f0f54152ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f0f5966f640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0f5406c970 0x7f0f540b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f0f5966f640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0f5406c970 0x7f0f540b3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3301/0 says I am v2:172.21.15.72:57760/0 (socket says 172.21.15.72:57760) 2024-04-15T15:15:10.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f0f5966f640 1 -- 172.21.15.72:0/3512745541 learned_addr learned my addr 172.21.15.72:0/3512745541 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:10.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.665+0000 7ffa5ced3640 1 -- 172.21.15.72:0/1504474686 shutdown_connections 2024-04-15T15:15:10.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.665+0000 7ffa5ced3640 1 -- 172.21.15.72:0/1504474686 wait complete. 2024-04-15T15:15:10.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.666+0000 7fdd46ffd640 1 -- 172.21.15.72:0/769761281 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd34035070 con 0x7fdd380039d0 2024-04-15T15:15:10.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.666+0000 7fdd46ffd640 1 -- 172.21.15.72:0/769761281 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd34030660 con 0x7fdd380039d0 2024-04-15T15:15:10.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.666+0000 7fdd46ffd640 1 -- 172.21.15.72:0/769761281 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3400b600 con 0x7fdd380039d0 2024-04-15T15:15:10.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.665+0000 7f403e7fc640 1 -- 172.21.15.72:0/4025438024 <== mon.2 v2:172.21.15.175:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f404c129270 con 0x7f40440eb9b0 2024-04-15T15:15:10.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.665+0000 7f403e7fc640 1 --2- 172.21.15.72:0/4025438024 >> [v2:172.21.15.72:6816/4041627082,v1:172.21.15.72:6817/4041627082] conn(0x7f4030076040 0x7f4030078480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.666+0000 7f405188d640 1 --2- 172.21.15.72:0/4025438024 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f403006c940 0x7f403006ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.666+0000 7f403e7fc640 1 -- 172.21.15.72:0/4025438024 --> [v2:172.21.15.72:6816/4041627082,v1:172.21.15.72:6817/4041627082] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f404c0af440 con 0x7f4030076040 2024-04-15T15:15:10.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.666+0000 7f403e7fc640 1 -- 172.21.15.72:0/4025438024 <== mon.2 v2:172.21.15.175:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f404c130050 con 0x7f40440eb9b0 2024-04-15T15:15:10.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.666+0000 7f405188d640 1 --2- 172.21.15.72:0/4025438024 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f403006c940 0x7f403006ee00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4048003d90 tx=0x7f4048009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.666+0000 7f405108c640 1 --2- 172.21.15.72:0/4025438024 >> [v2:172.21.15.72:6816/4041627082,v1:172.21.15.72:6817/4041627082] conn(0x7f4030076040 0x7f4030078480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f54c4b17640 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f54c00af700 0x7f54c006e960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.72:53388/0 (socket says 172.21.15.72:53388) 2024-04-15T15:15:10.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f54c4b17640 1 -- 172.21.15.72:0/2826732973 learned_addr learned my addr 172.21.15.72:0/2826732973 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:10.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f0f59e70640 1 --2- 172.21.15.72:0/3512745541 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0f5415aaf0 0x7f0f541515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f40aeb52640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f40a806b960 0x7f40a8150b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f40aeb52640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f40a8072fa0 0x7f40a81f7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f40aeb52640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40a815aaf0 0x7f40a81fbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f40aeb52640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f40a80b0a80 con 0x7f40a8072fa0 2024-04-15T15:15:10.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f40aeb52640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f40a80b1920 con 0x7f40a815aaf0 2024-04-15T15:15:10.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f40aeb52640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f40a8073c20 con 0x7f40a806b960 2024-04-15T15:15:10.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f54c5b19640 1 --2- 172.21.15.72:0/2826732973 >> v2:172.21.15.175:3300/0 conn(0x7f54c006b960 0x7f54c006bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f7b43fff640 1 --2- 172.21.15.72:0/1532730000 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b3c0df830 0x7f7b3c007920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f7b437fe640 1 --2- 172.21.15.72:0/1532730000 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7b3c0366d0 0x7f7b3c04d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f54c5318640 1 --2- 172.21.15.72:0/2826732973 >> v2:172.21.15.72:3300/0 conn(0x7f54c015c7c0 0x7f54c006e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f54b7fff640 1 -- 172.21.15.72:0/2826732973 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2564380352 0 0) 0x7f54c00cffd0 con 0x7f54c00af700 2024-04-15T15:15:10.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f54b7fff640 1 -- 172.21.15.72:0/2826732973 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54ac003520 con 0x7f54c00af700 2024-04-15T15:15:10.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f0f58e6e640 1 -- 172.21.15.72:0/3512745541 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0f5406c970 msgr2=0x7f0f540b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f0f58e6e640 1 --2- 172.21.15.72:0/3512745541 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0f5406c970 0x7f0f540b3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f0f58e6e640 1 -- 172.21.15.72:0/3512745541 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0f5415aaf0 msgr2=0x7f0f541515e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f0f58e6e640 1 --2- 172.21.15.72:0/3512745541 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0f5415aaf0 0x7f0f541515e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f0f58e6e640 1 -- 172.21.15.72:0/3512745541 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f54146f80 con 0x7f0f54072fa0 2024-04-15T15:15:10.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f40a7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f40a8072fa0 0x7f40a81f7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f40a7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f40a8072fa0 0x7f40a81f7f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44728/0 (socket says 172.21.15.72:44728) 2024-04-15T15:15:10.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f40a7fff640 1 -- 172.21.15.72:0/371671587 learned_addr learned my addr 172.21.15.72:0/371671587 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:10.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f7b42ffd640 1 -- 172.21.15.72:0/1532730000 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7b3c0366d0 msgr2=0x7f7b3c04d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f7b42ffd640 1 --2- 172.21.15.72:0/1532730000 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7b3c0366d0 0x7f7b3c04d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f7b42ffd640 1 -- 172.21.15.72:0/1532730000 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b3c0df830 msgr2=0x7f7b3c007920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f40ad0ef640 1 --2- 172.21.15.72:0/371671587 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40a815aaf0 0x7f40a81fbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7f40ac8ee640 1 --2- 172.21.15.72:0/371671587 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f40a806b960 0x7f40a8150b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7fdd4e5a8640 1 -- 172.21.15.72:0/769761281 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdd380039d0 msgr2=0x7fdd38003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7fdd4e5a8640 1 --2- 172.21.15.72:0/769761281 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdd380039d0 0x7fdd38003e10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fdd3400ba30 tx=0x7fdd38005830 comp rx=0 tx=0).stop 2024-04-15T15:15:10.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7efcaf7fe640 1 --2- 172.21.15.72:0/3779405512 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcb00d1480 0x7efcb00d2e60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7efca800b1a0 tx=0x7efca802c290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5dfc4449420910c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f0f58e6e640 1 --2- 172.21.15.72:0/3512745541 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f54072fa0 0x7f0f54152ec0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0f4400c480 tx=0x7f0f4400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7fdd4cb45640 1 -- 172.21.15.72:0/769761281 reap_dead start 2024-04-15T15:15:10.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f54b7fff640 1 -- 172.21.15.72:0/2826732973 <== mon.1 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3513028413 0 0) 0x7f54ac003520 con 0x7f54c00af700 2024-04-15T15:15:10.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f54b7fff640 1 -- 172.21.15.72:0/2826732973 >> v2:172.21.15.175:3300/0 conn(0x7f54c006b960 msgr2=0x7f54c006bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:10.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f54b7fff640 1 --2- 172.21.15.72:0/2826732973 >> v2:172.21.15.175:3300/0 conn(0x7f54c006b960 0x7f54c006bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f54b7fff640 1 -- 172.21.15.72:0/2826732973 >> v2:172.21.15.72:3300/0 conn(0x7f54c015c7c0 msgr2=0x7f54c006e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:10.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f54b7fff640 1 --2- 172.21.15.72:0/2826732973 >> v2:172.21.15.72:3300/0 conn(0x7f54c015c7c0 0x7f54c006e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f54b7fff640 1 -- 172.21.15.72:0/2826732973 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54c006de60 con 0x7f54c00af700 2024-04-15T15:15:10.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.667+0000 7f54b7fff640 1 -- 172.21.15.72:0/2826732973 <== mon.1 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f54b8001670 con 0x7f54c00af700 2024-04-15T15:15:10.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7f54b7fff640 1 -- 172.21.15.72:0/2826732973 >> v1:172.21.15.175:6789/0 conn(0x7f54c00af700 legacy=0x7f54c006e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7f54b7fff640 1 --2- 172.21.15.72:0/2826732973 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f54ac0039b0 0x7f54ac003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7f54b7fff640 1 --2- 172.21.15.72:0/2826732973 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54ac004800 0x7f54ac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7f54b7fff640 1 --2- 172.21.15.72:0/2826732973 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f54ac0051c0 0x7f54ac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7f54b7fff640 1 -- 172.21.15.72:0/2826732973 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f54c015cba0 con 0x7f54ac004800 2024-04-15T15:15:10.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7f54b7fff640 1 -- 172.21.15.72:0/2826732973 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f54c006dc10 con 0x7f54ac0039b0 2024-04-15T15:15:10.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7f54b7fff640 1 -- 172.21.15.72:0/2826732973 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f54ac00bf50 con 0x7f54ac0051c0 2024-04-15T15:15:10.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7f40a7fff640 1 -- 172.21.15.72:0/371671587 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f40a806b960 msgr2=0x7f40a8150b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7f40a7fff640 1 --2- 172.21.15.72:0/371671587 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f40a806b960 0x7f40a8150b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7f40a7fff640 1 -- 172.21.15.72:0/371671587 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40a815aaf0 msgr2=0x7f40a81fbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7f40a7fff640 1 --2- 172.21.15.72:0/371671587 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f40a815aaf0 0x7f40a81fbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7f0f49ffb640 1 -- 172.21.15.72:0/3512745541 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f44018070 con 0x7f0f54072fa0 2024-04-15T15:15:10.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7f0f49ffb640 1 -- 172.21.15.72:0/3512745541 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f44004d40 con 0x7f0f54072fa0 2024-04-15T15:15:10.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7fdd4e5a8640 1 -- 172.21.15.72:0/769761281 shutdown_connections 2024-04-15T15:15:10.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7f40a7fff640 1 -- 172.21.15.72:0/371671587 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40a814e520 con 0x7f40a8072fa0 2024-04-15T15:15:10.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7f0f49ffb640 1 -- 172.21.15.72:0/3512745541 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f44007310 con 0x7f0f54072fa0 2024-04-15T15:15:10.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7ffa5ced3640 1 Processor -- start 2024-04-15T15:15:10.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7fdd4e5a8640 1 -- 172.21.15.72:0/769761281 >> 172.21.15.72:0/769761281 conn(0x7fdd40055a10 msgr2=0x7fdd40055e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:10.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7f7b42ffd640 1 --2- 172.21.15.72:0/1532730000 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7b3c0df830 0x7f7b3c007920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7f7b42ffd640 1 -- 172.21.15.72:0/1532730000 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b3c00bf10 con 0x7f7b3c03a0d0 2024-04-15T15:15:10.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7f54c4b17640 1 --2- 172.21.15.72:0/2826732973 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54ac004800 0x7f54ac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7f54c5b19640 1 --2- 172.21.15.72:0/2826732973 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f54ac0051c0 0x7f54ac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7f54c5318640 1 --2- 172.21.15.72:0/2826732973 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f54ac0039b0 0x7f54ac003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7ffa5ced3640 1 -- start start 2024-04-15T15:15:10.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7efcae7fc640 1 -- 172.21.15.72:0/3779405512 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca802cc30 con 0x7efcb00d1480 2024-04-15T15:15:10.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7efcae7fc640 1 -- 172.21.15.72:0/3779405512 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efca802cdd0 con 0x7efcb00d1480 2024-04-15T15:15:10.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7f0f5b8d3640 1 -- 172.21.15.72:0/3512745541 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f54145980 con 0x7f0f54072fa0 2024-04-15T15:15:10.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7efcae7fc640 1 -- 172.21.15.72:0/3779405512 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca8034610 con 0x7efcb00d1480 2024-04-15T15:15:10.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7f7b42ffd640 1 --2- 172.21.15.72:0/1532730000 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b3c03a0d0 0x7f7b3c0dd360 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7b3000c4a0 tx=0x7f7b3000c960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7f40a7fff640 1 --2- 172.21.15.72:0/371671587 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f40a8072fa0 0x7f40a81f7f20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4098007320 tx=0x7f40980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.669+0000 7f40a57fa640 1 -- 172.21.15.72:0/371671587 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f409802f710 con 0x7f40a8072fa0 2024-04-15T15:15:10.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.669+0000 7f40a57fa640 1 -- 172.21.15.72:0/371671587 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f409802f8b0 con 0x7f40a8072fa0 2024-04-15T15:15:10.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.669+0000 7f405108c640 1 --2- 172.21.15.72:0/4025438024 >> [v2:172.21.15.72:6816/4041627082,v1:172.21.15.72:6817/4041627082] conn(0x7f4030076040 0x7f4030078480 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.669+0000 7efcb58c8640 1 -- 172.21.15.72:0/3779405512 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcb00d1480 msgr2=0x7efcb00d2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.669+0000 7efcb58c8640 1 --2- 172.21.15.72:0/3779405512 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcb00d1480 0x7efcb00d2e60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7efca800b1a0 tx=0x7efca802c290 comp rx=0 tx=0).stop 2024-04-15T15:15:10.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.669+0000 7efcb58c8640 1 -- 172.21.15.72:0/3779405512 shutdown_connections 2024-04-15T15:15:10.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.670+0000 7efcb58c8640 1 --2- 172.21.15.72:0/3779405512 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcb00d1480 0x7efcb00d2e60 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.670+0000 7efcb58c8640 1 -- 172.21.15.72:0/3779405512 >> 172.21.15.72:0/3779405512 conn(0x7efcb009d950 msgr2=0x7efcb00b9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:10.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7f0f5b8d3640 1 -- 172.21.15.72:0/3512745541 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f54145f40 con 0x7f0f54072fa0 2024-04-15T15:15:10.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.669+0000 7f7b2ffff640 1 -- 172.21.15.72:0/1532730000 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b30018070 con 0x7f7b3c03a0d0 2024-04-15T15:15:10.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.669+0000 7f7b2ffff640 1 -- 172.21.15.72:0/1532730000 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b300075a0 con 0x7f7b3c03a0d0 2024-04-15T15:15:10.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.669+0000 7f7b2ffff640 1 -- 172.21.15.72:0/1532730000 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b30004a20 con 0x7f7b3c03a0d0 2024-04-15T15:15:10.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.670+0000 7f7b49ddb640 1 -- 172.21.15.72:0/1532730000 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b3c00c1f0 con 0x7f7b3c03a0d0 2024-04-15T15:15:10.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.670+0000 7f54c5318640 1 -- 172.21.15.72:0/2826732973 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f54ac0051c0 msgr2=0x7f54ac00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:10.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.670+0000 7f7b49ddb640 1 -- 172.21.15.72:0/1532730000 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b3c04d300 con 0x7f7b3c03a0d0 2024-04-15T15:15:10.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.670+0000 7f403e7fc640 1 -- 172.21.15.72:0/4025438024 <== osd.2 v2:172.21.15.72:6816/4041627082 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f404c0af440 con 0x7f4030076040 2024-04-15T15:15:10.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.670+0000 7f40aeb52640 1 -- 172.21.15.72:0/371671587 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40a81fca40 con 0x7f40a8072fa0 2024-04-15T15:15:10.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.670+0000 7f40a57fa640 1 -- 172.21.15.72:0/371671587 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f409802fca0 con 0x7f40a8072fa0 2024-04-15T15:15:10.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.670+0000 7f40aeb52640 1 -- 172.21.15.72:0/371671587 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40a81f8460 con 0x7f40a8072fa0 2024-04-15T15:15:10.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.670+0000 7efcb58c8640 1 -- 172.21.15.72:0/3779405512 shutdown_connections 2024-04-15T15:15:10.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.670+0000 7efcb58c8640 1 -- 172.21.15.72:0/3779405512 wait complete. 2024-04-15T15:15:10.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.670+0000 7f54c5318640 1 --2- 172.21.15.72:0/2826732973 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f54ac0051c0 0x7f54ac00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.670+0000 7f54c5318640 1 -- 172.21.15.72:0/2826732973 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54ac004800 msgr2=0x7f54ac004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:10.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.670+0000 7f54c5318640 1 --2- 172.21.15.72:0/2826732973 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54ac004800 0x7f54ac004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.670+0000 7f54c5318640 1 -- 172.21.15.72:0/2826732973 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54c006de60 con 0x7f54ac0039b0 2024-04-15T15:15:10.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.670+0000 7f54c5b19640 1 --2- 172.21.15.72:0/2826732973 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f54ac0051c0 0x7f54ac00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:10.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.670+0000 7f54c5b19640 1 -- 172.21.15.72:0/2826732973 reap_dead start 2024-04-15T15:15:10.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.670+0000 7f0f377fe640 1 -- 172.21.15.72:0/3512745541 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0f541544f0 con 0x7f0f54072fa0 2024-04-15T15:15:10.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.669+0000 7ffa5ced3640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffa58161f80 0x7ffa580b8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.669+0000 7ffa5ced3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffa580b7760 0x7ffa581ec850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.670+0000 7ffa5659c640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffa58161f80 0x7ffa580b8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.670+0000 7ffa5659c640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffa58161f80 0x7ffa580b8380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3301/0 says I am v2:172.21.15.72:57792/0 (socket says 172.21.15.72:57792) 2024-04-15T15:15:10.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.670+0000 7ffa5659c640 1 -- 172.21.15.72:0/2777229421 learned_addr learned my addr 172.21.15.72:0/2777229421 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:10.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.670+0000 7ffa5ced3640 1 --2- 172.21.15.72:0/2777229421 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffa581ecd90 0x7ffa580b6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.670+0000 7ffa5ced3640 1 -- 172.21.15.72:0/2777229421 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ffa5806bea0 con 0x7ffa580b7760 2024-04-15T15:15:10.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.670+0000 7ffa5ced3640 1 -- 172.21.15.72:0/2777229421 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7ffa580d1340 con 0x7ffa581ecd90 2024-04-15T15:15:10.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.670+0000 7ffa5ced3640 1 -- 172.21.15.72:0/2777229421 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7ffa5806c020 con 0x7ffa58161f80 2024-04-15T15:15:10.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.670+0000 7ffa55d9b640 1 --2- 172.21.15.72:0/2777229421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffa580b7760 0x7ffa581ec850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.670+0000 7ffa5659c640 1 -- 172.21.15.72:0/2777229421 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffa581ecd90 msgr2=0x7ffa580b6630 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-15T15:15:10.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.670+0000 7ffa5659c640 1 --2- 172.21.15.72:0/2777229421 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffa581ecd90 0x7ffa580b6630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.670+0000 7ffa5659c640 1 -- 172.21.15.72:0/2777229421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffa580b7760 msgr2=0x7ffa581ec850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7fdd4e5a8640 1 -- 172.21.15.72:0/769761281 shutdown_connections 2024-04-15T15:15:10.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.668+0000 7fdd4e5a8640 1 -- 172.21.15.72:0/769761281 wait complete. 2024-04-15T15:15:10.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.670+0000 7fdd4e5a8640 1 Processor -- start 2024-04-15T15:15:10.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.670+0000 7fdd4e5a8640 1 -- start start 2024-04-15T15:15:10.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.671+0000 7f0f49ffb640 1 -- 172.21.15.72:0/3512745541 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0f4401b070 con 0x7f0f54072fa0 2024-04-15T15:15:10.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.671+0000 7f7b49ddb640 1 -- 172.21.15.72:0/1532730000 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7b3c00c430 con 0x7f7b3c03a0d0 2024-04-15T15:15:10.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.671+0000 7f40aeb52640 1 -- 172.21.15.72:0/371671587 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f40a814e060 con 0x7f40a8072fa0 2024-04-15T15:15:10.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.672+0000 7f0f49ffb640 1 --2- 172.21.15.72:0/3512745541 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0f3806c970 0x7f0f3806ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.672+0000 7fdd4e5a8640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdd380039d0 0x7fdd400d7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.672+0000 7fdd4e5a8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd4004f020 0x7fdd400e9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.672+0000 7fdd47fff640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdd380039d0 0x7fdd400d7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.672+0000 7fdd47fff640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdd380039d0 0x7fdd400d7170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3301/0 says I am v2:172.21.15.72:57798/0 (socket says 172.21.15.72:57798) 2024-04-15T15:15:10.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.672+0000 7fdd4e5a8640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdd400ea2d0 0x7fdd400db0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.672+0000 7fdd47fff640 1 -- 172.21.15.72:0/3903159186 learned_addr learned my addr 172.21.15.72:0/3903159186 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:10.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.672+0000 7fdd4e5a8640 1 -- 172.21.15.72:0/3903159186 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fdd40058d10 con 0x7fdd4004f020 2024-04-15T15:15:10.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.672+0000 7fdd4e5a8640 1 -- 172.21.15.72:0/3903159186 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fdd400ec1d0 con 0x7fdd400ea2d0 2024-04-15T15:15:10.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.672+0000 7fdd4e5a8640 1 -- 172.21.15.72:0/3903159186 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fdd40054720 con 0x7fdd380039d0 2024-04-15T15:15:10.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.672+0000 7efcb58c8640 1 Processor -- start 2024-04-15T15:15:10.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.672+0000 7fdd4cb45640 1 --2- 172.21.15.72:0/3903159186 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdd400ea2d0 0x7fdd400db0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.672+0000 7f7b2ffff640 1 -- 172.21.15.72:0/1532730000 <== mon.1 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7b3001b070 con 0x7f7b3c03a0d0 2024-04-15T15:15:10.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.672+0000 7f0f5966f640 1 --2- 172.21.15.72:0/3512745541 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0f3806c970 0x7f0f3806ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.672+0000 7efcb58c8640 1 -- start start 2024-04-15T15:15:10.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.672+0000 7f0f49ffb640 1 -- 172.21.15.72:0/3512745541 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f0f4408aea0 con 0x7f0f54072fa0 2024-04-15T15:15:10.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7f0f5966f640 1 --2- 172.21.15.72:0/3512745541 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0f3806c970 0x7f0f3806ee30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0f5415f3e0 tx=0x7f0f4c0099d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7f0f49ffb640 1 --2- 172.21.15.72:0/3512745541 >> [v2:172.21.15.175:6800/1465263431,v1:172.21.15.175:6801/1465263431] conn(0x7f0f380760e0 0x7f0f38078520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7f0f49ffb640 1 -- 172.21.15.72:0/3512745541 --> [v2:172.21.15.175:6800/1465263431,v1:172.21.15.175:6801/1465263431] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0f38078bf0 con 0x7f0f380760e0 2024-04-15T15:15:10.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7f0f49ffb640 1 -- 172.21.15.72:0/3512745541 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f0f44090060 con 0x7f0f54072fa0 2024-04-15T15:15:10.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7ffa5659c640 1 --2- 172.21.15.72:0/2777229421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffa580b7760 0x7ffa581ec850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7ffa5659c640 1 -- 172.21.15.72:0/2777229421 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa580b8960 con 0x7ffa58161f80 2024-04-15T15:15:10.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7ffa55d9b640 1 --2- 172.21.15.72:0/2777229421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffa580b7760 0x7ffa581ec850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:10.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7f54c5318640 1 --2- 172.21.15.72:0/2826732973 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f54ac0039b0 0x7f54ac003dd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f54b0009650 tx=0x7f54ac005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ff36ed88e17daa0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7f0f59e70640 1 --2- 172.21.15.72:0/3512745541 >> [v2:172.21.15.175:6800/1465263431,v1:172.21.15.175:6801/1465263431] conn(0x7f0f380760e0 0x7f0f38078520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7f7b2ffff640 1 --2- 172.21.15.72:0/1532730000 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7b2806c9c0 0x7f7b2806ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7fdd477fe640 1 --2- 172.21.15.72:0/3903159186 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd4004f020 0x7fdd400e9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7f7b2ffff640 1 -- 172.21.15.72:0/1532730000 <== mon.1 v2:172.21.15.175:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f7b3008b340 con 0x7f7b3c03a0d0 2024-04-15T15:15:10.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7f40a57fa640 1 -- 172.21.15.72:0/371671587 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4098038480 con 0x7f40a8072fa0 2024-04-15T15:15:10.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7efcb58c8640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efcb00b3590 0x7efcb01f4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7efcb58c8640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efcb00d1480 0x7efcb01f1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7f7b437fe640 1 --2- 172.21.15.72:0/1532730000 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7b2806c9c0 0x7f7b2806ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7f7b2ffff640 1 --2- 172.21.15.72:0/1532730000 >> [v2:172.21.15.72:6800/1858587354,v1:172.21.15.72:6801/1858587354] conn(0x7f7b28076140 0x7f7b28078580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7f7b2ffff640 1 -- 172.21.15.72:0/1532730000 --> [v2:172.21.15.72:6800/1858587354,v1:172.21.15.72:6801/1858587354] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7b30010e50 con 0x7f7b28076140 2024-04-15T15:15:10.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7f7b2ffff640 1 -- 172.21.15.72:0/1532730000 <== mon.1 v2:172.21.15.175:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f7b30092050 con 0x7f7b3c03a0d0 2024-04-15T15:15:10.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7efcb58c8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcb00d4900 0x7efcb00b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7fdd4cb45640 1 -- 172.21.15.72:0/3903159186 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdd380039d0 msgr2=0x7fdd400d7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7fdd4cb45640 1 --2- 172.21.15.72:0/3903159186 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdd380039d0 0x7fdd400d7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7fdd4cb45640 1 -- 172.21.15.72:0/3903159186 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd4004f020 msgr2=0x7fdd400e9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7fdd4cb45640 1 --2- 172.21.15.72:0/3903159186 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd4004f020 0x7fdd400e9d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7fdd4cb45640 1 -- 172.21.15.72:0/3903159186 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd400df770 con 0x7fdd400ea2d0 2024-04-15T15:15:10.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7efcb58c8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7efcb015ed20 con 0x7efcb00d4900 2024-04-15T15:15:10.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7efcb58c8640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7efcb00d2230 con 0x7efcb00b3590 2024-04-15T15:15:10.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7efcb58c8640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7efcb015eea0 con 0x7efcb00d1480 2024-04-15T15:15:10.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7fdd4cb45640 1 --2- 172.21.15.72:0/3903159186 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdd400ea2d0 0x7fdd400db0d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdd4806a310 tx=0x7fdd4806dd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7efcaeffd640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efcb00b3590 0x7efcb01f4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7efcaf7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcb00d4900 0x7efcb00b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7efcaeffd640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efcb00b3590 0x7efcb01f4d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.72:60038/0 (socket says 172.21.15.72:60038) 2024-04-15T15:15:10.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7efcaeffd640 1 -- 172.21.15.72:0/2863945513 learned_addr learned my addr 172.21.15.72:0/2863945513 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:10.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7f54b7fff640 1 -- 172.21.15.72:0/2826732973 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b003d070 con 0x7f54ac0039b0 2024-04-15T15:15:10.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7f54b7fff640 1 -- 172.21.15.72:0/2826732973 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54b0030ac0 con 0x7f54ac0039b0 2024-04-15T15:15:10.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7f54b7fff640 1 -- 172.21.15.72:0/2826732973 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54b0009690 con 0x7f54ac0039b0 2024-04-15T15:15:10.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.673+0000 7ffa5659c640 1 --2- 172.21.15.72:0/2777229421 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffa58161f80 0x7ffa580b8380 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c000c00 tx=0x7ffa4c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.674+0000 7efca7fff640 1 --2- 172.21.15.72:0/2863945513 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efcb00d1480 0x7efcb01f1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.674+0000 7f0f59e70640 1 --2- 172.21.15.72:0/3512745541 >> [v2:172.21.15.175:6800/1465263431,v1:172.21.15.175:6801/1465263431] conn(0x7f0f380760e0 0x7f0f38078520 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.674+0000 7f7b43fff640 1 --2- 172.21.15.72:0/1532730000 >> [v2:172.21.15.72:6800/1858587354,v1:172.21.15.72:6801/1858587354] conn(0x7f7b28076140 0x7f7b28078580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.674+0000 7ffa46ffd640 1 -- 172.21.15.72:0/2777229421 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c044020 con 0x7ffa58161f80 2024-04-15T15:15:10.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.674+0000 7ffa46ffd640 1 -- 172.21.15.72:0/2777229421 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa4c02de10 con 0x7ffa58161f80 2024-04-15T15:15:10.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.674+0000 7ffa46ffd640 1 -- 172.21.15.72:0/2777229421 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c039040 con 0x7ffa58161f80 2024-04-15T15:15:10.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.674+0000 7fdd44ff9640 1 -- 172.21.15.72:0/3903159186 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4806b960 con 0x7fdd400ea2d0 2024-04-15T15:15:10.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.674+0000 7fdd44ff9640 1 -- 172.21.15.72:0/3903159186 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd48091070 con 0x7fdd400ea2d0 2024-04-15T15:15:10.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.674+0000 7efcaeffd640 1 -- 172.21.15.72:0/2863945513 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efcb00d1480 msgr2=0x7efcb01f1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.674+0000 7efcaeffd640 1 --2- 172.21.15.72:0/2863945513 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efcb00d1480 0x7efcb01f1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.674+0000 7efcaeffd640 1 -- 172.21.15.72:0/2863945513 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcb00d4900 msgr2=0x7efcb00b4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.674+0000 7efcaeffd640 1 --2- 172.21.15.72:0/2863945513 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcb00d4900 0x7efcb00b4f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.674+0000 7efcaeffd640 1 -- 172.21.15.72:0/2863945513 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcb01f5250 con 0x7efcb00b3590 2024-04-15T15:15:10.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.674+0000 7fdd44ff9640 1 -- 172.21.15.72:0/3903159186 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4806bd00 con 0x7fdd400ea2d0 2024-04-15T15:15:10.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.674+0000 7fdd4e5a8640 1 -- 172.21.15.72:0/3903159186 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd400ea8e0 con 0x7fdd400ea2d0 2024-04-15T15:15:10.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.674+0000 7fdd4e5a8640 1 -- 172.21.15.72:0/3903159186 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd400d6970 con 0x7fdd400ea2d0 2024-04-15T15:15:10.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.674+0000 7f7b437fe640 1 --2- 172.21.15.72:0/1532730000 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7b2806c9c0 0x7f7b2806ee80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c006de0 tx=0x7f7b440ab000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.674+0000 7f54c757c640 1 -- 172.21.15.72:0/2826732973 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f54ac0039b0 msgr2=0x7f54ac003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.674+0000 7f54c757c640 1 --2- 172.21.15.72:0/2826732973 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f54ac0039b0 0x7f54ac003dd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f54b0009650 tx=0x7f54ac005830 comp rx=0 tx=0).stop 2024-04-15T15:15:10.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.674+0000 7f0f49ffb640 1 -- 172.21.15.72:0/3512745541 <== osd.5 v2:172.21.15.175:6800/1465263431 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0f38078bf0 con 0x7f0f380760e0 2024-04-15T15:15:10.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.675+0000 7f54c757c640 1 -- 172.21.15.72:0/2826732973 shutdown_connections 2024-04-15T15:15:10.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.675+0000 7f54c757c640 1 --2- 172.21.15.72:0/2826732973 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f54ac0039b0 0x7f54ac003dd0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.675+0000 7f54c757c640 1 -- 172.21.15.72:0/2826732973 >> 172.21.15.72:0/2826732973 conn(0x7f54c00b1dd0 msgr2=0x7f54c00b21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:10.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.674+0000 7f7b43fff640 1 --2- 172.21.15.72:0/1532730000 >> [v2:172.21.15.72:6800/1858587354,v1:172.21.15.72:6801/1858587354] conn(0x7f7b28076140 0x7f7b28078580 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.675+0000 7ffa5ced3640 1 -- 172.21.15.72:0/2777229421 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa580b8c40 con 0x7ffa58161f80 2024-04-15T15:15:10.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.675+0000 7efcaeffd640 1 --2- 172.21.15.72:0/2863945513 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efcb00b3590 0x7efcb01f4d10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efca000c500 tx=0x7efca000c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.675+0000 7f54c757c640 1 -- 172.21.15.72:0/2826732973 shutdown_connections 2024-04-15T15:15:10.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.675+0000 7ffa5ced3640 1 -- 172.21.15.72:0/2777229421 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa580c3c30 con 0x7ffa58161f80 2024-04-15T15:15:10.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.675+0000 7fdd4e5a8640 1 -- 172.21.15.72:0/3903159186 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdd40005810 con 0x7fdd400ea2d0 2024-04-15T15:15:10.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.675+0000 7efcacff9640 1 -- 172.21.15.72:0/2863945513 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca00052d0 con 0x7efcb00b3590 2024-04-15T15:15:10.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.675+0000 7efcacff9640 1 -- 172.21.15.72:0/2863945513 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efca0005470 con 0x7efcb00b3590 2024-04-15T15:15:10.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.676+0000 7ffa5ced3640 1 -- 172.21.15.72:0/2777229421 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ffa58075b10 con 0x7ffa58161f80 2024-04-15T15:15:10.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.676+0000 7f7b2ffff640 1 -- 172.21.15.72:0/1532730000 <== osd.0 v2:172.21.15.72:6800/1858587354 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7b30010e50 con 0x7f7b28076140 2024-04-15T15:15:10.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.676+0000 7efcacff9640 1 -- 172.21.15.72:0/2863945513 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca0012830 con 0x7efcb00b3590 2024-04-15T15:15:10.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.675+0000 7f54c757c640 1 -- 172.21.15.72:0/2826732973 wait complete. 2024-04-15T15:15:10.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.676+0000 7fdd44ff9640 1 -- 172.21.15.72:0/3903159186 <== mon.1 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdd4806d110 con 0x7fdd400ea2d0 2024-04-15T15:15:10.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.676+0000 7fdd44ff9640 1 --2- 172.21.15.72:0/3903159186 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fdd2c06c940 0x7fdd2c06ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.677+0000 7efcb58c8640 1 -- 172.21.15.72:0/2863945513 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcb01f1ca0 con 0x7efcb00b3590 2024-04-15T15:15:10.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.677+0000 7fdd44ff9640 1 -- 172.21.15.72:0/3903159186 <== mon.1 v2:172.21.15.175:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fdd48106970 con 0x7fdd400ea2d0 2024-04-15T15:15:10.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.677+0000 7efcb58c8640 1 -- 172.21.15.72:0/2863945513 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcb015f620 con 0x7efcb00b3590 2024-04-15T15:15:10.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.677+0000 7fdd47fff640 1 --2- 172.21.15.72:0/3903159186 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fdd2c06c940 0x7fdd2c06ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.677+0000 7fdd44ff9640 1 --2- 172.21.15.72:0/3903159186 >> [v2:172.21.15.175:6816/2672274294,v1:172.21.15.175:6817/2672274294] conn(0x7fdd2c076040 0x7fdd2c078480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.677+0000 7fdd44ff9640 1 -- 172.21.15.72:0/3903159186 --> [v2:172.21.15.175:6816/2672274294,v1:172.21.15.175:6817/2672274294] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdd4808ee50 con 0x7fdd2c076040 2024-04-15T15:15:10.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.677+0000 7fdd44ff9640 1 -- 172.21.15.72:0/3903159186 <== mon.1 v2:172.21.15.175:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fdd480d0670 con 0x7fdd400ea2d0 2024-04-15T15:15:10.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.677+0000 7f54c757c640 1 Processor -- start 2024-04-15T15:15:10.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.675+0000 7f40a57fa640 1 --2- 172.21.15.72:0/371671587 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f409406c9f0 0x7f409406eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.675+0000 7f40a57fa640 1 -- 172.21.15.72:0/371671587 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f40980ad9e0 con 0x7f40a8072fa0 2024-04-15T15:15:10.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.677+0000 7fdd477fe640 1 --2- 172.21.15.72:0/3903159186 >> [v2:172.21.15.175:6816/2672274294,v1:172.21.15.175:6817/2672274294] conn(0x7fdd2c076040 0x7fdd2c078480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.675+0000 7f40ac8ee640 1 --2- 172.21.15.72:0/371671587 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f409406c9f0 0x7f409406eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.676+0000 7f40a57fa640 1 --2- 172.21.15.72:0/371671587 >> [v2:172.21.15.72:6824/3675605364,v1:172.21.15.72:6825/3675605364] conn(0x7f4094076160 0x7f40940785a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.676+0000 7f40ad0ef640 1 --2- 172.21.15.72:0/371671587 >> [v2:172.21.15.72:6824/3675605364,v1:172.21.15.72:6825/3675605364] conn(0x7f4094076160 0x7f40940785a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.676+0000 7f40ac8ee640 1 --2- 172.21.15.72:0/371671587 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f409406c9f0 0x7f409406eeb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f409c000f60 tx=0x7f409c007b70 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.677+0000 7f40a57fa640 1 -- 172.21.15.72:0/371671587 --> [v2:172.21.15.72:6824/3675605364,v1:172.21.15.72:6825/3675605364] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4098001d30 con 0x7f4094076160 2024-04-15T15:15:10.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.677+0000 7f40a57fa640 1 -- 172.21.15.72:0/371671587 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f40980b22a0 con 0x7f40a8072fa0 2024-04-15T15:15:10.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.677+0000 7f40ad0ef640 1 --2- 172.21.15.72:0/371671587 >> [v2:172.21.15.72:6824/3675605364,v1:172.21.15.72:6825/3675605364] conn(0x7f4094076160 0x7f40940785a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.678+0000 7f54c757c640 1 -- start start 2024-04-15T15:15:10.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.678+0000 7ffa46ffd640 1 -- 172.21.15.72:0/2777229421 <== mon.2 v2:172.21.15.175:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ffa4c047020 con 0x7ffa58161f80 2024-04-15T15:15:10.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.678+0000 7efcb58c8640 1 -- 172.21.15.72:0/2863945513 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7efcb00690f0 con 0x7efcb00b3590 2024-04-15T15:15:10.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.678+0000 7fdd47fff640 1 --2- 172.21.15.72:0/3903159186 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fdd2c06c940 0x7fdd2c06ee00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fdd3400ba30 tx=0x7fdd3403d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.679+0000 7f40a57fa640 1 -- 172.21.15.72:0/371671587 <== osd.1 v2:172.21.15.72:6824/3675605364 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4098001d30 con 0x7f4094076160 2024-04-15T15:15:10.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.679+0000 7fdd477fe640 1 --2- 172.21.15.72:0/3903159186 >> [v2:172.21.15.175:6816/2672274294,v1:172.21.15.175:6817/2672274294] conn(0x7fdd2c076040 0x7fdd2c078480 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.679+0000 7ffa46ffd640 1 --2- 172.21.15.72:0/2777229421 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ffa3806c9c0 0x7ffa3806ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.679+0000 7f54c757c640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f54ac0039b0 0x7f54c0157410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.679+0000 7ffa46ffd640 1 -- 172.21.15.72:0/2777229421 <== mon.2 v2:172.21.15.175:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ffa4c0ae950 con 0x7ffa58161f80 2024-04-15T15:15:10.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.679+0000 7f54c757c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54c0158020 0x7f54c01584a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.679+0000 7ffa55d9b640 1 --2- 172.21.15.72:0/2777229421 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ffa3806c9c0 0x7ffa3806ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.679+0000 7f54c5318640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f54ac0039b0 0x7f54c0157410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.679+0000 7f54c757c640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f54c01529f0 0x7f54c0152e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.679+0000 7f54c757c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f54c00cfab0 con 0x7f54c0158020 2024-04-15T15:15:10.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.679+0000 7f54c757c640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f54c00cf7d0 con 0x7f54ac0039b0 2024-04-15T15:15:10.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.679+0000 7f54c757c640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f54c00cdc60 con 0x7f54c01529f0 2024-04-15T15:15:10.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.679+0000 7ffa46ffd640 1 --2- 172.21.15.72:0/2777229421 >> [v2:172.21.15.175:6808/1973126001,v1:172.21.15.175:6809/1973126001] conn(0x7ffa38076140 0x7ffa38078580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.679+0000 7ffa46ffd640 1 -- 172.21.15.72:0/2777229421 --> [v2:172.21.15.175:6808/1973126001,v1:172.21.15.175:6809/1973126001] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ffa4c00e020 con 0x7ffa38076140 2024-04-15T15:15:10.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.679+0000 7ffa46ffd640 1 -- 172.21.15.72:0/2777229421 <== mon.2 v2:172.21.15.175:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ffa4c0aed40 con 0x7ffa58161f80 2024-04-15T15:15:10.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.679+0000 7f54c5318640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f54ac0039b0 0x7f54c0157410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.72:60040/0 (socket says 172.21.15.72:60040) 2024-04-15T15:15:10.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.679+0000 7f54c5318640 1 -- 172.21.15.72:0/2672574873 learned_addr learned my addr 172.21.15.72:0/2672574873 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:10.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.679+0000 7f54c5b19640 1 --2- 172.21.15.72:0/2672574873 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f54c01529f0 0x7f54c0152e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.680+0000 7fdd44ff9640 1 -- 172.21.15.72:0/3903159186 <== osd.7 v2:172.21.15.175:6816/2672274294 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdd4808ee50 con 0x7fdd2c076040 2024-04-15T15:15:10.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.679+0000 7ffa56d9d640 1 --2- 172.21.15.72:0/2777229421 >> [v2:172.21.15.175:6808/1973126001,v1:172.21.15.175:6809/1973126001] conn(0x7ffa38076140 0x7ffa38078580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.680+0000 7f54c4b17640 1 --2- 172.21.15.72:0/2672574873 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54c0158020 0x7f54c01584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.680+0000 7efcacff9640 1 -- 172.21.15.72:0/2863945513 <== mon.1 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efca00129d0 con 0x7efcb00b3590 2024-04-15T15:15:10.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.680+0000 7ffa56d9d640 1 --2- 172.21.15.72:0/2777229421 >> [v2:172.21.15.175:6808/1973126001,v1:172.21.15.175:6809/1973126001] conn(0x7ffa38076140 0x7ffa38078580 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.680+0000 7ffa55d9b640 1 --2- 172.21.15.72:0/2777229421 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ffa3806c9c0 0x7ffa3806ee80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ffa50000ba0 tx=0x7ffa5000b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.680+0000 7f54c5b19640 1 -- 172.21.15.72:0/2672574873 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f54ac0039b0 msgr2=0x7f54c0157410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.680+0000 7f54c5b19640 1 --2- 172.21.15.72:0/2672574873 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f54ac0039b0 0x7f54c0157410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.680+0000 7f54c5b19640 1 -- 172.21.15.72:0/2672574873 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54c0158020 msgr2=0x7f54c01584a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.680+0000 7f54c5b19640 1 --2- 172.21.15.72:0/2672574873 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54c0158020 0x7f54c01584a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.680+0000 7f54c5b19640 1 -- 172.21.15.72:0/2672574873 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54c0143c80 con 0x7f54c01529f0 2024-04-15T15:15:10.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.680+0000 7f54c5b19640 1 --2- 172.21.15.72:0/2672574873 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f54c01529f0 0x7f54c0152e70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f54c0068540 tx=0x7f54bc010590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.680+0000 7f54b5ffb640 1 -- 172.21.15.72:0/2672574873 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54bc003990 con 0x7f54c01529f0 2024-04-15T15:15:10.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.680+0000 7f54b5ffb640 1 -- 172.21.15.72:0/2672574873 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54bc01b070 con 0x7f54c01529f0 2024-04-15T15:15:10.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.680+0000 7f54b5ffb640 1 -- 172.21.15.72:0/2672574873 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54bc003d90 con 0x7f54c01529f0 2024-04-15T15:15:10.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.680+0000 7efcacff9640 1 --2- 172.21.15.72:0/2863945513 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7efc9006c9c0 0x7efc9006ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.681+0000 7efcacff9640 1 -- 172.21.15.72:0/2863945513 <== mon.1 v2:172.21.15.175:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7efca008ba30 con 0x7efcb00b3590 2024-04-15T15:15:10.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.681+0000 7f54c5318640 1 --2- 172.21.15.72:0/2672574873 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f54ac0039b0 0x7f54c0157410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:10.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.681+0000 7efcacff9640 1 --2- 172.21.15.72:0/2863945513 >> [v2:172.21.15.72:6808/3973649661,v1:172.21.15.72:6809/3973649661] conn(0x7efc90076130 0x7efc90078570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.681+0000 7efcacff9640 1 -- 172.21.15.72:0/2863945513 --> [v2:172.21.15.72:6808/3973649661,v1:172.21.15.72:6809/3973649661] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efca0020e50 con 0x7efc90076130 2024-04-15T15:15:10.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.681+0000 7efcacff9640 1 -- 172.21.15.72:0/2863945513 <== mon.1 v2:172.21.15.175:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7efca00902a0 con 0x7efcb00b3590 2024-04-15T15:15:10.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.681+0000 7efcaf7fe640 1 --2- 172.21.15.72:0/2863945513 >> [v2:172.21.15.72:6808/3973649661,v1:172.21.15.72:6809/3973649661] conn(0x7efc90076130 0x7efc90078570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.681+0000 7f54c757c640 1 -- 172.21.15.72:0/2672574873 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54c0157950 con 0x7f54c01529f0 2024-04-15T15:15:10.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.681+0000 7f54c4b17640 1 --2- 172.21.15.72:0/2672574873 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54c0158020 0x7f54c01584a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:10.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.681+0000 7f54c757c640 1 -- 172.21.15.72:0/2672574873 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f54c01539b0 con 0x7f54c01529f0 2024-04-15T15:15:10.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.682+0000 7ffa46ffd640 1 -- 172.21.15.72:0/2777229421 <== osd.6 v2:172.21.15.175:6808/1973126001 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ffa4c00e020 con 0x7ffa38076140 2024-04-15T15:15:10.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.681+0000 7efca7fff640 1 --2- 172.21.15.72:0/2863945513 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7efc9006c9c0 0x7efc9006ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.682+0000 7efcaf7fe640 1 --2- 172.21.15.72:0/2863945513 >> [v2:172.21.15.72:6808/3973649661,v1:172.21.15.72:6809/3973649661] conn(0x7efc90076130 0x7efc90078570 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.684+0000 7efca7fff640 1 --2- 172.21.15.72:0/2863945513 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7efc9006c9c0 0x7efc9006ee80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7efcb00b1d50 tx=0x7efc98009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.683+0000 7f54c757c640 1 -- 172.21.15.72:0/2672574873 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f54c0151390 con 0x7f54c01529f0 2024-04-15T15:15:10.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.683+0000 7f54b5ffb640 1 -- 172.21.15.72:0/2672574873 <== mon.2 v2:172.21.15.175:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f54bc004030 con 0x7f54c01529f0 2024-04-15T15:15:10.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.684+0000 7f54b5ffb640 1 --2- 172.21.15.72:0/2672574873 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f54a40751d0 0x7f54a4077690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.684+0000 7efcacff9640 1 -- 172.21.15.72:0/2863945513 <== osd.3 v2:172.21.15.72:6808/3973649661 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efca0020e50 con 0x7efc90076130 2024-04-15T15:15:10.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.684+0000 7f54b5ffb640 1 -- 172.21.15.72:0/2672574873 <== mon.2 v2:172.21.15.175:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f54bc08cb10 con 0x7f54c01529f0 2024-04-15T15:15:10.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.684+0000 7f54b5ffb640 1 --2- 172.21.15.72:0/2672574873 >> [v2:172.21.15.175:6823/626414407,v1:172.21.15.175:6825/626414407] conn(0x7f54a407e8d0 0x7f54a4080d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.684+0000 7f54b5ffb640 1 -- 172.21.15.72:0/2672574873 --> [v2:172.21.15.175:6823/626414407,v1:172.21.15.175:6825/626414407] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f54bc009200 con 0x7f54a407e8d0 2024-04-15T15:15:10.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.684+0000 7f54b5ffb640 1 -- 172.21.15.72:0/2672574873 <== mon.2 v2:172.21.15.175:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f54bc056810 con 0x7f54c01529f0 2024-04-15T15:15:10.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.684+0000 7f54c5318640 1 --2- 172.21.15.72:0/2672574873 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f54a40751d0 0x7f54a4077690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.684+0000 7f54c4b17640 1 --2- 172.21.15.72:0/2672574873 >> [v2:172.21.15.175:6823/626414407,v1:172.21.15.175:6825/626414407] conn(0x7f54a407e8d0 0x7f54a4080d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.685+0000 7f54c5318640 1 --2- 172.21.15.72:0/2672574873 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f54a40751d0 0x7f54a4077690 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f54b0000c00 tx=0x7f54b0034000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.686+0000 7f54c4b17640 1 --2- 172.21.15.72:0/2672574873 >> [v2:172.21.15.175:6823/626414407,v1:172.21.15.175:6825/626414407] conn(0x7f54a407e8d0 0x7f54a4080d10 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.687+0000 7f54b5ffb640 1 -- 172.21.15.72:0/2672574873 <== osd.4 v2:172.21.15.175:6823/626414407 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f54bc009200 con 0x7f54a407e8d0 2024-04-15T15:15:10.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.703+0000 7f4053af1640 1 -- 172.21.15.72:0/4025438024 --> [v2:172.21.15.72:6816/4041627082,v1:172.21.15.72:6817/4041627082] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4044050230 con 0x7f4030076040 2024-04-15T15:15:10.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.704+0000 7f403e7fc640 1 -- 172.21.15.72:0/4025438024 <== osd.2 v2:172.21.15.72:6816/4041627082 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f4044050230 con 0x7f4030076040 2024-04-15T15:15:10.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.705+0000 7f402ffff640 1 -- 172.21.15.72:0/4025438024 >> [v2:172.21.15.72:6816/4041627082,v1:172.21.15.72:6817/4041627082] conn(0x7f4030076040 msgr2=0x7f4030078480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.705+0000 7f402ffff640 1 --2- 172.21.15.72:0/4025438024 >> [v2:172.21.15.72:6816/4041627082,v1:172.21.15.72:6817/4041627082] conn(0x7f4030076040 0x7f4030078480 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.706+0000 7f402ffff640 1 -- 172.21.15.72:0/4025438024 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f403006c940 msgr2=0x7f403006ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.706+0000 7f402ffff640 1 --2- 172.21.15.72:0/4025438024 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f403006c940 0x7f403006ee00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4048003d90 tx=0x7f4048009040 comp rx=0 tx=0).stop 2024-04-15T15:15:10.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.706+0000 7f402ffff640 1 -- 172.21.15.72:0/4025438024 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f40440eb9b0 msgr2=0x7f40440e50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.706+0000 7f402ffff640 1 --2- 172.21.15.72:0/4025438024 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f40440eb9b0 0x7f40440e50a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f404c06a3d0 tx=0x7f404c06b770 comp rx=0 tx=0).stop 2024-04-15T15:15:10.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.706+0000 7f405208e640 1 -- 172.21.15.72:0/4025438024 reap_dead start 2024-04-15T15:15:10.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.706+0000 7f402ffff640 1 -- 172.21.15.72:0/4025438024 shutdown_connections 2024-04-15T15:15:10.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.706+0000 7f402ffff640 1 -- 172.21.15.72:0/4025438024 >> 172.21.15.72:0/4025438024 conn(0x7f4044055c40 msgr2=0x7f4044056020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:10.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.707+0000 7f402ffff640 1 -- 172.21.15.72:0/4025438024 shutdown_connections 2024-04-15T15:15:10.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.707+0000 7f402ffff640 1 -- 172.21.15.72:0/4025438024 wait complete. 2024-04-15T15:15:10.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.707+0000 7f0f377fe640 1 -- 172.21.15.72:0/3512745541 --> [v2:172.21.15.175:6800/1465263431,v1:172.21.15.175:6801/1465263431] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0f540b05f0 con 0x7f0f380760e0 2024-04-15T15:15:10.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.708+0000 7f0f49ffb640 1 -- 172.21.15.72:0/3512745541 <== osd.5 v2:172.21.15.175:6800/1465263431 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0f540b05f0 con 0x7f0f380760e0 2024-04-15T15:15:10.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.708+0000 7f7b49ddb640 1 -- 172.21.15.72:0/1532730000 --> [v2:172.21.15.72:6800/1858587354,v1:172.21.15.72:6801/1858587354] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7b3c04d500 con 0x7f7b28076140 2024-04-15T15:15:10.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.708+0000 7f0f5b8d3640 1 -- 172.21.15.72:0/3512745541 >> [v2:172.21.15.175:6800/1465263431,v1:172.21.15.175:6801/1465263431] conn(0x7f0f380760e0 msgr2=0x7f0f38078520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.708+0000 7f0f5b8d3640 1 --2- 172.21.15.72:0/3512745541 >> [v2:172.21.15.175:6800/1465263431,v1:172.21.15.175:6801/1465263431] conn(0x7f0f380760e0 0x7f0f38078520 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.708+0000 7f0f5b8d3640 1 -- 172.21.15.72:0/3512745541 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0f3806c970 msgr2=0x7f0f3806ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.708+0000 7f0f5b8d3640 1 --2- 172.21.15.72:0/3512745541 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0f3806c970 0x7f0f3806ee30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0f5415f3e0 tx=0x7f0f4c0099d0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.708+0000 7f0f5b8d3640 1 -- 172.21.15.72:0/3512745541 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f54072fa0 msgr2=0x7f0f54152ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.708+0000 7f0f5b8d3640 1 --2- 172.21.15.72:0/3512745541 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0f54072fa0 0x7f0f54152ec0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0f4400c480 tx=0x7f0f4400c940 comp rx=0 tx=0).stop 2024-04-15T15:15:10.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.709+0000 7f0f59e70640 1 -- 172.21.15.72:0/3512745541 reap_dead start 2024-04-15T15:15:10.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.709+0000 7f7b2ffff640 1 -- 172.21.15.72:0/1532730000 <== osd.0 v2:172.21.15.72:6800/1858587354 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7b3c04d500 con 0x7f7b28076140 2024-04-15T15:15:10.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.709+0000 7fdd4e5a8640 1 -- 172.21.15.72:0/3903159186 --> [v2:172.21.15.175:6816/2672274294,v1:172.21.15.175:6817/2672274294] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdd40056d70 con 0x7fdd2c076040 2024-04-15T15:15:10.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.709+0000 7f0f5b8d3640 1 -- 172.21.15.72:0/3512745541 shutdown_connections 2024-04-15T15:15:10.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.709+0000 7f0f5b8d3640 1 -- 172.21.15.72:0/3512745541 >> 172.21.15.72:0/3512745541 conn(0x7f0f540b5af0 msgr2=0x7f0f541f2c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:10.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.709+0000 7f0f5b8d3640 1 -- 172.21.15.72:0/3512745541 shutdown_connections 2024-04-15T15:15:10.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.709+0000 7f0f5b8d3640 1 -- 172.21.15.72:0/3512745541 wait complete. 2024-04-15T15:15:10.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.709+0000 7f7b2dffb640 1 -- 172.21.15.72:0/1532730000 >> [v2:172.21.15.72:6800/1858587354,v1:172.21.15.72:6801/1858587354] conn(0x7f7b28076140 msgr2=0x7f7b28078580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.710+0000 7f7b2dffb640 1 --2- 172.21.15.72:0/1532730000 >> [v2:172.21.15.72:6800/1858587354,v1:172.21.15.72:6801/1858587354] conn(0x7f7b28076140 0x7f7b28078580 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.710+0000 7f7b2dffb640 1 -- 172.21.15.72:0/1532730000 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7b2806c9c0 msgr2=0x7f7b2806ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.710+0000 7f7b2dffb640 1 --2- 172.21.15.72:0/1532730000 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7b2806c9c0 0x7f7b2806ee80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7b3c006de0 tx=0x7f7b440ab000 comp rx=0 tx=0).stop 2024-04-15T15:15:10.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.710+0000 7f7b2dffb640 1 -- 172.21.15.72:0/1532730000 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b3c03a0d0 msgr2=0x7f7b3c0dd360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.710+0000 7f7b2dffb640 1 --2- 172.21.15.72:0/1532730000 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7b3c03a0d0 0x7f7b3c0dd360 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7b3000c4a0 tx=0x7f7b3000c960 comp rx=0 tx=0).stop 2024-04-15T15:15:10.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.710+0000 7fdd44ff9640 1 -- 172.21.15.72:0/3903159186 <== osd.7 v2:172.21.15.175:6816/2672274294 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fdd40056d70 con 0x7fdd2c076040 2024-04-15T15:15:10.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.710+0000 7f7b43fff640 1 -- 172.21.15.72:0/1532730000 reap_dead start 2024-04-15T15:15:10.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.710+0000 7f7b2dffb640 1 -- 172.21.15.72:0/1532730000 shutdown_connections 2024-04-15T15:15:10.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.710+0000 7f7b2dffb640 1 -- 172.21.15.72:0/1532730000 >> 172.21.15.72:0/1532730000 conn(0x7f7b3c01e6f0 msgr2=0x7f7b3c00b500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:10.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.710+0000 7f7b2dffb640 1 -- 172.21.15.72:0/1532730000 shutdown_connections 2024-04-15T15:15:10.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.710+0000 7f7b2dffb640 1 -- 172.21.15.72:0/1532730000 wait complete. 2024-04-15T15:15:10.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.711+0000 7fdd4e5a8640 1 -- 172.21.15.72:0/3903159186 >> [v2:172.21.15.175:6816/2672274294,v1:172.21.15.175:6817/2672274294] conn(0x7fdd2c076040 msgr2=0x7fdd2c078480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.711+0000 7fdd4e5a8640 1 --2- 172.21.15.72:0/3903159186 >> [v2:172.21.15.175:6816/2672274294,v1:172.21.15.175:6817/2672274294] conn(0x7fdd2c076040 0x7fdd2c078480 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.711+0000 7fdd4e5a8640 1 -- 172.21.15.72:0/3903159186 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fdd2c06c940 msgr2=0x7fdd2c06ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.711+0000 7fdd4e5a8640 1 --2- 172.21.15.72:0/3903159186 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fdd2c06c940 0x7fdd2c06ee00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fdd3400ba30 tx=0x7fdd3403d040 comp rx=0 tx=0).stop 2024-04-15T15:15:10.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.711+0000 7fdd4e5a8640 1 -- 172.21.15.72:0/3903159186 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdd400ea2d0 msgr2=0x7fdd400db0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.711+0000 7fdd4e5a8640 1 --2- 172.21.15.72:0/3903159186 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdd400ea2d0 0x7fdd400db0d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fdd4806a310 tx=0x7fdd4806dd50 comp rx=0 tx=0).stop 2024-04-15T15:15:10.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.711+0000 7fdd4cb45640 1 -- 172.21.15.72:0/3903159186 reap_dead start 2024-04-15T15:15:10.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.711+0000 7fdd4e5a8640 1 -- 172.21.15.72:0/3903159186 shutdown_connections 2024-04-15T15:15:10.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.711+0000 7fdd4e5a8640 1 -- 172.21.15.72:0/3903159186 >> 172.21.15.72:0/3903159186 conn(0x7fdd40055a10 msgr2=0x7fdd40055e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:10.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.712+0000 7fdd4e5a8640 1 -- 172.21.15.72:0/3903159186 shutdown_connections 2024-04-15T15:15:10.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.712+0000 7fdd4e5a8640 1 -- 172.21.15.72:0/3903159186 wait complete. 2024-04-15T15:15:10.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.712+0000 7f40aeb52640 1 -- 172.21.15.72:0/371671587 --> [v2:172.21.15.72:6824/3675605364,v1:172.21.15.72:6825/3675605364] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f40a80689e0 con 0x7f4094076160 2024-04-15T15:15:10.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.712+0000 7f40a57fa640 1 -- 172.21.15.72:0/371671587 <== osd.1 v2:172.21.15.72:6824/3675605364 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f40a80689e0 con 0x7f4094076160 2024-04-15T15:15:10.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.713+0000 7ffa5ced3640 1 -- 172.21.15.72:0/2777229421 --> [v2:172.21.15.175:6808/1973126001,v1:172.21.15.175:6809/1973126001] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ffa58075b10 con 0x7ffa38076140 2024-04-15T15:15:10.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.713+0000 7f40aeb52640 1 -- 172.21.15.72:0/371671587 >> [v2:172.21.15.72:6824/3675605364,v1:172.21.15.72:6825/3675605364] conn(0x7f4094076160 msgr2=0x7f40940785a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.713+0000 7f40aeb52640 1 --2- 172.21.15.72:0/371671587 >> [v2:172.21.15.72:6824/3675605364,v1:172.21.15.72:6825/3675605364] conn(0x7f4094076160 0x7f40940785a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.713+0000 7f40aeb52640 1 -- 172.21.15.72:0/371671587 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f409406c9f0 msgr2=0x7f409406eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.713+0000 7f40aeb52640 1 --2- 172.21.15.72:0/371671587 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f409406c9f0 0x7f409406eeb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f409c000f60 tx=0x7f409c007b70 comp rx=0 tx=0).stop 2024-04-15T15:15:10.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.713+0000 7f40aeb52640 1 -- 172.21.15.72:0/371671587 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f40a8072fa0 msgr2=0x7f40a81f7f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.713+0000 7f40aeb52640 1 --2- 172.21.15.72:0/371671587 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f40a8072fa0 0x7f40a81f7f20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4098007320 tx=0x7f40980046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.713+0000 7f40ad0ef640 1 -- 172.21.15.72:0/371671587 reap_dead start 2024-04-15T15:15:10.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.714+0000 7f40aeb52640 1 -- 172.21.15.72:0/371671587 shutdown_connections 2024-04-15T15:15:10.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.714+0000 7f40aeb52640 1 -- 172.21.15.72:0/371671587 >> 172.21.15.72:0/371671587 conn(0x7f40a80b5af0 msgr2=0x7f40a814bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:10.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.714+0000 7ffa46ffd640 1 -- 172.21.15.72:0/2777229421 <== osd.6 v2:172.21.15.175:6808/1973126001 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ffa58075b10 con 0x7ffa38076140 2024-04-15T15:15:10.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.714+0000 7f40aeb52640 1 -- 172.21.15.72:0/371671587 shutdown_connections 2024-04-15T15:15:10.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.714+0000 7f40aeb52640 1 -- 172.21.15.72:0/371671587 wait complete. 2024-04-15T15:15:10.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.714+0000 7efcb58c8640 1 -- 172.21.15.72:0/2863945513 --> [v2:172.21.15.72:6808/3973649661,v1:172.21.15.72:6809/3973649661] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7efcb015d0e0 con 0x7efc90076130 2024-04-15T15:15:10.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.715+0000 7ffa5ced3640 1 -- 172.21.15.72:0/2777229421 >> [v2:172.21.15.175:6808/1973126001,v1:172.21.15.175:6809/1973126001] conn(0x7ffa38076140 msgr2=0x7ffa38078580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.715+0000 7ffa5ced3640 1 --2- 172.21.15.72:0/2777229421 >> [v2:172.21.15.175:6808/1973126001,v1:172.21.15.175:6809/1973126001] conn(0x7ffa38076140 0x7ffa38078580 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.715+0000 7ffa5ced3640 1 -- 172.21.15.72:0/2777229421 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ffa3806c9c0 msgr2=0x7ffa3806ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.715+0000 7ffa5ced3640 1 --2- 172.21.15.72:0/2777229421 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ffa3806c9c0 0x7ffa3806ee80 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ffa50000ba0 tx=0x7ffa5000b040 comp rx=0 tx=0).stop 2024-04-15T15:15:10.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.715+0000 7ffa5ced3640 1 -- 172.21.15.72:0/2777229421 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffa58161f80 msgr2=0x7ffa580b8380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.715+0000 7ffa5ced3640 1 --2- 172.21.15.72:0/2777229421 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffa58161f80 0x7ffa580b8380 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c000c00 tx=0x7ffa4c004730 comp rx=0 tx=0).stop 2024-04-15T15:15:10.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.715+0000 7ffa56d9d640 1 -- 172.21.15.72:0/2777229421 reap_dead start 2024-04-15T15:15:10.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.715+0000 7ffa5ced3640 1 -- 172.21.15.72:0/2777229421 shutdown_connections 2024-04-15T15:15:10.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.715+0000 7ffa5ced3640 1 -- 172.21.15.72:0/2777229421 >> 172.21.15.72:0/2777229421 conn(0x7ffa58073080 msgr2=0x7ffa580ce720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:10.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.715+0000 7efcacff9640 1 -- 172.21.15.72:0/2863945513 <== osd.3 v2:172.21.15.72:6808/3973649661 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7efcb015d0e0 con 0x7efc90076130 2024-04-15T15:15:10.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.715+0000 7ffa5ced3640 1 -- 172.21.15.72:0/2777229421 shutdown_connections 2024-04-15T15:15:10.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.716+0000 7ffa5ced3640 1 -- 172.21.15.72:0/2777229421 wait complete. 2024-04-15T15:15:10.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.716+0000 7efcb58c8640 1 -- 172.21.15.72:0/2863945513 >> [v2:172.21.15.72:6808/3973649661,v1:172.21.15.72:6809/3973649661] conn(0x7efc90076130 msgr2=0x7efc90078570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.716+0000 7efcb58c8640 1 --2- 172.21.15.72:0/2863945513 >> [v2:172.21.15.72:6808/3973649661,v1:172.21.15.72:6809/3973649661] conn(0x7efc90076130 0x7efc90078570 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.716+0000 7efcb58c8640 1 -- 172.21.15.72:0/2863945513 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7efc9006c9c0 msgr2=0x7efc9006ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.716+0000 7efcb58c8640 1 --2- 172.21.15.72:0/2863945513 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7efc9006c9c0 0x7efc9006ee80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7efcb00b1d50 tx=0x7efc98009040 comp rx=0 tx=0).stop 2024-04-15T15:15:10.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.717+0000 7efcb58c8640 1 -- 172.21.15.72:0/2863945513 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efcb00b3590 msgr2=0x7efcb01f4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.717+0000 7efcb58c8640 1 --2- 172.21.15.72:0/2863945513 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efcb00b3590 0x7efcb01f4d10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efca000c500 tx=0x7efca000c9c0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.717+0000 7efcaf7fe640 1 -- 172.21.15.72:0/2863945513 reap_dead start 2024-04-15T15:15:10.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.717+0000 7efcb58c8640 1 -- 172.21.15.72:0/2863945513 shutdown_connections 2024-04-15T15:15:10.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.717+0000 7efcb58c8640 1 -- 172.21.15.72:0/2863945513 >> 172.21.15.72:0/2863945513 conn(0x7efcb009d950 msgr2=0x7efcb015d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:10.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.717+0000 7efcb58c8640 1 -- 172.21.15.72:0/2863945513 shutdown_connections 2024-04-15T15:15:10.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.717+0000 7efcb58c8640 1 -- 172.21.15.72:0/2863945513 wait complete. 2024-04-15T15:15:10.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.718+0000 7f54c757c640 1 -- 172.21.15.72:0/2672574873 --> [v2:172.21.15.175:6823/626414407,v1:172.21.15.175:6825/626414407] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f54c00689e0 con 0x7f54a407e8d0 2024-04-15T15:15:10.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.718+0000 7f54b5ffb640 1 -- 172.21.15.72:0/2672574873 <== osd.4 v2:172.21.15.175:6823/626414407 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f54c00689e0 con 0x7f54a407e8d0 2024-04-15T15:15:10.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.719+0000 7f54c757c640 1 -- 172.21.15.72:0/2672574873 >> [v2:172.21.15.175:6823/626414407,v1:172.21.15.175:6825/626414407] conn(0x7f54a407e8d0 msgr2=0x7f54a4080d10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.719+0000 7f54c757c640 1 --2- 172.21.15.72:0/2672574873 >> [v2:172.21.15.175:6823/626414407,v1:172.21.15.175:6825/626414407] conn(0x7f54a407e8d0 0x7f54a4080d10 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.719+0000 7f54c757c640 1 -- 172.21.15.72:0/2672574873 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f54a40751d0 msgr2=0x7f54a4077690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.720+0000 7f54c757c640 1 --2- 172.21.15.72:0/2672574873 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f54a40751d0 0x7f54a4077690 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f54b0000c00 tx=0x7f54b0034000 comp rx=0 tx=0).stop 2024-04-15T15:15:10.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.720+0000 7f54c757c640 1 -- 172.21.15.72:0/2672574873 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f54c01529f0 msgr2=0x7f54c0152e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.720+0000 7f54c757c640 1 --2- 172.21.15.72:0/2672574873 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f54c01529f0 0x7f54c0152e70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f54c0068540 tx=0x7f54bc010590 comp rx=0 tx=0).stop 2024-04-15T15:15:10.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.721+0000 7f54c5b19640 1 -- 172.21.15.72:0/2672574873 reap_dead start 2024-04-15T15:15:10.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.721+0000 7f54c757c640 1 -- 172.21.15.72:0/2672574873 shutdown_connections 2024-04-15T15:15:10.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.721+0000 7f54c757c640 1 -- 172.21.15.72:0/2672574873 >> 172.21.15.72:0/2672574873 conn(0x7f54c00b1dd0 msgr2=0x7f54c00cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:10.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.721+0000 7f54c757c640 1 -- 172.21.15.72:0/2672574873 shutdown_connections 2024-04-15T15:15:10.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:10.721+0000 7f54c757c640 1 -- 172.21.15.72:0/2672574873 wait complete. 2024-04-15T15:15:10.788 INFO:teuthology.orchestra.run.smithi072.stdout:55834574857 2024-04-15T15:15:10.788 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-15T15:15:10.789 INFO:teuthology.orchestra.run.smithi072.stdout:55834574857 2024-04-15T15:15:10.790 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-15T15:15:10.790 INFO:teuthology.orchestra.run.smithi072.stdout:55834574858 2024-04-15T15:15:10.791 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-15T15:15:10.791 INFO:teuthology.orchestra.run.smithi072.stdout:55834574856 2024-04-15T15:15:10.792 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-15T15:15:10.792 INFO:teuthology.orchestra.run.smithi072.stdout:55834574858 2024-04-15T15:15:10.793 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-15T15:15:10.793 INFO:teuthology.orchestra.run.smithi072.stdout:55834574858 2024-04-15T15:15:10.794 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-15T15:15:10.795 INFO:teuthology.orchestra.run.smithi072.stdout:55834574858 2024-04-15T15:15:10.795 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-15T15:15:10.796 INFO:teuthology.orchestra.run.smithi072.stdout:55834574857 2024-04-15T15:15:10.796 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-15T15:15:11.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.000+0000 7f53d9e28640 1 Processor -- start 2024-04-15T15:15:11.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.000+0000 7f53d9e28640 1 -- start start 2024-04-15T15:15:11.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.002+0000 7f53d9e28640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f53d40aced0 0x7f53d40ad2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.002+0000 7f53d9e28640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53d40ad820 con 0x7f53d406f640 2024-04-15T15:15:11.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.002+0000 7f53d9e28640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53d415aaf0 con 0x7f53d415a710 2024-04-15T15:15:11.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.002+0000 7f53d9e28640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f53d40cf5e0 con 0x7f53d40aced0 2024-04-15T15:15:11.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.002+0000 7f53d3fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f53d40aced0 0x7f53d40ad2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.002+0000 7f53d3fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f53d40aced0 0x7f53d40ad2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44786/0 (socket says 172.21.15.72:44786) 2024-04-15T15:15:11.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.002+0000 7f53d2ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f53d406f640 0x7f53d40adcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45154/0 (socket says 172.21.15.72:45154) 2024-04-15T15:15:11.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.002+0000 7f53d3fff640 1 -- 172.21.15.72:0/1294398756 learned_addr learned my addr 172.21.15.72:0/1294398756 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:11.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.003+0000 7f53d27fc640 1 -- 172.21.15.72:0/1294398756 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3057817956 0 0) 0x7f53cc0013e0 con 0x7f53d406f640 2024-04-15T15:15:11.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.003+0000 7f53d3fff640 1 -- 172.21.15.72:0/1294398756 >> v1:172.21.15.175:6789/0 conn(0x7f53d415a710 legacy=0x7f53d415eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.003+0000 7f53d3fff640 1 -- 172.21.15.72:0/1294398756 >> v1:172.21.15.72:6789/0 conn(0x7f53d406f640 legacy=0x7f53d40adcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.003+0000 7f53d3fff640 1 -- 172.21.15.72:0/1294398756 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53d40cfdd0 con 0x7f53d40aced0 2024-04-15T15:15:11.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.004+0000 7f2b90cdb640 1 Processor -- start 2024-04-15T15:15:11.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.004+0000 7f2b90cdb640 1 -- start start 2024-04-15T15:15:11.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.004+0000 7f1d87fff640 1 Processor -- start 2024-04-15T15:15:11.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.004+0000 7f1d87fff640 1 -- start start 2024-04-15T15:15:11.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.005+0000 7f53d3fff640 1 --2- 172.21.15.72:0/1294398756 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53d40aced0 0x7f53d40ad2e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f53c800bb10 tx=0x7f53c802e1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72fb66f4625c9e5d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:11.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.005+0000 7fa6494f2640 1 Processor -- start 2024-04-15T15:15:11.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.005+0000 7f53d27fc640 1 -- 172.21.15.72:0/1294398756 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c803b070 con 0x7f53d40aced0 2024-04-15T15:15:11.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.005+0000 7f53d27fc640 1 -- 172.21.15.72:0/1294398756 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53c8036440 con 0x7f53d40aced0 2024-04-15T15:15:11.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.005+0000 7f1d86ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f1d8000c0a0 0x7f1d800defb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45168/0 (socket says 172.21.15.72:45168) 2024-04-15T15:15:11.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.006+0000 7f1d86ffd640 1 -- 172.21.15.72:0/2723978071 learned_addr learned my addr 172.21.15.72:0/2723978071 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:11.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.006+0000 7f1d87fff640 1 --2- 172.21.15.72:0/2723978071 >> v2:172.21.15.72:3300/0 conn(0x7f1d80053c70 0x7f1d80038080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.006+0000 7f1d87fff640 1 --2- 172.21.15.72:0/2723978071 >> v2:172.21.15.175:3300/0 conn(0x7f1d80033ef0 0x7f1d8004fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.006+0000 7f1d87fff640 1 -- 172.21.15.72:0/2723978071 --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d800541e0 con 0x7f1d8000c0a0 2024-04-15T15:15:11.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.005+0000 7f2b90cdb640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2b840e12b0 0x7f2b840534f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.006+0000 7f2b90cdb640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2b84054b50 0x7f2b84053a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.006+0000 7f2b90cdb640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b84054fe0 con 0x7f2b840e4d30 2024-04-15T15:15:11.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.006+0000 7f2b90cdb640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f2b840552c0 con 0x7f2b84054b50 2024-04-15T15:15:11.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.006+0000 7f2b90cdb640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f2b840555a0 con 0x7f2b840e12b0 2024-04-15T15:15:11.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.006+0000 7f2b8affd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2b84054b50 0x7f2b84053a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.006+0000 7f2b8affd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2b84054b50 0x7f2b84053a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44792/0 (socket says 172.21.15.72:44792) 2024-04-15T15:15:11.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.006+0000 7f2b8affd640 1 -- 172.21.15.72:0/3623601803 learned_addr learned my addr 172.21.15.72:0/3623601803 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:11.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.006+0000 7f2b8b7fe640 1 --2- 172.21.15.72:0/3623601803 >> v2:172.21.15.175:3300/0 conn(0x7f2b840e12b0 0x7f2b840534f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.006+0000 7f1d87fff640 1 -- 172.21.15.72:0/2723978071 --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f1d80005f30 con 0x7f1d80053c70 2024-04-15T15:15:11.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.006+0000 7f1d87fff640 1 -- 172.21.15.72:0/2723978071 --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f1d80006210 con 0x7f1d80033ef0 2024-04-15T15:15:11.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.006+0000 7f1d877fe640 1 --2- 172.21.15.72:0/2723978071 >> v2:172.21.15.175:3300/0 conn(0x7f1d80033ef0 0x7f1d8004fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.006+0000 7f53d9e28640 1 -- 172.21.15.72:0/1294398756 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53d40aced0 msgr2=0x7f53d40ad2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.006+0000 7f53d9e28640 1 --2- 172.21.15.72:0/1294398756 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53d40aced0 0x7f53d40ad2e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f53c800bb10 tx=0x7f53c802e1f0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.005+0000 7fa6494f2640 1 -- start start 2024-04-15T15:15:11.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.006+0000 7fa6494f2640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa6440cfe30 0x7fa64406c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.006+0000 7f17add8e640 1 Processor -- start 2024-04-15T15:15:11.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.006+0000 7f1d867fc640 1 --2- 172.21.15.72:0/2723978071 >> v2:172.21.15.72:3300/0 conn(0x7f1d80053c70 0x7f1d80038080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.006+0000 7f53d9e28640 1 -- 172.21.15.72:0/1294398756 shutdown_connections 2024-04-15T15:15:11.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.006+0000 7f53d9e28640 1 --2- 172.21.15.72:0/1294398756 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53d40aced0 0x7f53d40ad2e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.006+0000 7f53d9e28640 1 -- 172.21.15.72:0/1294398756 >> 172.21.15.72:0/1294398756 conn(0x7f53d40b2220 msgr2=0x7f53d40d0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:11.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.006+0000 7fa642ffd640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa6440cfe30 0x7fa64406c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.006+0000 7fa642ffd640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa6440cfe30 0x7fa64406c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.72:60060/0 (socket says 172.21.15.72:60060) 2024-04-15T15:15:11.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.006+0000 7ffacdaf7640 1 Processor -- start 2024-04-15T15:15:11.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.006+0000 7ffacdaf7640 1 -- start start 2024-04-15T15:15:11.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.006+0000 7fa6427fc640 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fa6440d0210 0x7fa644159080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.72:53404/0 (socket says 172.21.15.72:53404) 2024-04-15T15:15:11.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.006+0000 7f1d877fe640 1 -- 172.21.15.72:0/2723978071 >> v2:172.21.15.72:3300/0 conn(0x7f1d80053c70 msgr2=0x7f1d80038080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:11.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.006+0000 7f1d877fe640 1 --2- 172.21.15.72:0/2723978071 >> v2:172.21.15.72:3300/0 conn(0x7f1d80053c70 0x7f1d80038080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.006+0000 7f1d877fe640 1 -- 172.21.15.72:0/2723978071 >> v1:172.21.15.72:6789/0 conn(0x7f1d8000c0a0 legacy=0x7f1d800defb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.007+0000 7fa642ffd640 1 -- 172.21.15.72:0/2318298730 learned_addr learned my addr 172.21.15.72:0/2318298730 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:11.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.006+0000 7f17add8e640 1 -- start start 2024-04-15T15:15:11.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.007+0000 7f53d9e28640 1 -- 172.21.15.72:0/1294398756 shutdown_connections 2024-04-15T15:15:11.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.007+0000 7fa6494f2640 1 -- 172.21.15.72:0/2318298730 --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa644163100 con 0x7fa6440afbe0 2024-04-15T15:15:11.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.007+0000 7fa641ffb640 1 -- 172.21.15.72:0/2318298730 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4186032128 0 0) 0x7fa644163100 con 0x7fa6440afbe0 2024-04-15T15:15:11.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.007+0000 7fa6494f2640 1 -- 172.21.15.72:0/2318298730 --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6441633e0 con 0x7fa6440d0210 2024-04-15T15:15:11.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.007+0000 7fa6494f2640 1 -- 172.21.15.72:0/2318298730 --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fa6441636c0 con 0x7fa6440cfe30 2024-04-15T15:15:11.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.007+0000 7f53d9e28640 1 -- 172.21.15.72:0/1294398756 wait complete. 2024-04-15T15:15:11.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.007+0000 7f1d877fe640 1 -- 172.21.15.72:0/2723978071 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d800064f0 con 0x7f1d80033ef0 2024-04-15T15:15:11.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.008+0000 7f10b97db640 1 Processor -- start 2024-04-15T15:15:11.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.008+0000 7f17add8e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f17a0050bf0 0x7f17a00510a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.008+0000 7f17add8e640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f17a0039c10 0x7f17a003a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.008+0000 7ffacdaf7640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ffac00e67f0 0x7ffac00e4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.008+0000 7f17add8e640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17a0009100 con 0x7f17a00e3820 2024-04-15T15:15:11.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.008+0000 7f17add8e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f17a00093e0 con 0x7f17a0050bf0 2024-04-15T15:15:11.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.008+0000 7f17add8e640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f17a0039410 con 0x7f17a0039c10 2024-04-15T15:15:11.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.008+0000 7ffacdaf7640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffac0050ce0 0x7ffac0051160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.008+0000 7ffacdaf7640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffac0053130 con 0x7ffac00e3ad0 2024-04-15T15:15:11.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.008+0000 7f2712aab640 1 Processor -- start 2024-04-15T15:15:11.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.008+0000 7f17ad58d640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f17a0039c10 0x7f17a003a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.008+0000 7f17acd8c640 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f17a00e3820 0x7f17a0008590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.72:53410/0 (socket says 172.21.15.72:53410) 2024-04-15T15:15:11.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.008+0000 7f17acd8c640 1 -- 172.21.15.72:0/1938286510 learned_addr learned my addr 172.21.15.72:0/1938286510 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:11.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.007+0000 7f2b8affd640 1 -- 172.21.15.72:0/3623601803 >> v2:172.21.15.175:3300/0 conn(0x7f2b840e12b0 msgr2=0x7f2b840534f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:11.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.007+0000 7f2b8affd640 1 --2- 172.21.15.72:0/3623601803 >> v2:172.21.15.175:3300/0 conn(0x7f2b840e12b0 0x7f2b840534f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.007+0000 7f2b8affd640 1 -- 172.21.15.72:0/3623601803 >> v1:172.21.15.72:6789/0 conn(0x7f2b840e4d30 legacy=0x7f2b840e1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.007+0000 7f2b8affd640 1 -- 172.21.15.72:0/3623601803 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b84055880 con 0x7f2b84054b50 2024-04-15T15:15:11.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.008+0000 7f17a7fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f17a0050bf0 0x7f17a00510a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.008+0000 7ffacdaf7640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7ffac0053410 con 0x7ffac00e67f0 2024-04-15T15:15:11.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.008+0000 7ffacdaf7640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7ffac00536f0 con 0x7ffac0050ce0 2024-04-15T15:15:11.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.008+0000 7ffac7fff640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffac0050ce0 0x7ffac0051160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.008+0000 7f1d877fe640 1 --2- 172.21.15.72:0/2723978071 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d80033ef0 0x7f1d8004fa90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1d880654e0 tx=0x7f1d88065780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=368092d40c4da87c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:11.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.008+0000 7fa641ffb640 1 -- 172.21.15.72:0/2318298730 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa62c003520 con 0x7fa6440afbe0 2024-04-15T15:15:11.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.008+0000 7fa641ffb640 1 -- 172.21.15.72:0/2318298730 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3386447762 0 0) 0x7fa6441633e0 con 0x7fa6440d0210 2024-04-15T15:15:11.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.008+0000 7fa641ffb640 1 -- 172.21.15.72:0/2318298730 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa644163100 con 0x7fa6440d0210 2024-04-15T15:15:11.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.008+0000 7f2b8affd640 1 --2- 172.21.15.72:0/3623601803 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b84054b50 0x7f2b84053a30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2b8400c020 tx=0x7f2b7802b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc4330866d443c64 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:11.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.008+0000 7f17ad58d640 1 -- 172.21.15.72:0/1938286510 >> v2:172.21.15.72:3300/0 conn(0x7f17a0050bf0 msgr2=0x7f17a00510a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:11.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.008+0000 7f17ad58d640 1 --2- 172.21.15.72:0/1938286510 >> v2:172.21.15.72:3300/0 conn(0x7f17a0050bf0 0x7f17a00510a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.008+0000 7f17ad58d640 1 -- 172.21.15.72:0/1938286510 >> v1:172.21.15.175:6789/0 conn(0x7f17a00e3820 legacy=0x7f17a0008590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.008+0000 7f17ad58d640 1 -- 172.21.15.72:0/1938286510 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17a00396f0 con 0x7f17a0039c10 2024-04-15T15:15:11.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.008+0000 7ffaccaf5640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ffac00e67f0 0x7ffac00e4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.008+0000 7ffacd2f6640 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7ffac00e3ad0 0x7ffac00e3f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.72:53426/0 (socket says 172.21.15.72:53426) 2024-04-15T15:15:11.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.008+0000 7ffacd2f6640 1 -- 172.21.15.72:0/2052567075 learned_addr learned my addr 172.21.15.72:0/2052567075 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:11.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.008+0000 7ffac7fff640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffac0050ce0 0x7ffac0051160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3301/0 says I am v2:172.21.15.72:57814/0 (socket says 172.21.15.72:57814) 2024-04-15T15:15:11.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.008+0000 7f2712aab640 1 -- start start 2024-04-15T15:15:11.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.008+0000 7f10b97db640 1 -- start start 2024-04-15T15:15:11.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.009+0000 7fa641ffb640 1 -- 172.21.15.72:0/2318298730 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 644359334 0 0) 0x7fa62c003520 con 0x7fa6440afbe0 2024-04-15T15:15:11.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.009+0000 7fa641ffb640 1 -- 172.21.15.72:0/2318298730 >> v2:172.21.15.175:3300/0 conn(0x7fa6440cfe30 msgr2=0x7fa64406c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:11.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.009+0000 7fa641ffb640 1 --2- 172.21.15.72:0/2318298730 >> v2:172.21.15.175:3300/0 conn(0x7fa6440cfe30 0x7fa64406c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.009+0000 7fa641ffb640 1 -- 172.21.15.72:0/2318298730 >> v1:172.21.15.175:6789/0 conn(0x7fa6440d0210 legacy=0x7fa644159080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.009+0000 7fa641ffb640 1 -- 172.21.15.72:0/2318298730 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6441639a0 con 0x7fa6440afbe0 2024-04-15T15:15:11.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.009+0000 7f53d9e28640 1 Processor -- start 2024-04-15T15:15:11.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.009+0000 7f1d85ffb640 1 -- 172.21.15.72:0/2723978071 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d880b4070 con 0x7f1d80033ef0 2024-04-15T15:15:11.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.009+0000 7f1d85ffb640 1 -- 172.21.15.72:0/2723978071 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d880af640 con 0x7f1d80033ef0 2024-04-15T15:15:11.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.009+0000 7f1d85ffb640 1 -- 172.21.15.72:0/2723978071 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d880b4070 con 0x7f1d80033ef0 2024-04-15T15:15:11.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.009+0000 7f53d9e28640 1 -- start start 2024-04-15T15:15:11.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.009+0000 7fa641ffb640 1 -- 172.21.15.72:0/2318298730 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7fa638001670 con 0x7fa6440afbe0 2024-04-15T15:15:11.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.009+0000 7fa641ffb640 1 -- 172.21.15.72:0/2318298730 >> v1:172.21.15.72:6789/0 conn(0x7fa6440afbe0 legacy=0x7fa644159790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.009+0000 7fa641ffb640 1 --2- 172.21.15.72:0/2318298730 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa62c004bd0 0x7fa62c004fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.009+0000 7fa641ffb640 1 --2- 172.21.15.72:0/2318298730 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa62c0037a0 0x7fa62c003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.009+0000 7fa641ffb640 1 --2- 172.21.15.72:0/2318298730 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa62c004160 0x7fa62c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.009+0000 7fa641ffb640 1 -- 172.21.15.72:0/2318298730 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fa6441636c0 con 0x7fa62c004bd0 2024-04-15T15:15:11.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.009+0000 7fa641ffb640 1 -- 172.21.15.72:0/2318298730 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c00c010 con 0x7fa62c0037a0 2024-04-15T15:15:11.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.009+0000 7fa641ffb640 1 -- 172.21.15.72:0/2318298730 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fa62c00c210 con 0x7fa62c004160 2024-04-15T15:15:11.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.009+0000 7f2712aab640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f270c0d1d50 0x7f270c1606d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.009+0000 7f2712aab640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f270c160c10 0x7f270c0cff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.009+0000 7f2712aab640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f270c0cc880 con 0x7f270c145e90 2024-04-15T15:15:11.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.009+0000 7f2712aab640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f270c0b2740 con 0x7f270c0d1d50 2024-04-15T15:15:11.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.009+0000 7f2712aab640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f270c0cce90 con 0x7f270c160c10 2024-04-15T15:15:11.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.009+0000 7ffac7fff640 1 -- 172.21.15.72:0/2052567075 >> v2:172.21.15.175:3300/0 conn(0x7ffac00e67f0 msgr2=0x7ffac00e4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:11.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7ffac7fff640 1 --2- 172.21.15.72:0/2052567075 >> v2:172.21.15.175:3300/0 conn(0x7ffac00e67f0 0x7ffac00e4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7f2711aa9640 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f270c145e90 0x7f270c15ffc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.72:53442/0 (socket says 172.21.15.72:53442) 2024-04-15T15:15:11.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7f2711aa9640 1 -- 172.21.15.72:0/2316409899 learned_addr learned my addr 172.21.15.72:0/2316409899 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:11.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7ffac7fff640 1 -- 172.21.15.72:0/2052567075 >> v1:172.21.15.175:6789/0 conn(0x7ffac00e3ad0 legacy=0x7ffac00e3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7f27122aa640 1 --2- 172.21.15.72:0/2316409899 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f270c160c10 0x7f270c0cff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7ffac7fff640 1 -- 172.21.15.72:0/2052567075 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffac00539d0 con 0x7ffac0050ce0 2024-04-15T15:15:11.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7f10b97db640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f10ac0392d0 0x7f10ac0df1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7f27112a8640 1 --2- 172.21.15.72:0/2316409899 >> v2:172.21.15.175:3300/0 conn(0x7f270c0d1d50 0x7f270c1606d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7f10b97db640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10ac006dd0 con 0x7f10ac00aaa0 2024-04-15T15:15:11.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.009+0000 7fa642ffd640 1 --2- 172.21.15.72:0/2318298730 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa62c004bd0 0x7fa62c004fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7fa6427fc640 1 --2- 172.21.15.72:0/2318298730 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa62c0037a0 0x7fa62c003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7fa643fff640 1 --2- 172.21.15.72:0/2318298730 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa62c004160 0x7fa62c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7f1d87fff640 1 -- 172.21.15.72:0/2723978071 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d80033ef0 msgr2=0x7f1d8004fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7f1d87fff640 1 --2- 172.21.15.72:0/2723978071 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d80033ef0 0x7f1d8004fa90 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1d880654e0 tx=0x7f1d88065780 comp rx=0 tx=0).stop 2024-04-15T15:15:11.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7f17ad58d640 1 --2- 172.21.15.72:0/1938286510 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f17a0039c10 0x7f17a003a020 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f17a80654e0 tx=0x7f17a8065780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b8cfd0fdc411702 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:11.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.009+0000 7f2b8a7fc640 1 -- 172.21.15.72:0/3623601803 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b78034070 con 0x7f2b84054b50 2024-04-15T15:15:11.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7f2b8a7fc640 1 -- 172.21.15.72:0/3623601803 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b78030870 con 0x7f2b84054b50 2024-04-15T15:15:11.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7f2b8a7fc640 1 -- 172.21.15.72:0/3623601803 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b78030c40 con 0x7f2b84054b50 2024-04-15T15:15:11.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7fa6427fc640 1 -- 172.21.15.72:0/2318298730 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa62c004160 msgr2=0x7fa62c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:11.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7fa6427fc640 1 --2- 172.21.15.72:0/2318298730 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa62c004160 0x7fa62c00b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7fa6427fc640 1 -- 172.21.15.72:0/2318298730 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa62c004bd0 msgr2=0x7fa62c004fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:11.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7f1d87fff640 1 -- 172.21.15.72:0/2723978071 shutdown_connections 2024-04-15T15:15:11.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7f1d87fff640 1 --2- 172.21.15.72:0/2723978071 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d80033ef0 0x7f1d8004fa90 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7f1d87fff640 1 --2- 172.21.15.72:0/2723978071 >> v2:172.21.15.72:3300/0 conn(0x7f1d80053c70 0x7f1d80038080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7f1d87fff640 1 -- 172.21.15.72:0/2723978071 >> 172.21.15.72:0/2723978071 conn(0x7f1d80009fb0 msgr2=0x7f1d8000a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:11.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7f10b97db640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10ac0070b0 con 0x7f10ac0501a0 2024-04-15T15:15:11.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7f10b97db640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f10ac0068a0 con 0x7f10ac0392d0 2024-04-15T15:15:11.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7f10b37fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f10ac0392d0 0x7f10ac0df1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7f10b2ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f10ac00aaa0 0x7f10ac04fa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:45208/0 (socket says 172.21.15.72:45208) 2024-04-15T15:15:11.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7f53d9e28640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53d406f640 0x7f53d41f4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7f17a77fe640 1 -- 172.21.15.72:0/1938286510 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a80b4070 con 0x7f17a0039c10 2024-04-15T15:15:11.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7f17a77fe640 1 -- 172.21.15.72:0/1938286510 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17a80af440 con 0x7f17a0039c10 2024-04-15T15:15:11.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7f17a77fe640 1 -- 172.21.15.72:0/1938286510 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a80af8d0 con 0x7f17a0039c10 2024-04-15T15:15:11.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7f10b2ffd640 1 -- 172.21.15.72:0/2421237271 learned_addr learned my addr 172.21.15.72:0/2421237271 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:11.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.011+0000 7f1d87fff640 1 -- 172.21.15.72:0/2723978071 shutdown_connections 2024-04-15T15:15:11.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.011+0000 7f10b37fe640 1 -- 172.21.15.72:0/2421237271 >> v1:172.21.15.175:6789/0 conn(0x7f10ac0501a0 legacy=0x7f10ac053260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.011+0000 7f10b37fe640 1 -- 172.21.15.72:0/2421237271 >> v1:172.21.15.72:6789/0 conn(0x7f10ac00aaa0 legacy=0x7f10ac04fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.011+0000 7f10b37fe640 1 -- 172.21.15.72:0/2421237271 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10ac007390 con 0x7f10ac0392d0 2024-04-15T15:15:11.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.011+0000 7fa6427fc640 1 --2- 172.21.15.72:0/2318298730 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa62c004bd0 0x7fa62c004fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.011+0000 7fa6427fc640 1 -- 172.21.15.72:0/2318298730 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa62c003520 con 0x7fa62c0037a0 2024-04-15T15:15:11.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.011+0000 7f53d9e28640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f53d40aced0 0x7f53d409d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.039 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.011+0000 7f53d9e28640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f53d415a710 0x7f53d40c47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.011+0000 7f53d9e28640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f53d415e2f0 con 0x7f53d406f640 2024-04-15T15:15:11.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.011+0000 7f53d37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53d406f640 0x7f53d41f4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.011+0000 7f53d37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53d406f640 0x7f53d41f4300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44836/0 (socket says 172.21.15.72:44836) 2024-04-15T15:15:11.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.011+0000 7f53d37fe640 1 -- 172.21.15.72:0/2739721631 learned_addr learned my addr 172.21.15.72:0/2739721631 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:11.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.011+0000 7f53d9e28640 1 -- 172.21.15.72:0/2739721631 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f53d40cfaf0 con 0x7f53d40aced0 2024-04-15T15:15:11.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.011+0000 7f53d9e28640 1 -- 172.21.15.72:0/2739721631 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f53d415e5d0 con 0x7f53d415a710 2024-04-15T15:15:11.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.011+0000 7fa643fff640 1 -- 172.21.15.72:0/2318298730 reap_dead start 2024-04-15T15:15:11.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.011+0000 7ffac7fff640 1 --2- 172.21.15.72:0/2052567075 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffac0050ce0 0x7ffac0051160 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ffab40097a0 tx=0x7ffab402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d919dac2fc8505a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:11.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.011+0000 7f53d3fff640 1 --2- 172.21.15.72:0/2739721631 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f53d415a710 0x7f53d40c47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.011+0000 7f2b90cdb640 1 -- 172.21.15.72:0/3623601803 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b84054b50 msgr2=0x7f2b84053a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.011+0000 7fa642ffd640 1 --2- 172.21.15.72:0/2318298730 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa62c004bd0 0x7fa62c004fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:11.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7f2710aa7640 1 -- 172.21.15.72:0/2316409899 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2218907855 0 0) 0x7f270c0cc880 con 0x7f270c145e90 2024-04-15T15:15:11.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7f2710aa7640 1 -- 172.21.15.72:0/2316409899 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26f8003520 con 0x7f270c145e90 2024-04-15T15:15:11.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7f27112a8640 1 -- 172.21.15.72:0/2316409899 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f270c160c10 msgr2=0x7f270c0cff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:11.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7f27112a8640 1 --2- 172.21.15.72:0/2316409899 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f270c160c10 0x7f270c0cff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7f27112a8640 1 -- 172.21.15.72:0/2316409899 >> v1:172.21.15.175:6789/0 conn(0x7f270c145e90 legacy=0x7f270c15ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.010+0000 7f27112a8640 1 -- 172.21.15.72:0/2316409899 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f270c0b2990 con 0x7f270c0d1d50 2024-04-15T15:15:11.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.011+0000 7f2b90cdb640 1 --2- 172.21.15.72:0/3623601803 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b84054b50 0x7f2b84053a30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2b8400c020 tx=0x7f2b7802b9b0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.011+0000 7ffac77fe640 1 -- 172.21.15.72:0/2052567075 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab4035020 con 0x7ffac0050ce0 2024-04-15T15:15:11.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.011+0000 7f53d37fe640 1 -- 172.21.15.72:0/2739721631 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f53d415a710 msgr2=0x7f53d40c47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.011+0000 7ffac77fe640 1 -- 172.21.15.72:0/2052567075 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffab4031810 con 0x7ffac0050ce0 2024-04-15T15:15:11.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.011+0000 7f53d37fe640 1 --2- 172.21.15.72:0/2739721631 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f53d415a710 0x7f53d40c47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.011+0000 7f53d37fe640 1 -- 172.21.15.72:0/2739721631 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f53d40aced0 msgr2=0x7f53d409d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.011+0000 7f53d37fe640 1 --2- 172.21.15.72:0/2739721631 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f53d40aced0 0x7f53d409d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.011+0000 7f53d37fe640 1 -- 172.21.15.72:0/2739721631 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53d40729c0 con 0x7f53d406f640 2024-04-15T15:15:11.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.011+0000 7f53d3fff640 1 --2- 172.21.15.72:0/2739721631 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f53d415a710 0x7f53d40c47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:11.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.011+0000 7f17add8e640 1 -- 172.21.15.72:0/1938286510 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f17a0039c10 msgr2=0x7f17a003a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.011+0000 7f17add8e640 1 --2- 172.21.15.72:0/1938286510 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f17a0039c10 0x7f17a003a020 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f17a80654e0 tx=0x7f17a8065780 comp rx=0 tx=0).stop 2024-04-15T15:15:11.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.012+0000 7f17add8e640 1 -- 172.21.15.72:0/1938286510 shutdown_connections 2024-04-15T15:15:11.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.012+0000 7f17add8e640 1 --2- 172.21.15.72:0/1938286510 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f17a0039c10 0x7f17a003a020 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.012+0000 7f17add8e640 1 --2- 172.21.15.72:0/1938286510 >> v2:172.21.15.72:3300/0 conn(0x7f17a0050bf0 0x7f17a00510a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.012+0000 7f17add8e640 1 -- 172.21.15.72:0/1938286510 >> 172.21.15.72:0/1938286510 conn(0x7f17a0055bd0 msgr2=0x7f17a0055fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:11.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.012+0000 7ffac77fe640 1 -- 172.21.15.72:0/2052567075 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffab4035020 con 0x7ffac0050ce0 2024-04-15T15:15:11.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.011+0000 7f1d87fff640 1 -- 172.21.15.72:0/2723978071 wait complete. 2024-04-15T15:15:11.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.012+0000 7f17add8e640 1 -- 172.21.15.72:0/1938286510 shutdown_connections 2024-04-15T15:15:11.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.012+0000 7f10b37fe640 1 --2- 172.21.15.72:0/2421237271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10ac0392d0 0x7f10ac0df1b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f10b4050270 tx=0x7f10b4069760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef41d3a3d8571c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:11.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.012+0000 7f53d37fe640 1 --2- 172.21.15.72:0/2739721631 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53d406f640 0x7f53d41f4300 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f53c4001a30 tx=0x7f53c4014420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:11.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.012+0000 7f17add8e640 1 -- 172.21.15.72:0/1938286510 wait complete. 2024-04-15T15:15:11.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.012+0000 7f53bbfff640 1 -- 172.21.15.72:0/2739721631 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c4014ad0 con 0x7f53d406f640 2024-04-15T15:15:11.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.012+0000 7f53d9e28640 1 -- 172.21.15.72:0/2739721631 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53d4072ca0 con 0x7f53d406f640 2024-04-15T15:15:11.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.012+0000 7f53d9e28640 1 -- 172.21.15.72:0/2739721631 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53d4073260 con 0x7f53d406f640 2024-04-15T15:15:11.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.012+0000 7f53bbfff640 1 -- 172.21.15.72:0/2739721631 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53c4014c70 con 0x7f53d406f640 2024-04-15T15:15:11.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.012+0000 7f53bbfff640 1 -- 172.21.15.72:0/2739721631 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c4005310 con 0x7f53d406f640 2024-04-15T15:15:11.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.012+0000 7ffacdaf7640 1 -- 172.21.15.72:0/2052567075 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffac0050ce0 msgr2=0x7ffac0051160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.012+0000 7ffacdaf7640 1 --2- 172.21.15.72:0/2052567075 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffac0050ce0 0x7ffac0051160 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ffab40097a0 tx=0x7ffab402b8b0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.012+0000 7ffacdaf7640 1 -- 172.21.15.72:0/2052567075 shutdown_connections 2024-04-15T15:15:11.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.012+0000 7ffacdaf7640 1 --2- 172.21.15.72:0/2052567075 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffac0050ce0 0x7ffac0051160 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.012+0000 7ffacdaf7640 1 --2- 172.21.15.72:0/2052567075 >> v2:172.21.15.175:3300/0 conn(0x7ffac00e67f0 0x7ffac00e4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.012+0000 7ffacdaf7640 1 -- 172.21.15.72:0/2052567075 >> 172.21.15.72:0/2052567075 conn(0x7ffac0055350 msgr2=0x7ffac0055760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:11.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.012+0000 7ffacdaf7640 1 -- 172.21.15.72:0/2052567075 shutdown_connections 2024-04-15T15:15:11.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.013+0000 7ffacdaf7640 1 -- 172.21.15.72:0/2052567075 wait complete. 2024-04-15T15:15:11.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.012+0000 7f27112a8640 1 --2- 172.21.15.72:0/2316409899 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f270c0d1d50 0x7f270c1606d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f27040097a0 tx=0x7f270402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=642de0e7ef5b6a11 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:11.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.013+0000 7fa6427fc640 1 --2- 172.21.15.72:0/2318298730 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa62c0037a0 0x7fa62c003c20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa63400c160 tx=0x7fa62c005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4a8285e4547d3e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:11.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.013+0000 7f10b1ffb640 1 -- 172.21.15.72:0/2421237271 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b40b8070 con 0x7f10ac0392d0 2024-04-15T15:15:11.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.013+0000 7f10b1ffb640 1 -- 172.21.15.72:0/2421237271 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10b40b3640 con 0x7f10ac0392d0 2024-04-15T15:15:11.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.013+0000 7f10b1ffb640 1 -- 172.21.15.72:0/2421237271 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b40b3ab0 con 0x7f10ac0392d0 2024-04-15T15:15:11.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.012+0000 7f2b90cdb640 1 -- 172.21.15.72:0/3623601803 shutdown_connections 2024-04-15T15:15:11.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.012+0000 7f2b90cdb640 1 --2- 172.21.15.72:0/3623601803 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b84054b50 0x7f2b84053a30 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.012+0000 7f2b90cdb640 1 --2- 172.21.15.72:0/3623601803 >> v2:172.21.15.175:3300/0 conn(0x7f2b840e12b0 0x7f2b840534f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.012+0000 7f2b90cdb640 1 -- 172.21.15.72:0/3623601803 >> 172.21.15.72:0/3623601803 conn(0x7f2b84020040 msgr2=0x7f2b84056d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:11.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.013+0000 7f2b90cdb640 1 -- 172.21.15.72:0/3623601803 shutdown_connections 2024-04-15T15:15:11.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.013+0000 7f2b90cdb640 1 -- 172.21.15.72:0/3623601803 wait complete. 2024-04-15T15:15:11.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.013+0000 7f2710aa7640 1 -- 172.21.15.72:0/2316409899 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2704035020 con 0x7f270c0d1d50 2024-04-15T15:15:11.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.013+0000 7f2710aa7640 1 -- 172.21.15.72:0/2316409899 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2704031810 con 0x7f270c0d1d50 2024-04-15T15:15:11.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.013+0000 7f1d87fff640 1 Processor -- start 2024-04-15T15:15:11.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.013+0000 7f17add8e640 1 Processor -- start 2024-04-15T15:15:11.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.013+0000 7f17add8e640 1 -- start start 2024-04-15T15:15:11.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.014+0000 7f2712aab640 1 -- 172.21.15.72:0/2316409899 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f270c0d1d50 msgr2=0x7f270c1606d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.014+0000 7f2710aa7640 1 -- 172.21.15.72:0/2316409899 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2704035020 con 0x7f270c0d1d50 2024-04-15T15:15:11.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.014+0000 7f2712aab640 1 --2- 172.21.15.72:0/2316409899 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f270c0d1d50 0x7f270c1606d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f27040097a0 tx=0x7f270402b8b0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.013+0000 7f1d87fff640 1 -- start start 2024-04-15T15:15:11.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.014+0000 7fa641ffb640 1 -- 172.21.15.72:0/2318298730 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa634007c20 con 0x7fa62c0037a0 2024-04-15T15:15:11.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.014+0000 7fa641ffb640 1 -- 172.21.15.72:0/2318298730 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa634030030 con 0x7fa62c0037a0 2024-04-15T15:15:11.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.014+0000 7fa641ffb640 1 -- 172.21.15.72:0/2318298730 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6340075b0 con 0x7fa62c0037a0 2024-04-15T15:15:11.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.014+0000 7f10b97db640 1 -- 172.21.15.72:0/2421237271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10ac0392d0 msgr2=0x7f10ac0df1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.014+0000 7f10b97db640 1 --2- 172.21.15.72:0/2421237271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10ac0392d0 0x7f10ac0df1b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f10b4050270 tx=0x7f10b4069760 comp rx=0 tx=0).stop 2024-04-15T15:15:11.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.014+0000 7f2712aab640 1 -- 172.21.15.72:0/2316409899 shutdown_connections 2024-04-15T15:15:11.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.014+0000 7f2712aab640 1 --2- 172.21.15.72:0/2316409899 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f270c160c10 0x7f270c0cff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.014+0000 7f2712aab640 1 --2- 172.21.15.72:0/2316409899 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f270c0d1d50 0x7f270c1606d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.014+0000 7f2712aab640 1 -- 172.21.15.72:0/2316409899 >> 172.21.15.72:0/2316409899 conn(0x7f270c0b3ec0 msgr2=0x7f270c0b42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:11.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7f10b97db640 1 -- 172.21.15.72:0/2421237271 shutdown_connections 2024-04-15T15:15:11.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7f10b97db640 1 --2- 172.21.15.72:0/2421237271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10ac0392d0 0x7f10ac0df1b0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7f10b97db640 1 -- 172.21.15.72:0/2421237271 >> 172.21.15.72:0/2421237271 conn(0x7f10ac009fb0 msgr2=0x7f10ac00a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:11.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7ffacdaf7640 1 Processor -- start 2024-04-15T15:15:11.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7f17add8e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17a0039c10 0x7f17a00514a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7f1d87fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d8000c0a0 0x7f1d8004a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7f17add8e640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f17a0050bf0 0x7f17a00d8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7f10b97db640 1 -- 172.21.15.72:0/2421237271 shutdown_connections 2024-04-15T15:15:11.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7fa6494f2640 1 -- 172.21.15.72:0/2318298730 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa62c0037a0 msgr2=0x7fa62c003c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7fa6494f2640 1 --2- 172.21.15.72:0/2318298730 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa62c0037a0 0x7fa62c003c20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa63400c160 tx=0x7fa62c005830 comp rx=0 tx=0).stop 2024-04-15T15:15:11.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7ffacdaf7640 1 -- start start 2024-04-15T15:15:11.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7f17add8e640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f17a00e3820 0x7f17a01813d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7f10b97db640 1 -- 172.21.15.72:0/2421237271 wait complete. 2024-04-15T15:15:11.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7fa6494f2640 1 -- 172.21.15.72:0/2318298730 shutdown_connections 2024-04-15T15:15:11.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7fa6494f2640 1 --2- 172.21.15.72:0/2318298730 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa62c0037a0 0x7fa62c003c20 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7fa6494f2640 1 -- 172.21.15.72:0/2318298730 >> 172.21.15.72:0/2318298730 conn(0x7fa644071fe0 msgr2=0x7fa6440723f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:11.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7f2712aab640 1 -- 172.21.15.72:0/2316409899 shutdown_connections 2024-04-15T15:15:11.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7f17add8e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f17a00391d0 con 0x7f17a0039c10 2024-04-15T15:15:11.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7f17add8e640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f17a0039050 con 0x7f17a0050bf0 2024-04-15T15:15:11.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7f17add8e640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f17a0008ca0 con 0x7f17a00e3820 2024-04-15T15:15:11.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7f17ad58d640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f17a00e3820 0x7f17a01813d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7f17ad58d640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f17a00e3820 0x7f17a01813d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3301/0 says I am v2:172.21.15.72:57846/0 (socket says 172.21.15.72:57846) 2024-04-15T15:15:11.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7fa6494f2640 1 -- 172.21.15.72:0/2318298730 shutdown_connections 2024-04-15T15:15:11.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7f2b90cdb640 1 Processor -- start 2024-04-15T15:15:11.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7f17ad58d640 1 -- 172.21.15.72:0/1348663825 learned_addr learned my addr 172.21.15.72:0/1348663825 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:11.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7f53bbfff640 1 -- 172.21.15.72:0/2739721631 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f53c40120d0 con 0x7f53d406f640 2024-04-15T15:15:11.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7f17a7fff640 1 --2- 172.21.15.72:0/1348663825 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f17a0050bf0 0x7f17a00d8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7f1d87fff640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d80033ef0 0x7f1d8004e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7f1d87fff640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1d80053c70 0x7f1d801897f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7f1d87fff640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f1d80037a90 con 0x7f1d8000c0a0 2024-04-15T15:15:11.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7f1d87fff640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f1d80037910 con 0x7f1d80033ef0 2024-04-15T15:15:11.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7f1d87fff640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f1d8004ffd0 con 0x7f1d80053c70 2024-04-15T15:15:11.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7f1d86ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d8000c0a0 0x7f1d8004a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7f1d867fc640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d80033ef0 0x7f1d8004e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7f1d877fe640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1d80053c70 0x7f1d801897f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7f1d877fe640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1d80053c70 0x7f1d801897f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3301/0 says I am v2:172.21.15.72:57854/0 (socket says 172.21.15.72:57854) 2024-04-15T15:15:11.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7f1d867fc640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d80033ef0 0x7f1d8004e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.72:60094/0 (socket says 172.21.15.72:60094) 2024-04-15T15:15:11.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7f1d877fe640 1 -- 172.21.15.72:0/1028536126 learned_addr learned my addr 172.21.15.72:0/1028536126 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:11.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7f2b90cdb640 1 -- start start 2024-04-15T15:15:11.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.016+0000 7f1d877fe640 1 -- 172.21.15.72:0/1028536126 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d80033ef0 msgr2=0x7f1d8004e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.016+0000 7f1d877fe640 1 --2- 172.21.15.72:0/1028536126 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d80033ef0 0x7f1d8004e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.016+0000 7f1d877fe640 1 -- 172.21.15.72:0/1028536126 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d8000c0a0 msgr2=0x7f1d8004a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.016+0000 7f1d867fc640 1 --2- 172.21.15.72:0/1028536126 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d80033ef0 0x7f1d8004e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-04-15T15:15:11.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.016+0000 7f1d877fe640 1 --2- 172.21.15.72:0/1028536126 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d8000c0a0 0x7f1d8004a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.016+0000 7f1d877fe640 1 -- 172.21.15.72:0/1028536126 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d8004a750 con 0x7f1d80053c70 2024-04-15T15:15:11.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.016+0000 7f1d86ffd640 1 --2- 172.21.15.72:0/1028536126 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d8000c0a0 0x7f1d8004a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:11.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7f2712aab640 1 -- 172.21.15.72:0/2316409899 wait complete. 2024-04-15T15:15:11.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.015+0000 7fa6494f2640 1 -- 172.21.15.72:0/2318298730 wait complete. 2024-04-15T15:15:11.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.016+0000 7f1d877fe640 1 --2- 172.21.15.72:0/1028536126 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1d80053c70 0x7f1d801897f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1d8806a0f0 tx=0x7f1d88075700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:11.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.016+0000 7f17a7fff640 1 -- 172.21.15.72:0/1348663825 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f17a00e3820 msgr2=0x7f17a01813d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.016+0000 7f17a7fff640 1 --2- 172.21.15.72:0/1348663825 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f17a00e3820 0x7f17a01813d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.016+0000 7f17a7fff640 1 -- 172.21.15.72:0/1348663825 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17a0039c10 msgr2=0x7f17a00514a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-15T15:15:11.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.016+0000 7f17a7fff640 1 --2- 172.21.15.72:0/1348663825 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17a0039c10 0x7f17a00514a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.016+0000 7f17a7fff640 1 -- 172.21.15.72:0/1348663825 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17a00d9050 con 0x7f17a0050bf0 2024-04-15T15:15:11.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.017+0000 7ffacdaf7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffac0050ce0 0x7ffac00dbea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.017+0000 7f2b90cdb640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2b840e12b0 0x7f2b840d1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.017+0000 7f10b97db640 1 Processor -- start 2024-04-15T15:15:11.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.016+0000 7f53bbfff640 1 --2- 172.21.15.72:0/2739721631 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f53c006c920 0x7f53c006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.016+0000 7f53bbfff640 1 -- 172.21.15.72:0/2739721631 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f53c4093c50 con 0x7f53d406f640 2024-04-15T15:15:11.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.017+0000 7f53d2ffd640 1 --2- 172.21.15.72:0/2739721631 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f53c006c920 0x7f53c006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.017+0000 7f2b90cdb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b840e4d30 0x7f2b84050830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.016+0000 7f1d6f7fe640 1 -- 172.21.15.72:0/1028536126 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d880b4070 con 0x7f1d80053c70 2024-04-15T15:15:11.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.016+0000 7f1d6f7fe640 1 -- 172.21.15.72:0/1028536126 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d880ae9e0 con 0x7f1d80053c70 2024-04-15T15:15:11.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.016+0000 7f1d6f7fe640 1 -- 172.21.15.72:0/1028536126 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d880aebd0 con 0x7f1d80053c70 2024-04-15T15:15:11.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.017+0000 7f2b8b7fe640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2b840e12b0 0x7f2b840d1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.017+0000 7f2b8b7fe640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2b840e12b0 0x7f2b840d1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3301/0 says I am v2:172.21.15.72:57856/0 (socket says 172.21.15.72:57856) 2024-04-15T15:15:11.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.017+0000 7f53b9ffb640 1 -- 172.21.15.72:0/2739721631 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53d4075aa0 con 0x7f53d406f640 2024-04-15T15:15:11.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.017+0000 7f2b8b7fe640 1 -- 172.21.15.72:0/2853332298 learned_addr learned my addr 172.21.15.72:0/2853332298 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:11.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.017+0000 7f2b90cdb640 1 --2- 172.21.15.72:0/2853332298 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2b84050d70 0x7f2b8404c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.017+0000 7fa6494f2640 1 Processor -- start 2024-04-15T15:15:11.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.017+0000 7f17a7fff640 1 --2- 172.21.15.72:0/1348663825 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f17a0050bf0 0x7f17a00d8670 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f179c00e850 tx=0x7f179c00ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:11.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.017+0000 7ffaccaf5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffac0050ce0 0x7ffac00dbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.017+0000 7ffaccaf5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffac0050ce0 0x7ffac00dbea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44852/0 (socket says 172.21.15.72:44852) 2024-04-15T15:15:11.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.017+0000 7f2b8affd640 1 --2- 172.21.15.72:0/2853332298 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b840e4d30 0x7f2b84050830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.017+0000 7ffaccaf5640 1 -- 172.21.15.72:0/3646882716 learned_addr learned my addr 172.21.15.72:0/3646882716 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:11.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.017+0000 7f2b90cdb640 1 -- 172.21.15.72:0/2853332298 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2b84057820 con 0x7f2b840e4d30 2024-04-15T15:15:11.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.017+0000 7fa6494f2640 1 -- start start 2024-04-15T15:15:11.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.017+0000 7ffacdaf7640 1 --2- 172.21.15.72:0/3646882716 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffac00e3ad0 0x7ffac0034cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.017+0000 7f2b90cdb640 1 -- 172.21.15.72:0/2853332298 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f2b840576a0 con 0x7f2b84050d70 2024-04-15T15:15:11.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.017+0000 7ffacdaf7640 1 --2- 172.21.15.72:0/3646882716 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffac00e67f0 0x7ffac003a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.017+0000 7f2b90cdb640 1 -- 172.21.15.72:0/2853332298 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f2b840e1da0 con 0x7f2b840e12b0 2024-04-15T15:15:11.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.018+0000 7ffacdaf7640 1 -- 172.21.15.72:0/3646882716 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ffac00dd310 con 0x7ffac0050ce0 2024-04-15T15:15:11.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.018+0000 7f2b8bfff640 1 --2- 172.21.15.72:0/2853332298 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2b84050d70 0x7f2b8404c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.018+0000 7ffacdaf7640 1 -- 172.21.15.72:0/3646882716 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7ffac00dd190 con 0x7ffac00e67f0 2024-04-15T15:15:11.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.018+0000 7ffacdaf7640 1 -- 172.21.15.72:0/3646882716 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7ffac00dd490 con 0x7ffac00e3ad0 2024-04-15T15:15:11.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.018+0000 7ffac7fff640 1 --2- 172.21.15.72:0/3646882716 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffac00e3ad0 0x7ffac0034cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.018+0000 7f2b8affd640 1 -- 172.21.15.72:0/2853332298 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2b840e12b0 msgr2=0x7f2b840d1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.018+0000 7f2b8affd640 1 --2- 172.21.15.72:0/2853332298 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2b840e12b0 0x7f2b840d1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.018+0000 7f53d2ffd640 1 --2- 172.21.15.72:0/2739721631 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f53c006c920 0x7f53c006ede0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f53cc000f80 tx=0x7f53cc001570 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:11.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.018+0000 7f1d87fff640 1 -- 172.21.15.72:0/1028536126 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d8004aa30 con 0x7f1d80053c70 2024-04-15T15:15:11.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.018+0000 7f1d87fff640 1 -- 172.21.15.72:0/1028536126 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d80170610 con 0x7f1d80053c70 2024-04-15T15:15:11.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.018+0000 7f2b8affd640 1 -- 172.21.15.72:0/2853332298 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2b84050d70 msgr2=0x7f2b8404c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.017+0000 7f10b97db640 1 -- start start 2024-04-15T15:15:11.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.018+0000 7f10b97db640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f10ac00aaa0 0x7f10ac16e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.018+0000 7f10b97db640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10ac0392d0 0x7f10ac0e1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.018+0000 7f10b97db640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f10ac0501a0 0x7f10ac0dfef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.018+0000 7fa6494f2640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa62c0037a0 0x7fa644145700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.018+0000 7fa6494f2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa6441eab00 0x7fa6441eb390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.018+0000 7f10b97db640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f10ac0e3e90 con 0x7f10ac0392d0 2024-04-15T15:15:11.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.018+0000 7f10b97db640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f10ac0e3d10 con 0x7f10ac00aaa0 2024-04-15T15:15:11.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.018+0000 7f10b97db640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f10ac0e4010 con 0x7f10ac0501a0 2024-04-15T15:15:11.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.018+0000 7fa6494f2640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa6441eb8d0 0x7fa6441e8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.018+0000 7fa6494f2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fa64406f3a0 con 0x7fa6441eab00 2024-04-15T15:15:11.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.018+0000 7fa6494f2640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fa64406f220 con 0x7fa62c0037a0 2024-04-15T15:15:11.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.018+0000 7fa6494f2640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fa64406f520 con 0x7fa6441eb8d0 2024-04-15T15:15:11.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.018+0000 7f2b8affd640 1 --2- 172.21.15.72:0/2853332298 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2b84050d70 0x7f2b8404c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.018+0000 7f17a57fa640 1 -- 172.21.15.72:0/1348663825 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f179c017070 con 0x7f17a0050bf0 2024-04-15T15:15:11.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.018+0000 7f2b8affd640 1 -- 172.21.15.72:0/2853332298 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b8404f860 con 0x7f2b840e4d30 2024-04-15T15:15:11.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.018+0000 7f2b8bfff640 1 --2- 172.21.15.72:0/2853332298 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2b84050d70 0x7f2b8404c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:11.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.018+0000 7f2b8affd640 1 --2- 172.21.15.72:0/2853332298 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b840e4d30 0x7f2b84050830 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2b78033000 tx=0x7f2b78004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:11.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.019+0000 7f17a57fa640 1 -- 172.21.15.72:0/1348663825 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f179c00c940 con 0x7f17a0050bf0 2024-04-15T15:15:11.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.019+0000 7ffaccaf5640 1 -- 172.21.15.72:0/3646882716 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffac00e3ad0 msgr2=0x7ffac0034cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.019+0000 7ffaccaf5640 1 --2- 172.21.15.72:0/3646882716 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffac00e3ad0 0x7ffac0034cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.019+0000 7ffaccaf5640 1 -- 172.21.15.72:0/3646882716 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffac00e67f0 msgr2=0x7ffac003a140 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-15T15:15:11.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.019+0000 7ffaccaf5640 1 --2- 172.21.15.72:0/3646882716 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffac00e67f0 0x7ffac003a140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.019+0000 7ffaccaf5640 1 -- 172.21.15.72:0/3646882716 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffac00dc3e0 con 0x7ffac0050ce0 2024-04-15T15:15:11.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.019+0000 7fa642ffd640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa62c0037a0 0x7fa644145700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.019+0000 7ffac7fff640 1 --2- 172.21.15.72:0/3646882716 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffac00e3ad0 0x7ffac0034cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:11.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.019+0000 7fa643fff640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa6441eb8d0 0x7fa6441e8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.019+0000 7fa643fff640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa6441eb8d0 0x7fa6441e8d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3301/0 says I am v2:172.21.15.72:57880/0 (socket says 172.21.15.72:57880) 2024-04-15T15:15:11.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.019+0000 7ffaccaf5640 1 --2- 172.21.15.72:0/3646882716 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffac0050ce0 0x7ffac00dbea0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ffabc00ebe0 tx=0x7ffabc00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:11.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.019+0000 7fa643fff640 1 -- 172.21.15.72:0/531527626 learned_addr learned my addr 172.21.15.72:0/531527626 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:11.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.017+0000 7f2712aab640 1 Processor -- start 2024-04-15T15:15:11.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.017+0000 7f2712aab640 1 -- start start 2024-04-15T15:15:11.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.019+0000 7f17a57fa640 1 -- 172.21.15.72:0/1348663825 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f179c00cd10 con 0x7f17a0050bf0 2024-04-15T15:15:11.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.018+0000 7f10b2ffd640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f10ac00aaa0 0x7f10ac16e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.018+0000 7f10b27fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10ac0392d0 0x7f10ac0e1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.019+0000 7f10b27fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10ac0392d0 0x7f10ac0e1f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44860/0 (socket says 172.21.15.72:44860) 2024-04-15T15:15:11.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.019+0000 7f10b27fc640 1 -- 172.21.15.72:0/4193862566 learned_addr learned my addr 172.21.15.72:0/4193862566 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:11.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.019+0000 7f10b37fe640 1 --2- 172.21.15.72:0/4193862566 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f10ac0501a0 0x7f10ac0dfef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.019+0000 7f10b27fc640 1 -- 172.21.15.72:0/4193862566 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f10ac0501a0 msgr2=0x7f10ac0dfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.019+0000 7f10b27fc640 1 --2- 172.21.15.72:0/4193862566 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f10ac0501a0 0x7f10ac0dfef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.019+0000 7f10b27fc640 1 -- 172.21.15.72:0/4193862566 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f10ac00aaa0 msgr2=0x7f10ac16e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.019+0000 7f10b27fc640 1 --2- 172.21.15.72:0/4193862566 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f10ac00aaa0 0x7f10ac16e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.019+0000 7f10b27fc640 1 -- 172.21.15.72:0/4193862566 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10ac0df2c0 con 0x7f10ac0392d0 2024-04-15T15:15:11.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.019+0000 7f10b37fe640 1 --2- 172.21.15.72:0/4193862566 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f10ac0501a0 0x7f10ac0dfef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:11.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.019+0000 7f10b2ffd640 1 --2- 172.21.15.72:0/4193862566 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f10ac00aaa0 0x7f10ac16e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:11.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.019+0000 7f10b27fc640 1 --2- 172.21.15.72:0/4193862566 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10ac0392d0 0x7f10ac0e1f50 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f10a800c4e0 tx=0x7f10a800c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:11.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.019+0000 7fa643fff640 1 -- 172.21.15.72:0/531527626 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa62c0037a0 msgr2=0x7fa644145700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.019+0000 7fa643fff640 1 --2- 172.21.15.72:0/531527626 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa62c0037a0 0x7fa644145700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.020+0000 7f17add8e640 1 -- 172.21.15.72:0/1348663825 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17a00d9330 con 0x7f17a0050bf0 2024-04-15T15:15:11.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.019+0000 7fa643fff640 1 -- 172.21.15.72:0/531527626 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa6441eab00 msgr2=0x7fa6441eb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.019+0000 7fa643fff640 1 --2- 172.21.15.72:0/531527626 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa6441eab00 0x7fa6441eb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.019+0000 7fa643fff640 1 -- 172.21.15.72:0/531527626 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6441e9a60 con 0x7fa6441eb8d0 2024-04-15T15:15:11.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.020+0000 7f17add8e640 1 -- 172.21.15.72:0/1348663825 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17a00d98f0 con 0x7f17a0050bf0 2024-04-15T15:15:11.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.020+0000 7fa643fff640 1 --2- 172.21.15.72:0/531527626 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa6441eb8d0 0x7fa6441e8d30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa638000c00 tx=0x7fa63800db40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:11.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.020+0000 7f1d87fff640 1 -- 172.21.15.72:0/1028536126 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d8004af50 con 0x7f1d80053c70 2024-04-15T15:15:11.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.020+0000 7fa62b7fe640 1 -- 172.21.15.72:0/531527626 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa638018070 con 0x7fa6441eb8d0 2024-04-15T15:15:11.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.020+0000 7f2712aab640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f270c0d1d50 0x7f270c1f5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.020+0000 7fa62b7fe640 1 -- 172.21.15.72:0/531527626 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa638005270 con 0x7fa6441eb8d0 2024-04-15T15:15:11.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.020+0000 7fa6494f2640 1 -- 172.21.15.72:0/531527626 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6441e9d40 con 0x7fa6441eb8d0 2024-04-15T15:15:11.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.020+0000 7f2712aab640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f270c145e90 0x7f270c13cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.020+0000 7f2712aab640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f270c13d180 0x7f270c13e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.020+0000 7f2712aab640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f270c0ccb20 con 0x7f270c145e90 2024-04-15T15:15:11.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.020+0000 7f2712aab640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f270c0cc530 con 0x7f270c13d180 2024-04-15T15:15:11.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.020+0000 7f2712aab640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f270c0d0440 con 0x7f270c0d1d50 2024-04-15T15:15:11.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.020+0000 7fa62b7fe640 1 -- 172.21.15.72:0/531527626 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63800de80 con 0x7fa6441eb8d0 2024-04-15T15:15:11.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.020+0000 7fa6494f2640 1 -- 172.21.15.72:0/531527626 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6441ea300 con 0x7fa6441eb8d0 2024-04-15T15:15:11.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.020+0000 7f2711aa9640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f270c0d1d50 0x7f270c1f5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.020+0000 7f2711aa9640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f270c0d1d50 0x7f270c1f5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3301/0 says I am v2:172.21.15.72:57896/0 (socket says 172.21.15.72:57896) 2024-04-15T15:15:11.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.020+0000 7f2711aa9640 1 -- 172.21.15.72:0/1102960819 learned_addr learned my addr 172.21.15.72:0/1102960819 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:11.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.020+0000 7f1d6f7fe640 1 -- 172.21.15.72:0/1028536126 <== mon.2 v2:172.21.15.175:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1d880aed70 con 0x7f1d80053c70 2024-04-15T15:15:11.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.021+0000 7f2711aa9640 1 -- 172.21.15.72:0/1102960819 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f270c13d180 msgr2=0x7f270c13e950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-15T15:15:11.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.021+0000 7f2711aa9640 1 --2- 172.21.15.72:0/1102960819 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f270c13d180 0x7f270c13e950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.021+0000 7f2711aa9640 1 -- 172.21.15.72:0/1102960819 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f270c145e90 msgr2=0x7f270c13cc40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-15T15:15:11.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.021+0000 7f2711aa9640 1 --2- 172.21.15.72:0/1102960819 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f270c145e90 0x7f270c13cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.021+0000 7f2711aa9640 1 -- 172.21.15.72:0/1102960819 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f270c1f2880 con 0x7f270c0d1d50 2024-04-15T15:15:11.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.021+0000 7f1d6f7fe640 1 --2- 172.21.15.72:0/1028536126 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1d7406c940 0x7f1d7406ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.021+0000 7f2711aa9640 1 --2- 172.21.15.72:0/1102960819 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f270c0d1d50 0x7f270c1f5290 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f270800c7c0 tx=0x7f270800cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:11.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.021+0000 7f17add8e640 1 -- 172.21.15.72:0/1348663825 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17a00057e0 con 0x7f17a0050bf0 2024-04-15T15:15:11.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.021+0000 7f1d6f7fe640 1 -- 172.21.15.72:0/1028536126 <== mon.2 v2:172.21.15.175:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f1d881288c0 con 0x7f1d80053c70 2024-04-15T15:15:11.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.021+0000 7f1d86ffd640 1 --2- 172.21.15.72:0/1028536126 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1d7406c940 0x7f1d7406ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.022+0000 7fa6494f2640 1 -- 172.21.15.72:0/531527626 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6440690f0 con 0x7fa6441eb8d0 2024-04-15T15:15:11.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.022+0000 7f1d86ffd640 1 --2- 172.21.15.72:0/1028536126 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1d7406c940 0x7f1d7406ee00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1d78005880 tx=0x7f1d78009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:11.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.022+0000 7f17a57fa640 1 -- 172.21.15.72:0/1348663825 <== mon.1 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f179c01a070 con 0x7f17a0050bf0 2024-04-15T15:15:11.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.022+0000 7fa62b7fe640 1 -- 172.21.15.72:0/531527626 <== mon.2 v2:172.21.15.175:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa63801b070 con 0x7fa6441eb8d0 2024-04-15T15:15:11.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.022+0000 7f17a57fa640 1 --2- 172.21.15.72:0/1348663825 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f178c06c940 0x7f178c06ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.023+0000 7fa62b7fe640 1 --2- 172.21.15.72:0/531527626 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa63006c940 0x7fa63006ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.023+0000 7f17acd8c640 1 --2- 172.21.15.72:0/1348663825 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f178c06c940 0x7f178c06ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.023+0000 7f17a57fa640 1 -- 172.21.15.72:0/1348663825 <== mon.1 v2:172.21.15.175:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f179c08bfe0 con 0x7f17a0050bf0 2024-04-15T15:15:11.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.023+0000 7fa642ffd640 1 --2- 172.21.15.72:0/531527626 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa63006c940 0x7fa63006ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.023+0000 7f17acd8c640 1 --2- 172.21.15.72:0/1348663825 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f178c06c940 0x7f178c06ee00 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1794005880 tx=0x7f1794007a00 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:11.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.023+0000 7fa642ffd640 1 --2- 172.21.15.72:0/531527626 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa63006c940 0x7fa63006ee00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa63c000ba0 tx=0x7fa63c00b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:11.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.024+0000 7f53bbfff640 1 -- 172.21.15.72:0/2739721631 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f53c4023340 con 0x7f53d406f640 2024-04-15T15:15:11.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.024+0000 7f2b77fff640 1 -- 172.21.15.72:0/2853332298 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b78048070 con 0x7f2b840e4d30 2024-04-15T15:15:11.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.024+0000 7ffac57fa640 1 -- 172.21.15.72:0/3646882716 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffabc017070 con 0x7ffac0050ce0 2024-04-15T15:15:11.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.024+0000 7f2b77fff640 1 -- 172.21.15.72:0/2853332298 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b78030670 con 0x7f2b840e4d30 2024-04-15T15:15:11.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.024+0000 7f2b90cdb640 1 -- 172.21.15.72:0/2853332298 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b8404ece0 con 0x7f2b840e4d30 2024-04-15T15:15:11.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.024+0000 7ffacdaf7640 1 -- 172.21.15.72:0/3646882716 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffac00dc900 con 0x7ffac0050ce0 2024-04-15T15:15:11.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.024+0000 7f109b7fe640 1 -- 172.21.15.72:0/4193862566 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a8017070 con 0x7f10ac0392d0 2024-04-15T15:15:11.093 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.025+0000 7ffacdaf7640 1 -- 172.21.15.72:0/3646882716 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffac0037b40 con 0x7ffac0050ce0 2024-04-15T15:15:11.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.025+0000 7f2b90cdb640 1 -- 172.21.15.72:0/2853332298 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b8404f2a0 con 0x7f2b840e4d30 2024-04-15T15:15:11.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.025+0000 7f2b77fff640 1 -- 172.21.15.72:0/2853332298 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b78034040 con 0x7f2b840e4d30 2024-04-15T15:15:11.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.025+0000 7f10b97db640 1 -- 172.21.15.72:0/4193862566 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10ac0df5a0 con 0x7f10ac0392d0 2024-04-15T15:15:11.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.025+0000 7f10b97db640 1 -- 172.21.15.72:0/4193862566 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10ac16ee80 con 0x7f10ac0392d0 2024-04-15T15:15:11.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.025+0000 7f109b7fe640 1 -- 172.21.15.72:0/4193862566 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10a8005480 con 0x7f10ac0392d0 2024-04-15T15:15:11.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.025+0000 7ffac57fa640 1 -- 172.21.15.72:0/3646882716 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffabc00ecd0 con 0x7ffac0050ce0 2024-04-15T15:15:11.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.025+0000 7ffac57fa640 1 -- 172.21.15.72:0/3646882716 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffabc00ce50 con 0x7ffac0050ce0 2024-04-15T15:15:11.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.026+0000 7f109b7fe640 1 -- 172.21.15.72:0/4193862566 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a8017070 con 0x7f10ac0392d0 2024-04-15T15:15:11.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.026+0000 7f2b90cdb640 1 -- 172.21.15.72:0/2853332298 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b840057e0 con 0x7f2b840e4d30 2024-04-15T15:15:11.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.026+0000 7f1d6f7fe640 1 -- 172.21.15.72:0/1028536126 <== mon.2 v2:172.21.15.175:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1d8004af50 con 0x7f1d80053c70 2024-04-15T15:15:11.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.026+0000 7ffacdaf7640 1 -- 172.21.15.72:0/3646882716 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffaa8005290 con 0x7ffac0050ce0 2024-04-15T15:15:11.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.027+0000 7f2b77fff640 1 -- 172.21.15.72:0/2853332298 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2b7804b070 con 0x7f2b840e4d30 2024-04-15T15:15:11.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.027+0000 7fa62b7fe640 1 -- 172.21.15.72:0/531527626 <== mon.2 v2:172.21.15.175:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa638056f00 con 0x7fa6441eb8d0 2024-04-15T15:15:11.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.026+0000 7f27027fc640 1 -- 172.21.15.72:0/1102960819 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2708018070 con 0x7f270c0d1d50 2024-04-15T15:15:11.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.026+0000 7f27027fc640 1 -- 172.21.15.72:0/1102960819 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f270800a640 con 0x7f270c0d1d50 2024-04-15T15:15:11.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.026+0000 7f27027fc640 1 -- 172.21.15.72:0/1102960819 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f270800aa10 con 0x7f270c0d1d50 2024-04-15T15:15:11.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.026+0000 7f2712aab640 1 -- 172.21.15.72:0/1102960819 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f270c1f2b60 con 0x7f270c0d1d50 2024-04-15T15:15:11.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.026+0000 7f2712aab640 1 -- 172.21.15.72:0/1102960819 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f270c1f3120 con 0x7f270c0d1d50 2024-04-15T15:15:11.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.027+0000 7f2b77fff640 1 --2- 172.21.15.72:0/2853332298 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2b7006c8f0 0x7f2b7006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.028+0000 7ffac57fa640 1 -- 172.21.15.72:0/3646882716 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ffabc01a070 con 0x7ffac0050ce0 2024-04-15T15:15:11.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.028+0000 7f2712aab640 1 -- 172.21.15.72:0/1102960819 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f270c075aa0 con 0x7f270c0d1d50 2024-04-15T15:15:11.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.028+0000 7f2b77fff640 1 -- 172.21.15.72:0/2853332298 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f2b780adf40 con 0x7f2b840e4d30 2024-04-15T15:15:11.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.028+0000 7f2b8b7fe640 1 --2- 172.21.15.72:0/2853332298 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2b7006c8f0 0x7f2b7006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.027+0000 7f10997fa640 1 -- 172.21.15.72:0/4193862566 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f108c005290 con 0x7f10ac0392d0 2024-04-15T15:15:11.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.028+0000 7ffac57fa640 1 --2- 172.21.15.72:0/3646882716 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ffaac06c8f0 0x7ffaac06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.028+0000 7ffac57fa640 1 -- 172.21.15.72:0/3646882716 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ffabc08c450 con 0x7ffac0050ce0 2024-04-15T15:15:11.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.028+0000 7ffac7fff640 1 --2- 172.21.15.72:0/3646882716 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ffaac06c8f0 0x7ffaac06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.099 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.028+0000 7f17a57fa640 1 -- 172.21.15.72:0/1348663825 <== mon.1 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f179c059b60 con 0x7f17a0050bf0 2024-04-15T15:15:11.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.029+0000 7f2b8b7fe640 1 --2- 172.21.15.72:0/2853332298 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2b7006c8f0 0x7f2b7006edb0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2b80006fd0 tx=0x7f2b80008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:11.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.029+0000 7ffac7fff640 1 --2- 172.21.15.72:0/3646882716 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ffaac06c8f0 0x7ffaac06edb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ffab4036000 tx=0x7ffab4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:11.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.029+0000 7f109b7fe640 1 -- 172.21.15.72:0/4193862566 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f10a80035f0 con 0x7f10ac0392d0 2024-04-15T15:15:11.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.030+0000 7f109b7fe640 1 --2- 172.21.15.72:0/4193862566 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f10a006ccf0 0x7f10a006f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.030+0000 7f10b2ffd640 1 --2- 172.21.15.72:0/4193862566 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f10a006ccf0 0x7f10a006f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.030+0000 7f10b2ffd640 1 --2- 172.21.15.72:0/4193862566 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f10a006ccf0 0x7f10a006f1b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f10a4007fd0 tx=0x7f10a4007d40 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:11.101 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.031+0000 7fa62b7fe640 1 -- 172.21.15.72:0/531527626 <== mon.2 v2:172.21.15.175:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa638059560 con 0x7fa6441eb8d0 2024-04-15T15:15:11.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.033+0000 7f27027fc640 1 -- 172.21.15.72:0/1102960819 <== mon.2 v2:172.21.15.175:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f270801b070 con 0x7f270c0d1d50 2024-04-15T15:15:11.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.034+0000 7f27027fc640 1 --2- 172.21.15.72:0/1102960819 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f26f006c9c0 0x7f26f006ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.034+0000 7f27027fc640 1 -- 172.21.15.72:0/1102960819 <== mon.2 v2:172.21.15.175:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f270808bb70 con 0x7f270c0d1d50 2024-04-15T15:15:11.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.035+0000 7f27112a8640 1 --2- 172.21.15.72:0/1102960819 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f26f006c9c0 0x7f26f006ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.036+0000 7f27027fc640 1 -- 172.21.15.72:0/1102960819 <== mon.2 v2:172.21.15.175:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2708022020 con 0x7f270c0d1d50 2024-04-15T15:15:11.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.036+0000 7f27112a8640 1 --2- 172.21.15.72:0/1102960819 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f26f006c9c0 0x7f26f006ee80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2704036000 tx=0x7f2704030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:11.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.037+0000 7f2b77fff640 1 -- 172.21.15.72:0/2853332298 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2b7807b8b0 con 0x7f2b840e4d30 2024-04-15T15:15:11.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.046+0000 7ffac57fa640 1 -- 172.21.15.72:0/3646882716 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ffac00dc900 con 0x7ffac0050ce0 2024-04-15T15:15:11.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.055+0000 7f109b7fe640 1 -- 172.21.15.72:0/4193862566 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f10a8005280 con 0x7f10ac0392d0 2024-04-15T15:15:11.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.056+0000 7f109b7fe640 1 -- 172.21.15.72:0/4193862566 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f10a8020080 con 0x7f10ac0392d0 2024-04-15T15:15:11.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.433+0000 7fa6494f2640 1 -- 172.21.15.72:0/531527626 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fa6440b0340 con 0x7fa6441eb8d0 2024-04-15T15:15:11.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.434+0000 7fa62b7fe640 1 -- 172.21.15.72:0/531527626 <== mon.2 v2:172.21.15.175:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa638058e60 con 0x7fa6441eb8d0 2024-04-15T15:15:11.436 INFO:teuthology.orchestra.run.smithi072.stdout:55834574855 2024-04-15T15:15:11.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.437+0000 7fa6297fa640 1 -- 172.21.15.72:0/531527626 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa63006c940 msgr2=0x7fa63006ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.437+0000 7fa6297fa640 1 --2- 172.21.15.72:0/531527626 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa63006c940 0x7fa63006ee00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa63c000ba0 tx=0x7fa63c00b040 comp rx=0 tx=0).stop 2024-04-15T15:15:11.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.437+0000 7fa6297fa640 1 -- 172.21.15.72:0/531527626 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa6441eb8d0 msgr2=0x7fa6441e8d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.437+0000 7fa6297fa640 1 --2- 172.21.15.72:0/531527626 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa6441eb8d0 0x7fa6441e8d30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa638000c00 tx=0x7fa63800db40 comp rx=0 tx=0).stop 2024-04-15T15:15:11.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.438+0000 7fa6297fa640 1 -- 172.21.15.72:0/531527626 shutdown_connections 2024-04-15T15:15:11.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.438+0000 7fa6297fa640 1 --2- 172.21.15.72:0/531527626 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa63006c940 0x7fa63006ee00 secure :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa63c000ba0 tx=0x7fa63c00b040 comp rx=0 tx=0).stop 2024-04-15T15:15:11.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.438+0000 7fa6297fa640 1 --2- 172.21.15.72:0/531527626 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa6441eb8d0 0x7fa6441e8d30 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.438+0000 7fa6297fa640 1 --2- 172.21.15.72:0/531527626 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa6441eab00 0x7fa6441eb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.438+0000 7fa6297fa640 1 --2- 172.21.15.72:0/531527626 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa62c0037a0 0x7fa644145700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.438+0000 7fa6297fa640 1 -- 172.21.15.72:0/531527626 >> 172.21.15.72:0/531527626 conn(0x7fa644071fe0 msgr2=0x7fa64406e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:11.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.438+0000 7fa6297fa640 1 -- 172.21.15.72:0/531527626 shutdown_connections 2024-04-15T15:15:11.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.438+0000 7fa6297fa640 1 -- 172.21.15.72:0/531527626 wait complete. 2024-04-15T15:15:11.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.443+0000 7f53b9ffb640 1 -- 172.21.15.72:0/2739721631 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f53d40ae170 con 0x7f53d406f640 2024-04-15T15:15:11.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.443+0000 7f53bbfff640 1 -- 172.21.15.72:0/2739721631 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f53c4014e20 con 0x7f53d406f640 2024-04-15T15:15:11.446 INFO:teuthology.orchestra.run.smithi072.stdout:55834574855 2024-04-15T15:15:11.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.446+0000 7f17add8e640 1 -- 172.21.15.72:0/1348663825 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f17a00d7140 con 0x7f17a0050bf0 2024-04-15T15:15:11.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.447+0000 7f53d9e28640 1 -- 172.21.15.72:0/2739721631 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f53c006c920 msgr2=0x7f53c006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.447+0000 7f53d9e28640 1 --2- 172.21.15.72:0/2739721631 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f53c006c920 0x7f53c006ede0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f53cc000f80 tx=0x7f53cc001570 comp rx=0 tx=0).stop 2024-04-15T15:15:11.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.447+0000 7f17a57fa640 1 -- 172.21.15.72:0/1348663825 <== mon.1 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f179c05de60 con 0x7f17a0050bf0 2024-04-15T15:15:11.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.447+0000 7f53d9e28640 1 -- 172.21.15.72:0/2739721631 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53d406f640 msgr2=0x7f53d41f4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.447+0000 7f53d9e28640 1 --2- 172.21.15.72:0/2739721631 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53d406f640 0x7f53d41f4300 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f53c4001a30 tx=0x7f53c4014420 comp rx=0 tx=0).stop 2024-04-15T15:15:11.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.447+0000 7f53d9e28640 1 -- 172.21.15.72:0/2739721631 shutdown_connections 2024-04-15T15:15:11.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.447+0000 7f53d9e28640 1 --2- 172.21.15.72:0/2739721631 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f53c006c920 0x7f53c006ede0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.447+0000 7f53d9e28640 1 --2- 172.21.15.72:0/2739721631 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f53d415a710 0x7f53d40c47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.447+0000 7f53d9e28640 1 --2- 172.21.15.72:0/2739721631 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f53d40aced0 0x7f53d409d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.447+0000 7f53d9e28640 1 --2- 172.21.15.72:0/2739721631 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53d406f640 0x7f53d41f4300 secure :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f53c4001a30 tx=0x7f53c4014420 comp rx=0 tx=0).stop 2024-04-15T15:15:11.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.447+0000 7f53d9e28640 1 -- 172.21.15.72:0/2739721631 >> 172.21.15.72:0/2739721631 conn(0x7f53d40b2220 msgr2=0x7f53d40afbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:11.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.448+0000 7f53d9e28640 1 -- 172.21.15.72:0/2739721631 shutdown_connections 2024-04-15T15:15:11.450 INFO:teuthology.orchestra.run.smithi072.stdout:55834574856 2024-04-15T15:15:11.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.448+0000 7f53d9e28640 1 -- 172.21.15.72:0/2739721631 wait complete. 2024-04-15T15:15:11.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.451+0000 7f17add8e640 1 -- 172.21.15.72:0/1348663825 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f178c06c940 msgr2=0x7f178c06ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.451+0000 7f17add8e640 1 --2- 172.21.15.72:0/1348663825 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f178c06c940 0x7f178c06ee00 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1794005880 tx=0x7f1794007a00 comp rx=0 tx=0).stop 2024-04-15T15:15:11.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.450+0000 7f1d87fff640 1 -- 172.21.15.72:0/1028536126 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f1d80005390 con 0x7f1d80053c70 2024-04-15T15:15:11.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.451+0000 7f17add8e640 1 -- 172.21.15.72:0/1348663825 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f17a0050bf0 msgr2=0x7f17a00d8670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.453 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.451+0000 7f17add8e640 1 --2- 172.21.15.72:0/1348663825 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f17a0050bf0 0x7f17a00d8670 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f179c00e850 tx=0x7f179c00ed10 comp rx=0 tx=0).stop 2024-04-15T15:15:11.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.451+0000 7f17add8e640 1 -- 172.21.15.72:0/1348663825 shutdown_connections 2024-04-15T15:15:11.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.451+0000 7f17add8e640 1 --2- 172.21.15.72:0/1348663825 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f178c06c940 0x7f178c06ee00 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.451+0000 7f17add8e640 1 --2- 172.21.15.72:0/1348663825 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f17a00e3820 0x7f17a01813d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.451+0000 7f17add8e640 1 --2- 172.21.15.72:0/1348663825 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f17a0050bf0 0x7f17a00d8670 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.451+0000 7f17add8e640 1 --2- 172.21.15.72:0/1348663825 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17a0039c10 0x7f17a00514a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.451+0000 7f17add8e640 1 -- 172.21.15.72:0/1348663825 >> 172.21.15.72:0/1348663825 conn(0x7f17a0055bd0 msgr2=0x7f17a00d7660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:11.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.451+0000 7f1d6f7fe640 1 -- 172.21.15.72:0/1028536126 <== mon.2 v2:172.21.15.175:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1d880f62b0 con 0x7f1d80053c70 2024-04-15T15:15:11.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.451+0000 7f17add8e640 1 -- 172.21.15.72:0/1348663825 shutdown_connections 2024-04-15T15:15:11.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.451+0000 7f17add8e640 1 -- 172.21.15.72:0/1348663825 wait complete. 2024-04-15T15:15:11.455 INFO:teuthology.orchestra.run.smithi072.stdout:55834574854 2024-04-15T15:15:11.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.452+0000 7f2712aab640 1 -- 172.21.15.72:0/1102960819 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f270c0b1b10 con 0x7f270c0d1d50 2024-04-15T15:15:11.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.453+0000 7f27027fc640 1 -- 172.21.15.72:0/1102960819 <== mon.2 v2:172.21.15.175:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2708059bb0 con 0x7f270c0d1d50 2024-04-15T15:15:11.456 INFO:teuthology.orchestra.run.smithi072.stdout:55834574856 2024-04-15T15:15:11.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.456+0000 7f1d87fff640 1 -- 172.21.15.72:0/1028536126 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1d7406c940 msgr2=0x7f1d7406ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.456+0000 7f1d87fff640 1 --2- 172.21.15.72:0/1028536126 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1d7406c940 0x7f1d7406ee00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1d78005880 tx=0x7f1d78009040 comp rx=0 tx=0).stop 2024-04-15T15:15:11.460 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574855 for osd.5 2024-04-15T15:15:11.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.456+0000 7f1d87fff640 1 -- 172.21.15.72:0/1028536126 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1d80053c70 msgr2=0x7f1d801897f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.456+0000 7f1d87fff640 1 --2- 172.21.15.72:0/1028536126 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1d80053c70 0x7f1d801897f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1d8806a0f0 tx=0x7f1d88075700 comp rx=0 tx=0).stop 2024-04-15T15:15:11.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.456+0000 7f2712aab640 1 -- 172.21.15.72:0/1102960819 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f26f006c9c0 msgr2=0x7f26f006ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.456+0000 7f2712aab640 1 --2- 172.21.15.72:0/1102960819 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f26f006c9c0 0x7f26f006ee80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2704036000 tx=0x7f2704030000 comp rx=0 tx=0).stop 2024-04-15T15:15:11.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.456+0000 7f2712aab640 1 -- 172.21.15.72:0/1102960819 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f270c0d1d50 msgr2=0x7f270c1f5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.456+0000 7f2712aab640 1 --2- 172.21.15.72:0/1102960819 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f270c0d1d50 0x7f270c1f5290 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f270800c7c0 tx=0x7f270800cc80 comp rx=0 tx=0).stop 2024-04-15T15:15:11.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.456+0000 7f2712aab640 1 -- 172.21.15.72:0/1102960819 shutdown_connections 2024-04-15T15:15:11.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.456+0000 7f2712aab640 1 --2- 172.21.15.72:0/1102960819 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f26f006c9c0 0x7f26f006ee80 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.456+0000 7f2712aab640 1 --2- 172.21.15.72:0/1102960819 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f270c13d180 0x7f270c13e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.456+0000 7f2712aab640 1 --2- 172.21.15.72:0/1102960819 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f270c145e90 0x7f270c13cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.456+0000 7f2712aab640 1 --2- 172.21.15.72:0/1102960819 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f270c0d1d50 0x7f270c1f5290 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.456+0000 7f2712aab640 1 -- 172.21.15.72:0/1102960819 >> 172.21.15.72:0/1102960819 conn(0x7f270c0b3ec0 msgr2=0x7f270c0ce500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:11.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.457+0000 7f2712aab640 1 -- 172.21.15.72:0/1102960819 shutdown_connections 2024-04-15T15:15:11.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.457+0000 7f1d87fff640 1 -- 172.21.15.72:0/1028536126 shutdown_connections 2024-04-15T15:15:11.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.457+0000 7f1d87fff640 1 --2- 172.21.15.72:0/1028536126 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1d7406c940 0x7f1d7406ee00 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.457+0000 7f1d87fff640 1 --2- 172.21.15.72:0/1028536126 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1d80053c70 0x7f1d801897f0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.457+0000 7f1d87fff640 1 --2- 172.21.15.72:0/1028536126 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d80033ef0 0x7f1d8004e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.457+0000 7f1d87fff640 1 --2- 172.21.15.72:0/1028536126 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d8000c0a0 0x7f1d8004a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.457+0000 7f1d87fff640 1 -- 172.21.15.72:0/1028536126 >> 172.21.15.72:0/1028536126 conn(0x7f1d80009fb0 msgr2=0x7f1d800342d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:11.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.457+0000 7f2712aab640 1 -- 172.21.15.72:0/1102960819 wait complete. 2024-04-15T15:15:11.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.457+0000 7f1d87fff640 1 -- 172.21.15.72:0/1028536126 shutdown_connections 2024-04-15T15:15:11.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.457+0000 7f1d87fff640 1 -- 172.21.15.72:0/1028536126 wait complete. 2024-04-15T15:15:11.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.457+0000 7f2b90cdb640 1 -- 172.21.15.72:0/2853332298 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f2b84057480 con 0x7f2b840e4d30 2024-04-15T15:15:11.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.458+0000 7f2b77fff640 1 -- 172.21.15.72:0/2853332298 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2b7807fbb0 con 0x7f2b840e4d30 2024-04-15T15:15:11.465 INFO:teuthology.orchestra.run.smithi072.stdout:55834574855 2024-04-15T15:15:11.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.461+0000 7f2b90cdb640 1 -- 172.21.15.72:0/2853332298 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2b7006c8f0 msgr2=0x7f2b7006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.461+0000 7f2b90cdb640 1 --2- 172.21.15.72:0/2853332298 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2b7006c8f0 0x7f2b7006edb0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2b80006fd0 tx=0x7f2b80008040 comp rx=0 tx=0).stop 2024-04-15T15:15:11.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.461+0000 7f2b90cdb640 1 -- 172.21.15.72:0/2853332298 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b840e4d30 msgr2=0x7f2b84050830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.461+0000 7f2b90cdb640 1 --2- 172.21.15.72:0/2853332298 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b840e4d30 0x7f2b84050830 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2b78033000 tx=0x7f2b78004730 comp rx=0 tx=0).stop 2024-04-15T15:15:11.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.461+0000 7f2b90cdb640 1 -- 172.21.15.72:0/2853332298 shutdown_connections 2024-04-15T15:15:11.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.462+0000 7f2b90cdb640 1 --2- 172.21.15.72:0/2853332298 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2b7006c8f0 0x7f2b7006edb0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.462+0000 7f2b90cdb640 1 --2- 172.21.15.72:0/2853332298 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2b84050d70 0x7f2b8404c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.462+0000 7f2b90cdb640 1 --2- 172.21.15.72:0/2853332298 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b840e4d30 0x7f2b84050830 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.461+0000 7ffacdaf7640 1 -- 172.21.15.72:0/3646882716 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7ffaa8005850 con 0x7ffac0050ce0 2024-04-15T15:15:11.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.462+0000 7f2b90cdb640 1 --2- 172.21.15.72:0/2853332298 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2b840e12b0 0x7f2b840d1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.462+0000 7f2b90cdb640 1 -- 172.21.15.72:0/2853332298 >> 172.21.15.72:0/2853332298 conn(0x7f2b84020040 msgr2=0x7f2b840d8b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:11.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.462+0000 7f2b90cdb640 1 -- 172.21.15.72:0/2853332298 shutdown_connections 2024-04-15T15:15:11.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.462+0000 7f2b90cdb640 1 -- 172.21.15.72:0/2853332298 wait complete. 2024-04-15T15:15:11.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.462+0000 7ffac57fa640 1 -- 172.21.15.72:0/3646882716 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ffabc059fd0 con 0x7ffac0050ce0 2024-04-15T15:15:11.468 INFO:teuthology.orchestra.run.smithi072.stdout:55834574856 2024-04-15T15:15:11.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.465+0000 7ffacdaf7640 1 -- 172.21.15.72:0/3646882716 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ffaac06c8f0 msgr2=0x7ffaac06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.465+0000 7ffacdaf7640 1 --2- 172.21.15.72:0/3646882716 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ffaac06c8f0 0x7ffaac06edb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ffab4036000 tx=0x7ffab4008030 comp rx=0 tx=0).stop 2024-04-15T15:15:11.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.465+0000 7ffacdaf7640 1 -- 172.21.15.72:0/3646882716 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffac0050ce0 msgr2=0x7ffac00dbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.465+0000 7ffacdaf7640 1 --2- 172.21.15.72:0/3646882716 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffac0050ce0 0x7ffac00dbea0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ffabc00ebe0 tx=0x7ffabc00c520 comp rx=0 tx=0).stop 2024-04-15T15:15:11.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.465+0000 7ffacdaf7640 1 -- 172.21.15.72:0/3646882716 shutdown_connections 2024-04-15T15:15:11.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.465+0000 7ffacdaf7640 1 --2- 172.21.15.72:0/3646882716 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ffaac06c8f0 0x7ffaac06edb0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.465+0000 7ffacdaf7640 1 --2- 172.21.15.72:0/3646882716 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffac00e67f0 0x7ffac003a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.465+0000 7ffacdaf7640 1 --2- 172.21.15.72:0/3646882716 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffac00e3ad0 0x7ffac0034cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.465+0000 7ffacdaf7640 1 --2- 172.21.15.72:0/3646882716 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffac0050ce0 0x7ffac00dbea0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.465+0000 7ffacdaf7640 1 -- 172.21.15.72:0/3646882716 >> 172.21.15.72:0/3646882716 conn(0x7ffac0055350 msgr2=0x7ffac004b830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:11.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.466+0000 7ffacdaf7640 1 -- 172.21.15.72:0/3646882716 shutdown_connections 2024-04-15T15:15:11.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.466+0000 7ffacdaf7640 1 -- 172.21.15.72:0/3646882716 wait complete. 2024-04-15T15:15:11.471 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574855 for osd.2 2024-04-15T15:15:11.474 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574856 for osd.7 2024-04-15T15:15:11.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.473+0000 7f10997fa640 1 -- 172.21.15.72:0/4193862566 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f108c005850 con 0x7f10ac0392d0 2024-04-15T15:15:11.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.473+0000 7f109b7fe640 1 -- 172.21.15.72:0/4193862566 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f10a8020080 con 0x7f10ac0392d0 2024-04-15T15:15:11.475 INFO:teuthology.orchestra.run.smithi072.stdout:55834574856 2024-04-15T15:15:11.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.476+0000 7f10997fa640 1 -- 172.21.15.72:0/4193862566 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f10a006ccf0 msgr2=0x7f10a006f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.476+0000 7f10997fa640 1 --2- 172.21.15.72:0/4193862566 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f10a006ccf0 0x7f10a006f1b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f10a4007fd0 tx=0x7f10a4007d40 comp rx=0 tx=0).stop 2024-04-15T15:15:11.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.476+0000 7f10997fa640 1 -- 172.21.15.72:0/4193862566 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10ac0392d0 msgr2=0x7f10ac0e1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.479 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574856 for osd.6 2024-04-15T15:15:11.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.476+0000 7f10997fa640 1 --2- 172.21.15.72:0/4193862566 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10ac0392d0 0x7f10ac0e1f50 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f10a800c4e0 tx=0x7f10a800c9a0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.476+0000 7f10997fa640 1 -- 172.21.15.72:0/4193862566 shutdown_connections 2024-04-15T15:15:11.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.476+0000 7f10997fa640 1 --2- 172.21.15.72:0/4193862566 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f10a006ccf0 0x7f10a006f1b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.476+0000 7f10997fa640 1 --2- 172.21.15.72:0/4193862566 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f10ac0501a0 0x7f10ac0dfef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.476+0000 7f10997fa640 1 --2- 172.21.15.72:0/4193862566 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10ac0392d0 0x7f10ac0e1f50 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.476+0000 7f10997fa640 1 --2- 172.21.15.72:0/4193862566 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f10ac00aaa0 0x7f10ac16e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.476+0000 7f10997fa640 1 -- 172.21.15.72:0/4193862566 >> 172.21.15.72:0/4193862566 conn(0x7f10ac009fb0 msgr2=0x7f10ac052d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:11.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.477+0000 7f10997fa640 1 -- 172.21.15.72:0/4193862566 shutdown_connections 2024-04-15T15:15:11.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:11.477+0000 7f10997fa640 1 -- 172.21.15.72:0/4193862566 wait complete. 2024-04-15T15:15:11.481 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574854 for osd.0 2024-04-15T15:15:11.481 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574855 for osd.4 2024-04-15T15:15:11.482 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574856 for osd.1 2024-04-15T15:15:11.490 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574856 for osd.3 2024-04-15T15:15:12.461 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-15T15:15:12.472 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-15T15:15:12.475 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-15T15:15:12.480 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-15T15:15:12.481 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-15T15:15:12.483 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-15T15:15:12.483 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-15T15:15:12.490 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-15T15:15:12.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.627+0000 7fcd3aa22640 1 Processor -- start 2024-04-15T15:15:12.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.627+0000 7fcd3aa22640 1 -- start start 2024-04-15T15:15:12.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.628+0000 7fcd3aa22640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcd3415b180 0x7fcd3406eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.628+0000 7fcd3aa22640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcd3415b560 0x7fcd3406f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.628+0000 7fcd3aa22640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd340cf910 con 0x7fcd34073a20 2024-04-15T15:15:12.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.628+0000 7fcd3aa22640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fcd340b4130 con 0x7fcd3415b560 2024-04-15T15:15:12.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.628+0000 7fcd3aa22640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fcd340d0cc0 con 0x7fcd3415b180 2024-04-15T15:15:12.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.629+0000 7fcd38fbf640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fcd34073a20 0x7fcd3415eee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37840/0 (socket says 172.21.15.72:37840) 2024-04-15T15:15:12.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.629+0000 7fcd38fbf640 1 -- 172.21.15.72:0/1675179634 learned_addr learned my addr 172.21.15.72:0/1675179634 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:12.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.629+0000 7fcd337fe640 1 --2- 172.21.15.72:0/1675179634 >> v2:172.21.15.72:3300/0 conn(0x7fcd3415b560 0x7fcd3406f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.630+0000 7fcd337fe640 1 -- 172.21.15.72:0/1675179634 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcd3415b180 msgr2=0x7fcd3406eca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-15T15:15:12.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.630+0000 7fcd337fe640 1 --2- 172.21.15.72:0/1675179634 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcd3415b180 0x7fcd3406eca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.630+0000 7fcd337fe640 1 -- 172.21.15.72:0/1675179634 >> v1:172.21.15.72:6789/0 conn(0x7fcd34073a20 legacy=0x7fcd3415eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:12.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.630+0000 7fcd337fe640 1 -- 172.21.15.72:0/1675179634 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd340cfbf0 con 0x7fcd3415b560 2024-04-15T15:15:12.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.631+0000 7fcd337fe640 1 --2- 172.21.15.72:0/1675179634 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcd3415b560 0x7fcd3406f1e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fcd28009a00 tx=0x7fcd2802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec3b7b86b96b4cd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:12.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.632+0000 7fcd32ffd640 1 -- 172.21.15.72:0/1675179634 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd28035020 con 0x7fcd3415b560 2024-04-15T15:15:12.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.632+0000 7fcd32ffd640 1 -- 172.21.15.72:0/1675179634 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd2802d040 con 0x7fcd3415b560 2024-04-15T15:15:12.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.633+0000 7fcd3aa22640 1 -- 172.21.15.72:0/1675179634 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcd3415b560 msgr2=0x7fcd3406f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:12.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.633+0000 7fcd3aa22640 1 --2- 172.21.15.72:0/1675179634 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcd3415b560 0x7fcd3406f1e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fcd28009a00 tx=0x7fcd2802b980 comp rx=0 tx=0).stop 2024-04-15T15:15:12.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.633+0000 7fcd3aa22640 1 -- 172.21.15.72:0/1675179634 shutdown_connections 2024-04-15T15:15:12.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.633+0000 7fcd3aa22640 1 --2- 172.21.15.72:0/1675179634 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcd3415b560 0x7fcd3406f1e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.633+0000 7fcd3aa22640 1 --2- 172.21.15.72:0/1675179634 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcd3415b180 0x7fcd3406eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.633+0000 7fcd3aa22640 1 -- 172.21.15.72:0/1675179634 >> 172.21.15.72:0/1675179634 conn(0x7fcd34072a10 msgr2=0x7fcd34072e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:12.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.635+0000 7fcd3aa22640 1 -- 172.21.15.72:0/1675179634 shutdown_connections 2024-04-15T15:15:12.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.635+0000 7fcd3aa22640 1 -- 172.21.15.72:0/1675179634 wait complete. 2024-04-15T15:15:12.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.636+0000 7fcd3aa22640 1 Processor -- start 2024-04-15T15:15:12.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.637+0000 7fcd3aa22640 1 -- start start 2024-04-15T15:15:12.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.638+0000 7fcd3aa22640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcd34073a20 0x7fcd341453f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.638+0000 7fcd33fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcd34073a20 0x7fcd341453f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.638+0000 7fcd33fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcd34073a20 0x7fcd341453f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45298/0 (socket says 172.21.15.72:45298) 2024-04-15T15:15:12.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.638+0000 7fcd3aa22640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcd3415b180 0x7fcd34159e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.638+0000 7fcd3aa22640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcd341414c0 0x7fcd3414f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.638+0000 7fcd3aa22640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fcd340d0920 con 0x7fcd34073a20 2024-04-15T15:15:12.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.638+0000 7fcd3aa22640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fcd3406d300 con 0x7fcd3415b180 2024-04-15T15:15:12.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.638+0000 7fcd3aa22640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fcd340d0aa0 con 0x7fcd341414c0 2024-04-15T15:15:12.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.639+0000 7fcd33fff640 1 -- 172.21.15.72:0/3279367777 learned_addr learned my addr 172.21.15.72:0/3279367777 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:12.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.639+0000 7fcd337fe640 1 --2- 172.21.15.72:0/3279367777 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcd3415b180 0x7fcd34159e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.639+0000 7fcd38fbf640 1 --2- 172.21.15.72:0/3279367777 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcd341414c0 0x7fcd3414f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.639+0000 7fcd33fff640 1 -- 172.21.15.72:0/3279367777 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcd341414c0 msgr2=0x7fcd3414f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:12.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.639+0000 7fcd33fff640 1 --2- 172.21.15.72:0/3279367777 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcd341414c0 0x7fcd3414f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.639+0000 7fcd33fff640 1 -- 172.21.15.72:0/3279367777 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcd3415b180 msgr2=0x7fcd34159e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:12.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.639+0000 7fcd33fff640 1 --2- 172.21.15.72:0/3279367777 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcd3415b180 0x7fcd34159e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.639+0000 7fcd33fff640 1 -- 172.21.15.72:0/3279367777 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd34141a40 con 0x7fcd34073a20 2024-04-15T15:15:12.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.640+0000 7fcd33fff640 1 --2- 172.21.15.72:0/3279367777 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcd34073a20 0x7fcd341453f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fcd2000b780 tx=0x7fcd2000bc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:12.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.641+0000 7fcd30ff9640 1 -- 172.21.15.72:0/3279367777 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd20019070 con 0x7fcd34073a20 2024-04-15T15:15:12.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.642+0000 7fcd3aa22640 1 -- 172.21.15.72:0/3279367777 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd34141d20 con 0x7fcd34073a20 2024-04-15T15:15:12.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.642+0000 7fcd3aa22640 1 -- 172.21.15.72:0/3279367777 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd341422e0 con 0x7fcd34073a20 2024-04-15T15:15:12.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.646+0000 7fcd30ff9640 1 -- 172.21.15.72:0/3279367777 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd2000c740 con 0x7fcd34073a20 2024-04-15T15:15:12.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.646+0000 7fcd30ff9640 1 -- 172.21.15.72:0/3279367777 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd2000cb30 con 0x7fcd34073a20 2024-04-15T15:15:12.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.646+0000 7fcd30ff9640 1 -- 172.21.15.72:0/3279367777 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcd2001c070 con 0x7fcd34073a20 2024-04-15T15:15:12.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.647+0000 7fcd30ff9640 1 --2- 172.21.15.72:0/3279367777 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcd1806c9f0 0x7fcd1806eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.647+0000 7fcd337fe640 1 --2- 172.21.15.72:0/3279367777 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcd1806c9f0 0x7fcd1806eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.648+0000 7fcd337fe640 1 --2- 172.21.15.72:0/3279367777 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcd1806c9f0 0x7fcd1806eeb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcd28036000 tx=0x7fcd28008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:12.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.648+0000 7fcd30ff9640 1 -- 172.21.15.72:0/3279367777 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fcd2008b440 con 0x7fcd34073a20 2024-04-15T15:15:12.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.650+0000 7fcd3aa22640 1 -- 172.21.15.72:0/3279367777 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd34075b10 con 0x7fcd34073a20 2024-04-15T15:15:12.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.659+0000 7fcd30ff9640 1 -- 172.21.15.72:0/3279367777 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcd200595a0 con 0x7fcd34073a20 2024-04-15T15:15:12.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.682+0000 7f5263fff640 1 Processor -- start 2024-04-15T15:15:12.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.682+0000 7f5263fff640 1 -- start start 2024-04-15T15:15:12.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.683+0000 7fe7bd8ab640 1 Processor -- start 2024-04-15T15:15:12.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.683+0000 7fe7bd8ab640 1 -- start start 2024-04-15T15:15:12.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.684+0000 7f5263fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f525c049070 0x7f525c03b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.684+0000 7f5263fff640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f525c009570 con 0x7f525c009190 2024-04-15T15:15:12.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.684+0000 7f5263fff640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f525c035970 con 0x7f525c048c40 2024-04-15T15:15:12.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.684+0000 7f5263fff640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f525c035c30 con 0x7f525c049070 2024-04-15T15:15:12.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.682+0000 7fdc75c4a640 1 Processor -- start 2024-04-15T15:15:12.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.683+0000 7fdc75c4a640 1 -- start start 2024-04-15T15:15:12.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.684+0000 7fdc75c4a640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fdc6803eaa0 0x7fdc680f3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.684+0000 7f52627fc640 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f525c048c40 0x7f525c03aa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.72:34012/0 (socket says 172.21.15.72:34012) 2024-04-15T15:15:12.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.684+0000 7f52627fc640 1 -- 172.21.15.72:0/543268169 learned_addr learned my addr 172.21.15.72:0/543268169 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:12.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.684+0000 7fdc75c4a640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdc68038050 0x7fdc68057570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.684+0000 7fdc75c4a640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc680f49c0 con 0x7fdc68037b00 2024-04-15T15:15:12.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.684+0000 7fdc75c4a640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fdc68054580 con 0x7fdc6803eaa0 2024-04-15T15:15:12.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.684+0000 7fdc75c4a640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fdc68053f30 con 0x7fdc68038050 2024-04-15T15:15:12.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.684+0000 7f52637fe640 1 --2- 172.21.15.72:0/543268169 >> v2:172.21.15.72:3300/0 conn(0x7f525c049070 0x7f525c03b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.684+0000 7f5261ffb640 1 -- 172.21.15.72:0/543268169 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 349732576 0 0) 0x7f525c035970 con 0x7f525c048c40 2024-04-15T15:15:12.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.685+0000 7f5261ffb640 1 -- 172.21.15.72:0/543268169 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f524c003520 con 0x7f525c048c40 2024-04-15T15:15:12.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.685+0000 7f52637fe640 1 -- 172.21.15.72:0/543268169 >> v1:172.21.15.175:6789/0 conn(0x7f525c048c40 legacy=0x7f525c03aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:12.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.685+0000 7f52637fe640 1 -- 172.21.15.72:0/543268169 >> v1:172.21.15.72:6789/0 conn(0x7f525c009190 legacy=0x7f525c052f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:12.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.685+0000 7fdc6effd640 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fdc68037b00 0x7fdc680f40f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.72:34028/0 (socket says 172.21.15.72:34028) 2024-04-15T15:15:12.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.685+0000 7fdc6effd640 1 -- 172.21.15.72:0/1118402973 learned_addr learned my addr 172.21.15.72:0/1118402973 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:12.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.685+0000 7fdc6ffff640 1 --2- 172.21.15.72:0/1118402973 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdc68038050 0x7fdc68057570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.685+0000 7fdc6f7fe640 1 --2- 172.21.15.72:0/1118402973 >> v2:172.21.15.175:3300/0 conn(0x7fdc6803eaa0 0x7fdc680f3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.685+0000 7fdc6ffff640 1 -- 172.21.15.72:0/1118402973 >> v2:172.21.15.175:3300/0 conn(0x7fdc6803eaa0 msgr2=0x7fdc680f3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:12.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.685+0000 7fdc6ffff640 1 --2- 172.21.15.72:0/1118402973 >> v2:172.21.15.175:3300/0 conn(0x7fdc6803eaa0 0x7fdc680f3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.685+0000 7fdc6ffff640 1 -- 172.21.15.72:0/1118402973 >> v1:172.21.15.175:6789/0 conn(0x7fdc68037b00 legacy=0x7fdc680f40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:12.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.685+0000 7fdc6ffff640 1 -- 172.21.15.72:0/1118402973 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc68054210 con 0x7fdc68038050 2024-04-15T15:15:12.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.685+0000 7f7bfa1bd640 1 Processor -- start 2024-04-15T15:15:12.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.685+0000 7f4ac8255640 1 Processor -- start 2024-04-15T15:15:12.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.685+0000 7fe7bd8ab640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe7b0048c40 0x7fe7b0049090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.685+0000 7fe7bd8ab640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe7b003aa20 0x7fe7b003ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.685+0000 7fe7bd8ab640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7b0009570 con 0x7fe7b0009190 2024-04-15T15:15:12.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.685+0000 7fe7bd8ab640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fe7b00495d0 con 0x7fe7b003aa20 2024-04-15T15:15:12.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.685+0000 7fe7bd8ab640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fe7b0035970 con 0x7fe7b0048c40 2024-04-15T15:15:12.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.685+0000 7fe7bc8a9640 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fe7b0009190 0x7fe7b0052f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.72:34030/0 (socket says 172.21.15.72:34030) 2024-04-15T15:15:12.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.685+0000 7fe7bc8a9640 1 -- 172.21.15.72:0/2704731299 learned_addr learned my addr 172.21.15.72:0/2704731299 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:12.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.685+0000 7f52637fe640 1 -- 172.21.15.72:0/543268169 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f525c035f10 con 0x7f525c049070 2024-04-15T15:15:12.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.686+0000 7f52637fe640 1 --2- 172.21.15.72:0/543268169 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f525c049070 0x7f525c03b130 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f526406a870 tx=0x7f5264069760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=301bd44c25b0809 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:12.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.686+0000 7f5261ffb640 1 -- 172.21.15.72:0/543268169 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52640b4070 con 0x7f525c049070 2024-04-15T15:15:12.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.686+0000 7fe7bd0aa640 1 --2- 172.21.15.72:0/2704731299 >> v2:172.21.15.175:3300/0 conn(0x7fe7b003aa20 0x7fe7b003ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.686+0000 7f5261ffb640 1 -- 172.21.15.72:0/543268169 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52640a7e20 con 0x7f525c049070 2024-04-15T15:15:12.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.686+0000 7fe7b7fff640 1 --2- 172.21.15.72:0/2704731299 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe7b0048c40 0x7fe7b0049090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.686+0000 7fdc6ffff640 1 --2- 172.21.15.72:0/1118402973 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdc68038050 0x7fdc68057570 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdc70050270 tx=0x7fdc70069760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efd8a5603143b5e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:12.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.685+0000 7f7bfa1bd640 1 -- start start 2024-04-15T15:15:12.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.686+0000 7f7bfa1bd640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f7bf40d0490 0x7f7bf406ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.686+0000 7f7bfa1bd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7bf40d0920 0x7f7bf406f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.686+0000 7f7bfa1bd640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7bf415b9b0 con 0x7f7bf415b5d0 2024-04-15T15:15:12.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.686+0000 7f7bfa1bd640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f7bf40cf1a0 con 0x7f7bf40d0920 2024-04-15T15:15:12.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.686+0000 7f7bfa1bd640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f7bf40d0000 con 0x7f7bf40d0490 2024-04-15T15:15:12.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.687+0000 7f5261ffb640 1 -- 172.21.15.72:0/543268169 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52640af640 con 0x7f525c049070 2024-04-15T15:15:12.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.686+0000 7fe7bd0aa640 1 -- 172.21.15.72:0/2704731299 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe7b0048c40 msgr2=0x7fe7b0049090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:12.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.686+0000 7fe7bd0aa640 1 --2- 172.21.15.72:0/2704731299 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe7b0048c40 0x7fe7b0049090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.686+0000 7fe7bd0aa640 1 -- 172.21.15.72:0/2704731299 >> v1:172.21.15.175:6789/0 conn(0x7fe7b0009190 legacy=0x7fe7b0052f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:12.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.687+0000 7fe7bd0aa640 1 -- 172.21.15.72:0/2704731299 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7b0035c30 con 0x7fe7b003aa20 2024-04-15T15:15:12.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.687+0000 7fdc6e7fc640 1 -- 172.21.15.72:0/1118402973 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc700b4070 con 0x7fdc68038050 2024-04-15T15:15:12.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.687+0000 7fdc6e7fc640 1 -- 172.21.15.72:0/1118402973 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc700af640 con 0x7fdc68038050 2024-04-15T15:15:12.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.687+0000 7f7bf37fe640 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f7bf415b5d0 0x7f7bf406dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.72:34042/0 (socket says 172.21.15.72:34042) 2024-04-15T15:15:12.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.687+0000 7f23fffff640 1 Processor -- start 2024-04-15T15:15:12.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.687+0000 7f7bf37fe640 1 -- 172.21.15.72:0/1122853329 learned_addr learned my addr 172.21.15.72:0/1122853329 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:12.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.687+0000 7f23fffff640 1 -- start start 2024-04-15T15:15:12.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.686+0000 7f4ac8255640 1 -- start start 2024-04-15T15:15:12.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.687+0000 7f4ac8255640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4ac006e5e0 0x7f4ac00b3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.687+0000 7f4ac8255640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4ac006c970 0x7f4ac006cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.687+0000 7f4ac8255640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ac006d2c0 con 0x7f4ac0162d30 2024-04-15T15:15:12.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.687+0000 7f4ac8255640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f4ac00b3aa0 con 0x7f4ac006c970 2024-04-15T15:15:12.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.687+0000 7f4ac8255640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f4ac00b5480 con 0x7f4ac006e5e0 2024-04-15T15:15:12.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.687+0000 7f4ac67f2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4ac006c970 0x7f4ac006cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.687+0000 7f4ac67f2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4ac006c970 0x7f4ac006cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45330/0 (socket says 172.21.15.72:45330) 2024-04-15T15:15:12.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.687+0000 7f4ac67f2640 1 -- 172.21.15.72:0/85275783 learned_addr learned my addr 172.21.15.72:0/85275783 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:12.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.687+0000 7f5263fff640 1 -- 172.21.15.72:0/543268169 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f525c049070 msgr2=0x7f525c03b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:12.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.687+0000 7f5263fff640 1 --2- 172.21.15.72:0/543268169 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f525c049070 0x7f525c03b130 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f526406a870 tx=0x7f5264069760 comp rx=0 tx=0).stop 2024-04-15T15:15:12.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.687+0000 7f4ac57f0640 1 --2- 172.21.15.72:0/85275783 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4ac006e5e0 0x7f4ac00b3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.688+0000 7f5263fff640 1 -- 172.21.15.72:0/543268169 shutdown_connections 2024-04-15T15:15:12.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.688+0000 7f5263fff640 1 --2- 172.21.15.72:0/543268169 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f525c049070 0x7f525c03b130 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.688+0000 7f5263fff640 1 -- 172.21.15.72:0/543268169 >> 172.21.15.72:0/543268169 conn(0x7f525c01ef40 msgr2=0x7f525c055c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:12.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.688+0000 7fdc75c4a640 1 -- 172.21.15.72:0/1118402973 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdc68038050 msgr2=0x7fdc68057570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:12.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.687+0000 7f7bf2ffd640 1 --2- 172.21.15.72:0/1122853329 >> v2:172.21.15.175:3300/0 conn(0x7f7bf40d0490 0x7f7bf406ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.688+0000 7fdc75c4a640 1 --2- 172.21.15.72:0/1118402973 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdc68038050 0x7fdc68057570 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdc70050270 tx=0x7fdc70069760 comp rx=0 tx=0).stop 2024-04-15T15:15:12.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.688+0000 7f4ac67f2640 1 -- 172.21.15.72:0/85275783 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4ac006e5e0 msgr2=0x7f4ac00b3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:12.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.688+0000 7f4ac67f2640 1 --2- 172.21.15.72:0/85275783 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4ac006e5e0 0x7f4ac00b3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.686+0000 7fd7320ba640 1 Processor -- start 2024-04-15T15:15:12.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.686+0000 7fd7320ba640 1 -- start start 2024-04-15T15:15:12.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.688+0000 7fd7320ba640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd72c06d200 0x7fd72c06e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.688+0000 7fd7320ba640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd72c0ce710 con 0x7fd72c0cf4d0 2024-04-15T15:15:12.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.688+0000 7fd7320ba640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd72c0ce9f0 con 0x7fd72c0b65d0 2024-04-15T15:15:12.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.688+0000 7fd7320ba640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fd72c0cecd0 con 0x7fd72c06d200 2024-04-15T15:15:12.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.688+0000 7f5263fff640 1 -- 172.21.15.72:0/543268169 shutdown_connections 2024-04-15T15:15:12.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.688+0000 7f5263fff640 1 -- 172.21.15.72:0/543268169 wait complete. 2024-04-15T15:15:12.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.688+0000 7f7bf2ffd640 1 -- 172.21.15.72:0/1122853329 >> v2:172.21.15.72:3300/0 conn(0x7f7bf40d0920 msgr2=0x7f7bf406f370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-15T15:15:12.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.688+0000 7f7bf27fc640 1 -- 172.21.15.72:0/1122853329 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3982668301 0 0) 0x7f7bf415b9b0 con 0x7f7bf415b5d0 2024-04-15T15:15:12.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.688+0000 7f4ac67f2640 1 -- 172.21.15.72:0/85275783 >> v1:172.21.15.72:6789/0 conn(0x7f4ac0162d30 legacy=0x7f4ac00b4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:12.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.688+0000 7f4ac67f2640 1 -- 172.21.15.72:0/85275783 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ac00b3cd0 con 0x7f4ac006c970 2024-04-15T15:15:12.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.688+0000 7fdc75c4a640 1 -- 172.21.15.72:0/1118402973 shutdown_connections 2024-04-15T15:15:12.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.688+0000 7fdc75c4a640 1 --2- 172.21.15.72:0/1118402973 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdc68038050 0x7fdc68057570 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.688+0000 7fdc75c4a640 1 --2- 172.21.15.72:0/1118402973 >> v2:172.21.15.175:3300/0 conn(0x7fdc6803eaa0 0x7fdc680f3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.688+0000 7fdc75c4a640 1 -- 172.21.15.72:0/1118402973 >> 172.21.15.72:0/1118402973 conn(0x7fdc680559e0 msgr2=0x7fdc68055df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:12.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.688+0000 7fdc75c4a640 1 -- 172.21.15.72:0/1118402973 shutdown_connections 2024-04-15T15:15:12.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.688+0000 7f7bf3fff640 1 --2- 172.21.15.72:0/1122853329 >> v2:172.21.15.72:3300/0 conn(0x7f7bf40d0920 0x7f7bf406f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.688+0000 7f7bf2ffd640 1 --2- 172.21.15.72:0/1122853329 >> v2:172.21.15.72:3300/0 conn(0x7f7bf40d0920 0x7f7bf406f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.688+0000 7f7bf2ffd640 1 -- 172.21.15.72:0/1122853329 >> v1:172.21.15.175:6789/0 conn(0x7f7bf415b5d0 legacy=0x7f7bf406dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:12.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.688+0000 7fd72affd640 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fd72c0b65d0 0x7fd72c06d5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.72:34052/0 (socket says 172.21.15.72:34052) 2024-04-15T15:15:12.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.688+0000 7fd72affd640 1 -- 172.21.15.72:0/510524330 learned_addr learned my addr 172.21.15.72:0/510524330 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:12.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.688+0000 7f7bf2ffd640 1 -- 172.21.15.72:0/1122853329 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bf40cf3f0 con 0x7f7bf40d0490 2024-04-15T15:15:12.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.688+0000 7f7bf3fff640 1 --2- 172.21.15.72:0/1122853329 >> v2:172.21.15.72:3300/0 conn(0x7f7bf40d0920 0x7f7bf406f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:12.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.688+0000 7f23fffff640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f23f80e1090 0x7f23f8038630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.688+0000 7f23fffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f23f8050dd0 0x7f23f8039ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.688+0000 7f23fffff640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23f804f540 con 0x7f23f803a400 2024-04-15T15:15:12.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.688+0000 7f23fffff640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f23f804f820 con 0x7f23f8050dd0 2024-04-15T15:15:12.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.688+0000 7f23fffff640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f23f804fb00 con 0x7f23f80e1090 2024-04-15T15:15:12.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.689+0000 7f23feffd640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f23f80e1090 0x7f23f8038630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.689+0000 7f23feffd640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f23f80e1090 0x7f23f8038630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3301/0 says I am v2:172.21.15.72:56798/0 (socket says 172.21.15.72:56798) 2024-04-15T15:15:12.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.689+0000 7f23feffd640 1 -- 172.21.15.72:0/3396478632 learned_addr learned my addr 172.21.15.72:0/3396478632 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:12.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.688+0000 7fdc75c4a640 1 -- 172.21.15.72:0/1118402973 wait complete. 2024-04-15T15:15:12.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.689+0000 7fd72a7fc640 1 -- 172.21.15.72:0/510524330 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 526862231 0 0) 0x7fd72c0ce9f0 con 0x7fd72c0b65d0 2024-04-15T15:15:12.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.689+0000 7fd72a7fc640 1 -- 172.21.15.72:0/510524330 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd71c003520 con 0x7fd72c0b65d0 2024-04-15T15:15:12.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.689+0000 7fe7bd0aa640 1 --2- 172.21.15.72:0/2704731299 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe7b003aa20 0x7fe7b003ae30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe7b806a870 tx=0x7fe7b8069760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b71b713bf8e3385b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:12.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.689+0000 7f23feffd640 1 -- 172.21.15.72:0/3396478632 >> v2:172.21.15.72:3300/0 conn(0x7f23f8050dd0 msgr2=0x7f23f8039ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:12.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.689+0000 7f23feffd640 1 --2- 172.21.15.72:0/3396478632 >> v2:172.21.15.72:3300/0 conn(0x7f23f8050dd0 0x7f23f8039ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.689+0000 7f23feffd640 1 -- 172.21.15.72:0/3396478632 >> v1:172.21.15.175:6789/0 conn(0x7f23f803a400 legacy=0x7f23f80e4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:12.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.689+0000 7f23feffd640 1 -- 172.21.15.72:0/3396478632 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23f804fde0 con 0x7f23f80e1090 2024-04-15T15:15:12.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.689+0000 7fd72b7fe640 1 --2- 172.21.15.72:0/510524330 >> v2:172.21.15.72:3300/0 conn(0x7fd72c06d200 0x7fd72c06e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.689+0000 7fe7b77fe640 1 -- 172.21.15.72:0/2704731299 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b80b4070 con 0x7fe7b003aa20 2024-04-15T15:15:12.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.689+0000 7fe7b77fe640 1 -- 172.21.15.72:0/2704731299 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7b80a7dc0 con 0x7fe7b003aa20 2024-04-15T15:15:12.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.689+0000 7fe7b77fe640 1 -- 172.21.15.72:0/2704731299 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b80af640 con 0x7fe7b003aa20 2024-04-15T15:15:12.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.689+0000 7fd72a7fc640 1 -- 172.21.15.72:0/510524330 <== mon.1 v1:172.21.15.175:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2367274722 0 0) 0x7fd71c003520 con 0x7fd72c0b65d0 2024-04-15T15:15:12.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.689+0000 7fd72a7fc640 1 -- 172.21.15.72:0/510524330 >> v2:172.21.15.72:3300/0 conn(0x7fd72c06d200 msgr2=0x7fd72c06e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:12.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.689+0000 7fd72a7fc640 1 --2- 172.21.15.72:0/510524330 >> v2:172.21.15.72:3300/0 conn(0x7fd72c06d200 0x7fd72c06e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.689+0000 7fd72a7fc640 1 -- 172.21.15.72:0/510524330 >> v1:172.21.15.72:6789/0 conn(0x7fd72c0cf4d0 legacy=0x7fd72c0cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:12.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.689+0000 7fd72a7fc640 1 -- 172.21.15.72:0/510524330 --> v1:172.21.15.175:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd72c0cefb0 con 0x7fd72c0b65d0 2024-04-15T15:15:12.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.690+0000 7fd72a7fc640 1 -- 172.21.15.72:0/510524330 <== mon.1 v1:172.21.15.175:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7fd718001650 con 0x7fd72c0b65d0 2024-04-15T15:15:12.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.690+0000 7fd72a7fc640 1 -- 172.21.15.72:0/510524330 >> v1:172.21.15.175:6789/0 conn(0x7fd72c0b65d0 legacy=0x7fd72c06d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:12.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.690+0000 7fdc75c4a640 1 Processor -- start 2024-04-15T15:15:12.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.690+0000 7fd72a7fc640 1 --2- 172.21.15.72:0/510524330 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd71c0039d0 0x7fd71c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.690+0000 7fd72a7fc640 1 --2- 172.21.15.72:0/510524330 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd71c004800 0x7fd71c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.690+0000 7fd72a7fc640 1 --2- 172.21.15.72:0/510524330 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd71c005210 0x7fd71c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.690+0000 7fd72a7fc640 1 -- 172.21.15.72:0/510524330 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c0cecd0 con 0x7fd71c005210 2024-04-15T15:15:12.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.690+0000 7fd72a7fc640 1 -- 172.21.15.72:0/510524330 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fd71c00c1f0 con 0x7fd71c0039d0 2024-04-15T15:15:12.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.690+0000 7fd72a7fc640 1 -- 172.21.15.72:0/510524330 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fd71c00c3f0 con 0x7fd71c004800 2024-04-15T15:15:12.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.690+0000 7fd72bfff640 1 --2- 172.21.15.72:0/510524330 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd71c005210 0x7fd71c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.690+0000 7f5263fff640 1 Processor -- start 2024-04-15T15:15:12.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.690+0000 7f4ac67f2640 1 --2- 172.21.15.72:0/85275783 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ac006c970 0x7f4ac006cd80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8009e50 tx=0x7f4ab802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54279060125238ab server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:12.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.690+0000 7f7bf2ffd640 1 --2- 172.21.15.72:0/1122853329 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7bf40d0490 0x7f7bf406ee30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7bf40b3f30 tx=0x7f7be402e0e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23acb4b13063760e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:12.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.690+0000 7f5263fff640 1 -- start start 2024-04-15T15:15:12.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.690+0000 7f7bf27fc640 1 -- 172.21.15.72:0/1122853329 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be403b070 con 0x7f7bf40d0490 2024-04-15T15:15:12.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.690+0000 7f23feffd640 1 --2- 172.21.15.72:0/3396478632 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f23f80e1090 0x7f23f8038630 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f23f4009870 tx=0x7f23f402b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ede07cdaa2e23de server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:12.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.690+0000 7f7bf27fc640 1 -- 172.21.15.72:0/1122853329 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7be4036440 con 0x7f7bf40d0490 2024-04-15T15:15:12.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.690+0000 7f7bf27fc640 1 -- 172.21.15.72:0/1122853329 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7be4036810 con 0x7f7bf40d0490 2024-04-15T15:15:12.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.690+0000 7fdc75c4a640 1 -- start start 2024-04-15T15:15:12.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.690+0000 7fd72bfff640 1 -- 172.21.15.72:0/510524330 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd71c004800 msgr2=0x7fd71c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:12.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.690+0000 7fd72bfff640 1 --2- 172.21.15.72:0/510524330 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd71c004800 0x7fd71c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.690+0000 7fd72bfff640 1 -- 172.21.15.72:0/510524330 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd71c0039d0 msgr2=0x7fd71c003e10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-15T15:15:12.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.690+0000 7fd72bfff640 1 --2- 172.21.15.72:0/510524330 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd71c0039d0 0x7fd71c003e10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.690+0000 7fd72bfff640 1 -- 172.21.15.72:0/510524330 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd72c0ce710 con 0x7fd71c005210 2024-04-15T15:15:12.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.691+0000 7fe7bd8ab640 1 -- 172.21.15.72:0/2704731299 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe7b003aa20 msgr2=0x7fe7b003ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:12.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.691+0000 7fe7bd8ab640 1 --2- 172.21.15.72:0/2704731299 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe7b003aa20 0x7fe7b003ae30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe7b806a870 tx=0x7fe7b8069760 comp rx=0 tx=0).stop 2024-04-15T15:15:12.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.691+0000 7f4ac4fef640 1 -- 172.21.15.72:0/85275783 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab802cb00 con 0x7f4ac006c970 2024-04-15T15:15:12.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.691+0000 7f4ac4fef640 1 -- 172.21.15.72:0/85275783 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ab802cca0 con 0x7f4ac006c970 2024-04-15T15:15:12.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.691+0000 7f4ac4fef640 1 -- 172.21.15.72:0/85275783 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab8034610 con 0x7f4ac006c970 2024-04-15T15:15:12.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.691+0000 7f23fdffb640 1 -- 172.21.15.72:0/3396478632 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f4035020 con 0x7f23f80e1090 2024-04-15T15:15:12.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.691+0000 7f23fdffb640 1 -- 172.21.15.72:0/3396478632 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23f4031870 con 0x7f23f80e1090 2024-04-15T15:15:12.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.691+0000 7f23fdffb640 1 -- 172.21.15.72:0/3396478632 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f4031c40 con 0x7f23f80e1090 2024-04-15T15:15:12.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.691+0000 7fdc75c4a640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdc68037b00 0x7fdc680edd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.691+0000 7fdc75c4a640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc68038050 0x7fdc6803a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.691+0000 7fdc75c4a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc6803eaa0 0x7fdc6800d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.691+0000 7fdc75c4a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fdc68038790 con 0x7fdc6803eaa0 2024-04-15T15:15:12.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.691+0000 7fdc75c4a640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fdc68038610 con 0x7fdc68038050 2024-04-15T15:15:12.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.691+0000 7fdc75c4a640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fdc68057ab0 con 0x7fdc68037b00 2024-04-15T15:15:12.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.691+0000 7fd72bfff640 1 -- 172.21.15.72:0/510524330 reap_dead start 2024-04-15T15:15:12.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.691+0000 7f5263fff640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f525c009190 0x7f525c0e7d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.691+0000 7f7bfa1bd640 1 -- 172.21.15.72:0/1122853329 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7bf40d0490 msgr2=0x7f7bf406ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:12.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.691+0000 7f7bfa1bd640 1 --2- 172.21.15.72:0/1122853329 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7bf40d0490 0x7f7bf406ee30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7bf40b3f30 tx=0x7f7be402e0e0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7fdc6effd640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc68038050 0x7fdc6803a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7f7bfa1bd640 1 -- 172.21.15.72:0/1122853329 shutdown_connections 2024-04-15T15:15:12.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7f7bfa1bd640 1 --2- 172.21.15.72:0/1122853329 >> v2:172.21.15.72:3300/0 conn(0x7f7bf40d0920 0x7f7bf406f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7f7bfa1bd640 1 --2- 172.21.15.72:0/1122853329 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7bf40d0490 0x7f7bf406ee30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7f7bfa1bd640 1 -- 172.21.15.72:0/1122853329 >> 172.21.15.72:0/1122853329 conn(0x7f7bf4072c80 msgr2=0x7f7bf4073090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:12.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7fd72bfff640 1 --2- 172.21.15.72:0/510524330 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd71c005210 0x7fd71c00b9c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd7240044f0 tx=0x7fd71c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3967ccacb5f6d9e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:12.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7fdc6ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc6803eaa0 0x7fdc6800d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7fdc6f7fe640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdc68037b00 0x7fdc680edd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7fdc6effd640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc68038050 0x7fdc6803a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.72:52588/0 (socket says 172.21.15.72:52588) 2024-04-15T15:15:12.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7fdc6ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc6803eaa0 0x7fdc6800d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45376/0 (socket says 172.21.15.72:45376) 2024-04-15T15:15:12.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7f5263fff640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f525c048c40 0x7f525c033cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7f5262ffd640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f525c009190 0x7f525c0e7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7f5262ffd640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f525c009190 0x7f525c0e7d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.72:52604/0 (socket says 172.21.15.72:52604) 2024-04-15T15:15:12.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7f4ac8255640 1 -- 172.21.15.72:0/85275783 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ac006c970 msgr2=0x7f4ac006cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:12.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7f4ac8255640 1 --2- 172.21.15.72:0/85275783 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ac006c970 0x7f4ac006cd80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8009e50 tx=0x7f4ab802bf60 comp rx=0 tx=0).stop 2024-04-15T15:15:12.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7f23fffff640 1 -- 172.21.15.72:0/3396478632 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f23f80e1090 msgr2=0x7f23f8038630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:12.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7f23fffff640 1 --2- 172.21.15.72:0/3396478632 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f23f80e1090 0x7f23f8038630 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f23f4009870 tx=0x7f23f402b910 comp rx=0 tx=0).stop 2024-04-15T15:15:12.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7f5262ffd640 1 -- 172.21.15.72:0/113132821 learned_addr learned my addr 172.21.15.72:0/113132821 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:12.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7f7bfa1bd640 1 -- 172.21.15.72:0/1122853329 shutdown_connections 2024-04-15T15:15:12.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7f23fffff640 1 -- 172.21.15.72:0/3396478632 shutdown_connections 2024-04-15T15:15:12.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7f23fffff640 1 --2- 172.21.15.72:0/3396478632 >> v2:172.21.15.72:3300/0 conn(0x7f23f8050dd0 0x7f23f8039ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7f23fffff640 1 --2- 172.21.15.72:0/3396478632 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f23f80e1090 0x7f23f8038630 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7f23fffff640 1 -- 172.21.15.72:0/3396478632 >> 172.21.15.72:0/3396478632 conn(0x7f23f8009fb0 msgr2=0x7f23f800a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:12.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7f4ac8255640 1 -- 172.21.15.72:0/85275783 shutdown_connections 2024-04-15T15:15:12.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7f4ac8255640 1 --2- 172.21.15.72:0/85275783 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ac006c970 0x7f4ac006cd80 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7f4ac8255640 1 --2- 172.21.15.72:0/85275783 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4ac006e5e0 0x7f4ac00b3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7f4ac8255640 1 -- 172.21.15.72:0/85275783 >> 172.21.15.72:0/85275783 conn(0x7f4ac0071f80 msgr2=0x7f4ac0072390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:12.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7fe7bd8ab640 1 -- 172.21.15.72:0/2704731299 shutdown_connections 2024-04-15T15:15:12.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7fe7bd8ab640 1 --2- 172.21.15.72:0/2704731299 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe7b003aa20 0x7fe7b003ae30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7fd72a7fc640 1 -- 172.21.15.72:0/510524330 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd724034070 con 0x7fd71c005210 2024-04-15T15:15:12.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7fd72a7fc640 1 -- 172.21.15.72:0/510524330 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd724030610 con 0x7fd71c005210 2024-04-15T15:15:12.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7fe7bd8ab640 1 --2- 172.21.15.72:0/2704731299 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe7b0048c40 0x7fe7b0049090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7fe7bd8ab640 1 -- 172.21.15.72:0/2704731299 >> 172.21.15.72:0/2704731299 conn(0x7fe7b001ef40 msgr2=0x7fe7b0055c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:12.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7f23fffff640 1 -- 172.21.15.72:0/3396478632 shutdown_connections 2024-04-15T15:15:12.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7f52627fc640 1 --2- 172.21.15.72:0/113132821 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f525c048c40 0x7f525c033cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7fe7bd8ab640 1 -- 172.21.15.72:0/2704731299 shutdown_connections 2024-04-15T15:15:12.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7f5263fff640 1 --2- 172.21.15.72:0/113132821 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f525c049070 0x7f525c008040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.693+0000 7f52637fe640 1 --2- 172.21.15.72:0/113132821 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f525c049070 0x7f525c008040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.693+0000 7f5263fff640 1 -- 172.21.15.72:0/113132821 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f525c0367b0 con 0x7f525c049070 2024-04-15T15:15:12.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.693+0000 7f5263fff640 1 -- 172.21.15.72:0/113132821 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f525c0364d0 con 0x7f525c009190 2024-04-15T15:15:12.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.693+0000 7f5263fff640 1 -- 172.21.15.72:0/113132821 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f525c0540a0 con 0x7f525c048c40 2024-04-15T15:15:12.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7fdc6effd640 1 -- 172.21.15.72:0/480856207 learned_addr learned my addr 172.21.15.72:0/480856207 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:12.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7fdc6ffff640 1 -- 172.21.15.72:0/480856207 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdc68037b00 msgr2=0x7fdc680edd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:12.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7fdc6ffff640 1 --2- 172.21.15.72:0/480856207 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdc68037b00 0x7fdc680edd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7fdc6ffff640 1 -- 172.21.15.72:0/480856207 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc68038050 msgr2=0x7fdc6803a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:12.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7fdc6ffff640 1 --2- 172.21.15.72:0/480856207 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc68038050 0x7fdc6803a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7fdc6ffff640 1 -- 172.21.15.72:0/480856207 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc6800da70 con 0x7fdc6803eaa0 2024-04-15T15:15:12.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.693+0000 7f4ac8255640 1 -- 172.21.15.72:0/85275783 shutdown_connections 2024-04-15T15:15:12.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7fe7bd8ab640 1 -- 172.21.15.72:0/2704731299 wait complete. 2024-04-15T15:15:12.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.693+0000 7fd72a7fc640 1 -- 172.21.15.72:0/510524330 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7240040f0 con 0x7fd71c005210 2024-04-15T15:15:12.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.693+0000 7fd7320ba640 1 -- 172.21.15.72:0/510524330 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd71c005210 msgr2=0x7fd71c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:12.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.693+0000 7fd7320ba640 1 --2- 172.21.15.72:0/510524330 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd71c005210 0x7fd71c00b9c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd7240044f0 tx=0x7fd71c005830 comp rx=0 tx=0).stop 2024-04-15T15:15:12.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.693+0000 7f52627fc640 1 -- 172.21.15.72:0/113132821 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f525c009190 msgr2=0x7f525c0e7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:12.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.693+0000 7f52627fc640 1 --2- 172.21.15.72:0/113132821 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f525c009190 0x7f525c0e7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.693+0000 7f52627fc640 1 -- 172.21.15.72:0/113132821 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f525c049070 msgr2=0x7f525c008040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:12.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.693+0000 7f52627fc640 1 --2- 172.21.15.72:0/113132821 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f525c049070 0x7f525c008040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.693+0000 7f52627fc640 1 -- 172.21.15.72:0/113132821 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f525c0e8340 con 0x7f525c048c40 2024-04-15T15:15:12.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.693+0000 7fd7320ba640 1 -- 172.21.15.72:0/510524330 shutdown_connections 2024-04-15T15:15:12.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.693+0000 7fd7320ba640 1 --2- 172.21.15.72:0/510524330 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd71c005210 0x7fd71c00b9c0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.693+0000 7fd7320ba640 1 -- 172.21.15.72:0/510524330 >> 172.21.15.72:0/510524330 conn(0x7fd72c072130 msgr2=0x7fd72c072540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:12.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.692+0000 7f7bfa1bd640 1 -- 172.21.15.72:0/1122853329 wait complete. 2024-04-15T15:15:12.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.693+0000 7f52627fc640 1 --2- 172.21.15.72:0/113132821 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f525c048c40 0x7f525c033cd0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f525800c430 tx=0x7f525800c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:12.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.693+0000 7f4ac8255640 1 -- 172.21.15.72:0/85275783 wait complete. 2024-04-15T15:15:12.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.693+0000 7f23fffff640 1 -- 172.21.15.72:0/3396478632 wait complete. 2024-04-15T15:15:12.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.694+0000 7fd7320ba640 1 -- 172.21.15.72:0/510524330 shutdown_connections 2024-04-15T15:15:12.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.694+0000 7fdc6ffff640 1 --2- 172.21.15.72:0/480856207 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc6803eaa0 0x7fdc6800d530 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdc70050b00 tx=0x7fdc7006b770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:12.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.694+0000 7fd7320ba640 1 -- 172.21.15.72:0/510524330 wait complete. 2024-04-15T15:15:12.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.694+0000 7f524b7fe640 1 -- 172.21.15.72:0/113132821 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5258017070 con 0x7f525c048c40 2024-04-15T15:15:12.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.694+0000 7f524b7fe640 1 -- 172.21.15.72:0/113132821 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5258005ce0 con 0x7f525c048c40 2024-04-15T15:15:12.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.694+0000 7fe7bd8ab640 1 Processor -- start 2024-04-15T15:15:12.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.694+0000 7f524b7fe640 1 -- 172.21.15.72:0/113132821 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52580072d0 con 0x7f525c048c40 2024-04-15T15:15:12.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.694+0000 7f5263fff640 1 -- 172.21.15.72:0/113132821 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f525c0e8620 con 0x7f525c048c40 2024-04-15T15:15:12.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.695+0000 7f5263fff640 1 -- 172.21.15.72:0/113132821 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f525c0e8be0 con 0x7f525c048c40 2024-04-15T15:15:12.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.694+0000 7fe7bd8ab640 1 -- start start 2024-04-15T15:15:12.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.695+0000 7f4ac8255640 1 Processor -- start 2024-04-15T15:15:12.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.695+0000 7f23fffff640 1 Processor -- start 2024-04-15T15:15:12.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.695+0000 7f23fffff640 1 -- start start 2024-04-15T15:15:12.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.696+0000 7fe7bd8ab640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe7b0009190 0x7fe7b00502a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.696+0000 7fe7bd8ab640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe7b003aa20 0x7fe7b0006ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.696+0000 7fe7bd8ab640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7b0048c40 0x7fe7b0006ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.694+0000 7f7bfa1bd640 1 Processor -- start 2024-04-15T15:15:12.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.694+0000 7f7bfa1bd640 1 -- start start 2024-04-15T15:15:12.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.695+0000 7f7bfa1bd640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7bf40d0490 0x7f7bf4146d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.695+0000 7f7bfa1bd640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7bf40d0920 0x7f7bf406dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.695+0000 7f7bfa1bd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7bf415b5d0 0x7f7bf414e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.696+0000 7f7bf3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7bf415b5d0 0x7f7bf414e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.696+0000 7f7bf3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7bf415b5d0 0x7f7bf414e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45396/0 (socket says 172.21.15.72:45396) 2024-04-15T15:15:12.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.696+0000 7f7bf37fe640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7bf40d0490 0x7f7bf4146d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.696+0000 7f7bf3fff640 1 -- 172.21.15.72:0/4135194527 learned_addr learned my addr 172.21.15.72:0/4135194527 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:12.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.696+0000 7f7bfa1bd640 1 -- 172.21.15.72:0/4135194527 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf40cfc90 con 0x7f7bf415b5d0 2024-04-15T15:15:12.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.696+0000 7f7bfa1bd640 1 -- 172.21.15.72:0/4135194527 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f7bf40cf9b0 con 0x7f7bf40d0920 2024-04-15T15:15:12.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.696+0000 7f7bfa1bd640 1 -- 172.21.15.72:0/4135194527 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f7bf40b0c20 con 0x7f7bf40d0490 2024-04-15T15:15:12.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.695+0000 7fdc5bfff640 1 -- 172.21.15.72:0/480856207 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc700b4050 con 0x7fdc6803eaa0 2024-04-15T15:15:12.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.695+0000 7fdc5bfff640 1 -- 172.21.15.72:0/480856207 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc700b5040 con 0x7fdc6803eaa0 2024-04-15T15:15:12.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.695+0000 7fdc5bfff640 1 -- 172.21.15.72:0/480856207 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc70067bf0 con 0x7fdc6803eaa0 2024-04-15T15:15:12.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.695+0000 7fdc75c4a640 1 -- 172.21.15.72:0/480856207 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc6804d660 con 0x7fdc6803eaa0 2024-04-15T15:15:12.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.695+0000 7fdc75c4a640 1 -- 172.21.15.72:0/480856207 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc6804dc20 con 0x7fdc6803eaa0 2024-04-15T15:15:12.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.695+0000 7fd7320ba640 1 Processor -- start 2024-04-15T15:15:12.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.696+0000 7fd7320ba640 1 -- start start 2024-04-15T15:15:12.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.696+0000 7f7bf2ffd640 1 --2- 172.21.15.72:0/4135194527 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7bf40d0920 0x7f7bf406dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.696+0000 7fe7bd0aa640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7b0048c40 0x7fe7b0006ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7f7bf37fe640 1 -- 172.21.15.72:0/4135194527 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7bf40d0920 msgr2=0x7f7bf406dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:12.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7f7bf37fe640 1 --2- 172.21.15.72:0/4135194527 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7bf40d0920 0x7f7bf406dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7f7bf37fe640 1 -- 172.21.15.72:0/4135194527 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7bf415b5d0 msgr2=0x7f7bf414e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:12.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7fdc75c4a640 1 -- 172.21.15.72:0/480856207 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc6804fc60 con 0x7fdc6803eaa0 2024-04-15T15:15:12.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7fd7320ba640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd71c005210 0x7fd72c1377f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.696+0000 7fe7bd0aa640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7b0048c40 0x7fe7b0006ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45410/0 (socket says 172.21.15.72:45410) 2024-04-15T15:15:12.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.696+0000 7fe7b7fff640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe7b003aa20 0x7fe7b0006ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7fe7bd8ab640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe7b00364d0 con 0x7fe7b0048c40 2024-04-15T15:15:12.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7fe7bd0aa640 1 -- 172.21.15.72:0/259147821 learned_addr learned my addr 172.21.15.72:0/259147821 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:12.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7fe7bd8ab640 1 -- 172.21.15.72:0/259147821 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fe7b00361f0 con 0x7fe7b003aa20 2024-04-15T15:15:12.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7fe7bd8ab640 1 -- 172.21.15.72:0/259147821 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fe7b00367b0 con 0x7fe7b0009190 2024-04-15T15:15:12.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7fe7bd0aa640 1 -- 172.21.15.72:0/259147821 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe7b0009190 msgr2=0x7fe7b00502a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-15T15:15:12.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7fe7bd0aa640 1 --2- 172.21.15.72:0/259147821 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe7b0009190 0x7fe7b00502a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7fe7bd0aa640 1 -- 172.21.15.72:0/259147821 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe7b003aa20 msgr2=0x7fe7b0006ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:12.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7fe7bd0aa640 1 --2- 172.21.15.72:0/259147821 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe7b003aa20 0x7fe7b0006ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7fe7bd0aa640 1 -- 172.21.15.72:0/259147821 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7b00507e0 con 0x7fe7b0048c40 2024-04-15T15:15:12.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7fe7bd0aa640 1 --2- 172.21.15.72:0/259147821 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7b0048c40 0x7fe7b0006ff0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe7b806a0f0 tx=0x7fe7b806b770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:12.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7fd7320ba640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd72c147a30 0x7fd72c147eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.698+0000 7fe7b7fff640 1 --2- 172.21.15.72:0/259147821 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe7b003aa20 0x7fe7b0006ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:12.735 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7fd7320ba640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd72c141e90 0x7fd72c142310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7fd7320ba640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c162e40 con 0x7fd72c147a30 2024-04-15T15:15:12.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7f7bf37fe640 1 --2- 172.21.15.72:0/4135194527 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7bf415b5d0 0x7f7bf414e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7f7bf37fe640 1 -- 172.21.15.72:0/4135194527 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bf406c780 con 0x7f7bf40d0490 2024-04-15T15:15:12.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7f7bf37fe640 1 --2- 172.21.15.72:0/4135194527 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7bf40d0490 0x7f7bf4146d20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7bec00c410 tx=0x7f7bec00c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:12.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7f7bebfff640 1 -- 172.21.15.72:0/4135194527 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bec0074e0 con 0x7f7bf40d0490 2024-04-15T15:15:12.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7f7bebfff640 1 -- 172.21.15.72:0/4135194527 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bec0047e0 con 0x7f7bf40d0490 2024-04-15T15:15:12.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7f7bebfff640 1 -- 172.21.15.72:0/4135194527 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bec004bb0 con 0x7f7bf40d0490 2024-04-15T15:15:12.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7f7bfa1bd640 1 -- 172.21.15.72:0/4135194527 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bf406ca20 con 0x7f7bf40d0490 2024-04-15T15:15:12.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7f7bfa1bd640 1 -- 172.21.15.72:0/4135194527 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bf406cfe0 con 0x7f7bf40d0490 2024-04-15T15:15:12.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7fd72b7fe640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd71c005210 0x7fd72c1377f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.698+0000 7f23fffff640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f23f803a400 0x7f23f80e2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.698+0000 7f23fffff640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f23f8050dd0 0x7f23f80e1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.698+0000 7f23fffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23f80e1090 0x7f23f80db7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.698+0000 7f23fffff640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f23f803a870 con 0x7f23f80e1090 2024-04-15T15:15:12.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.698+0000 7f23fffff640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f23f80503a0 con 0x7f23f803a400 2024-04-15T15:15:12.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.698+0000 7f23fffff640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f23f8006fa0 con 0x7f23f8050dd0 2024-04-15T15:15:12.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.698+0000 7f524b7fe640 1 -- 172.21.15.72:0/113132821 <== mon.2 v2:172.21.15.175:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f525801a070 con 0x7f525c048c40 2024-04-15T15:15:12.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.698+0000 7fe7b57fa640 1 -- 172.21.15.72:0/259147821 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b80bd020 con 0x7fe7b0048c40 2024-04-15T15:15:12.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.698+0000 7fe7b57fa640 1 -- 172.21.15.72:0/259147821 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7b80b5040 con 0x7fe7b0048c40 2024-04-15T15:15:12.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.698+0000 7f23feffd640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f23f803a400 0x7f23f80e2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.698+0000 7f23feffd640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f23f803a400 0x7f23f80e2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.72:52640/0 (socket says 172.21.15.72:52640) 2024-04-15T15:15:12.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.698+0000 7f23feffd640 1 -- 172.21.15.72:0/1362727111 learned_addr learned my addr 172.21.15.72:0/1362727111 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:12.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.698+0000 7fe7b57fa640 1 -- 172.21.15.72:0/259147821 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7b80b4070 con 0x7fe7b0048c40 2024-04-15T15:15:12.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.698+0000 7fe7bd8ab640 1 -- 172.21.15.72:0/259147821 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7b0050ab0 con 0x7fe7b0048c40 2024-04-15T15:15:12.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.699+0000 7fe7bd8ab640 1 -- 172.21.15.72:0/259147821 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7b0051070 con 0x7fe7b0048c40 2024-04-15T15:15:12.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7fd72b7fe640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd71c005210 0x7fd72c1377f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3301/0 says I am v2:172.21.15.72:56842/0 (socket says 172.21.15.72:56842) 2024-04-15T15:15:12.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7fd72b7fe640 1 -- 172.21.15.72:0/744839292 learned_addr learned my addr 172.21.15.72:0/744839292 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:12.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7fd7320ba640 1 -- 172.21.15.72:0/744839292 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fd72c162cc0 con 0x7fd72c141e90 2024-04-15T15:15:12.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7fd7320ba640 1 -- 172.21.15.72:0/744839292 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fd72c06e200 con 0x7fd71c005210 2024-04-15T15:15:12.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7fd72bfff640 1 --2- 172.21.15.72:0/744839292 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd72c141e90 0x7fd72c142310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7fd72affd640 1 --2- 172.21.15.72:0/744839292 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd72c147a30 0x7fd72c147eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7fd72b7fe640 1 -- 172.21.15.72:0/744839292 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd72c141e90 msgr2=0x7fd72c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:12.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7fd72b7fe640 1 --2- 172.21.15.72:0/744839292 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd72c141e90 0x7fd72c142310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.697+0000 7fd72b7fe640 1 -- 172.21.15.72:0/744839292 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd72c147a30 msgr2=0x7fd72c147eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:12.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.698+0000 7fd72b7fe640 1 --2- 172.21.15.72:0/744839292 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd72c147a30 0x7fd72c147eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.698+0000 7fd72b7fe640 1 -- 172.21.15.72:0/744839292 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd72c148560 con 0x7fd71c005210 2024-04-15T15:15:12.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.698+0000 7fd72b7fe640 1 --2- 172.21.15.72:0/744839292 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd71c005210 0x7fd72c1377f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd72000d860 tx=0x7fd72000dd20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:12.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.699+0000 7fd72affd640 1 --2- 172.21.15.72:0/744839292 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd72c147a30 0x7fd72c147eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:12.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.699+0000 7f23feffd640 1 -- 172.21.15.72:0/1362727111 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f23f8050dd0 msgr2=0x7f23f80e1e80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-15T15:15:12.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.699+0000 7f23feffd640 1 --2- 172.21.15.72:0/1362727111 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f23f8050dd0 0x7f23f80e1e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.699+0000 7f23feffd640 1 -- 172.21.15.72:0/1362727111 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23f80e1090 msgr2=0x7f23f80db7d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-15T15:15:12.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.699+0000 7f23feffd640 1 --2- 172.21.15.72:0/1362727111 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23f80e1090 0x7f23f80db7d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.699+0000 7f23feffd640 1 -- 172.21.15.72:0/1362727111 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23f80d7600 con 0x7f23f803a400 2024-04-15T15:15:12.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.699+0000 7f7bfa1bd640 1 -- 172.21.15.72:0/4135194527 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bd8005290 con 0x7f7bf40d0490 2024-04-15T15:15:12.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.699+0000 7f52497fa640 1 -- 172.21.15.72:0/113132821 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f525c0057e0 con 0x7f525c048c40 2024-04-15T15:15:12.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.698+0000 7fdc5bfff640 1 -- 172.21.15.72:0/480856207 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdc700c2030 con 0x7fdc6803eaa0 2024-04-15T15:15:12.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.699+0000 7fdc5bfff640 1 --2- 172.21.15.72:0/480856207 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fdc5406c9c0 0x7fdc5406ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.699+0000 7fdc5bfff640 1 -- 172.21.15.72:0/480856207 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fdc70128f40 con 0x7fdc6803eaa0 2024-04-15T15:15:12.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.699+0000 7fdc6f7fe640 1 --2- 172.21.15.72:0/480856207 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fdc5406c9c0 0x7fdc5406ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.700+0000 7f524b7fe640 1 --2- 172.21.15.72:0/113132821 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f525006c9c0 0x7f525006ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.700+0000 7fd717fff640 1 -- 172.21.15.72:0/744839292 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd720018020 con 0x7fd71c005210 2024-04-15T15:15:12.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.700+0000 7fd717fff640 1 -- 172.21.15.72:0/744839292 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd720009660 con 0x7fd71c005210 2024-04-15T15:15:12.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.700+0000 7fd717fff640 1 -- 172.21.15.72:0/744839292 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd720009a50 con 0x7fd71c005210 2024-04-15T15:15:12.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.700+0000 7f524b7fe640 1 -- 172.21.15.72:0/113132821 <== mon.2 v2:172.21.15.175:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f525808b1b0 con 0x7f525c048c40 2024-04-15T15:15:12.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.700+0000 7fe7bd8ab640 1 -- 172.21.15.72:0/259147821 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7b00057e0 con 0x7fe7b0048c40 2024-04-15T15:15:12.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.700+0000 7f5262ffd640 1 --2- 172.21.15.72:0/113132821 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f525006c9c0 0x7f525006ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.699+0000 7f4ac8255640 1 -- start start 2024-04-15T15:15:12.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.700+0000 7f4ac8255640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4ac006c970 0x7f4ac00aa370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.700+0000 7f7bebfff640 1 -- 172.21.15.72:0/4135194527 <== mon.2 v2:172.21.15.175:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7bec011070 con 0x7f7bf40d0490 2024-04-15T15:15:12.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.700+0000 7f4ac8255640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ac006e5e0 0x7f4ac0150fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.701+0000 7f4ac5ff1640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4ac006c970 0x7f4ac00aa370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.701+0000 7f4ac5ff1640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4ac006c970 0x7f4ac00aa370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.72:52648/0 (socket says 172.21.15.72:52648) 2024-04-15T15:15:12.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.701+0000 7f4ac57f0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ac006e5e0 0x7f4ac0150fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.701+0000 7f4ac57f0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ac006e5e0 0x7f4ac0150fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45438/0 (socket says 172.21.15.72:45438) 2024-04-15T15:15:12.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.701+0000 7f4ac8255640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4ac0162d30 0x7f4ac0152dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.701+0000 7f7bebfff640 1 --2- 172.21.15.72:0/4135194527 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7bd406c9c0 0x7f7bd406ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.701+0000 7f5262ffd640 1 --2- 172.21.15.72:0/113132821 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f525006c9c0 0x7f525006ee80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5254000f60 tx=0x7f5254009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:12.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.701+0000 7f7bebfff640 1 -- 172.21.15.72:0/4135194527 <== mon.2 v2:172.21.15.175:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f7bec08b850 con 0x7f7bf40d0490 2024-04-15T15:15:12.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.701+0000 7f7bf2ffd640 1 --2- 172.21.15.72:0/4135194527 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7bd406c9c0 0x7f7bd406ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.701+0000 7f4ac8255640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac00b4570 con 0x7f4ac006e5e0 2024-04-15T15:15:12.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.701+0000 7f4ac8255640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac00b4290 con 0x7f4ac006c970 2024-04-15T15:15:12.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.701+0000 7f4ac8255640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f4ac00b4850 con 0x7f4ac0162d30 2024-04-15T15:15:12.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.701+0000 7f4ac57f0640 1 -- 172.21.15.72:0/1478973791 learned_addr learned my addr 172.21.15.72:0/1478973791 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:12.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.701+0000 7fe7b57fa640 1 -- 172.21.15.72:0/259147821 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe7b80c0020 con 0x7fe7b0048c40 2024-04-15T15:15:12.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.701+0000 7f4ac67f2640 1 --2- 172.21.15.72:0/1478973791 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4ac0162d30 0x7f4ac0152dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.701+0000 7fd7320ba640 1 -- 172.21.15.72:0/744839292 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd72c148840 con 0x7fd71c005210 2024-04-15T15:15:12.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.701+0000 7fdc6f7fe640 1 --2- 172.21.15.72:0/480856207 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fdc5406c9c0 0x7fdc5406ee80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c009af0 tx=0x7fdc5c009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:12.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.701+0000 7fd7320ba640 1 -- 172.21.15.72:0/744839292 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd72c148e00 con 0x7fd71c005210 2024-04-15T15:15:12.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.702+0000 7f7bf2ffd640 1 --2- 172.21.15.72:0/4135194527 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7bd406c9c0 0x7f7bd406ee80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7be4038000 tx=0x7f7be4032000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:12.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.702+0000 7fe7b57fa640 1 --2- 172.21.15.72:0/259147821 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe79c06c8f0 0x7fe79c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.700+0000 7f23feffd640 1 --2- 172.21.15.72:0/1362727111 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f23f803a400 0x7f23f80e2e00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f23f40090b0 tx=0x7f23f40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:12.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.701+0000 7f23e7fff640 1 -- 172.21.15.72:0/1362727111 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f40443f0 con 0x7f23f803a400 2024-04-15T15:15:12.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.701+0000 7f23e7fff640 1 -- 172.21.15.72:0/1362727111 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23f4044590 con 0x7f23f803a400 2024-04-15T15:15:12.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.701+0000 7f23e7fff640 1 -- 172.21.15.72:0/1362727111 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f4044980 con 0x7f23f803a400 2024-04-15T15:15:12.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.701+0000 7f23fffff640 1 -- 172.21.15.72:0/1362727111 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23f80e33e0 con 0x7f23f803a400 2024-04-15T15:15:12.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.701+0000 7f23fffff640 1 -- 172.21.15.72:0/1362727111 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23f80d5c80 con 0x7f23f803a400 2024-04-15T15:15:12.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.702+0000 7f4ac5ff1640 1 -- 172.21.15.72:0/1478973791 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4ac0162d30 msgr2=0x7f4ac0152dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:12.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.702+0000 7f4ac5ff1640 1 --2- 172.21.15.72:0/1478973791 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4ac0162d30 0x7f4ac0152dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.702+0000 7f4ac5ff1640 1 -- 172.21.15.72:0/1478973791 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ac006e5e0 msgr2=0x7f4ac0150fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:12.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.702+0000 7f4ac5ff1640 1 --2- 172.21.15.72:0/1478973791 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ac006e5e0 0x7f4ac0150fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:12.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.702+0000 7f4ac5ff1640 1 -- 172.21.15.72:0/1478973791 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ac00ac6d0 con 0x7f4ac006c970 2024-04-15T15:15:12.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.702+0000 7fe7b57fa640 1 -- 172.21.15.72:0/259147821 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fe7b81289b0 con 0x7fe7b0048c40 2024-04-15T15:15:12.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.702+0000 7fe7bc8a9640 1 --2- 172.21.15.72:0/259147821 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe79c06c8f0 0x7fe79c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.702+0000 7f4ac5ff1640 1 --2- 172.21.15.72:0/1478973791 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4ac006c970 0x7f4ac00aa370 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4abc00c3c0 tx=0x7f4abc00c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:12.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.703+0000 7fd7320ba640 1 -- 172.21.15.72:0/744839292 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd72c075b10 con 0x7fd71c005210 2024-04-15T15:15:12.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.703+0000 7fe7bc8a9640 1 --2- 172.21.15.72:0/259147821 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe79c06c8f0 0x7fe79c06edb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4007fd0 tx=0x7fe7a40079a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:12.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.703+0000 7f4ab67fc640 1 -- 172.21.15.72:0/1478973791 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4abc011790 con 0x7f4ac006c970 2024-04-15T15:15:12.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.703+0000 7f4ab67fc640 1 -- 172.21.15.72:0/1478973791 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4abc017070 con 0x7f4ac006c970 2024-04-15T15:15:12.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.703+0000 7f23fffff640 1 -- 172.21.15.72:0/1362727111 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23f804b950 con 0x7f23f803a400 2024-04-15T15:15:12.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.704+0000 7f23e7fff640 1 -- 172.21.15.72:0/1362727111 <== mon.1 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f23f404b060 con 0x7f23f803a400 2024-04-15T15:15:12.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.704+0000 7f4ab67fc640 1 -- 172.21.15.72:0/1478973791 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4abc011b30 con 0x7f4ac006c970 2024-04-15T15:15:12.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.704+0000 7f4ac8255640 1 -- 172.21.15.72:0/1478973791 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ac00ac990 con 0x7f4ac006c970 2024-04-15T15:15:12.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.704+0000 7f4ac8255640 1 -- 172.21.15.72:0/1478973791 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ac00acf50 con 0x7f4ac006c970 2024-04-15T15:15:12.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.704+0000 7f23e7fff640 1 --2- 172.21.15.72:0/1362727111 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f23ec06c9f0 0x7f23ec06eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.704+0000 7f23e7fff640 1 -- 172.21.15.72:0/1362727111 <== mon.1 v2:172.21.15.175:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f23f40adfa0 con 0x7f23f803a400 2024-04-15T15:15:12.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.704+0000 7f23fe7fc640 1 --2- 172.21.15.72:0/1362727111 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f23ec06c9f0 0x7f23ec06eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.705+0000 7f23fe7fc640 1 --2- 172.21.15.72:0/1362727111 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f23ec06c9f0 0x7f23ec06eeb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2400051650 tx=0x7f2400065780 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:12.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.705+0000 7f4ac8255640 1 -- 172.21.15.72:0/1478973791 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ac00746c0 con 0x7f4ac006c970 2024-04-15T15:15:12.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.705+0000 7f524b7fe640 1 -- 172.21.15.72:0/113132821 <== mon.2 v2:172.21.15.175:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5258058cc0 con 0x7f525c048c40 2024-04-15T15:15:12.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.707+0000 7fdc5bfff640 1 -- 172.21.15.72:0/480856207 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdc6804fc60 con 0x7fdc6803eaa0 2024-04-15T15:15:12.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.710+0000 7f23e7fff640 1 -- 172.21.15.72:0/1362727111 <== mon.1 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f23f80d7600 con 0x7f23f803a400 2024-04-15T15:15:12.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.710+0000 7f7bebfff640 1 -- 172.21.15.72:0/4135194527 <== mon.2 v2:172.21.15.175:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7bec0599b0 con 0x7f7bf40d0490 2024-04-15T15:15:12.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.711+0000 7f4ab67fc640 1 -- 172.21.15.72:0/1478973791 <== mon.1 v2:172.21.15.175:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4abc011cd0 con 0x7f4ac006c970 2024-04-15T15:15:12.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.712+0000 7f4ab67fc640 1 --2- 172.21.15.72:0/1478973791 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4aa406c8f0 0x7f4aa406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.712+0000 7f4ab67fc640 1 -- 172.21.15.72:0/1478973791 <== mon.1 v2:172.21.15.175:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f4abc08bbf0 con 0x7f4ac006c970 2024-04-15T15:15:12.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.712+0000 7f4ac57f0640 1 --2- 172.21.15.72:0/1478973791 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4aa406c8f0 0x7f4aa406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.713+0000 7f4ac57f0640 1 --2- 172.21.15.72:0/1478973791 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4aa406c8f0 0x7f4aa406edb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4ac01602d0 tx=0x7f4ab0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:12.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.713+0000 7fe7b57fa640 1 -- 172.21.15.72:0/259147821 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe7b80c7070 con 0x7fe7b0048c40 2024-04-15T15:15:12.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.713+0000 7fd717fff640 1 -- 172.21.15.72:0/744839292 <== mon.2 v2:172.21.15.175:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd72001b020 con 0x7fd71c005210 2024-04-15T15:15:12.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.714+0000 7fd717fff640 1 --2- 172.21.15.72:0/744839292 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd71006c9f0 0x7fd71006eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:12.759 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.714+0000 7f4ab67fc640 1 -- 172.21.15.72:0/1478973791 <== mon.1 v2:172.21.15.175:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4ac00746c0 con 0x7f4ac006c970 2024-04-15T15:15:12.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.715+0000 7fd717fff640 1 -- 172.21.15.72:0/744839292 <== mon.2 v2:172.21.15.175:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fd72008c9e0 con 0x7fd71c005210 2024-04-15T15:15:12.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.715+0000 7fd717fff640 1 -- 172.21.15.72:0/744839292 <== mon.2 v2:172.21.15.175:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd720022020 con 0x7fd71c005210 2024-04-15T15:15:12.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.718+0000 7fd72affd640 1 --2- 172.21.15.72:0/744839292 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd71006c9f0 0x7fd71006eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:12.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:12.719+0000 7fd72affd640 1 --2- 172.21.15.72:0/744839292 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd71006c9f0 0x7fd71006eeb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd718001eb0 tx=0x7fd718009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:13.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.065+0000 7fcd3aa22640 1 -- 172.21.15.72:0/3279367777 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fcd340703d0 con 0x7fcd34073a20 2024-04-15T15:15:13.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.066+0000 7fcd30ff9640 1 -- 172.21.15.72:0/3279367777 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fcd20058ea0 con 0x7fcd34073a20 2024-04-15T15:15:13.067 INFO:teuthology.orchestra.run.smithi072.stdout:55834574857 2024-04-15T15:15:13.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.069+0000 7fcd1e7fc640 1 -- 172.21.15.72:0/3279367777 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcd1806c9f0 msgr2=0x7fcd1806eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:13.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.069+0000 7fcd1e7fc640 1 --2- 172.21.15.72:0/3279367777 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcd1806c9f0 0x7fcd1806eeb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcd28036000 tx=0x7fcd28008030 comp rx=0 tx=0).stop 2024-04-15T15:15:13.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.069+0000 7fcd1e7fc640 1 -- 172.21.15.72:0/3279367777 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcd34073a20 msgr2=0x7fcd341453f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:13.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.069+0000 7fcd1e7fc640 1 --2- 172.21.15.72:0/3279367777 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcd34073a20 0x7fcd341453f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fcd2000b780 tx=0x7fcd2000bc40 comp rx=0 tx=0).stop 2024-04-15T15:15:13.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.070+0000 7fcd1e7fc640 1 -- 172.21.15.72:0/3279367777 shutdown_connections 2024-04-15T15:15:13.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.070+0000 7fcd1e7fc640 1 --2- 172.21.15.72:0/3279367777 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcd1806c9f0 0x7fcd1806eeb0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.070+0000 7fcd1e7fc640 1 --2- 172.21.15.72:0/3279367777 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcd341414c0 0x7fcd3414f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.070+0000 7fcd1e7fc640 1 --2- 172.21.15.72:0/3279367777 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcd3415b180 0x7fcd34159e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.070+0000 7fcd1e7fc640 1 --2- 172.21.15.72:0/3279367777 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcd34073a20 0x7fcd341453f0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.070+0000 7fcd1e7fc640 1 -- 172.21.15.72:0/3279367777 >> 172.21.15.72:0/3279367777 conn(0x7fcd34072a10 msgr2=0x7fcd340b46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:13.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.070+0000 7fcd1e7fc640 1 -- 172.21.15.72:0/3279367777 shutdown_connections 2024-04-15T15:15:13.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.071+0000 7fcd1e7fc640 1 -- 172.21.15.72:0/3279367777 wait complete. 2024-04-15T15:15:13.091 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.5 2024-04-15T15:15:13.091 DEBUG:teuthology.parallel:result is None 2024-04-15T15:15:13.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.107+0000 7f4ac8255640 1 -- 172.21.15.72:0/1478973791 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f4ac0075c50 con 0x7f4ac006c970 2024-04-15T15:15:13.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.108+0000 7f4ab67fc640 1 -- 172.21.15.72:0/1478973791 <== mon.1 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4abc089e20 con 0x7f4ac006c970 2024-04-15T15:15:13.110 INFO:teuthology.orchestra.run.smithi072.stdout:55834574856 2024-04-15T15:15:13.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.110+0000 7f4ac8255640 1 -- 172.21.15.72:0/1478973791 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4aa406c8f0 msgr2=0x7f4aa406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:13.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.110+0000 7f4ac8255640 1 --2- 172.21.15.72:0/1478973791 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4aa406c8f0 0x7f4aa406edb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f4ac01602d0 tx=0x7f4ab0008040 comp rx=0 tx=0).stop 2024-04-15T15:15:13.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.110+0000 7f4ac8255640 1 -- 172.21.15.72:0/1478973791 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4ac006c970 msgr2=0x7f4ac00aa370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:13.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.110+0000 7f4ac8255640 1 --2- 172.21.15.72:0/1478973791 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4ac006c970 0x7f4ac00aa370 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4abc00c3c0 tx=0x7f4abc00c880 comp rx=0 tx=0).stop 2024-04-15T15:15:13.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.110+0000 7f4ac8255640 1 -- 172.21.15.72:0/1478973791 shutdown_connections 2024-04-15T15:15:13.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.110+0000 7f4ac8255640 1 --2- 172.21.15.72:0/1478973791 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4aa406c8f0 0x7f4aa406edb0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.110+0000 7f4ac8255640 1 --2- 172.21.15.72:0/1478973791 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4ac0162d30 0x7f4ac0152dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.110+0000 7f4ac8255640 1 --2- 172.21.15.72:0/1478973791 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4ac006e5e0 0x7f4ac0150fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.110+0000 7f4ac8255640 1 --2- 172.21.15.72:0/1478973791 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4ac006c970 0x7f4ac00aa370 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.110+0000 7f4ac8255640 1 -- 172.21.15.72:0/1478973791 >> 172.21.15.72:0/1478973791 conn(0x7f4ac0071f80 msgr2=0x7f4ac00724f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:13.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.111+0000 7f4ac8255640 1 -- 172.21.15.72:0/1478973791 shutdown_connections 2024-04-15T15:15:13.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.111+0000 7f4ac8255640 1 -- 172.21.15.72:0/1478973791 wait complete. 2024-04-15T15:15:13.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.121+0000 7f52497fa640 1 -- 172.21.15.72:0/113132821 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f525c035970 con 0x7f525c048c40 2024-04-15T15:15:13.124 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.0 2024-04-15T15:15:13.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.122+0000 7f524b7fe640 1 -- 172.21.15.72:0/113132821 <== mon.2 v2:172.21.15.175:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f525805cfc0 con 0x7f525c048c40 2024-04-15T15:15:13.125 INFO:teuthology.orchestra.run.smithi072.stdout:55834574857 2024-04-15T15:15:13.125 DEBUG:teuthology.parallel:result is None 2024-04-15T15:15:13.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.124+0000 7f52497fa640 1 -- 172.21.15.72:0/113132821 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f525006c9c0 msgr2=0x7f525006ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:13.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.124+0000 7f52497fa640 1 --2- 172.21.15.72:0/113132821 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f525006c9c0 0x7f525006ee80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5254000f60 tx=0x7f5254009040 comp rx=0 tx=0).stop 2024-04-15T15:15:13.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.124+0000 7f52497fa640 1 -- 172.21.15.72:0/113132821 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f525c048c40 msgr2=0x7f525c033cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:13.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.124+0000 7f52497fa640 1 --2- 172.21.15.72:0/113132821 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f525c048c40 0x7f525c033cd0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f525800c430 tx=0x7f525800c8f0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.124+0000 7f52497fa640 1 -- 172.21.15.72:0/113132821 shutdown_connections 2024-04-15T15:15:13.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.124+0000 7f52497fa640 1 --2- 172.21.15.72:0/113132821 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f525006c9c0 0x7f525006ee80 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.124+0000 7f52497fa640 1 --2- 172.21.15.72:0/113132821 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f525c049070 0x7f525c008040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.124+0000 7f52497fa640 1 --2- 172.21.15.72:0/113132821 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f525c048c40 0x7f525c033cd0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.124+0000 7f52497fa640 1 --2- 172.21.15.72:0/113132821 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f525c009190 0x7f525c0e7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.124+0000 7f52497fa640 1 -- 172.21.15.72:0/113132821 >> 172.21.15.72:0/113132821 conn(0x7f525c01ef40 msgr2=0x7f525c054310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:13.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.124+0000 7f52497fa640 1 -- 172.21.15.72:0/113132821 shutdown_connections 2024-04-15T15:15:13.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.124+0000 7f52497fa640 1 -- 172.21.15.72:0/113132821 wait complete. 2024-04-15T15:15:13.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.125+0000 7fdc75c4a640 1 -- 172.21.15.72:0/480856207 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fdc68057570 con 0x7fdc6803eaa0 2024-04-15T15:15:13.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.126+0000 7fdc5bfff640 1 -- 172.21.15.72:0/480856207 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fdc68057570 con 0x7fdc6803eaa0 2024-04-15T15:15:13.128 INFO:teuthology.orchestra.run.smithi072.stdout:55834574858 2024-04-15T15:15:13.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.129+0000 7fdc75c4a640 1 -- 172.21.15.72:0/480856207 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fdc5406c9c0 msgr2=0x7fdc5406ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:13.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.129+0000 7fdc75c4a640 1 --2- 172.21.15.72:0/480856207 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fdc5406c9c0 0x7fdc5406ee80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdc5c009af0 tx=0x7fdc5c009190 comp rx=0 tx=0).stop 2024-04-15T15:15:13.131 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.129+0000 7fdc75c4a640 1 -- 172.21.15.72:0/480856207 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc6803eaa0 msgr2=0x7fdc6800d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:13.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.129+0000 7fdc75c4a640 1 --2- 172.21.15.72:0/480856207 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc6803eaa0 0x7fdc6800d530 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdc70050b00 tx=0x7fdc7006b770 comp rx=0 tx=0).stop 2024-04-15T15:15:13.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.129+0000 7fdc75c4a640 1 -- 172.21.15.72:0/480856207 shutdown_connections 2024-04-15T15:15:13.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.129+0000 7fdc75c4a640 1 --2- 172.21.15.72:0/480856207 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fdc5406c9c0 0x7fdc5406ee80 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.129+0000 7fdc75c4a640 1 --2- 172.21.15.72:0/480856207 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdc6803eaa0 0x7fdc6800d530 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.129+0000 7fdc75c4a640 1 --2- 172.21.15.72:0/480856207 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdc68038050 0x7fdc6803a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.129+0000 7fdc75c4a640 1 --2- 172.21.15.72:0/480856207 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdc68037b00 0x7fdc680edd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.129+0000 7fdc75c4a640 1 -- 172.21.15.72:0/480856207 >> 172.21.15.72:0/480856207 conn(0x7fdc680559e0 msgr2=0x7fdc680066a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:13.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.130+0000 7fdc75c4a640 1 -- 172.21.15.72:0/480856207 shutdown_connections 2024-04-15T15:15:13.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.130+0000 7fdc75c4a640 1 -- 172.21.15.72:0/480856207 wait complete. 2024-04-15T15:15:13.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.131+0000 7fe7bd8ab640 1 -- 172.21.15.72:0/259147821 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fe7b000b5b0 con 0x7fe7b0048c40 2024-04-15T15:15:13.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.132+0000 7fe7b57fa640 1 -- 172.21.15.72:0/259147821 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe7b80f6a90 con 0x7fe7b0048c40 2024-04-15T15:15:13.133 INFO:teuthology.orchestra.run.smithi072.stdout:55834574858 2024-04-15T15:15:13.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.132+0000 7f23fffff640 1 -- 172.21.15.72:0/1362727111 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f23f80e55a0 con 0x7f23f803a400 2024-04-15T15:15:13.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.132+0000 7f23e7fff640 1 -- 172.21.15.72:0/1362727111 <== mon.1 v2:172.21.15.175:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f23f403f070 con 0x7f23f803a400 2024-04-15T15:15:13.134 INFO:teuthology.orchestra.run.smithi072.stdout:55834574858 2024-04-15T15:15:13.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.134+0000 7fe7bd8ab640 1 -- 172.21.15.72:0/259147821 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe79c06c8f0 msgr2=0x7fe79c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:13.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.134+0000 7fe7bd8ab640 1 --2- 172.21.15.72:0/259147821 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe79c06c8f0 0x7fe79c06edb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4007fd0 tx=0x7fe7a40079a0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.134+0000 7fe7bd8ab640 1 -- 172.21.15.72:0/259147821 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7b0048c40 msgr2=0x7fe7b0006ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:13.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.134+0000 7fe7bd8ab640 1 --2- 172.21.15.72:0/259147821 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7b0048c40 0x7fe7b0006ff0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe7b806a0f0 tx=0x7fe7b806b770 comp rx=0 tx=0).stop 2024-04-15T15:15:13.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.134+0000 7fe7bd8ab640 1 -- 172.21.15.72:0/259147821 shutdown_connections 2024-04-15T15:15:13.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.134+0000 7fe7bd8ab640 1 --2- 172.21.15.72:0/259147821 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe79c06c8f0 0x7fe79c06edb0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.136 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.134+0000 7fe7bd8ab640 1 --2- 172.21.15.72:0/259147821 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7b0048c40 0x7fe7b0006ff0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.134+0000 7fe7bd8ab640 1 --2- 172.21.15.72:0/259147821 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe7b003aa20 0x7fe7b0006ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.134+0000 7fe7bd8ab640 1 --2- 172.21.15.72:0/259147821 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe7b0009190 0x7fe7b00502a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.134+0000 7fe7bd8ab640 1 -- 172.21.15.72:0/259147821 >> 172.21.15.72:0/259147821 conn(0x7fe7b001ef40 msgr2=0x7fe7b00e5f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:13.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.135+0000 7fe7bd8ab640 1 -- 172.21.15.72:0/259147821 shutdown_connections 2024-04-15T15:15:13.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.135+0000 7fe7bd8ab640 1 -- 172.21.15.72:0/259147821 wait complete. 2024-04-15T15:15:13.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.135+0000 7f23fffff640 1 -- 172.21.15.72:0/1362727111 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f23ec06c9f0 msgr2=0x7f23ec06eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:13.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.135+0000 7f23fffff640 1 --2- 172.21.15.72:0/1362727111 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f23ec06c9f0 0x7f23ec06eeb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2400051650 tx=0x7f2400065780 comp rx=0 tx=0).stop 2024-04-15T15:15:13.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.135+0000 7f23fffff640 1 -- 172.21.15.72:0/1362727111 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f23f803a400 msgr2=0x7f23f80e2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:13.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.135+0000 7f23fffff640 1 --2- 172.21.15.72:0/1362727111 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f23f803a400 0x7f23f80e2e00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f23f40090b0 tx=0x7f23f40046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.135+0000 7f23fffff640 1 -- 172.21.15.72:0/1362727111 shutdown_connections 2024-04-15T15:15:13.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.135+0000 7f23fffff640 1 --2- 172.21.15.72:0/1362727111 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f23ec06c9f0 0x7f23ec06eeb0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.135+0000 7f23fffff640 1 --2- 172.21.15.72:0/1362727111 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23f80e1090 0x7f23f80db7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.135+0000 7f23fffff640 1 --2- 172.21.15.72:0/1362727111 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f23f8050dd0 0x7f23f80e1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.135+0000 7f23fffff640 1 --2- 172.21.15.72:0/1362727111 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f23f803a400 0x7f23f80e2e00 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.135+0000 7f23fffff640 1 -- 172.21.15.72:0/1362727111 >> 172.21.15.72:0/1362727111 conn(0x7f23f8009fb0 msgr2=0x7f23f8036590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:13.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.135+0000 7f23fffff640 1 -- 172.21.15.72:0/1362727111 shutdown_connections 2024-04-15T15:15:13.138 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.135+0000 7f23fffff640 1 -- 172.21.15.72:0/1362727111 wait complete. 2024-04-15T15:15:13.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.137+0000 7f7bfa1bd640 1 -- 172.21.15.72:0/4135194527 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f7bd8005580 con 0x7f7bf40d0490 2024-04-15T15:15:13.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.138+0000 7f7bebfff640 1 -- 172.21.15.72:0/4135194527 <== mon.2 v2:172.21.15.175:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7bec0592b0 con 0x7f7bf40d0490 2024-04-15T15:15:13.139 INFO:teuthology.orchestra.run.smithi072.stdout:55834574858 2024-04-15T15:15:13.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.138+0000 7fd7320ba640 1 -- 172.21.15.72:0/744839292 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fd72c06be10 con 0x7fd71c005210 2024-04-15T15:15:13.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.138+0000 7fd717fff640 1 -- 172.21.15.72:0/744839292 <== mon.2 v2:172.21.15.175:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd72005ab70 con 0x7fd71c005210 2024-04-15T15:15:13.140 INFO:teuthology.orchestra.run.smithi072.stdout:55834574857 2024-04-15T15:15:13.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.140+0000 7f7bfa1bd640 1 -- 172.21.15.72:0/4135194527 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7bd406c9c0 msgr2=0x7f7bd406ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:13.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.140+0000 7f7bfa1bd640 1 --2- 172.21.15.72:0/4135194527 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7bd406c9c0 0x7f7bd406ee80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7be4038000 tx=0x7f7be4032000 comp rx=0 tx=0).stop 2024-04-15T15:15:13.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.140+0000 7f7bfa1bd640 1 -- 172.21.15.72:0/4135194527 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7bf40d0490 msgr2=0x7f7bf4146d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:13.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.140+0000 7f7bfa1bd640 1 --2- 172.21.15.72:0/4135194527 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7bf40d0490 0x7f7bf4146d20 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f7bec00c410 tx=0x7f7bec00c8d0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.140+0000 7f7bfa1bd640 1 -- 172.21.15.72:0/4135194527 shutdown_connections 2024-04-15T15:15:13.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.140+0000 7f7bfa1bd640 1 --2- 172.21.15.72:0/4135194527 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7bd406c9c0 0x7f7bd406ee80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.140+0000 7f7bfa1bd640 1 --2- 172.21.15.72:0/4135194527 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7bf415b5d0 0x7f7bf414e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.140+0000 7f7bfa1bd640 1 --2- 172.21.15.72:0/4135194527 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7bf40d0920 0x7f7bf406dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.140+0000 7f7bfa1bd640 1 --2- 172.21.15.72:0/4135194527 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7bf40d0490 0x7f7bf4146d20 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.140+0000 7f7bfa1bd640 1 -- 172.21.15.72:0/4135194527 >> 172.21.15.72:0/4135194527 conn(0x7f7bf4072c80 msgr2=0x7f7bf41563b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:13.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.141+0000 7f7bfa1bd640 1 -- 172.21.15.72:0/4135194527 shutdown_connections 2024-04-15T15:15:13.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.141+0000 7f7bfa1bd640 1 -- 172.21.15.72:0/4135194527 wait complete. 2024-04-15T15:15:13.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.141+0000 7fd7320ba640 1 -- 172.21.15.72:0/744839292 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd71006c9f0 msgr2=0x7fd71006eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:13.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.141+0000 7fd7320ba640 1 --2- 172.21.15.72:0/744839292 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd71006c9f0 0x7fd71006eeb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd718001eb0 tx=0x7fd718009040 comp rx=0 tx=0).stop 2024-04-15T15:15:13.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.141+0000 7fd7320ba640 1 -- 172.21.15.72:0/744839292 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd71c005210 msgr2=0x7fd72c1377f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:13.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.141+0000 7fd7320ba640 1 --2- 172.21.15.72:0/744839292 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd71c005210 0x7fd72c1377f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd72000d860 tx=0x7fd72000dd20 comp rx=0 tx=0).stop 2024-04-15T15:15:13.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.141+0000 7fd7320ba640 1 -- 172.21.15.72:0/744839292 shutdown_connections 2024-04-15T15:15:13.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.141+0000 7fd7320ba640 1 --2- 172.21.15.72:0/744839292 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd71006c9f0 0x7fd71006eeb0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.141+0000 7fd7320ba640 1 --2- 172.21.15.72:0/744839292 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd72c141e90 0x7fd72c142310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.141+0000 7fd7320ba640 1 --2- 172.21.15.72:0/744839292 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd72c147a30 0x7fd72c147eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.141+0000 7fd7320ba640 1 --2- 172.21.15.72:0/744839292 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd71c005210 0x7fd72c1377f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.142+0000 7fd7320ba640 1 -- 172.21.15.72:0/744839292 >> 172.21.15.72:0/744839292 conn(0x7fd72c072130 msgr2=0x7fd72c14f950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:13.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.142+0000 7fd7320ba640 1 -- 172.21.15.72:0/744839292 shutdown_connections 2024-04-15T15:15:13.146 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.2 2024-04-15T15:15:13.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.142+0000 7fd7320ba640 1 -- 172.21.15.72:0/744839292 wait complete. 2024-04-15T15:15:13.146 DEBUG:teuthology.parallel:result is None 2024-04-15T15:15:13.149 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.6 2024-04-15T15:15:13.149 DEBUG:teuthology.parallel:result is None 2024-04-15T15:15:13.157 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.7 2024-04-15T15:15:13.157 DEBUG:teuthology.parallel:result is None 2024-04-15T15:15:13.158 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.1 2024-04-15T15:15:13.158 DEBUG:teuthology.parallel:result is None 2024-04-15T15:15:13.162 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.4 2024-04-15T15:15:13.162 DEBUG:teuthology.parallel:result is None 2024-04-15T15:15:13.162 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.3 2024-04-15T15:15:13.163 DEBUG:teuthology.parallel:result is None 2024-04-15T15:15:13.163 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-15T15:15:13.163 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:15:13.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.303+0000 7f14dfcc5640 1 Processor -- start 2024-04-15T15:15:13.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.303+0000 7f14dfcc5640 1 -- start start 2024-04-15T15:15:13.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.304+0000 7f14dfcc5640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f14d815eba0 0x7f14d806f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:13.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.304+0000 7f14dfcc5640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f14d80ccb90 0x7f14d80ccfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:13.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.304+0000 7f14dfcc5640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14d80afb10 con 0x7f14d806f720 2024-04-15T15:15:13.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.304+0000 7f14dfcc5640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f14d80cd520 con 0x7f14d815eba0 2024-04-15T15:15:13.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.304+0000 7f14dfcc5640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f14d815ef80 con 0x7f14d80ccb90 2024-04-15T15:15:13.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.304+0000 7f14dda61640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f14d815eba0 0x7f14d806f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:13.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.304+0000 7f14dda61640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f14d815eba0 0x7f14d806f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45442/0 (socket says 172.21.15.72:45442) 2024-04-15T15:15:13.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.304+0000 7f14dda61640 1 -- 172.21.15.72:0/1610539649 learned_addr learned my addr 172.21.15.72:0/1610539649 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:13.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.304+0000 7f14dd260640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f14d806f720 0x7f14d80af400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37880/0 (socket says 172.21.15.72:37880) 2024-04-15T15:15:13.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.304+0000 7f14de262640 1 --2- 172.21.15.72:0/1610539649 >> v2:172.21.15.175:3300/0 conn(0x7f14d80ccb90 0x7f14d80ccfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:13.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.304+0000 7f14dda61640 1 -- 172.21.15.72:0/1610539649 >> v2:172.21.15.175:3300/0 conn(0x7f14d80ccb90 msgr2=0x7f14d80ccfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:13.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.304+0000 7f14dda61640 1 --2- 172.21.15.72:0/1610539649 >> v2:172.21.15.175:3300/0 conn(0x7f14d80ccb90 0x7f14d80ccfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.304+0000 7f14dda61640 1 -- 172.21.15.72:0/1610539649 >> v1:172.21.15.72:6789/0 conn(0x7f14d806f720 legacy=0x7f14d80af400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:13.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.304+0000 7f14dca5f640 1 -- 172.21.15.72:0/1610539649 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 352908935 0 0) 0x7f14d80afb10 con 0x7f14d806f720 2024-04-15T15:15:13.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.304+0000 7f14dda61640 1 -- 172.21.15.72:0/1610539649 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14d80ce2b0 con 0x7f14d815eba0 2024-04-15T15:15:13.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.305+0000 7f14dda61640 1 --2- 172.21.15.72:0/1610539649 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14d815eba0 0x7f14d806f1e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f14d80b3180 tx=0x7f14d402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d681323b9e106437 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:13.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.305+0000 7f14dca5f640 1 -- 172.21.15.72:0/1610539649 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d4035020 con 0x7f14d815eba0 2024-04-15T15:15:13.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.305+0000 7f14dca5f640 1 -- 172.21.15.72:0/1610539649 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14d4031870 con 0x7f14d815eba0 2024-04-15T15:15:13.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.305+0000 7f14dca5f640 1 -- 172.21.15.72:0/1610539649 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d4031c40 con 0x7f14d815eba0 2024-04-15T15:15:13.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.306+0000 7f14dfcc5640 1 -- 172.21.15.72:0/1610539649 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14d815eba0 msgr2=0x7f14d806f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:13.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.306+0000 7f14dfcc5640 1 --2- 172.21.15.72:0/1610539649 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14d815eba0 0x7f14d806f1e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f14d80b3180 tx=0x7f14d402b910 comp rx=0 tx=0).stop 2024-04-15T15:15:13.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.306+0000 7f14dfcc5640 1 -- 172.21.15.72:0/1610539649 shutdown_connections 2024-04-15T15:15:13.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.306+0000 7f14dfcc5640 1 --2- 172.21.15.72:0/1610539649 >> v2:172.21.15.175:3300/0 conn(0x7f14d80ccb90 0x7f14d80ccfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.306+0000 7f14dfcc5640 1 --2- 172.21.15.72:0/1610539649 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14d815eba0 0x7f14d806f1e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.306+0000 7f14dfcc5640 1 -- 172.21.15.72:0/1610539649 >> 172.21.15.72:0/1610539649 conn(0x7f14d80b1dd0 msgr2=0x7f14d80b21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:13.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.306+0000 7f14dfcc5640 1 -- 172.21.15.72:0/1610539649 shutdown_connections 2024-04-15T15:15:13.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.306+0000 7f14dfcc5640 1 -- 172.21.15.72:0/1610539649 wait complete. 2024-04-15T15:15:13.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.307+0000 7f14dfcc5640 1 Processor -- start 2024-04-15T15:15:13.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.307+0000 7f14dfcc5640 1 -- start start 2024-04-15T15:15:13.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.308+0000 7f14dfcc5640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f14d80ccb90 0x7f14d81461f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:13.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.308+0000 7f14dfcc5640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f14d815eba0 0x7f14d81458a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:13.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.308+0000 7f14dfcc5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14d809d960 0x7f14d8154850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:13.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.308+0000 7f14dfcc5640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f14d806ecc0 con 0x7f14d809d960 2024-04-15T15:15:13.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.308+0000 7f14dfcc5640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f14d80ce870 con 0x7f14d815eba0 2024-04-15T15:15:13.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.308+0000 7f14dfcc5640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f14d80b2b80 con 0x7f14d80ccb90 2024-04-15T15:15:13.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.308+0000 7f14de262640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14d809d960 0x7f14d8154850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:13.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.308+0000 7f14de262640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14d809d960 0x7f14d8154850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45458/0 (socket says 172.21.15.72:45458) 2024-04-15T15:15:13.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.308+0000 7f14de262640 1 -- 172.21.15.72:0/217504600 learned_addr learned my addr 172.21.15.72:0/217504600 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:13.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.308+0000 7f14dd260640 1 --2- 172.21.15.72:0/217504600 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f14d815eba0 0x7f14d81458a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:13.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.308+0000 7f14de262640 1 -- 172.21.15.72:0/217504600 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f14d80ccb90 msgr2=0x7f14d81461f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:13.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.308+0000 7f14de262640 1 --2- 172.21.15.72:0/217504600 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f14d80ccb90 0x7f14d81461f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.308+0000 7f14de262640 1 -- 172.21.15.72:0/217504600 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f14d815eba0 msgr2=0x7f14d81458a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:13.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.308+0000 7f14de262640 1 --2- 172.21.15.72:0/217504600 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f14d815eba0 0x7f14d81458a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.308+0000 7f14de262640 1 -- 172.21.15.72:0/217504600 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14d809ac70 con 0x7f14d809d960 2024-04-15T15:15:13.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.308+0000 7f14de262640 1 --2- 172.21.15.72:0/217504600 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14d809d960 0x7f14d8154850 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f14d0007e60 tx=0x7f14d000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:13.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.308+0000 7f14ce7fc640 1 -- 172.21.15.72:0/217504600 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d0014420 con 0x7f14d809d960 2024-04-15T15:15:13.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.308+0000 7f14ce7fc640 1 -- 172.21.15.72:0/217504600 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14d0019070 con 0x7f14d809d960 2024-04-15T15:15:13.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.308+0000 7f14dfcc5640 1 -- 172.21.15.72:0/217504600 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14d809de00 con 0x7f14d809d960 2024-04-15T15:15:13.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.309+0000 7f14ce7fc640 1 -- 172.21.15.72:0/217504600 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d0014820 con 0x7f14d809d960 2024-04-15T15:15:13.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.309+0000 7f14dfcc5640 1 -- 172.21.15.72:0/217504600 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14d809e0e0 con 0x7f14d809d960 2024-04-15T15:15:13.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.310+0000 7f14dfcc5640 1 -- 172.21.15.72:0/217504600 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14d80b6180 con 0x7f14d809d960 2024-04-15T15:15:13.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.310+0000 7f14ce7fc640 1 -- 172.21.15.72:0/217504600 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f14d00149c0 con 0x7f14d809d960 2024-04-15T15:15:13.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.310+0000 7f14ce7fc640 1 --2- 172.21.15.72:0/217504600 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f14bc06c8f0 0x7f14bc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:13.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.311+0000 7f14ce7fc640 1 -- 172.21.15.72:0/217504600 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f14d008c620 con 0x7f14d809d960 2024-04-15T15:15:13.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.311+0000 7f14dda61640 1 --2- 172.21.15.72:0/217504600 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f14bc06c8f0 0x7f14bc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:13.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.311+0000 7f14dda61640 1 --2- 172.21.15.72:0/217504600 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f14bc06c8f0 0x7f14bc06edb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f14d4036000 tx=0x7f14d4030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:13.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.315+0000 7f14ce7fc640 1 -- 172.21.15.72:0/217504600 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f14d80b6180 con 0x7f14d809d960 2024-04-15T15:15:13.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.486+0000 7f14dfcc5640 1 -- 172.21.15.72:0/217504600 --> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f14d80689e0 con 0x7f14bc06c8f0 2024-04-15T15:15:13.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.494+0000 7f14ce7fc640 1 -- 172.21.15.72:0/217504600 <== mgr.4101 v2:172.21.15.72:6832/3503807805 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332534 (secure 0 0 0) 0x7f14d80689e0 con 0x7f14bc06c8f0 2024-04-15T15:15:13.499 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:13.502 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-04-15T15:15:13.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.497+0000 7f14dfcc5640 1 -- 172.21.15.72:0/217504600 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f14bc06c8f0 msgr2=0x7f14bc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:13.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.497+0000 7f14dfcc5640 1 --2- 172.21.15.72:0/217504600 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f14bc06c8f0 0x7f14bc06edb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f14d4036000 tx=0x7f14d4030000 comp rx=0 tx=0).stop 2024-04-15T15:15:13.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.497+0000 7f14dfcc5640 1 -- 172.21.15.72:0/217504600 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14d809d960 msgr2=0x7f14d8154850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:13.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.497+0000 7f14dfcc5640 1 --2- 172.21.15.72:0/217504600 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14d809d960 0x7f14d8154850 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f14d0007e60 tx=0x7f14d000e4b0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.497+0000 7f14dfcc5640 1 -- 172.21.15.72:0/217504600 shutdown_connections 2024-04-15T15:15:13.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.497+0000 7f14dfcc5640 1 --2- 172.21.15.72:0/217504600 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f14bc06c8f0 0x7f14bc06edb0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.498+0000 7f14dfcc5640 1 --2- 172.21.15.72:0/217504600 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14d809d960 0x7f14d8154850 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.498+0000 7f14dfcc5640 1 --2- 172.21.15.72:0/217504600 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f14d815eba0 0x7f14d81458a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.498+0000 7f14dfcc5640 1 --2- 172.21.15.72:0/217504600 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f14d80ccb90 0x7f14d81461f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.498+0000 7f14dfcc5640 1 -- 172.21.15.72:0/217504600 >> 172.21.15.72:0/217504600 conn(0x7f14d80b1dd0 msgr2=0x7f14d80af1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:13.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.498+0000 7f14dfcc5640 1 -- 172.21.15.72:0/217504600 shutdown_connections 2024-04-15T15:15:13.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.498+0000 7f14dfcc5640 1 -- 172.21.15.72:0/217504600 wait complete. 2024-04-15T15:15:13.511 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-04-15T15:15:13.348367+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219392,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755328,"statfs":{"total":773094113280,"available":772869455872,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002938"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137048+0000","last_change":"2024-04-15T15:14:51.075131+0000","last_active":"2024-04-15T15:14:57.137048+0000","last_peered":"2024-04-15T15:14:57.137048+0000","last_clean":"2024-04-15T15:14:57.137048+0000","last_became_active":"2024-04-15T15:14:51.074841+0000","last_became_peered":"2024-04-15T15:14:51.074841+0000","last_unstale":"2024-04-15T15:14:57.137048+0000","last_undegraded":"2024-04-15T15:14:57.137048+0000","last_fullsized":"2024-04-15T15:14:57.137048+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:19:56.231155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138223+0000","last_change":"2024-04-15T15:14:52.077480+0000","last_active":"2024-04-15T15:14:57.138223+0000","last_peered":"2024-04-15T15:14:57.138223+0000","last_clean":"2024-04-15T15:14:57.138223+0000","last_became_active":"2024-04-15T15:14:52.077166+0000","last_became_peered":"2024-04-15T15:14:52.077166+0000","last_unstale":"2024-04-15T15:14:57.138223+0000","last_undegraded":"2024-04-15T15:14:57.138223+0000","last_fullsized":"2024-04-15T15:14:57.138223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:13:01.642838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136181+0000","last_change":"2024-04-15T15:14:56.255424+0000","last_active":"2024-04-15T15:14:57.136181+0000","last_peered":"2024-04-15T15:14:57.136181+0000","last_clean":"2024-04-15T15:14:57.136181+0000","last_became_active":"2024-04-15T15:14:56.254987+0000","last_became_peered":"2024-04-15T15:14:56.254987+0000","last_unstale":"2024-04-15T15:14:57.136181+0000","last_undegraded":"2024-04-15T15:14:57.136181+0000","last_fullsized":"2024-04-15T15:14:57.136181+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:11:24.196989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138454+0000","last_change":"2024-04-15T15:14:51.071376+0000","last_active":"2024-04-15T15:14:57.138454+0000","last_peered":"2024-04-15T15:14:57.138454+0000","last_clean":"2024-04-15T15:14:57.138454+0000","last_became_active":"2024-04-15T15:14:51.071108+0000","last_became_peered":"2024-04-15T15:14:51.071108+0000","last_unstale":"2024-04-15T15:14:57.138454+0000","last_undegraded":"2024-04-15T15:14:57.138454+0000","last_fullsized":"2024-04-15T15:14:57.138454+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:50:01.422527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136732+0000","last_change":"2024-04-15T15:14:52.083839+0000","last_active":"2024-04-15T15:14:57.136732+0000","last_peered":"2024-04-15T15:14:57.136732+0000","last_clean":"2024-04-15T15:14:57.136732+0000","last_became_active":"2024-04-15T15:14:52.083533+0000","last_became_peered":"2024-04-15T15:14:52.083533+0000","last_unstale":"2024-04-15T15:14:57.136732+0000","last_undegraded":"2024-04-15T15:14:57.136732+0000","last_fullsized":"2024-04-15T15:14:57.136732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:34:51.775979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135872+0000","last_change":"2024-04-15T15:14:56.098375+0000","last_active":"2024-04-15T15:14:57.135872+0000","last_peered":"2024-04-15T15:14:57.135872+0000","last_clean":"2024-04-15T15:14:57.135872+0000","last_became_active":"2024-04-15T15:14:56.097969+0000","last_became_peered":"2024-04-15T15:14:56.097969+0000","last_unstale":"2024-04-15T15:14:57.135872+0000","last_undegraded":"2024-04-15T15:14:57.135872+0000","last_fullsized":"2024-04-15T15:14:57.135872+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:12:44.445524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136739+0000","last_change":"2024-04-15T15:14:51.079869+0000","last_active":"2024-04-15T15:14:57.136739+0000","last_peered":"2024-04-15T15:14:57.136739+0000","last_clean":"2024-04-15T15:14:57.136739+0000","last_became_active":"2024-04-15T15:14:51.079576+0000","last_became_peered":"2024-04-15T15:14:51.079576+0000","last_unstale":"2024-04-15T15:14:57.136739+0000","last_undegraded":"2024-04-15T15:14:57.136739+0000","last_fullsized":"2024-04-15T15:14:57.136739+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:58:40.440083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109631+0000","last_change":"2024-04-15T15:14:52.075307+0000","last_active":"2024-04-15T15:14:57.109631+0000","last_peered":"2024-04-15T15:14:57.109631+0000","last_clean":"2024-04-15T15:14:57.109631+0000","last_became_active":"2024-04-15T15:14:52.075001+0000","last_became_peered":"2024-04-15T15:14:52.075001+0000","last_unstale":"2024-04-15T15:14:57.109631+0000","last_undegraded":"2024-04-15T15:14:57.109631+0000","last_fullsized":"2024-04-15T15:14:57.109631+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:01:24.679360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136399+0000","last_change":"2024-04-15T15:14:56.098292+0000","last_active":"2024-04-15T15:14:57.136399+0000","last_peered":"2024-04-15T15:14:57.136399+0000","last_clean":"2024-04-15T15:14:57.136399+0000","last_became_active":"2024-04-15T15:14:56.097848+0000","last_became_peered":"2024-04-15T15:14:56.097848+0000","last_unstale":"2024-04-15T15:14:57.136399+0000","last_undegraded":"2024-04-15T15:14:57.136399+0000","last_fullsized":"2024-04-15T15:14:57.136399+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:26:49.486090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136078+0000","last_change":"2024-04-15T15:14:51.079754+0000","last_active":"2024-04-15T15:14:57.136078+0000","last_peered":"2024-04-15T15:14:57.136078+0000","last_clean":"2024-04-15T15:14:57.136078+0000","last_became_active":"2024-04-15T15:14:51.079452+0000","last_became_peered":"2024-04-15T15:14:51.079452+0000","last_unstale":"2024-04-15T15:14:57.136078+0000","last_undegraded":"2024-04-15T15:14:57.136078+0000","last_fullsized":"2024-04-15T15:14:57.136078+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:19:37.987965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.110221+0000","last_change":"2024-04-15T15:14:52.076654+0000","last_active":"2024-04-15T15:14:57.110221+0000","last_peered":"2024-04-15T15:14:57.110221+0000","last_clean":"2024-04-15T15:14:57.110221+0000","last_became_active":"2024-04-15T15:14:52.076283+0000","last_became_peered":"2024-04-15T15:14:52.076283+0000","last_unstale":"2024-04-15T15:14:57.110221+0000","last_undegraded":"2024-04-15T15:14:57.110221+0000","last_fullsized":"2024-04-15T15:14:57.110221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:57:57.665646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136259+0000","last_change":"2024-04-15T15:14:56.105257+0000","last_active":"2024-04-15T15:14:57.136259+0000","last_peered":"2024-04-15T15:14:57.136259+0000","last_clean":"2024-04-15T15:14:57.136259+0000","last_became_active":"2024-04-15T15:14:56.104856+0000","last_became_peered":"2024-04-15T15:14:56.104856+0000","last_unstale":"2024-04-15T15:14:57.136259+0000","last_undegraded":"2024-04-15T15:14:57.136259+0000","last_fullsized":"2024-04-15T15:14:57.136259+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:10:23.721558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108699+0000","last_change":"2024-04-15T15:14:52.086412+0000","last_active":"2024-04-15T15:14:57.108699+0000","last_peered":"2024-04-15T15:14:57.108699+0000","last_clean":"2024-04-15T15:14:57.108699+0000","last_became_active":"2024-04-15T15:14:52.085479+0000","last_became_peered":"2024-04-15T15:14:52.085479+0000","last_unstale":"2024-04-15T15:14:57.108699+0000","last_undegraded":"2024-04-15T15:14:57.108699+0000","last_fullsized":"2024-04-15T15:14:57.108699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:13:01.642838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136325+0000","last_change":"2024-04-15T15:14:51.080109+0000","last_active":"2024-04-15T15:14:57.136325+0000","last_peered":"2024-04-15T15:14:57.136325+0000","last_clean":"2024-04-15T15:14:57.136325+0000","last_became_active":"2024-04-15T15:14:51.079768+0000","last_became_peered":"2024-04-15T15:14:51.079768+0000","last_unstale":"2024-04-15T15:14:57.136325+0000","last_undegraded":"2024-04-15T15:14:57.136325+0000","last_fullsized":"2024-04-15T15:14:57.136325+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:07:48.416478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136732+0000","last_change":"2024-04-15T15:14:56.102424+0000","last_active":"2024-04-15T15:14:57.136732+0000","last_peered":"2024-04-15T15:14:57.136732+0000","last_clean":"2024-04-15T15:14:57.136732+0000","last_became_active":"2024-04-15T15:14:56.101985+0000","last_became_peered":"2024-04-15T15:14:56.101985+0000","last_unstale":"2024-04-15T15:14:57.136732+0000","last_undegraded":"2024-04-15T15:14:57.136732+0000","last_fullsized":"2024-04-15T15:14:57.136732+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:01:28.704663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136649+0000","last_change":"2024-04-15T15:14:52.084991+0000","last_active":"2024-04-15T15:14:57.136649+0000","last_peered":"2024-04-15T15:14:57.136649+0000","last_clean":"2024-04-15T15:14:57.136649+0000","last_became_active":"2024-04-15T15:14:52.082250+0000","last_became_peered":"2024-04-15T15:14:52.082250+0000","last_unstale":"2024-04-15T15:14:57.136649+0000","last_undegraded":"2024-04-15T15:14:57.136649+0000","last_fullsized":"2024-04-15T15:14:57.136649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:26:45.460787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137232+0000","last_change":"2024-04-15T15:14:51.071863+0000","last_active":"2024-04-15T15:14:57.137232+0000","last_peered":"2024-04-15T15:14:57.137232+0000","last_clean":"2024-04-15T15:14:57.137232+0000","last_became_active":"2024-04-15T15:14:51.071595+0000","last_became_peered":"2024-04-15T15:14:51.071595+0000","last_unstale":"2024-04-15T15:14:57.137232+0000","last_undegraded":"2024-04-15T15:14:57.137232+0000","last_fullsized":"2024-04-15T15:14:57.137232+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:19:37.987965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136625+0000","last_change":"2024-04-15T15:14:56.099259+0000","last_active":"2024-04-15T15:14:57.136625+0000","last_peered":"2024-04-15T15:14:57.136625+0000","last_clean":"2024-04-15T15:14:57.136625+0000","last_became_active":"2024-04-15T15:14:56.098862+0000","last_became_peered":"2024-04-15T15:14:56.098862+0000","last_unstale":"2024-04-15T15:14:57.136625+0000","last_undegraded":"2024-04-15T15:14:57.136625+0000","last_fullsized":"2024-04-15T15:14:57.136625+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:38:39.210922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136403+0000","last_change":"2024-04-15T15:14:52.080320+0000","last_active":"2024-04-15T15:14:57.136403+0000","last_peered":"2024-04-15T15:14:57.136403+0000","last_clean":"2024-04-15T15:14:57.136403+0000","last_became_active":"2024-04-15T15:14:52.080064+0000","last_became_peered":"2024-04-15T15:14:52.080064+0000","last_unstale":"2024-04-15T15:14:57.136403+0000","last_undegraded":"2024-04-15T15:14:57.136403+0000","last_fullsized":"2024-04-15T15:14:57.136403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:10:19.696254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136107+0000","last_change":"2024-04-15T15:14:51.075197+0000","last_active":"2024-04-15T15:14:57.136107+0000","last_peered":"2024-04-15T15:14:57.136107+0000","last_clean":"2024-04-15T15:14:57.136107+0000","last_became_active":"2024-04-15T15:14:51.074955+0000","last_became_peered":"2024-04-15T15:14:51.074955+0000","last_unstale":"2024-04-15T15:14:57.136107+0000","last_undegraded":"2024-04-15T15:14:57.136107+0000","last_fullsized":"2024-04-15T15:14:57.136107+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:18:41.399059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137490+0000","last_change":"2024-04-15T15:14:56.253169+0000","last_active":"2024-04-15T15:14:57.137490+0000","last_peered":"2024-04-15T15:14:57.137490+0000","last_clean":"2024-04-15T15:14:57.137490+0000","last_became_active":"2024-04-15T15:14:56.252823+0000","last_became_peered":"2024-04-15T15:14:56.252823+0000","last_unstale":"2024-04-15T15:14:57.137490+0000","last_undegraded":"2024-04-15T15:14:57.137490+0000","last_fullsized":"2024-04-15T15:14:57.137490+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:03:07.876012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136337+0000","last_change":"2024-04-15T15:14:52.075278+0000","last_active":"2024-04-15T15:14:57.136337+0000","last_peered":"2024-04-15T15:14:57.136337+0000","last_clean":"2024-04-15T15:14:57.136337+0000","last_became_active":"2024-04-15T15:14:52.075003+0000","last_became_peered":"2024-04-15T15:14:52.075003+0000","last_unstale":"2024-04-15T15:14:57.136337+0000","last_undegraded":"2024-04-15T15:14:57.136337+0000","last_fullsized":"2024-04-15T15:14:57.136337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:47:55.994716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135597+0000","last_change":"2024-04-15T15:14:51.076530+0000","last_active":"2024-04-15T15:14:57.135597+0000","last_peered":"2024-04-15T15:14:57.135597+0000","last_clean":"2024-04-15T15:14:57.135597+0000","last_became_active":"2024-04-15T15:14:51.076247+0000","last_became_peered":"2024-04-15T15:14:51.076247+0000","last_unstale":"2024-04-15T15:14:57.135597+0000","last_undegraded":"2024-04-15T15:14:57.135597+0000","last_fullsized":"2024-04-15T15:14:57.135597+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:47:54.990677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136888+0000","last_change":"2024-04-15T15:14:56.098477+0000","last_active":"2024-04-15T15:14:57.136888+0000","last_peered":"2024-04-15T15:14:57.136888+0000","last_clean":"2024-04-15T15:14:57.136888+0000","last_became_active":"2024-04-15T15:14:56.097839+0000","last_became_peered":"2024-04-15T15:14:56.097839+0000","last_unstale":"2024-04-15T15:14:57.136888+0000","last_undegraded":"2024-04-15T15:14:57.136888+0000","last_fullsized":"2024-04-15T15:14:57.136888+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:12.500763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138519+0000","last_change":"2024-04-15T15:14:52.079014+0000","last_active":"2024-04-15T15:14:57.138519+0000","last_peered":"2024-04-15T15:14:57.138519+0000","last_clean":"2024-04-15T15:14:57.138519+0000","last_became_active":"2024-04-15T15:14:52.078708+0000","last_became_peered":"2024-04-15T15:14:52.078708+0000","last_unstale":"2024-04-15T15:14:57.138519+0000","last_undegraded":"2024-04-15T15:14:57.138519+0000","last_fullsized":"2024-04-15T15:14:57.138519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:37:31.656434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109619+0000","last_change":"2024-04-15T15:14:51.075067+0000","last_active":"2024-04-15T15:14:57.109619+0000","last_peered":"2024-04-15T15:14:57.109619+0000","last_clean":"2024-04-15T15:14:57.109619+0000","last_became_active":"2024-04-15T15:14:51.074780+0000","last_became_peered":"2024-04-15T15:14:51.074780+0000","last_unstale":"2024-04-15T15:14:57.109619+0000","last_undegraded":"2024-04-15T15:14:57.109619+0000","last_fullsized":"2024-04-15T15:14:57.109619+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:47:54.990677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136173+0000","last_change":"2024-04-15T15:14:56.252842+0000","last_active":"2024-04-15T15:14:57.136173+0000","last_peered":"2024-04-15T15:14:57.136173+0000","last_clean":"2024-04-15T15:14:57.136173+0000","last_became_active":"2024-04-15T15:14:56.252432+0000","last_became_peered":"2024-04-15T15:14:56.252432+0000","last_unstale":"2024-04-15T15:14:57.136173+0000","last_undegraded":"2024-04-15T15:14:57.136173+0000","last_fullsized":"2024-04-15T15:14:57.136173+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:51:26.840184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138191+0000","last_change":"2024-04-15T15:14:52.082404+0000","last_active":"2024-04-15T15:14:57.138191+0000","last_peered":"2024-04-15T15:14:57.138191+0000","last_clean":"2024-04-15T15:14:57.138191+0000","last_became_active":"2024-04-15T15:14:52.082136+0000","last_became_peered":"2024-04-15T15:14:52.082136+0000","last_unstale":"2024-04-15T15:14:57.138191+0000","last_undegraded":"2024-04-15T15:14:57.138191+0000","last_fullsized":"2024-04-15T15:14:57.138191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:38:35.185619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137543+0000","last_change":"2024-04-15T15:14:51.073915+0000","last_active":"2024-04-15T15:14:57.137543+0000","last_peered":"2024-04-15T15:14:57.137543+0000","last_clean":"2024-04-15T15:14:57.137543+0000","last_became_active":"2024-04-15T15:14:51.073639+0000","last_became_peered":"2024-04-15T15:14:51.073639+0000","last_unstale":"2024-04-15T15:14:57.137543+0000","last_undegraded":"2024-04-15T15:14:57.137543+0000","last_fullsized":"2024-04-15T15:14:57.137543+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:41:20.990702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135980+0000","last_change":"2024-04-15T15:14:56.251221+0000","last_active":"2024-04-15T15:14:57.135980+0000","last_peered":"2024-04-15T15:14:57.135980+0000","last_clean":"2024-04-15T15:14:57.135980+0000","last_became_active":"2024-04-15T15:14:56.250888+0000","last_became_peered":"2024-04-15T15:14:56.250888+0000","last_unstale":"2024-04-15T15:14:57.135980+0000","last_undegraded":"2024-04-15T15:14:57.135980+0000","last_fullsized":"2024-04-15T15:14:57.135980+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:42:46.671056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137233+0000","last_change":"2024-04-15T15:14:52.077347+0000","last_active":"2024-04-15T15:14:57.137233+0000","last_peered":"2024-04-15T15:14:57.137233+0000","last_clean":"2024-04-15T15:14:57.137233+0000","last_became_active":"2024-04-15T15:14:52.077053+0000","last_became_peered":"2024-04-15T15:14:52.077053+0000","last_unstale":"2024-04-15T15:14:57.137233+0000","last_undegraded":"2024-04-15T15:14:57.137233+0000","last_fullsized":"2024-04-15T15:14:57.137233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:13:01.642838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136936+0000","last_change":"2024-04-15T15:14:51.076396+0000","last_active":"2024-04-15T15:14:57.136936+0000","last_peered":"2024-04-15T15:14:57.136936+0000","last_clean":"2024-04-15T15:14:57.136936+0000","last_became_active":"2024-04-15T15:14:51.076124+0000","last_became_peered":"2024-04-15T15:14:51.076124+0000","last_unstale":"2024-04-15T15:14:57.136936+0000","last_undegraded":"2024-04-15T15:14:57.136936+0000","last_fullsized":"2024-04-15T15:14:57.136936+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:31:57.289767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137379+0000","last_change":"2024-04-15T15:14:56.251375+0000","last_active":"2024-04-15T15:14:57.137379+0000","last_peered":"2024-04-15T15:14:57.137379+0000","last_clean":"2024-04-15T15:14:57.137379+0000","last_became_active":"2024-04-15T15:14:56.250973+0000","last_became_peered":"2024-04-15T15:14:56.250973+0000","last_unstale":"2024-04-15T15:14:57.137379+0000","last_undegraded":"2024-04-15T15:14:57.137379+0000","last_fullsized":"2024-04-15T15:14:57.137379+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:58:01.690949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.110314+0000","last_change":"2024-04-15T15:14:52.082863+0000","last_active":"2024-04-15T15:14:57.110314+0000","last_peered":"2024-04-15T15:14:57.110314+0000","last_clean":"2024-04-15T15:14:57.110314+0000","last_became_active":"2024-04-15T15:14:52.082370+0000","last_became_peered":"2024-04-15T15:14:52.082370+0000","last_unstale":"2024-04-15T15:14:57.110314+0000","last_undegraded":"2024-04-15T15:14:57.110314+0000","last_fullsized":"2024-04-15T15:14:57.110314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:08.475460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137335+0000","last_change":"2024-04-15T15:14:51.073016+0000","last_active":"2024-04-15T15:14:57.137335+0000","last_peered":"2024-04-15T15:14:57.137335+0000","last_clean":"2024-04-15T15:14:57.137335+0000","last_became_active":"2024-04-15T15:14:51.072736+0000","last_became_peered":"2024-04-15T15:14:51.072736+0000","last_unstale":"2024-04-15T15:14:57.137335+0000","last_undegraded":"2024-04-15T15:14:57.137335+0000","last_fullsized":"2024-04-15T15:14:57.137335+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:44:33.685298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136866+0000","last_change":"2024-04-15T15:14:56.105315+0000","last_active":"2024-04-15T15:14:57.136866+0000","last_peered":"2024-04-15T15:14:57.136866+0000","last_clean":"2024-04-15T15:14:57.136866+0000","last_became_active":"2024-04-15T15:14:56.104923+0000","last_became_peered":"2024-04-15T15:14:56.104923+0000","last_unstale":"2024-04-15T15:14:57.136866+0000","last_undegraded":"2024-04-15T15:14:57.136866+0000","last_fullsized":"2024-04-15T15:14:57.136866+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:01:28.704663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136866+0000","last_change":"2024-04-15T15:14:52.078353+0000","last_active":"2024-04-15T15:14:57.136866+0000","last_peered":"2024-04-15T15:14:57.136866+0000","last_clean":"2024-04-15T15:14:57.136866+0000","last_became_active":"2024-04-15T15:14:52.076556+0000","last_became_peered":"2024-04-15T15:14:52.076556+0000","last_unstale":"2024-04-15T15:14:57.136866+0000","last_undegraded":"2024-04-15T15:14:57.136866+0000","last_fullsized":"2024-04-15T15:14:57.136866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:19:38.992004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109062+0000","last_change":"2024-04-15T15:14:51.072875+0000","last_active":"2024-04-15T15:14:57.109062+0000","last_peered":"2024-04-15T15:14:57.109062+0000","last_clean":"2024-04-15T15:14:57.109062+0000","last_became_active":"2024-04-15T15:14:51.072114+0000","last_became_peered":"2024-04-15T15:14:51.072114+0000","last_unstale":"2024-04-15T15:14:57.109062+0000","last_undegraded":"2024-04-15T15:14:57.109062+0000","last_fullsized":"2024-04-15T15:14:57.109062+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:18:41.399059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137344+0000","last_change":"2024-04-15T15:14:56.098426+0000","last_active":"2024-04-15T15:14:57.137344+0000","last_peered":"2024-04-15T15:14:57.137344+0000","last_clean":"2024-04-15T15:14:57.137344+0000","last_became_active":"2024-04-15T15:14:56.098066+0000","last_became_peered":"2024-04-15T15:14:56.098066+0000","last_unstale":"2024-04-15T15:14:57.137344+0000","last_undegraded":"2024-04-15T15:14:57.137344+0000","last_fullsized":"2024-04-15T15:14:57.137344+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:12.500763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136427+0000","last_change":"2024-04-15T15:14:52.077474+0000","last_active":"2024-04-15T15:14:57.136427+0000","last_peered":"2024-04-15T15:14:57.136427+0000","last_clean":"2024-04-15T15:14:57.136427+0000","last_became_active":"2024-04-15T15:14:52.077206+0000","last_became_peered":"2024-04-15T15:14:52.077206+0000","last_unstale":"2024-04-15T15:14:57.136427+0000","last_undegraded":"2024-04-15T15:14:57.136427+0000","last_fullsized":"2024-04-15T15:14:57.136427+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:34:51.775979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.107727+0000","last_change":"2024-04-15T15:14:51.067544+0000","last_active":"2024-04-15T15:14:57.107727+0000","last_peered":"2024-04-15T15:14:57.107727+0000","last_clean":"2024-04-15T15:14:57.107727+0000","last_became_active":"2024-04-15T15:14:51.067233+0000","last_became_peered":"2024-04-15T15:14:51.067233+0000","last_unstale":"2024-04-15T15:14:57.107727+0000","last_undegraded":"2024-04-15T15:14:57.107727+0000","last_fullsized":"2024-04-15T15:14:57.107727+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:44:33.685298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137463+0000","last_change":"2024-04-15T15:14:56.103911+0000","last_active":"2024-04-15T15:14:57.137463+0000","last_peered":"2024-04-15T15:14:57.137463+0000","last_clean":"2024-04-15T15:14:57.137463+0000","last_became_active":"2024-04-15T15:14:56.103035+0000","last_became_peered":"2024-04-15T15:14:56.103035+0000","last_unstale":"2024-04-15T15:14:57.137463+0000","last_undegraded":"2024-04-15T15:14:57.137463+0000","last_fullsized":"2024-04-15T15:14:57.137463+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:21:52.495367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136227+0000","last_change":"2024-04-15T15:14:52.078870+0000","last_active":"2024-04-15T15:14:57.136227+0000","last_peered":"2024-04-15T15:14:57.136227+0000","last_clean":"2024-04-15T15:14:57.136227+0000","last_became_active":"2024-04-15T15:14:52.078573+0000","last_became_peered":"2024-04-15T15:14:52.078573+0000","last_unstale":"2024-04-15T15:14:57.136227+0000","last_undegraded":"2024-04-15T15:14:57.136227+0000","last_fullsized":"2024-04-15T15:14:57.136227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:51:36.740221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135636+0000","last_change":"2024-04-15T15:14:51.077104+0000","last_active":"2024-04-15T15:14:57.135636+0000","last_peered":"2024-04-15T15:14:57.135636+0000","last_clean":"2024-04-15T15:14:57.135636+0000","last_became_active":"2024-04-15T15:14:51.076857+0000","last_became_peered":"2024-04-15T15:14:51.076857+0000","last_unstale":"2024-04-15T15:14:57.135636+0000","last_undegraded":"2024-04-15T15:14:57.135636+0000","last_fullsized":"2024-04-15T15:14:57.135636+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:18:41.399059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138372+0000","last_change":"2024-04-15T15:14:56.251372+0000","last_active":"2024-04-15T15:14:57.138372+0000","last_peered":"2024-04-15T15:14:57.138372+0000","last_clean":"2024-04-15T15:14:57.138372+0000","last_became_active":"2024-04-15T15:14:56.250961+0000","last_became_peered":"2024-04-15T15:14:56.250961+0000","last_unstale":"2024-04-15T15:14:57.138372+0000","last_undegraded":"2024-04-15T15:14:57.138372+0000","last_fullsized":"2024-04-15T15:14:57.138372+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:13:03.031872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136619+0000","last_change":"2024-04-15T15:14:52.089029+0000","last_active":"2024-04-15T15:14:57.136619+0000","last_peered":"2024-04-15T15:14:57.136619+0000","last_clean":"2024-04-15T15:14:57.136619+0000","last_became_active":"2024-04-15T15:14:52.088646+0000","last_became_peered":"2024-04-15T15:14:52.088646+0000","last_unstale":"2024-04-15T15:14:57.136619+0000","last_undegraded":"2024-04-15T15:14:57.136619+0000","last_fullsized":"2024-04-15T15:14:57.136619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:37:31.656434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108973+0000","last_change":"2024-04-15T15:14:51.067071+0000","last_active":"2024-04-15T15:14:57.108973+0000","last_peered":"2024-04-15T15:14:57.108973+0000","last_clean":"2024-04-15T15:14:57.108973+0000","last_became_active":"2024-04-15T15:14:51.066382+0000","last_became_peered":"2024-04-15T15:14:51.066382+0000","last_unstale":"2024-04-15T15:14:57.108973+0000","last_undegraded":"2024-04-15T15:14:57.108973+0000","last_fullsized":"2024-04-15T15:14:57.108973+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:50:01.422527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135699+0000","last_change":"2024-04-15T15:14:40.494279+0000","last_active":"2024-04-15T15:14:57.135699+0000","last_peered":"2024-04-15T15:14:57.135699+0000","last_clean":"2024-04-15T15:14:57.135699+0000","last_became_active":"2024-04-15T15:14:40.493404+0000","last_became_peered":"2024-04-15T15:14:40.493404+0000","last_unstale":"2024-04-15T15:14:57.135699+0000","last_undegraded":"2024-04-15T15:14:57.135699+0000","last_fullsized":"2024-04-15T15:14:57.135699+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:39.418508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:39.418508+0000","last_clean_scrub_stamp":"2024-04-15T15:14:39.418508+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:01:51.789246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109541+0000","last_change":"2024-04-15T15:14:56.253853+0000","last_active":"2024-04-15T15:14:57.109541+0000","last_peered":"2024-04-15T15:14:57.109541+0000","last_clean":"2024-04-15T15:14:57.109541+0000","last_became_active":"2024-04-15T15:14:56.253323+0000","last_became_peered":"2024-04-15T15:14:56.253323+0000","last_unstale":"2024-04-15T15:14:57.109541+0000","last_undegraded":"2024-04-15T15:14:57.109541+0000","last_fullsized":"2024-04-15T15:14:57.109541+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:03:07.876012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136598+0000","last_change":"2024-04-15T15:14:52.093103+0000","last_active":"2024-04-15T15:14:57.136598+0000","last_peered":"2024-04-15T15:14:57.136598+0000","last_clean":"2024-04-15T15:14:57.136598+0000","last_became_active":"2024-04-15T15:14:52.092786+0000","last_became_peered":"2024-04-15T15:14:52.092786+0000","last_unstale":"2024-04-15T15:14:57.136598+0000","last_undegraded":"2024-04-15T15:14:57.136598+0000","last_fullsized":"2024-04-15T15:14:57.136598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:57:57.665646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108044+0000","last_change":"2024-04-15T15:14:51.067545+0000","last_active":"2024-04-15T15:14:57.108044+0000","last_peered":"2024-04-15T15:14:57.108044+0000","last_clean":"2024-04-15T15:14:57.108044+0000","last_became_active":"2024-04-15T15:14:51.067229+0000","last_became_peered":"2024-04-15T15:14:51.067229+0000","last_unstale":"2024-04-15T15:14:57.108044+0000","last_undegraded":"2024-04-15T15:14:57.108044+0000","last_fullsized":"2024-04-15T15:14:57.108044+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:41:20.990702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108641+0000","last_change":"2024-04-15T15:14:56.251780+0000","last_active":"2024-04-15T15:14:57.108641+0000","last_peered":"2024-04-15T15:14:57.108641+0000","last_clean":"2024-04-15T15:14:57.108641+0000","last_became_active":"2024-04-15T15:14:56.251079+0000","last_became_peered":"2024-04-15T15:14:56.251079+0000","last_unstale":"2024-04-15T15:14:57.108641+0000","last_undegraded":"2024-04-15T15:14:57.108641+0000","last_fullsized":"2024-04-15T15:14:57.108641+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:12.500763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109714+0000","last_change":"2024-04-15T15:14:52.074261+0000","last_active":"2024-04-15T15:14:57.109714+0000","last_peered":"2024-04-15T15:14:57.109714+0000","last_clean":"2024-04-15T15:14:57.109714+0000","last_became_active":"2024-04-15T15:14:52.073979+0000","last_became_peered":"2024-04-15T15:14:52.073979+0000","last_unstale":"2024-04-15T15:14:57.109714+0000","last_undegraded":"2024-04-15T15:14:57.109714+0000","last_fullsized":"2024-04-15T15:14:57.109714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:03:03.850709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137650+0000","last_change":"2024-04-15T15:14:51.074612+0000","last_active":"2024-04-15T15:14:57.137650+0000","last_peered":"2024-04-15T15:14:57.137650+0000","last_clean":"2024-04-15T15:14:57.137650+0000","last_became_active":"2024-04-15T15:14:51.074176+0000","last_became_peered":"2024-04-15T15:14:51.074176+0000","last_unstale":"2024-04-15T15:14:57.137650+0000","last_undegraded":"2024-04-15T15:14:57.137650+0000","last_fullsized":"2024-04-15T15:14:57.137650+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:50:01.422527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137604+0000","last_change":"2024-04-15T15:14:56.103129+0000","last_active":"2024-04-15T15:14:57.137604+0000","last_peered":"2024-04-15T15:14:57.137604+0000","last_clean":"2024-04-15T15:14:57.137604+0000","last_became_active":"2024-04-15T15:14:56.102757+0000","last_became_peered":"2024-04-15T15:14:56.102757+0000","last_unstale":"2024-04-15T15:14:57.137604+0000","last_undegraded":"2024-04-15T15:14:57.137604+0000","last_fullsized":"2024-04-15T15:14:57.137604+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:12.500763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136942+0000","last_change":"2024-04-15T15:14:52.087082+0000","last_active":"2024-04-15T15:14:57.136942+0000","last_peered":"2024-04-15T15:14:57.136942+0000","last_clean":"2024-04-15T15:14:57.136942+0000","last_became_active":"2024-04-15T15:14:52.086517+0000","last_became_peered":"2024-04-15T15:14:52.086517+0000","last_unstale":"2024-04-15T15:14:57.136942+0000","last_undegraded":"2024-04-15T15:14:57.136942+0000","last_fullsized":"2024-04-15T15:14:57.136942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:08:15.196752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.107969+0000","last_change":"2024-04-15T15:14:51.067941+0000","last_active":"2024-04-15T15:14:57.107969+0000","last_peered":"2024-04-15T15:14:57.107969+0000","last_clean":"2024-04-15T15:14:57.107969+0000","last_became_active":"2024-04-15T15:14:51.067669+0000","last_became_peered":"2024-04-15T15:14:51.067669+0000","last_unstale":"2024-04-15T15:14:57.107969+0000","last_undegraded":"2024-04-15T15:14:57.107969+0000","last_fullsized":"2024-04-15T15:14:57.107969+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:50:01.422527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108366+0000","last_change":"2024-04-15T15:14:56.252233+0000","last_active":"2024-04-15T15:14:57.108366+0000","last_peered":"2024-04-15T15:14:57.108366+0000","last_clean":"2024-04-15T15:14:57.108366+0000","last_became_active":"2024-04-15T15:14:56.251084+0000","last_became_peered":"2024-04-15T15:14:56.251084+0000","last_unstale":"2024-04-15T15:14:57.108366+0000","last_undegraded":"2024-04-15T15:14:57.108366+0000","last_fullsized":"2024-04-15T15:14:57.108366+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:42:46.671056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136568+0000","last_change":"2024-04-15T15:14:52.085987+0000","last_active":"2024-04-15T15:14:57.136568+0000","last_peered":"2024-04-15T15:14:57.136568+0000","last_clean":"2024-04-15T15:14:57.136568+0000","last_became_active":"2024-04-15T15:14:52.085579+0000","last_became_peered":"2024-04-15T15:14:52.085579+0000","last_unstale":"2024-04-15T15:14:57.136568+0000","last_undegraded":"2024-04-15T15:14:57.136568+0000","last_fullsized":"2024-04-15T15:14:57.136568+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:08.475460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136963+0000","last_change":"2024-04-15T15:14:51.066803+0000","last_active":"2024-04-15T15:14:57.136963+0000","last_peered":"2024-04-15T15:14:57.136963+0000","last_clean":"2024-04-15T15:14:57.136963+0000","last_became_active":"2024-04-15T15:14:51.066103+0000","last_became_peered":"2024-04-15T15:14:51.066103+0000","last_unstale":"2024-04-15T15:14:57.136963+0000","last_undegraded":"2024-04-15T15:14:57.136963+0000","last_fullsized":"2024-04-15T15:14:57.136963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:35:13.836817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135900+0000","last_change":"2024-04-15T15:14:56.098601+0000","last_active":"2024-04-15T15:14:57.135900+0000","last_peered":"2024-04-15T15:14:57.135900+0000","last_clean":"2024-04-15T15:14:57.135900+0000","last_became_active":"2024-04-15T15:14:56.098191+0000","last_became_peered":"2024-04-15T15:14:56.098191+0000","last_unstale":"2024-04-15T15:14:57.135900+0000","last_undegraded":"2024-04-15T15:14:57.135900+0000","last_fullsized":"2024-04-15T15:14:57.135900+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:12:11.797852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135812+0000","last_change":"2024-04-15T15:14:52.086933+0000","last_active":"2024-04-15T15:14:57.135812+0000","last_peered":"2024-04-15T15:14:57.135812+0000","last_clean":"2024-04-15T15:14:57.135812+0000","last_became_active":"2024-04-15T15:14:52.086546+0000","last_became_peered":"2024-04-15T15:14:52.086546+0000","last_unstale":"2024-04-15T15:14:57.135812+0000","last_undegraded":"2024-04-15T15:14:57.135812+0000","last_fullsized":"2024-04-15T15:14:57.135812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:10:19.696254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.107592+0000","last_change":"2024-04-15T15:14:51.071716+0000","last_active":"2024-04-15T15:14:57.107592+0000","last_peered":"2024-04-15T15:14:57.107592+0000","last_clean":"2024-04-15T15:14:57.107592+0000","last_became_active":"2024-04-15T15:14:51.071084+0000","last_became_peered":"2024-04-15T15:14:51.071084+0000","last_unstale":"2024-04-15T15:14:57.107592+0000","last_undegraded":"2024-04-15T15:14:57.107592+0000","last_fullsized":"2024-04-15T15:14:57.107592+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:01:16.772802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138211+0000","last_change":"2024-04-15T15:14:56.102060+0000","last_active":"2024-04-15T15:14:57.138211+0000","last_peered":"2024-04-15T15:14:57.138211+0000","last_clean":"2024-04-15T15:14:57.138211+0000","last_became_active":"2024-04-15T15:14:56.101644+0000","last_became_peered":"2024-04-15T15:14:56.101644+0000","last_unstale":"2024-04-15T15:14:57.138211+0000","last_undegraded":"2024-04-15T15:14:57.138211+0000","last_fullsized":"2024-04-15T15:14:57.138211+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:03:07.876012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109139+0000","last_change":"2024-04-15T15:14:52.088688+0000","last_active":"2024-04-15T15:14:57.109139+0000","last_peered":"2024-04-15T15:14:57.109139+0000","last_clean":"2024-04-15T15:14:57.109139+0000","last_became_active":"2024-04-15T15:14:52.088433+0000","last_became_peered":"2024-04-15T15:14:52.088433+0000","last_unstale":"2024-04-15T15:14:57.109139+0000","last_undegraded":"2024-04-15T15:14:57.109139+0000","last_fullsized":"2024-04-15T15:14:57.109139+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:24:30.714153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.110238+0000","last_change":"2024-04-15T15:14:51.073931+0000","last_active":"2024-04-15T15:14:57.110238+0000","last_peered":"2024-04-15T15:14:57.110238+0000","last_clean":"2024-04-15T15:14:57.110238+0000","last_became_active":"2024-04-15T15:14:51.073571+0000","last_became_peered":"2024-04-15T15:14:51.073571+0000","last_unstale":"2024-04-15T15:14:57.110238+0000","last_undegraded":"2024-04-15T15:14:57.110238+0000","last_fullsized":"2024-04-15T15:14:57.110238+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:44:33.685298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138190+0000","last_change":"2024-04-15T15:14:56.251177+0000","last_active":"2024-04-15T15:14:57.138190+0000","last_peered":"2024-04-15T15:14:57.138190+0000","last_clean":"2024-04-15T15:14:57.138190+0000","last_became_active":"2024-04-15T15:14:56.250803+0000","last_became_peered":"2024-04-15T15:14:56.250803+0000","last_unstale":"2024-04-15T15:14:57.138190+0000","last_undegraded":"2024-04-15T15:14:57.138190+0000","last_fullsized":"2024-04-15T15:14:57.138190+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:39:36.394347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136175+0000","last_change":"2024-04-15T15:14:52.086912+0000","last_active":"2024-04-15T15:14:57.136175+0000","last_peered":"2024-04-15T15:14:57.136175+0000","last_clean":"2024-04-15T15:14:57.136175+0000","last_became_active":"2024-04-15T15:14:52.085621+0000","last_became_peered":"2024-04-15T15:14:52.085621+0000","last_unstale":"2024-04-15T15:14:57.136175+0000","last_undegraded":"2024-04-15T15:14:57.136175+0000","last_fullsized":"2024-04-15T15:14:57.136175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:38:35.185619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108968+0000","last_change":"2024-04-15T15:14:51.068349+0000","last_active":"2024-04-15T15:14:57.108968+0000","last_peered":"2024-04-15T15:14:57.108968+0000","last_clean":"2024-04-15T15:14:57.108968+0000","last_became_active":"2024-04-15T15:14:51.068078+0000","last_became_peered":"2024-04-15T15:14:51.068078+0000","last_unstale":"2024-04-15T15:14:57.108968+0000","last_undegraded":"2024-04-15T15:14:57.108968+0000","last_fullsized":"2024-04-15T15:14:57.108968+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:47:54.990677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136871+0000","last_change":"2024-04-15T15:14:56.251089+0000","last_active":"2024-04-15T15:14:57.136871+0000","last_peered":"2024-04-15T15:14:57.136871+0000","last_clean":"2024-04-15T15:14:57.136871+0000","last_became_active":"2024-04-15T15:14:56.250735+0000","last_became_peered":"2024-04-15T15:14:56.250735+0000","last_unstale":"2024-04-15T15:14:57.136871+0000","last_undegraded":"2024-04-15T15:14:57.136871+0000","last_fullsized":"2024-04-15T15:14:57.136871+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:39:38.108475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138515+0000","last_change":"2024-04-15T15:14:52.076410+0000","last_active":"2024-04-15T15:14:57.138515+0000","last_peered":"2024-04-15T15:14:57.138515+0000","last_clean":"2024-04-15T15:14:57.138515+0000","last_became_active":"2024-04-15T15:14:52.076115+0000","last_became_peered":"2024-04-15T15:14:52.076115+0000","last_unstale":"2024-04-15T15:14:57.138515+0000","last_undegraded":"2024-04-15T15:14:57.138515+0000","last_fullsized":"2024-04-15T15:14:57.138515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:48:45.790927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137841+0000","last_change":"2024-04-15T15:14:51.072889+0000","last_active":"2024-04-15T15:14:57.137841+0000","last_peered":"2024-04-15T15:14:57.137841+0000","last_clean":"2024-04-15T15:14:57.137841+0000","last_became_active":"2024-04-15T15:14:51.072611+0000","last_became_peered":"2024-04-15T15:14:51.072611+0000","last_unstale":"2024-04-15T15:14:57.137841+0000","last_undegraded":"2024-04-15T15:14:57.137841+0000","last_fullsized":"2024-04-15T15:14:57.137841+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:20:40.577242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.110168+0000","last_change":"2024-04-15T15:14:56.253070+0000","last_active":"2024-04-15T15:14:57.110168+0000","last_peered":"2024-04-15T15:14:57.110168+0000","last_clean":"2024-04-15T15:14:57.110168+0000","last_became_active":"2024-04-15T15:14:56.252652+0000","last_became_peered":"2024-04-15T15:14:56.252652+0000","last_unstale":"2024-04-15T15:14:57.110168+0000","last_undegraded":"2024-04-15T15:14:57.110168+0000","last_fullsized":"2024-04-15T15:14:57.110168+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:34:50.166643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136518+0000","last_change":"2024-04-15T15:14:52.092734+0000","last_active":"2024-04-15T15:14:57.136518+0000","last_peered":"2024-04-15T15:14:57.136518+0000","last_clean":"2024-04-15T15:14:57.136518+0000","last_became_active":"2024-04-15T15:14:52.092105+0000","last_became_peered":"2024-04-15T15:14:52.092105+0000","last_unstale":"2024-04-15T15:14:57.136518+0000","last_undegraded":"2024-04-15T15:14:57.136518+0000","last_fullsized":"2024-04-15T15:14:57.136518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:01:24.679360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:15:01.576821+0000","last_change":"2024-04-15T15:14:51.072503+0000","last_active":"2024-04-15T15:15:01.576821+0000","last_peered":"2024-04-15T15:15:01.576821+0000","last_clean":"2024-04-15T15:15:01.576821+0000","last_became_active":"2024-04-15T15:14:51.072212+0000","last_became_peered":"2024-04-15T15:14:51.072212+0000","last_unstale":"2024-04-15T15:15:01.576821+0000","last_undegraded":"2024-04-15T15:15:01.576821+0000","last_fullsized":"2024-04-15T15:15:01.576821+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:18:41.399059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137069+0000","last_change":"2024-04-15T15:14:56.105406+0000","last_active":"2024-04-15T15:14:57.137069+0000","last_peered":"2024-04-15T15:14:57.137069+0000","last_clean":"2024-04-15T15:14:57.137069+0000","last_became_active":"2024-04-15T15:14:56.104951+0000","last_became_peered":"2024-04-15T15:14:56.104951+0000","last_unstale":"2024-04-15T15:14:57.137069+0000","last_undegraded":"2024-04-15T15:14:57.137069+0000","last_fullsized":"2024-04-15T15:14:57.137069+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:58:01.690949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136751+0000","last_change":"2024-04-15T15:14:52.083396+0000","last_active":"2024-04-15T15:14:57.136751+0000","last_peered":"2024-04-15T15:14:57.136751+0000","last_clean":"2024-04-15T15:14:57.136751+0000","last_became_active":"2024-04-15T15:14:52.082661+0000","last_became_peered":"2024-04-15T15:14:52.082661+0000","last_unstale":"2024-04-15T15:14:57.136751+0000","last_undegraded":"2024-04-15T15:14:57.136751+0000","last_fullsized":"2024-04-15T15:14:57.136751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:07:49.420517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137624+0000","last_change":"2024-04-15T15:14:51.074372+0000","last_active":"2024-04-15T15:14:57.137624+0000","last_peered":"2024-04-15T15:14:57.137624+0000","last_clean":"2024-04-15T15:14:57.137624+0000","last_became_active":"2024-04-15T15:14:51.074030+0000","last_became_peered":"2024-04-15T15:14:51.074030+0000","last_unstale":"2024-04-15T15:14:57.137624+0000","last_undegraded":"2024-04-15T15:14:57.137624+0000","last_fullsized":"2024-04-15T15:14:57.137624+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:47:54.990677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136760+0000","last_change":"2024-04-15T15:14:56.099051+0000","last_active":"2024-04-15T15:14:57.136760+0000","last_peered":"2024-04-15T15:14:57.136760+0000","last_clean":"2024-04-15T15:14:57.136760+0000","last_became_active":"2024-04-15T15:14:56.098034+0000","last_became_peered":"2024-04-15T15:14:56.098034+0000","last_unstale":"2024-04-15T15:14:57.136760+0000","last_undegraded":"2024-04-15T15:14:57.136760+0000","last_fullsized":"2024-04-15T15:14:57.136760+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:58:01.690949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136894+0000","last_change":"2024-04-15T15:14:52.084508+0000","last_active":"2024-04-15T15:14:57.136894+0000","last_peered":"2024-04-15T15:14:57.136894+0000","last_clean":"2024-04-15T15:14:57.136894+0000","last_became_active":"2024-04-15T15:14:52.083550+0000","last_became_peered":"2024-04-15T15:14:52.083550+0000","last_unstale":"2024-04-15T15:14:57.136894+0000","last_undegraded":"2024-04-15T15:14:57.136894+0000","last_fullsized":"2024-04-15T15:14:57.136894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:12:40.420221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108919+0000","last_change":"2024-04-15T15:14:51.071093+0000","last_active":"2024-04-15T15:14:57.108919+0000","last_peered":"2024-04-15T15:14:57.108919+0000","last_clean":"2024-04-15T15:14:57.108919+0000","last_became_active":"2024-04-15T15:14:51.070810+0000","last_became_peered":"2024-04-15T15:14:51.070810+0000","last_unstale":"2024-04-15T15:14:57.108919+0000","last_undegraded":"2024-04-15T15:14:57.108919+0000","last_fullsized":"2024-04-15T15:14:57.108919+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:41:20.990702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136272+0000","last_change":"2024-04-15T15:14:56.092919+0000","last_active":"2024-04-15T15:14:57.136272+0000","last_peered":"2024-04-15T15:14:57.136272+0000","last_clean":"2024-04-15T15:14:57.136272+0000","last_became_active":"2024-04-15T15:14:56.092459+0000","last_became_peered":"2024-04-15T15:14:56.092459+0000","last_unstale":"2024-04-15T15:14:57.136272+0000","last_undegraded":"2024-04-15T15:14:57.136272+0000","last_fullsized":"2024-04-15T15:14:57.136272+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:46:38.024003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108815+0000","last_change":"2024-04-15T15:14:52.079880+0000","last_active":"2024-04-15T15:14:57.108815+0000","last_peered":"2024-04-15T15:14:57.108815+0000","last_clean":"2024-04-15T15:14:57.108815+0000","last_became_active":"2024-04-15T15:14:52.079619+0000","last_became_peered":"2024-04-15T15:14:52.079619+0000","last_unstale":"2024-04-15T15:14:57.108815+0000","last_undegraded":"2024-04-15T15:14:57.108815+0000","last_fullsized":"2024-04-15T15:14:57.108815+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:21:48.470063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109438+0000","last_change":"2024-04-15T15:14:51.075415+0000","last_active":"2024-04-15T15:14:57.109438+0000","last_peered":"2024-04-15T15:14:57.109438+0000","last_clean":"2024-04-15T15:14:57.109438+0000","last_became_active":"2024-04-15T15:14:51.075014+0000","last_became_peered":"2024-04-15T15:14:51.075014+0000","last_unstale":"2024-04-15T15:14:57.109438+0000","last_undegraded":"2024-04-15T15:14:57.109438+0000","last_fullsized":"2024-04-15T15:14:57.109438+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:41:20.990702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135982+0000","last_change":"2024-04-15T15:14:56.253630+0000","last_active":"2024-04-15T15:14:57.135982+0000","last_peered":"2024-04-15T15:14:57.135982+0000","last_clean":"2024-04-15T15:14:57.135982+0000","last_became_active":"2024-04-15T15:14:56.253253+0000","last_became_peered":"2024-04-15T15:14:56.253253+0000","last_unstale":"2024-04-15T15:14:57.135982+0000","last_undegraded":"2024-04-15T15:14:57.135982+0000","last_fullsized":"2024-04-15T15:14:57.135982+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:34:50.166643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138051+0000","last_change":"2024-04-15T15:14:56.253163+0000","last_active":"2024-04-15T15:14:57.138051+0000","last_peered":"2024-04-15T15:14:57.138051+0000","last_clean":"2024-04-15T15:14:57.138051+0000","last_became_active":"2024-04-15T15:14:56.252769+0000","last_became_peered":"2024-04-15T15:14:56.252769+0000","last_unstale":"2024-04-15T15:14:57.138051+0000","last_undegraded":"2024-04-15T15:14:57.138051+0000","last_fullsized":"2024-04-15T15:14:57.138051+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:21:52.495367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135905+0000","last_change":"2024-04-15T15:14:52.086725+0000","last_active":"2024-04-15T15:14:57.135905+0000","last_peered":"2024-04-15T15:14:57.135905+0000","last_clean":"2024-04-15T15:14:57.135905+0000","last_became_active":"2024-04-15T15:14:52.086220+0000","last_became_peered":"2024-04-15T15:14:52.086220+0000","last_unstale":"2024-04-15T15:14:57.135905+0000","last_undegraded":"2024-04-15T15:14:57.135905+0000","last_fullsized":"2024-04-15T15:14:57.135905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:21:50.111549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108433+0000","last_change":"2024-04-15T15:14:51.069916+0000","last_active":"2024-04-15T15:14:57.108433+0000","last_peered":"2024-04-15T15:14:57.108433+0000","last_clean":"2024-04-15T15:14:57.108433+0000","last_became_active":"2024-04-15T15:14:51.069631+0000","last_became_peered":"2024-04-15T15:14:51.069631+0000","last_unstale":"2024-04-15T15:14:57.108433+0000","last_undegraded":"2024-04-15T15:14:57.108433+0000","last_fullsized":"2024-04-15T15:14:57.108433+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:44:33.685298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109306+0000","last_change":"2024-04-15T15:14:56.102205+0000","last_active":"2024-04-15T15:14:57.109306+0000","last_peered":"2024-04-15T15:14:57.109306+0000","last_clean":"2024-04-15T15:14:57.109306+0000","last_became_active":"2024-04-15T15:14:56.101827+0000","last_became_peered":"2024-04-15T15:14:56.101827+0000","last_unstale":"2024-04-15T15:14:57.109306+0000","last_undegraded":"2024-04-15T15:14:57.109306+0000","last_fullsized":"2024-04-15T15:14:57.109306+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:21:52.495367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136314+0000","last_change":"2024-04-15T15:14:52.079898+0000","last_active":"2024-04-15T15:14:57.136314+0000","last_peered":"2024-04-15T15:14:57.136314+0000","last_clean":"2024-04-15T15:14:57.136314+0000","last_became_active":"2024-04-15T15:14:52.079430+0000","last_became_peered":"2024-04-15T15:14:52.079430+0000","last_unstale":"2024-04-15T15:14:57.136314+0000","last_undegraded":"2024-04-15T15:14:57.136314+0000","last_fullsized":"2024-04-15T15:14:57.136314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:12:07.772549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136279+0000","last_change":"2024-04-15T15:14:51.069720+0000","last_active":"2024-04-15T15:14:57.136279+0000","last_peered":"2024-04-15T15:14:57.136279+0000","last_clean":"2024-04-15T15:14:57.136279+0000","last_became_active":"2024-04-15T15:14:51.069416+0000","last_became_peered":"2024-04-15T15:14:51.069416+0000","last_unstale":"2024-04-15T15:14:57.136279+0000","last_undegraded":"2024-04-15T15:14:57.136279+0000","last_fullsized":"2024-04-15T15:14:57.136279+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:20:40.577242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108868+0000","last_change":"2024-04-15T15:14:51.070913+0000","last_active":"2024-04-15T15:14:57.108868+0000","last_peered":"2024-04-15T15:14:57.108868+0000","last_clean":"2024-04-15T15:14:57.108868+0000","last_became_active":"2024-04-15T15:14:51.070640+0000","last_became_peered":"2024-04-15T15:14:51.070640+0000","last_unstale":"2024-04-15T15:14:57.108868+0000","last_undegraded":"2024-04-15T15:14:57.108868+0000","last_fullsized":"2024-04-15T15:14:57.108868+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:19:37.987965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137741+0000","last_change":"2024-04-15T15:14:52.078627+0000","last_active":"2024-04-15T15:14:57.137741+0000","last_peered":"2024-04-15T15:14:57.137741+0000","last_clean":"2024-04-15T15:14:57.137741+0000","last_became_active":"2024-04-15T15:14:52.078333+0000","last_became_peered":"2024-04-15T15:14:52.078333+0000","last_unstale":"2024-04-15T15:14:57.137741+0000","last_undegraded":"2024-04-15T15:14:57.137741+0000","last_fullsized":"2024-04-15T15:14:57.137741+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:50:41.263447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108003+0000","last_change":"2024-04-15T15:14:56.093858+0000","last_active":"2024-04-15T15:14:57.108003+0000","last_peered":"2024-04-15T15:14:57.108003+0000","last_clean":"2024-04-15T15:14:57.108003+0000","last_became_active":"2024-04-15T15:14:56.093481+0000","last_became_peered":"2024-04-15T15:14:56.093481+0000","last_unstale":"2024-04-15T15:14:57.108003+0000","last_undegraded":"2024-04-15T15:14:57.108003+0000","last_fullsized":"2024-04-15T15:14:57.108003+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:48:49.816230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137031+0000","last_change":"2024-04-15T15:14:51.073707+0000","last_active":"2024-04-15T15:14:57.137031+0000","last_peered":"2024-04-15T15:14:57.137031+0000","last_clean":"2024-04-15T15:14:57.137031+0000","last_became_active":"2024-04-15T15:14:51.073451+0000","last_became_peered":"2024-04-15T15:14:51.073451+0000","last_unstale":"2024-04-15T15:14:57.137031+0000","last_undegraded":"2024-04-15T15:14:57.137031+0000","last_fullsized":"2024-04-15T15:14:57.137031+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:01:16.772802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136421+0000","last_change":"2024-04-15T15:14:52.078644+0000","last_active":"2024-04-15T15:14:57.136421+0000","last_peered":"2024-04-15T15:14:57.136421+0000","last_clean":"2024-04-15T15:14:57.136421+0000","last_became_active":"2024-04-15T15:14:52.078177+0000","last_became_peered":"2024-04-15T15:14:52.078177+0000","last_unstale":"2024-04-15T15:14:57.136421+0000","last_undegraded":"2024-04-15T15:14:57.136421+0000","last_fullsized":"2024-04-15T15:14:57.136421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:18:42.403098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136932+0000","last_change":"2024-04-15T15:14:56.092926+0000","last_active":"2024-04-15T15:14:57.136932+0000","last_peered":"2024-04-15T15:14:57.136932+0000","last_clean":"2024-04-15T15:14:57.136932+0000","last_became_active":"2024-04-15T15:14:56.092506+0000","last_became_peered":"2024-04-15T15:14:56.092506+0000","last_unstale":"2024-04-15T15:14:57.136932+0000","last_undegraded":"2024-04-15T15:14:57.136932+0000","last_fullsized":"2024-04-15T15:14:57.136932+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:55:14.590939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108967+0000","last_change":"2024-04-15T15:14:52.082533+0000","last_active":"2024-04-15T15:14:57.108967+0000","last_peered":"2024-04-15T15:14:57.108967+0000","last_clean":"2024-04-15T15:14:57.108967+0000","last_became_active":"2024-04-15T15:14:52.082257+0000","last_became_peered":"2024-04-15T15:14:52.082257+0000","last_unstale":"2024-04-15T15:14:57.108967+0000","last_undegraded":"2024-04-15T15:14:57.108967+0000","last_fullsized":"2024-04-15T15:14:57.108967+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:34:51.775979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.107671+0000","last_change":"2024-04-15T15:14:52.084944+0000","last_active":"2024-04-15T15:14:57.107671+0000","last_peered":"2024-04-15T15:14:57.107671+0000","last_clean":"2024-04-15T15:14:57.107671+0000","last_became_active":"2024-04-15T15:14:52.084665+0000","last_became_peered":"2024-04-15T15:14:52.084665+0000","last_unstale":"2024-04-15T15:14:57.107671+0000","last_undegraded":"2024-04-15T15:14:57.107671+0000","last_fullsized":"2024-04-15T15:14:57.107671+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:58:41.444122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108873+0000","last_change":"2024-04-15T15:14:52.082741+0000","last_active":"2024-04-15T15:14:57.108873+0000","last_peered":"2024-04-15T15:14:57.108873+0000","last_clean":"2024-04-15T15:14:57.108873+0000","last_became_active":"2024-04-15T15:14:52.082494+0000","last_became_peered":"2024-04-15T15:14:52.082494+0000","last_unstale":"2024-04-15T15:14:57.108873+0000","last_undegraded":"2024-04-15T15:14:57.108873+0000","last_fullsized":"2024-04-15T15:14:57.108873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:38:35.185619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137649+0000","last_change":"2024-04-15T15:14:52.076826+0000","last_active":"2024-04-15T15:14:57.137649+0000","last_peered":"2024-04-15T15:14:57.137649+0000","last_clean":"2024-04-15T15:14:57.137649+0000","last_became_active":"2024-04-15T15:14:52.076541+0000","last_became_peered":"2024-04-15T15:14:52.076541+0000","last_unstale":"2024-04-15T15:14:57.137649+0000","last_undegraded":"2024-04-15T15:14:57.137649+0000","last_fullsized":"2024-04-15T15:14:57.137649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:13:01.642838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137024+0000","last_change":"2024-04-15T15:14:52.075468+0000","last_active":"2024-04-15T15:14:57.137024+0000","last_peered":"2024-04-15T15:14:57.137024+0000","last_clean":"2024-04-15T15:14:57.137024+0000","last_became_active":"2024-04-15T15:14:52.075173+0000","last_became_peered":"2024-04-15T15:14:52.075173+0000","last_unstale":"2024-04-15T15:14:57.137024+0000","last_undegraded":"2024-04-15T15:14:57.137024+0000","last_fullsized":"2024-04-15T15:14:57.137024+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:50:41.263447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137555+0000","last_change":"2024-04-15T15:14:52.086866+0000","last_active":"2024-04-15T15:14:57.137555+0000","last_peered":"2024-04-15T15:14:57.137555+0000","last_clean":"2024-04-15T15:14:57.137555+0000","last_became_active":"2024-04-15T15:14:52.086547+0000","last_became_peered":"2024-04-15T15:14:52.086547+0000","last_unstale":"2024-04-15T15:14:57.137555+0000","last_undegraded":"2024-04-15T15:14:57.137555+0000","last_fullsized":"2024-04-15T15:14:57.137555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:37:31.656434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.110000+0000","last_change":"2024-04-15T15:14:52.074917+0000","last_active":"2024-04-15T15:14:57.110000+0000","last_peered":"2024-04-15T15:14:57.110000+0000","last_clean":"2024-04-15T15:14:57.110000+0000","last_became_active":"2024-04-15T15:14:52.074643+0000","last_became_peered":"2024-04-15T15:14:52.074643+0000","last_unstale":"2024-04-15T15:14:57.110000+0000","last_undegraded":"2024-04-15T15:14:57.110000+0000","last_fullsized":"2024-04-15T15:14:57.110000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:12:07.772549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138133+0000","last_change":"2024-04-15T15:14:52.076263+0000","last_active":"2024-04-15T15:14:57.138133+0000","last_peered":"2024-04-15T15:14:57.138133+0000","last_clean":"2024-04-15T15:14:57.138133+0000","last_became_active":"2024-04-15T15:14:52.075961+0000","last_became_peered":"2024-04-15T15:14:52.075961+0000","last_unstale":"2024-04-15T15:14:57.138133+0000","last_undegraded":"2024-04-15T15:14:57.138133+0000","last_fullsized":"2024-04-15T15:14:57.138133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:12:40.420221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108421+0000","last_change":"2024-04-15T15:14:52.074353+0000","last_active":"2024-04-15T15:14:57.108421+0000","last_peered":"2024-04-15T15:14:57.108421+0000","last_clean":"2024-04-15T15:14:57.108421+0000","last_became_active":"2024-04-15T15:14:52.074067+0000","last_became_peered":"2024-04-15T15:14:52.074067+0000","last_unstale":"2024-04-15T15:14:57.108421+0000","last_undegraded":"2024-04-15T15:14:57.108421+0000","last_fullsized":"2024-04-15T15:14:57.108421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:26:45.460787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136083+0000","last_change":"2024-04-15T15:14:52.075541+0000","last_active":"2024-04-15T15:14:57.136083+0000","last_peered":"2024-04-15T15:14:57.136083+0000","last_clean":"2024-04-15T15:14:57.136083+0000","last_became_active":"2024-04-15T15:14:52.075227+0000","last_became_peered":"2024-04-15T15:14:52.075227+0000","last_unstale":"2024-04-15T15:14:57.136083+0000","last_undegraded":"2024-04-15T15:14:57.136083+0000","last_fullsized":"2024-04-15T15:14:57.136083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:24:30.714153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108791+0000","last_change":"2024-04-15T15:14:52.080306+0000","last_active":"2024-04-15T15:14:57.108791+0000","last_peered":"2024-04-15T15:14:57.108791+0000","last_clean":"2024-04-15T15:14:57.108791+0000","last_became_active":"2024-04-15T15:14:52.079784+0000","last_became_peered":"2024-04-15T15:14:52.079784+0000","last_unstale":"2024-04-15T15:14:57.108791+0000","last_undegraded":"2024-04-15T15:14:57.108791+0000","last_fullsized":"2024-04-15T15:14:57.108791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:10:19.696254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137426+0000","last_change":"2024-04-15T15:14:52.081224+0000","last_active":"2024-04-15T15:14:57.137426+0000","last_peered":"2024-04-15T15:14:57.137426+0000","last_clean":"2024-04-15T15:14:57.137426+0000","last_became_active":"2024-04-15T15:14:52.080976+0000","last_became_peered":"2024-04-15T15:14:52.080976+0000","last_unstale":"2024-04-15T15:14:57.137426+0000","last_undegraded":"2024-04-15T15:14:57.137426+0000","last_fullsized":"2024-04-15T15:14:57.137426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:21:50.111549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109464+0000","last_change":"2024-04-15T15:14:52.085463+0000","last_active":"2024-04-15T15:14:57.109464+0000","last_peered":"2024-04-15T15:14:57.109464+0000","last_clean":"2024-04-15T15:14:57.109464+0000","last_became_active":"2024-04-15T15:14:52.085175+0000","last_became_peered":"2024-04-15T15:14:52.085175+0000","last_unstale":"2024-04-15T15:14:57.109464+0000","last_undegraded":"2024-04-15T15:14:57.109464+0000","last_fullsized":"2024-04-15T15:14:57.109464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:50:41.263447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136838+0000","last_change":"2024-04-15T15:14:52.092434+0000","last_active":"2024-04-15T15:14:57.136838+0000","last_peered":"2024-04-15T15:14:57.136838+0000","last_clean":"2024-04-15T15:14:57.136838+0000","last_became_active":"2024-04-15T15:14:52.088973+0000","last_became_peered":"2024-04-15T15:14:52.088973+0000","last_unstale":"2024-04-15T15:14:57.136838+0000","last_undegraded":"2024-04-15T15:14:57.136838+0000","last_fullsized":"2024-04-15T15:14:57.136838+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:03:03.850709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.107875+0000","last_change":"2024-04-15T15:14:52.076802+0000","last_active":"2024-04-15T15:14:57.107875+0000","last_peered":"2024-04-15T15:14:57.107875+0000","last_clean":"2024-04-15T15:14:57.107875+0000","last_became_active":"2024-04-15T15:14:52.076523+0000","last_became_peered":"2024-04-15T15:14:52.076523+0000","last_unstale":"2024-04-15T15:14:57.107875+0000","last_undegraded":"2024-04-15T15:14:57.107875+0000","last_fullsized":"2024-04-15T15:14:57.107875+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:07:49.420517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109362+0000","last_change":"2024-04-15T15:14:52.084418+0000","last_active":"2024-04-15T15:14:57.109362+0000","last_peered":"2024-04-15T15:14:57.109362+0000","last_clean":"2024-04-15T15:14:57.109362+0000","last_became_active":"2024-04-15T15:14:52.083428+0000","last_became_peered":"2024-04-15T15:14:52.083428+0000","last_unstale":"2024-04-15T15:14:57.109362+0000","last_undegraded":"2024-04-15T15:14:57.109362+0000","last_fullsized":"2024-04-15T15:14:57.109362+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:12:59.006568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108279+0000","last_change":"2024-04-15T15:14:52.083894+0000","last_active":"2024-04-15T15:14:57.108279+0000","last_peered":"2024-04-15T15:14:57.108279+0000","last_clean":"2024-04-15T15:14:57.108279+0000","last_became_active":"2024-04-15T15:14:52.083636+0000","last_became_peered":"2024-04-15T15:14:52.083636+0000","last_unstale":"2024-04-15T15:14:57.108279+0000","last_undegraded":"2024-04-15T15:14:57.108279+0000","last_fullsized":"2024-04-15T15:14:57.108279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:12:07.772549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136007+0000","last_change":"2024-04-15T15:14:52.087934+0000","last_active":"2024-04-15T15:14:57.136007+0000","last_peered":"2024-04-15T15:14:57.136007+0000","last_clean":"2024-04-15T15:14:57.136007+0000","last_became_active":"2024-04-15T15:14:52.087040+0000","last_became_peered":"2024-04-15T15:14:52.087040+0000","last_unstale":"2024-04-15T15:14:57.136007+0000","last_undegraded":"2024-04-15T15:14:57.136007+0000","last_fullsized":"2024-04-15T15:14:57.136007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:24:30.714153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138511+0000","last_change":"2024-04-15T15:14:52.079696+0000","last_active":"2024-04-15T15:14:57.138511+0000","last_peered":"2024-04-15T15:14:57.138511+0000","last_clean":"2024-04-15T15:14:57.138511+0000","last_became_active":"2024-04-15T15:14:52.079438+0000","last_became_peered":"2024-04-15T15:14:52.079438+0000","last_unstale":"2024-04-15T15:14:57.138511+0000","last_undegraded":"2024-04-15T15:14:57.138511+0000","last_fullsized":"2024-04-15T15:14:57.138511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:27:02.854265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136589+0000","last_change":"2024-04-15T15:14:52.076841+0000","last_active":"2024-04-15T15:14:57.136589+0000","last_peered":"2024-04-15T15:14:57.136589+0000","last_clean":"2024-04-15T15:14:57.136589+0000","last_became_active":"2024-04-15T15:14:52.076567+0000","last_became_peered":"2024-04-15T15:14:52.076567+0000","last_unstale":"2024-04-15T15:14:57.136589+0000","last_undegraded":"2024-04-15T15:14:57.136589+0000","last_fullsized":"2024-04-15T15:14:57.136589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:51:22.814881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136685+0000","last_change":"2024-04-15T15:14:52.078599+0000","last_active":"2024-04-15T15:14:57.136685+0000","last_peered":"2024-04-15T15:14:57.136685+0000","last_clean":"2024-04-15T15:14:57.136685+0000","last_became_active":"2024-04-15T15:14:52.078231+0000","last_became_peered":"2024-04-15T15:14:52.078231+0000","last_unstale":"2024-04-15T15:14:57.136685+0000","last_undegraded":"2024-04-15T15:14:57.136685+0000","last_fullsized":"2024-04-15T15:14:57.136685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:58:41.444122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109712+0000","last_change":"2024-04-15T15:14:52.080747+0000","last_active":"2024-04-15T15:14:57.109712+0000","last_peered":"2024-04-15T15:14:57.109712+0000","last_clean":"2024-04-15T15:14:57.109712+0000","last_became_active":"2024-04-15T15:14:52.080475+0000","last_became_peered":"2024-04-15T15:14:52.080475+0000","last_unstale":"2024-04-15T15:14:57.109712+0000","last_undegraded":"2024-04-15T15:14:57.109712+0000","last_fullsized":"2024-04-15T15:14:57.109712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:37:31.656434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109996+0000","last_change":"2024-04-15T15:14:52.081405+0000","last_active":"2024-04-15T15:14:57.109996+0000","last_peered":"2024-04-15T15:14:57.109996+0000","last_clean":"2024-04-15T15:14:57.109996+0000","last_became_active":"2024-04-15T15:14:52.081120+0000","last_became_peered":"2024-04-15T15:14:52.081120+0000","last_unstale":"2024-04-15T15:14:57.109996+0000","last_undegraded":"2024-04-15T15:14:57.109996+0000","last_fullsized":"2024-04-15T15:14:57.109996+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:12:40.420221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136834+0000","last_change":"2024-04-15T15:14:52.092337+0000","last_active":"2024-04-15T15:14:57.136834+0000","last_peered":"2024-04-15T15:14:57.136834+0000","last_clean":"2024-04-15T15:14:57.136834+0000","last_became_active":"2024-04-15T15:14:52.091800+0000","last_became_peered":"2024-04-15T15:14:52.091800+0000","last_unstale":"2024-04-15T15:14:57.136834+0000","last_undegraded":"2024-04-15T15:14:57.136834+0000","last_fullsized":"2024-04-15T15:14:57.136834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:12:59.006568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136083+0000","last_change":"2024-04-15T15:14:52.079078+0000","last_active":"2024-04-15T15:14:57.136083+0000","last_peered":"2024-04-15T15:14:57.136083+0000","last_clean":"2024-04-15T15:14:57.136083+0000","last_became_active":"2024-04-15T15:14:52.078796+0000","last_became_peered":"2024-04-15T15:14:52.078796+0000","last_unstale":"2024-04-15T15:14:57.136083+0000","last_undegraded":"2024-04-15T15:14:57.136083+0000","last_fullsized":"2024-04-15T15:14:57.136083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:34:51.775979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135766+0000","last_change":"2024-04-15T15:14:52.091742+0000","last_active":"2024-04-15T15:14:57.135766+0000","last_peered":"2024-04-15T15:14:57.135766+0000","last_clean":"2024-04-15T15:14:57.135766+0000","last_became_active":"2024-04-15T15:14:52.087579+0000","last_became_peered":"2024-04-15T15:14:52.087579+0000","last_unstale":"2024-04-15T15:14:57.135766+0000","last_undegraded":"2024-04-15T15:14:57.135766+0000","last_fullsized":"2024-04-15T15:14:57.135766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:51:22.814881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108707+0000","last_change":"2024-04-15T15:14:52.084878+0000","last_active":"2024-04-15T15:14:57.108707+0000","last_peered":"2024-04-15T15:14:57.108707+0000","last_clean":"2024-04-15T15:14:57.108707+0000","last_became_active":"2024-04-15T15:14:52.084602+0000","last_became_peered":"2024-04-15T15:14:52.084602+0000","last_unstale":"2024-04-15T15:14:57.108707+0000","last_undegraded":"2024-04-15T15:14:57.108707+0000","last_fullsized":"2024-04-15T15:14:57.108707+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:51:22.814881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136011+0000","last_change":"2024-04-15T15:14:52.085909+0000","last_active":"2024-04-15T15:14:57.136011+0000","last_peered":"2024-04-15T15:14:57.136011+0000","last_clean":"2024-04-15T15:14:57.136011+0000","last_became_active":"2024-04-15T15:14:52.085266+0000","last_became_peered":"2024-04-15T15:14:52.085266+0000","last_unstale":"2024-04-15T15:14:57.136011+0000","last_undegraded":"2024-04-15T15:14:57.136011+0000","last_fullsized":"2024-04-15T15:14:57.136011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:26:45.460787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109544+0000","last_change":"2024-04-15T15:14:52.084086+0000","last_active":"2024-04-15T15:14:57.109544+0000","last_peered":"2024-04-15T15:14:57.109544+0000","last_clean":"2024-04-15T15:14:57.109544+0000","last_became_active":"2024-04-15T15:14:52.083193+0000","last_became_peered":"2024-04-15T15:14:52.083193+0000","last_unstale":"2024-04-15T15:14:57.109544+0000","last_undegraded":"2024-04-15T15:14:57.109544+0000","last_fullsized":"2024-04-15T15:14:57.109544+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:12:40.420221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136555+0000","last_change":"2024-04-15T15:14:52.072448+0000","last_active":"2024-04-15T15:14:57.136555+0000","last_peered":"2024-04-15T15:14:57.136555+0000","last_clean":"2024-04-15T15:14:57.136555+0000","last_became_active":"2024-04-15T15:14:52.072162+0000","last_became_peered":"2024-04-15T15:14:52.072162+0000","last_unstale":"2024-04-15T15:14:57.136555+0000","last_undegraded":"2024-04-15T15:14:57.136555+0000","last_fullsized":"2024-04-15T15:14:57.136555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:24:30.714153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137467+0000","last_change":"2024-04-15T15:14:52.087484+0000","last_active":"2024-04-15T15:14:57.137467+0000","last_peered":"2024-04-15T15:14:57.137467+0000","last_clean":"2024-04-15T15:14:57.137467+0000","last_became_active":"2024-04-15T15:14:52.087195+0000","last_became_peered":"2024-04-15T15:14:52.087195+0000","last_unstale":"2024-04-15T15:14:57.137467+0000","last_undegraded":"2024-04-15T15:14:57.137467+0000","last_fullsized":"2024-04-15T15:14:57.137467+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:50:41.263447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136721+0000","last_change":"2024-04-15T15:14:52.083998+0000","last_active":"2024-04-15T15:14:57.136721+0000","last_peered":"2024-04-15T15:14:57.136721+0000","last_clean":"2024-04-15T15:14:57.136721+0000","last_became_active":"2024-04-15T15:14:52.083702+0000","last_became_peered":"2024-04-15T15:14:52.083702+0000","last_unstale":"2024-04-15T15:14:57.136721+0000","last_undegraded":"2024-04-15T15:14:57.136721+0000","last_fullsized":"2024-04-15T15:14:57.136721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:21:48.470063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:15:13.517 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:15:13.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.652+0000 7f0bf76d5640 1 Processor -- start 2024-04-15T15:15:13.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.652+0000 7f0bf76d5640 1 -- start start 2024-04-15T15:15:13.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.653+0000 7f0bf76d5640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0bf00c9570 0x7f0bf0162b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:13.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.653+0000 7f0bf76d5640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bf00ccd40 con 0x7f0bf00d12f0 2024-04-15T15:15:13.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.653+0000 7f0bf76d5640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bf00cd020 con 0x7f0bf0161fb0 2024-04-15T15:15:13.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.653+0000 7f0bf76d5640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f0bf00ccaf0 con 0x7f0bf00c9570 2024-04-15T15:15:13.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.653+0000 7f0bf5471640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f0bf00d12f0 0x7f0bf01618a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37894/0 (socket says 172.21.15.72:37894) 2024-04-15T15:15:13.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.653+0000 7f0bf5471640 1 -- 172.21.15.72:0/2130862905 learned_addr learned my addr 172.21.15.72:0/2130862905 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:13.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.653+0000 7f0bf5c72640 1 --2- 172.21.15.72:0/2130862905 >> v2:172.21.15.175:3300/0 conn(0x7f0bf00c9570 0x7f0bf0162b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:13.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.653+0000 7f0be7fff640 1 -- 172.21.15.72:0/2130862905 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 479542153 0 0) 0x7f0bf00ccd40 con 0x7f0bf00d12f0 2024-04-15T15:15:13.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.653+0000 7f0be7fff640 1 -- 172.21.15.72:0/2130862905 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0bdc003520 con 0x7f0bf00d12f0 2024-04-15T15:15:13.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.654+0000 7f0be7fff640 1 -- 172.21.15.72:0/2130862905 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3035604965 0 0) 0x7f0bdc003520 con 0x7f0bf00d12f0 2024-04-15T15:15:13.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.654+0000 7f0be7fff640 1 -- 172.21.15.72:0/2130862905 >> v2:172.21.15.175:3300/0 conn(0x7f0bf00c9570 msgr2=0x7f0bf0162b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:13.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.654+0000 7f0be7fff640 1 --2- 172.21.15.72:0/2130862905 >> v2:172.21.15.175:3300/0 conn(0x7f0bf00c9570 0x7f0bf0162b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.654+0000 7f0be7fff640 1 -- 172.21.15.72:0/2130862905 >> v1:172.21.15.175:6789/0 conn(0x7f0bf0161fb0 legacy=0x7f0bf0162450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:13.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.654+0000 7f0be7fff640 1 -- 172.21.15.72:0/2130862905 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bf00cd300 con 0x7f0bf00d12f0 2024-04-15T15:15:13.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.654+0000 7f0be7fff640 1 -- 172.21.15.72:0/2130862905 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f0bec001670 con 0x7f0bf00d12f0 2024-04-15T15:15:13.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.654+0000 7f0be7fff640 1 -- 172.21.15.72:0/2130862905 >> v1:172.21.15.72:6789/0 conn(0x7f0bf00d12f0 legacy=0x7f0bf01618a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:13.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.654+0000 7f0be7fff640 1 --2- 172.21.15.72:0/2130862905 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0bdc0039b0 0x7f0bdc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:13.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.654+0000 7f0be7fff640 1 --2- 172.21.15.72:0/2130862905 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0bdc0043b0 0x7f0bdc004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:13.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.654+0000 7f0be7fff640 1 --2- 172.21.15.72:0/2130862905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bdc005080 0x7f0bdc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:13.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.654+0000 7f0be7fff640 1 -- 172.21.15.72:0/2130862905 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf00ccaf0 con 0x7f0bdc005080 2024-04-15T15:15:13.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.654+0000 7f0be7fff640 1 -- 172.21.15.72:0/2130862905 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f0bdc00bfe0 con 0x7f0bdc0043b0 2024-04-15T15:15:13.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.654+0000 7f0be7fff640 1 -- 172.21.15.72:0/2130862905 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f0bdc00c1e0 con 0x7f0bdc0039b0 2024-04-15T15:15:13.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.654+0000 7f0bf5471640 1 --2- 172.21.15.72:0/2130862905 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0bdc0039b0 0x7f0bdc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:13.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.654+0000 7f0bf5c72640 1 --2- 172.21.15.72:0/2130862905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bdc005080 0x7f0bdc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:13.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.654+0000 7f0bf4c70640 1 --2- 172.21.15.72:0/2130862905 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0bdc0043b0 0x7f0bdc004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:13.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.654+0000 7f0bf5c72640 1 -- 172.21.15.72:0/2130862905 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0bdc0039b0 msgr2=0x7f0bdc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:13.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.654+0000 7f0bf5c72640 1 --2- 172.21.15.72:0/2130862905 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0bdc0039b0 0x7f0bdc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.654+0000 7f0bf5c72640 1 -- 172.21.15.72:0/2130862905 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0bdc0043b0 msgr2=0x7f0bdc004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:13.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.654+0000 7f0bf5c72640 1 --2- 172.21.15.72:0/2130862905 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0bdc0043b0 0x7f0bdc004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.654+0000 7f0bf5c72640 1 -- 172.21.15.72:0/2130862905 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bdc003520 con 0x7f0bdc005080 2024-04-15T15:15:13.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.655+0000 7f0bf5c72640 1 -- 172.21.15.72:0/2130862905 reap_dead start 2024-04-15T15:15:13.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.655+0000 7f0bf5c72640 1 --2- 172.21.15.72:0/2130862905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bdc005080 0x7f0bdc00b9c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0be8009f90 tx=0x7f0bdc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19437d36569570d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:13.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.655+0000 7f0be7fff640 1 -- 172.21.15.72:0/2130862905 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be8039070 con 0x7f0bdc005080 2024-04-15T15:15:13.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.655+0000 7f0be7fff640 1 -- 172.21.15.72:0/2130862905 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0be802cc20 con 0x7f0bdc005080 2024-04-15T15:15:13.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.655+0000 7f0be7fff640 1 -- 172.21.15.72:0/2130862905 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be8009c10 con 0x7f0bdc005080 2024-04-15T15:15:13.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.656+0000 7f0bf76d5640 1 -- 172.21.15.72:0/2130862905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bdc005080 msgr2=0x7f0bdc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:13.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.656+0000 7f0bf76d5640 1 --2- 172.21.15.72:0/2130862905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bdc005080 0x7f0bdc00b9c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0be8009f90 tx=0x7f0bdc005830 comp rx=0 tx=0).stop 2024-04-15T15:15:13.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.656+0000 7f0bf76d5640 1 -- 172.21.15.72:0/2130862905 shutdown_connections 2024-04-15T15:15:13.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.656+0000 7f0bf76d5640 1 --2- 172.21.15.72:0/2130862905 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bdc005080 0x7f0bdc00b9c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.656+0000 7f0bf76d5640 1 -- 172.21.15.72:0/2130862905 >> 172.21.15.72:0/2130862905 conn(0x7f0bf00b3f90 msgr2=0x7f0bf00b43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:13.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.656+0000 7f0bf76d5640 1 -- 172.21.15.72:0/2130862905 shutdown_connections 2024-04-15T15:15:13.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.656+0000 7f0bf76d5640 1 -- 172.21.15.72:0/2130862905 wait complete. 2024-04-15T15:15:13.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.657+0000 7f0bf76d5640 1 Processor -- start 2024-04-15T15:15:13.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.657+0000 7f0bf76d5640 1 -- start start 2024-04-15T15:15:13.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.658+0000 7f0bf76d5640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0bdc005080 0x7f0bf01f4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:13.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.658+0000 7f0bf76d5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bf00c72c0 0x7f0bf00c85f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:13.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.658+0000 7f0bf76d5640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0bf00c7740 0x7f0bf00c8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:13.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.658+0000 7f0bf76d5640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf00d2210 con 0x7f0bf00c72c0 2024-04-15T15:15:13.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.658+0000 7f0bf76d5640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf00d2090 con 0x7f0bdc005080 2024-04-15T15:15:13.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.658+0000 7f0bf76d5640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f0bf00b68e0 con 0x7f0bf00c7740 2024-04-15T15:15:13.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.658+0000 7f0bf4c70640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bf00c72c0 0x7f0bf00c85f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:13.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.658+0000 7f0bf4c70640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bf00c72c0 0x7f0bf00c85f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45478/0 (socket says 172.21.15.72:45478) 2024-04-15T15:15:13.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.658+0000 7f0bf4c70640 1 -- 172.21.15.72:0/216818221 learned_addr learned my addr 172.21.15.72:0/216818221 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:13.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.658+0000 7f0bf4c70640 1 -- 172.21.15.72:0/216818221 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0bf00c7740 msgr2=0x7f0bf00c8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:13.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.658+0000 7f0bf4c70640 1 --2- 172.21.15.72:0/216818221 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0bf00c7740 0x7f0bf00c8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.658+0000 7f0bf4c70640 1 -- 172.21.15.72:0/216818221 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0bdc005080 msgr2=0x7f0bf01f4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:13.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.658+0000 7f0bf4c70640 1 --2- 172.21.15.72:0/216818221 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0bdc005080 0x7f0bf01f4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.658+0000 7f0bf4c70640 1 -- 172.21.15.72:0/216818221 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bf0145dd0 con 0x7f0bf00c72c0 2024-04-15T15:15:13.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.658+0000 7f0bf4c70640 1 --2- 172.21.15.72:0/216818221 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bf00c72c0 0x7f0bf00c85f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0be000f820 tx=0x7f0be000fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:13.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.658+0000 7f0be5ffb640 1 -- 172.21.15.72:0/216818221 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be00050d0 con 0x7f0bf00c72c0 2024-04-15T15:15:13.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.658+0000 7f0be5ffb640 1 -- 172.21.15.72:0/216818221 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0be0018070 con 0x7f0bf00c72c0 2024-04-15T15:15:13.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.658+0000 7f0bf76d5640 1 -- 172.21.15.72:0/216818221 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bf00c6220 con 0x7f0bf00c72c0 2024-04-15T15:15:13.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.658+0000 7f0be5ffb640 1 -- 172.21.15.72:0/216818221 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be000d6d0 con 0x7f0bf00c72c0 2024-04-15T15:15:13.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.658+0000 7f0bf76d5640 1 -- 172.21.15.72:0/216818221 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bf00c67e0 con 0x7f0bf00c72c0 2024-04-15T15:15:13.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.659+0000 7f0bf76d5640 1 -- 172.21.15.72:0/216818221 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bf00689e0 con 0x7f0bf00c72c0 2024-04-15T15:15:13.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.660+0000 7f0be5ffb640 1 -- 172.21.15.72:0/216818221 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0be000d870 con 0x7f0bf00c72c0 2024-04-15T15:15:13.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.660+0000 7f0be5ffb640 1 --2- 172.21.15.72:0/216818221 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0bd406c920 0x7f0bd406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:13.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.660+0000 7f0be5ffb640 1 -- 172.21.15.72:0/216818221 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f0be008b880 con 0x7f0bf00c72c0 2024-04-15T15:15:13.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.660+0000 7f0bf5471640 1 --2- 172.21.15.72:0/216818221 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0bd406c920 0x7f0bd406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:13.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.661+0000 7f0bf5471640 1 --2- 172.21.15.72:0/216818221 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0bd406c920 0x7f0bd406ede0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0bec001ba0 tx=0x7f0bec001b10 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:13.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.664+0000 7f0be5ffb640 1 -- 172.21.15.72:0/216818221 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0be0059970 con 0x7f0bf00c72c0 2024-04-15T15:15:13.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.838+0000 7f0bf76d5640 1 -- 172.21.15.72:0/216818221 --> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0bf00ce730 con 0x7f0bd406c920 2024-04-15T15:15:13.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.845+0000 7f0be5ffb640 1 -- 172.21.15.72:0/216818221 <== mgr.4101 v2:172.21.15.72:6832/3503807805 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332534 (secure 0 0 0) 0x7f0bf00ce730 con 0x7f0bd406c920 2024-04-15T15:15:13.850 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:13.852 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-04-15T15:15:13.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.849+0000 7f0bf76d5640 1 -- 172.21.15.72:0/216818221 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0bd406c920 msgr2=0x7f0bd406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:13.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.849+0000 7f0bf76d5640 1 --2- 172.21.15.72:0/216818221 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0bd406c920 0x7f0bd406ede0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0bec001ba0 tx=0x7f0bec001b10 comp rx=0 tx=0).stop 2024-04-15T15:15:13.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.849+0000 7f0bf76d5640 1 -- 172.21.15.72:0/216818221 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bf00c72c0 msgr2=0x7f0bf00c85f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:13.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.849+0000 7f0bf76d5640 1 --2- 172.21.15.72:0/216818221 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bf00c72c0 0x7f0bf00c85f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f0be000f820 tx=0x7f0be000fce0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.849+0000 7f0bf76d5640 1 -- 172.21.15.72:0/216818221 shutdown_connections 2024-04-15T15:15:13.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.849+0000 7f0bf76d5640 1 --2- 172.21.15.72:0/216818221 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0bd406c920 0x7f0bd406ede0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.849+0000 7f0bf76d5640 1 --2- 172.21.15.72:0/216818221 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0bf00c7740 0x7f0bf00c8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.849+0000 7f0bf76d5640 1 --2- 172.21.15.72:0/216818221 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0bf00c72c0 0x7f0bf00c85f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.849+0000 7f0bf76d5640 1 --2- 172.21.15.72:0/216818221 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0bdc005080 0x7f0bf01f4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.849+0000 7f0bf76d5640 1 -- 172.21.15.72:0/216818221 >> 172.21.15.72:0/216818221 conn(0x7f0bf00b3f90 msgr2=0x7f0bf00c9d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:13.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.849+0000 7f0bf76d5640 1 -- 172.21.15.72:0/216818221 shutdown_connections 2024-04-15T15:15:13.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:13.849+0000 7f0bf76d5640 1 -- 172.21.15.72:0/216818221 wait complete. 2024-04-15T15:15:13.861 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-04-15T15:15:13.348367+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219392,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755328,"statfs":{"total":773094113280,"available":772869455872,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002938"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":26,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137048+0000","last_change":"2024-04-15T15:14:51.075131+0000","last_active":"2024-04-15T15:14:57.137048+0000","last_peered":"2024-04-15T15:14:57.137048+0000","last_clean":"2024-04-15T15:14:57.137048+0000","last_became_active":"2024-04-15T15:14:51.074841+0000","last_became_peered":"2024-04-15T15:14:51.074841+0000","last_unstale":"2024-04-15T15:14:57.137048+0000","last_undegraded":"2024-04-15T15:14:57.137048+0000","last_fullsized":"2024-04-15T15:14:57.137048+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:19:56.231155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138223+0000","last_change":"2024-04-15T15:14:52.077480+0000","last_active":"2024-04-15T15:14:57.138223+0000","last_peered":"2024-04-15T15:14:57.138223+0000","last_clean":"2024-04-15T15:14:57.138223+0000","last_became_active":"2024-04-15T15:14:52.077166+0000","last_became_peered":"2024-04-15T15:14:52.077166+0000","last_unstale":"2024-04-15T15:14:57.138223+0000","last_undegraded":"2024-04-15T15:14:57.138223+0000","last_fullsized":"2024-04-15T15:14:57.138223+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:13:01.642838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136181+0000","last_change":"2024-04-15T15:14:56.255424+0000","last_active":"2024-04-15T15:14:57.136181+0000","last_peered":"2024-04-15T15:14:57.136181+0000","last_clean":"2024-04-15T15:14:57.136181+0000","last_became_active":"2024-04-15T15:14:56.254987+0000","last_became_peered":"2024-04-15T15:14:56.254987+0000","last_unstale":"2024-04-15T15:14:57.136181+0000","last_undegraded":"2024-04-15T15:14:57.136181+0000","last_fullsized":"2024-04-15T15:14:57.136181+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:11:24.196989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138454+0000","last_change":"2024-04-15T15:14:51.071376+0000","last_active":"2024-04-15T15:14:57.138454+0000","last_peered":"2024-04-15T15:14:57.138454+0000","last_clean":"2024-04-15T15:14:57.138454+0000","last_became_active":"2024-04-15T15:14:51.071108+0000","last_became_peered":"2024-04-15T15:14:51.071108+0000","last_unstale":"2024-04-15T15:14:57.138454+0000","last_undegraded":"2024-04-15T15:14:57.138454+0000","last_fullsized":"2024-04-15T15:14:57.138454+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:50:01.422527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136732+0000","last_change":"2024-04-15T15:14:52.083839+0000","last_active":"2024-04-15T15:14:57.136732+0000","last_peered":"2024-04-15T15:14:57.136732+0000","last_clean":"2024-04-15T15:14:57.136732+0000","last_became_active":"2024-04-15T15:14:52.083533+0000","last_became_peered":"2024-04-15T15:14:52.083533+0000","last_unstale":"2024-04-15T15:14:57.136732+0000","last_undegraded":"2024-04-15T15:14:57.136732+0000","last_fullsized":"2024-04-15T15:14:57.136732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:34:51.775979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135872+0000","last_change":"2024-04-15T15:14:56.098375+0000","last_active":"2024-04-15T15:14:57.135872+0000","last_peered":"2024-04-15T15:14:57.135872+0000","last_clean":"2024-04-15T15:14:57.135872+0000","last_became_active":"2024-04-15T15:14:56.097969+0000","last_became_peered":"2024-04-15T15:14:56.097969+0000","last_unstale":"2024-04-15T15:14:57.135872+0000","last_undegraded":"2024-04-15T15:14:57.135872+0000","last_fullsized":"2024-04-15T15:14:57.135872+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:12:44.445524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136739+0000","last_change":"2024-04-15T15:14:51.079869+0000","last_active":"2024-04-15T15:14:57.136739+0000","last_peered":"2024-04-15T15:14:57.136739+0000","last_clean":"2024-04-15T15:14:57.136739+0000","last_became_active":"2024-04-15T15:14:51.079576+0000","last_became_peered":"2024-04-15T15:14:51.079576+0000","last_unstale":"2024-04-15T15:14:57.136739+0000","last_undegraded":"2024-04-15T15:14:57.136739+0000","last_fullsized":"2024-04-15T15:14:57.136739+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:58:40.440083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109631+0000","last_change":"2024-04-15T15:14:52.075307+0000","last_active":"2024-04-15T15:14:57.109631+0000","last_peered":"2024-04-15T15:14:57.109631+0000","last_clean":"2024-04-15T15:14:57.109631+0000","last_became_active":"2024-04-15T15:14:52.075001+0000","last_became_peered":"2024-04-15T15:14:52.075001+0000","last_unstale":"2024-04-15T15:14:57.109631+0000","last_undegraded":"2024-04-15T15:14:57.109631+0000","last_fullsized":"2024-04-15T15:14:57.109631+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:01:24.679360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136399+0000","last_change":"2024-04-15T15:14:56.098292+0000","last_active":"2024-04-15T15:14:57.136399+0000","last_peered":"2024-04-15T15:14:57.136399+0000","last_clean":"2024-04-15T15:14:57.136399+0000","last_became_active":"2024-04-15T15:14:56.097848+0000","last_became_peered":"2024-04-15T15:14:56.097848+0000","last_unstale":"2024-04-15T15:14:57.136399+0000","last_undegraded":"2024-04-15T15:14:57.136399+0000","last_fullsized":"2024-04-15T15:14:57.136399+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:26:49.486090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136078+0000","last_change":"2024-04-15T15:14:51.079754+0000","last_active":"2024-04-15T15:14:57.136078+0000","last_peered":"2024-04-15T15:14:57.136078+0000","last_clean":"2024-04-15T15:14:57.136078+0000","last_became_active":"2024-04-15T15:14:51.079452+0000","last_became_peered":"2024-04-15T15:14:51.079452+0000","last_unstale":"2024-04-15T15:14:57.136078+0000","last_undegraded":"2024-04-15T15:14:57.136078+0000","last_fullsized":"2024-04-15T15:14:57.136078+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:19:37.987965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.110221+0000","last_change":"2024-04-15T15:14:52.076654+0000","last_active":"2024-04-15T15:14:57.110221+0000","last_peered":"2024-04-15T15:14:57.110221+0000","last_clean":"2024-04-15T15:14:57.110221+0000","last_became_active":"2024-04-15T15:14:52.076283+0000","last_became_peered":"2024-04-15T15:14:52.076283+0000","last_unstale":"2024-04-15T15:14:57.110221+0000","last_undegraded":"2024-04-15T15:14:57.110221+0000","last_fullsized":"2024-04-15T15:14:57.110221+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:57:57.665646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136259+0000","last_change":"2024-04-15T15:14:56.105257+0000","last_active":"2024-04-15T15:14:57.136259+0000","last_peered":"2024-04-15T15:14:57.136259+0000","last_clean":"2024-04-15T15:14:57.136259+0000","last_became_active":"2024-04-15T15:14:56.104856+0000","last_became_peered":"2024-04-15T15:14:56.104856+0000","last_unstale":"2024-04-15T15:14:57.136259+0000","last_undegraded":"2024-04-15T15:14:57.136259+0000","last_fullsized":"2024-04-15T15:14:57.136259+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:10:23.721558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108699+0000","last_change":"2024-04-15T15:14:52.086412+0000","last_active":"2024-04-15T15:14:57.108699+0000","last_peered":"2024-04-15T15:14:57.108699+0000","last_clean":"2024-04-15T15:14:57.108699+0000","last_became_active":"2024-04-15T15:14:52.085479+0000","last_became_peered":"2024-04-15T15:14:52.085479+0000","last_unstale":"2024-04-15T15:14:57.108699+0000","last_undegraded":"2024-04-15T15:14:57.108699+0000","last_fullsized":"2024-04-15T15:14:57.108699+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:13:01.642838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136325+0000","last_change":"2024-04-15T15:14:51.080109+0000","last_active":"2024-04-15T15:14:57.136325+0000","last_peered":"2024-04-15T15:14:57.136325+0000","last_clean":"2024-04-15T15:14:57.136325+0000","last_became_active":"2024-04-15T15:14:51.079768+0000","last_became_peered":"2024-04-15T15:14:51.079768+0000","last_unstale":"2024-04-15T15:14:57.136325+0000","last_undegraded":"2024-04-15T15:14:57.136325+0000","last_fullsized":"2024-04-15T15:14:57.136325+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:07:48.416478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136732+0000","last_change":"2024-04-15T15:14:56.102424+0000","last_active":"2024-04-15T15:14:57.136732+0000","last_peered":"2024-04-15T15:14:57.136732+0000","last_clean":"2024-04-15T15:14:57.136732+0000","last_became_active":"2024-04-15T15:14:56.101985+0000","last_became_peered":"2024-04-15T15:14:56.101985+0000","last_unstale":"2024-04-15T15:14:57.136732+0000","last_undegraded":"2024-04-15T15:14:57.136732+0000","last_fullsized":"2024-04-15T15:14:57.136732+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:01:28.704663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136649+0000","last_change":"2024-04-15T15:14:52.084991+0000","last_active":"2024-04-15T15:14:57.136649+0000","last_peered":"2024-04-15T15:14:57.136649+0000","last_clean":"2024-04-15T15:14:57.136649+0000","last_became_active":"2024-04-15T15:14:52.082250+0000","last_became_peered":"2024-04-15T15:14:52.082250+0000","last_unstale":"2024-04-15T15:14:57.136649+0000","last_undegraded":"2024-04-15T15:14:57.136649+0000","last_fullsized":"2024-04-15T15:14:57.136649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:26:45.460787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137232+0000","last_change":"2024-04-15T15:14:51.071863+0000","last_active":"2024-04-15T15:14:57.137232+0000","last_peered":"2024-04-15T15:14:57.137232+0000","last_clean":"2024-04-15T15:14:57.137232+0000","last_became_active":"2024-04-15T15:14:51.071595+0000","last_became_peered":"2024-04-15T15:14:51.071595+0000","last_unstale":"2024-04-15T15:14:57.137232+0000","last_undegraded":"2024-04-15T15:14:57.137232+0000","last_fullsized":"2024-04-15T15:14:57.137232+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:19:37.987965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136625+0000","last_change":"2024-04-15T15:14:56.099259+0000","last_active":"2024-04-15T15:14:57.136625+0000","last_peered":"2024-04-15T15:14:57.136625+0000","last_clean":"2024-04-15T15:14:57.136625+0000","last_became_active":"2024-04-15T15:14:56.098862+0000","last_became_peered":"2024-04-15T15:14:56.098862+0000","last_unstale":"2024-04-15T15:14:57.136625+0000","last_undegraded":"2024-04-15T15:14:57.136625+0000","last_fullsized":"2024-04-15T15:14:57.136625+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:38:39.210922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136403+0000","last_change":"2024-04-15T15:14:52.080320+0000","last_active":"2024-04-15T15:14:57.136403+0000","last_peered":"2024-04-15T15:14:57.136403+0000","last_clean":"2024-04-15T15:14:57.136403+0000","last_became_active":"2024-04-15T15:14:52.080064+0000","last_became_peered":"2024-04-15T15:14:52.080064+0000","last_unstale":"2024-04-15T15:14:57.136403+0000","last_undegraded":"2024-04-15T15:14:57.136403+0000","last_fullsized":"2024-04-15T15:14:57.136403+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:10:19.696254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136107+0000","last_change":"2024-04-15T15:14:51.075197+0000","last_active":"2024-04-15T15:14:57.136107+0000","last_peered":"2024-04-15T15:14:57.136107+0000","last_clean":"2024-04-15T15:14:57.136107+0000","last_became_active":"2024-04-15T15:14:51.074955+0000","last_became_peered":"2024-04-15T15:14:51.074955+0000","last_unstale":"2024-04-15T15:14:57.136107+0000","last_undegraded":"2024-04-15T15:14:57.136107+0000","last_fullsized":"2024-04-15T15:14:57.136107+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:18:41.399059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137490+0000","last_change":"2024-04-15T15:14:56.253169+0000","last_active":"2024-04-15T15:14:57.137490+0000","last_peered":"2024-04-15T15:14:57.137490+0000","last_clean":"2024-04-15T15:14:57.137490+0000","last_became_active":"2024-04-15T15:14:56.252823+0000","last_became_peered":"2024-04-15T15:14:56.252823+0000","last_unstale":"2024-04-15T15:14:57.137490+0000","last_undegraded":"2024-04-15T15:14:57.137490+0000","last_fullsized":"2024-04-15T15:14:57.137490+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:03:07.876012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136337+0000","last_change":"2024-04-15T15:14:52.075278+0000","last_active":"2024-04-15T15:14:57.136337+0000","last_peered":"2024-04-15T15:14:57.136337+0000","last_clean":"2024-04-15T15:14:57.136337+0000","last_became_active":"2024-04-15T15:14:52.075003+0000","last_became_peered":"2024-04-15T15:14:52.075003+0000","last_unstale":"2024-04-15T15:14:57.136337+0000","last_undegraded":"2024-04-15T15:14:57.136337+0000","last_fullsized":"2024-04-15T15:14:57.136337+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:47:55.994716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135597+0000","last_change":"2024-04-15T15:14:51.076530+0000","last_active":"2024-04-15T15:14:57.135597+0000","last_peered":"2024-04-15T15:14:57.135597+0000","last_clean":"2024-04-15T15:14:57.135597+0000","last_became_active":"2024-04-15T15:14:51.076247+0000","last_became_peered":"2024-04-15T15:14:51.076247+0000","last_unstale":"2024-04-15T15:14:57.135597+0000","last_undegraded":"2024-04-15T15:14:57.135597+0000","last_fullsized":"2024-04-15T15:14:57.135597+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:47:54.990677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136888+0000","last_change":"2024-04-15T15:14:56.098477+0000","last_active":"2024-04-15T15:14:57.136888+0000","last_peered":"2024-04-15T15:14:57.136888+0000","last_clean":"2024-04-15T15:14:57.136888+0000","last_became_active":"2024-04-15T15:14:56.097839+0000","last_became_peered":"2024-04-15T15:14:56.097839+0000","last_unstale":"2024-04-15T15:14:57.136888+0000","last_undegraded":"2024-04-15T15:14:57.136888+0000","last_fullsized":"2024-04-15T15:14:57.136888+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:12.500763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138519+0000","last_change":"2024-04-15T15:14:52.079014+0000","last_active":"2024-04-15T15:14:57.138519+0000","last_peered":"2024-04-15T15:14:57.138519+0000","last_clean":"2024-04-15T15:14:57.138519+0000","last_became_active":"2024-04-15T15:14:52.078708+0000","last_became_peered":"2024-04-15T15:14:52.078708+0000","last_unstale":"2024-04-15T15:14:57.138519+0000","last_undegraded":"2024-04-15T15:14:57.138519+0000","last_fullsized":"2024-04-15T15:14:57.138519+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:37:31.656434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109619+0000","last_change":"2024-04-15T15:14:51.075067+0000","last_active":"2024-04-15T15:14:57.109619+0000","last_peered":"2024-04-15T15:14:57.109619+0000","last_clean":"2024-04-15T15:14:57.109619+0000","last_became_active":"2024-04-15T15:14:51.074780+0000","last_became_peered":"2024-04-15T15:14:51.074780+0000","last_unstale":"2024-04-15T15:14:57.109619+0000","last_undegraded":"2024-04-15T15:14:57.109619+0000","last_fullsized":"2024-04-15T15:14:57.109619+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:47:54.990677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136173+0000","last_change":"2024-04-15T15:14:56.252842+0000","last_active":"2024-04-15T15:14:57.136173+0000","last_peered":"2024-04-15T15:14:57.136173+0000","last_clean":"2024-04-15T15:14:57.136173+0000","last_became_active":"2024-04-15T15:14:56.252432+0000","last_became_peered":"2024-04-15T15:14:56.252432+0000","last_unstale":"2024-04-15T15:14:57.136173+0000","last_undegraded":"2024-04-15T15:14:57.136173+0000","last_fullsized":"2024-04-15T15:14:57.136173+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:51:26.840184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138191+0000","last_change":"2024-04-15T15:14:52.082404+0000","last_active":"2024-04-15T15:14:57.138191+0000","last_peered":"2024-04-15T15:14:57.138191+0000","last_clean":"2024-04-15T15:14:57.138191+0000","last_became_active":"2024-04-15T15:14:52.082136+0000","last_became_peered":"2024-04-15T15:14:52.082136+0000","last_unstale":"2024-04-15T15:14:57.138191+0000","last_undegraded":"2024-04-15T15:14:57.138191+0000","last_fullsized":"2024-04-15T15:14:57.138191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:38:35.185619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137543+0000","last_change":"2024-04-15T15:14:51.073915+0000","last_active":"2024-04-15T15:14:57.137543+0000","last_peered":"2024-04-15T15:14:57.137543+0000","last_clean":"2024-04-15T15:14:57.137543+0000","last_became_active":"2024-04-15T15:14:51.073639+0000","last_became_peered":"2024-04-15T15:14:51.073639+0000","last_unstale":"2024-04-15T15:14:57.137543+0000","last_undegraded":"2024-04-15T15:14:57.137543+0000","last_fullsized":"2024-04-15T15:14:57.137543+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:41:20.990702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135980+0000","last_change":"2024-04-15T15:14:56.251221+0000","last_active":"2024-04-15T15:14:57.135980+0000","last_peered":"2024-04-15T15:14:57.135980+0000","last_clean":"2024-04-15T15:14:57.135980+0000","last_became_active":"2024-04-15T15:14:56.250888+0000","last_became_peered":"2024-04-15T15:14:56.250888+0000","last_unstale":"2024-04-15T15:14:57.135980+0000","last_undegraded":"2024-04-15T15:14:57.135980+0000","last_fullsized":"2024-04-15T15:14:57.135980+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:42:46.671056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137233+0000","last_change":"2024-04-15T15:14:52.077347+0000","last_active":"2024-04-15T15:14:57.137233+0000","last_peered":"2024-04-15T15:14:57.137233+0000","last_clean":"2024-04-15T15:14:57.137233+0000","last_became_active":"2024-04-15T15:14:52.077053+0000","last_became_peered":"2024-04-15T15:14:52.077053+0000","last_unstale":"2024-04-15T15:14:57.137233+0000","last_undegraded":"2024-04-15T15:14:57.137233+0000","last_fullsized":"2024-04-15T15:14:57.137233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:13:01.642838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136936+0000","last_change":"2024-04-15T15:14:51.076396+0000","last_active":"2024-04-15T15:14:57.136936+0000","last_peered":"2024-04-15T15:14:57.136936+0000","last_clean":"2024-04-15T15:14:57.136936+0000","last_became_active":"2024-04-15T15:14:51.076124+0000","last_became_peered":"2024-04-15T15:14:51.076124+0000","last_unstale":"2024-04-15T15:14:57.136936+0000","last_undegraded":"2024-04-15T15:14:57.136936+0000","last_fullsized":"2024-04-15T15:14:57.136936+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:31:57.289767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137379+0000","last_change":"2024-04-15T15:14:56.251375+0000","last_active":"2024-04-15T15:14:57.137379+0000","last_peered":"2024-04-15T15:14:57.137379+0000","last_clean":"2024-04-15T15:14:57.137379+0000","last_became_active":"2024-04-15T15:14:56.250973+0000","last_became_peered":"2024-04-15T15:14:56.250973+0000","last_unstale":"2024-04-15T15:14:57.137379+0000","last_undegraded":"2024-04-15T15:14:57.137379+0000","last_fullsized":"2024-04-15T15:14:57.137379+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:58:01.690949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.110314+0000","last_change":"2024-04-15T15:14:52.082863+0000","last_active":"2024-04-15T15:14:57.110314+0000","last_peered":"2024-04-15T15:14:57.110314+0000","last_clean":"2024-04-15T15:14:57.110314+0000","last_became_active":"2024-04-15T15:14:52.082370+0000","last_became_peered":"2024-04-15T15:14:52.082370+0000","last_unstale":"2024-04-15T15:14:57.110314+0000","last_undegraded":"2024-04-15T15:14:57.110314+0000","last_fullsized":"2024-04-15T15:14:57.110314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:08.475460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137335+0000","last_change":"2024-04-15T15:14:51.073016+0000","last_active":"2024-04-15T15:14:57.137335+0000","last_peered":"2024-04-15T15:14:57.137335+0000","last_clean":"2024-04-15T15:14:57.137335+0000","last_became_active":"2024-04-15T15:14:51.072736+0000","last_became_peered":"2024-04-15T15:14:51.072736+0000","last_unstale":"2024-04-15T15:14:57.137335+0000","last_undegraded":"2024-04-15T15:14:57.137335+0000","last_fullsized":"2024-04-15T15:14:57.137335+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:44:33.685298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136866+0000","last_change":"2024-04-15T15:14:56.105315+0000","last_active":"2024-04-15T15:14:57.136866+0000","last_peered":"2024-04-15T15:14:57.136866+0000","last_clean":"2024-04-15T15:14:57.136866+0000","last_became_active":"2024-04-15T15:14:56.104923+0000","last_became_peered":"2024-04-15T15:14:56.104923+0000","last_unstale":"2024-04-15T15:14:57.136866+0000","last_undegraded":"2024-04-15T15:14:57.136866+0000","last_fullsized":"2024-04-15T15:14:57.136866+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:01:28.704663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136866+0000","last_change":"2024-04-15T15:14:52.078353+0000","last_active":"2024-04-15T15:14:57.136866+0000","last_peered":"2024-04-15T15:14:57.136866+0000","last_clean":"2024-04-15T15:14:57.136866+0000","last_became_active":"2024-04-15T15:14:52.076556+0000","last_became_peered":"2024-04-15T15:14:52.076556+0000","last_unstale":"2024-04-15T15:14:57.136866+0000","last_undegraded":"2024-04-15T15:14:57.136866+0000","last_fullsized":"2024-04-15T15:14:57.136866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:19:38.992004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109062+0000","last_change":"2024-04-15T15:14:51.072875+0000","last_active":"2024-04-15T15:14:57.109062+0000","last_peered":"2024-04-15T15:14:57.109062+0000","last_clean":"2024-04-15T15:14:57.109062+0000","last_became_active":"2024-04-15T15:14:51.072114+0000","last_became_peered":"2024-04-15T15:14:51.072114+0000","last_unstale":"2024-04-15T15:14:57.109062+0000","last_undegraded":"2024-04-15T15:14:57.109062+0000","last_fullsized":"2024-04-15T15:14:57.109062+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:18:41.399059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137344+0000","last_change":"2024-04-15T15:14:56.098426+0000","last_active":"2024-04-15T15:14:57.137344+0000","last_peered":"2024-04-15T15:14:57.137344+0000","last_clean":"2024-04-15T15:14:57.137344+0000","last_became_active":"2024-04-15T15:14:56.098066+0000","last_became_peered":"2024-04-15T15:14:56.098066+0000","last_unstale":"2024-04-15T15:14:57.137344+0000","last_undegraded":"2024-04-15T15:14:57.137344+0000","last_fullsized":"2024-04-15T15:14:57.137344+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:12.500763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136427+0000","last_change":"2024-04-15T15:14:52.077474+0000","last_active":"2024-04-15T15:14:57.136427+0000","last_peered":"2024-04-15T15:14:57.136427+0000","last_clean":"2024-04-15T15:14:57.136427+0000","last_became_active":"2024-04-15T15:14:52.077206+0000","last_became_peered":"2024-04-15T15:14:52.077206+0000","last_unstale":"2024-04-15T15:14:57.136427+0000","last_undegraded":"2024-04-15T15:14:57.136427+0000","last_fullsized":"2024-04-15T15:14:57.136427+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:34:51.775979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.107727+0000","last_change":"2024-04-15T15:14:51.067544+0000","last_active":"2024-04-15T15:14:57.107727+0000","last_peered":"2024-04-15T15:14:57.107727+0000","last_clean":"2024-04-15T15:14:57.107727+0000","last_became_active":"2024-04-15T15:14:51.067233+0000","last_became_peered":"2024-04-15T15:14:51.067233+0000","last_unstale":"2024-04-15T15:14:57.107727+0000","last_undegraded":"2024-04-15T15:14:57.107727+0000","last_fullsized":"2024-04-15T15:14:57.107727+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:44:33.685298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137463+0000","last_change":"2024-04-15T15:14:56.103911+0000","last_active":"2024-04-15T15:14:57.137463+0000","last_peered":"2024-04-15T15:14:57.137463+0000","last_clean":"2024-04-15T15:14:57.137463+0000","last_became_active":"2024-04-15T15:14:56.103035+0000","last_became_peered":"2024-04-15T15:14:56.103035+0000","last_unstale":"2024-04-15T15:14:57.137463+0000","last_undegraded":"2024-04-15T15:14:57.137463+0000","last_fullsized":"2024-04-15T15:14:57.137463+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:21:52.495367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136227+0000","last_change":"2024-04-15T15:14:52.078870+0000","last_active":"2024-04-15T15:14:57.136227+0000","last_peered":"2024-04-15T15:14:57.136227+0000","last_clean":"2024-04-15T15:14:57.136227+0000","last_became_active":"2024-04-15T15:14:52.078573+0000","last_became_peered":"2024-04-15T15:14:52.078573+0000","last_unstale":"2024-04-15T15:14:57.136227+0000","last_undegraded":"2024-04-15T15:14:57.136227+0000","last_fullsized":"2024-04-15T15:14:57.136227+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:51:36.740221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135636+0000","last_change":"2024-04-15T15:14:51.077104+0000","last_active":"2024-04-15T15:14:57.135636+0000","last_peered":"2024-04-15T15:14:57.135636+0000","last_clean":"2024-04-15T15:14:57.135636+0000","last_became_active":"2024-04-15T15:14:51.076857+0000","last_became_peered":"2024-04-15T15:14:51.076857+0000","last_unstale":"2024-04-15T15:14:57.135636+0000","last_undegraded":"2024-04-15T15:14:57.135636+0000","last_fullsized":"2024-04-15T15:14:57.135636+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:18:41.399059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138372+0000","last_change":"2024-04-15T15:14:56.251372+0000","last_active":"2024-04-15T15:14:57.138372+0000","last_peered":"2024-04-15T15:14:57.138372+0000","last_clean":"2024-04-15T15:14:57.138372+0000","last_became_active":"2024-04-15T15:14:56.250961+0000","last_became_peered":"2024-04-15T15:14:56.250961+0000","last_unstale":"2024-04-15T15:14:57.138372+0000","last_undegraded":"2024-04-15T15:14:57.138372+0000","last_fullsized":"2024-04-15T15:14:57.138372+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:13:03.031872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136619+0000","last_change":"2024-04-15T15:14:52.089029+0000","last_active":"2024-04-15T15:14:57.136619+0000","last_peered":"2024-04-15T15:14:57.136619+0000","last_clean":"2024-04-15T15:14:57.136619+0000","last_became_active":"2024-04-15T15:14:52.088646+0000","last_became_peered":"2024-04-15T15:14:52.088646+0000","last_unstale":"2024-04-15T15:14:57.136619+0000","last_undegraded":"2024-04-15T15:14:57.136619+0000","last_fullsized":"2024-04-15T15:14:57.136619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:37:31.656434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108973+0000","last_change":"2024-04-15T15:14:51.067071+0000","last_active":"2024-04-15T15:14:57.108973+0000","last_peered":"2024-04-15T15:14:57.108973+0000","last_clean":"2024-04-15T15:14:57.108973+0000","last_became_active":"2024-04-15T15:14:51.066382+0000","last_became_peered":"2024-04-15T15:14:51.066382+0000","last_unstale":"2024-04-15T15:14:57.108973+0000","last_undegraded":"2024-04-15T15:14:57.108973+0000","last_fullsized":"2024-04-15T15:14:57.108973+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:50:01.422527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":178,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135699+0000","last_change":"2024-04-15T15:14:40.494279+0000","last_active":"2024-04-15T15:14:57.135699+0000","last_peered":"2024-04-15T15:14:57.135699+0000","last_clean":"2024-04-15T15:14:57.135699+0000","last_became_active":"2024-04-15T15:14:40.493404+0000","last_became_peered":"2024-04-15T15:14:40.493404+0000","last_unstale":"2024-04-15T15:14:57.135699+0000","last_undegraded":"2024-04-15T15:14:57.135699+0000","last_fullsized":"2024-04-15T15:14:57.135699+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:39.418508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:39.418508+0000","last_clean_scrub_stamp":"2024-04-15T15:14:39.418508+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:01:51.789246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109541+0000","last_change":"2024-04-15T15:14:56.253853+0000","last_active":"2024-04-15T15:14:57.109541+0000","last_peered":"2024-04-15T15:14:57.109541+0000","last_clean":"2024-04-15T15:14:57.109541+0000","last_became_active":"2024-04-15T15:14:56.253323+0000","last_became_peered":"2024-04-15T15:14:56.253323+0000","last_unstale":"2024-04-15T15:14:57.109541+0000","last_undegraded":"2024-04-15T15:14:57.109541+0000","last_fullsized":"2024-04-15T15:14:57.109541+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:03:07.876012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136598+0000","last_change":"2024-04-15T15:14:52.093103+0000","last_active":"2024-04-15T15:14:57.136598+0000","last_peered":"2024-04-15T15:14:57.136598+0000","last_clean":"2024-04-15T15:14:57.136598+0000","last_became_active":"2024-04-15T15:14:52.092786+0000","last_became_peered":"2024-04-15T15:14:52.092786+0000","last_unstale":"2024-04-15T15:14:57.136598+0000","last_undegraded":"2024-04-15T15:14:57.136598+0000","last_fullsized":"2024-04-15T15:14:57.136598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:57:57.665646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108044+0000","last_change":"2024-04-15T15:14:51.067545+0000","last_active":"2024-04-15T15:14:57.108044+0000","last_peered":"2024-04-15T15:14:57.108044+0000","last_clean":"2024-04-15T15:14:57.108044+0000","last_became_active":"2024-04-15T15:14:51.067229+0000","last_became_peered":"2024-04-15T15:14:51.067229+0000","last_unstale":"2024-04-15T15:14:57.108044+0000","last_undegraded":"2024-04-15T15:14:57.108044+0000","last_fullsized":"2024-04-15T15:14:57.108044+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:41:20.990702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108641+0000","last_change":"2024-04-15T15:14:56.251780+0000","last_active":"2024-04-15T15:14:57.108641+0000","last_peered":"2024-04-15T15:14:57.108641+0000","last_clean":"2024-04-15T15:14:57.108641+0000","last_became_active":"2024-04-15T15:14:56.251079+0000","last_became_peered":"2024-04-15T15:14:56.251079+0000","last_unstale":"2024-04-15T15:14:57.108641+0000","last_undegraded":"2024-04-15T15:14:57.108641+0000","last_fullsized":"2024-04-15T15:14:57.108641+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:12.500763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109714+0000","last_change":"2024-04-15T15:14:52.074261+0000","last_active":"2024-04-15T15:14:57.109714+0000","last_peered":"2024-04-15T15:14:57.109714+0000","last_clean":"2024-04-15T15:14:57.109714+0000","last_became_active":"2024-04-15T15:14:52.073979+0000","last_became_peered":"2024-04-15T15:14:52.073979+0000","last_unstale":"2024-04-15T15:14:57.109714+0000","last_undegraded":"2024-04-15T15:14:57.109714+0000","last_fullsized":"2024-04-15T15:14:57.109714+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:03:03.850709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137650+0000","last_change":"2024-04-15T15:14:51.074612+0000","last_active":"2024-04-15T15:14:57.137650+0000","last_peered":"2024-04-15T15:14:57.137650+0000","last_clean":"2024-04-15T15:14:57.137650+0000","last_became_active":"2024-04-15T15:14:51.074176+0000","last_became_peered":"2024-04-15T15:14:51.074176+0000","last_unstale":"2024-04-15T15:14:57.137650+0000","last_undegraded":"2024-04-15T15:14:57.137650+0000","last_fullsized":"2024-04-15T15:14:57.137650+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:50:01.422527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137604+0000","last_change":"2024-04-15T15:14:56.103129+0000","last_active":"2024-04-15T15:14:57.137604+0000","last_peered":"2024-04-15T15:14:57.137604+0000","last_clean":"2024-04-15T15:14:57.137604+0000","last_became_active":"2024-04-15T15:14:56.102757+0000","last_became_peered":"2024-04-15T15:14:56.102757+0000","last_unstale":"2024-04-15T15:14:57.137604+0000","last_undegraded":"2024-04-15T15:14:57.137604+0000","last_fullsized":"2024-04-15T15:14:57.137604+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:12.500763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136942+0000","last_change":"2024-04-15T15:14:52.087082+0000","last_active":"2024-04-15T15:14:57.136942+0000","last_peered":"2024-04-15T15:14:57.136942+0000","last_clean":"2024-04-15T15:14:57.136942+0000","last_became_active":"2024-04-15T15:14:52.086517+0000","last_became_peered":"2024-04-15T15:14:52.086517+0000","last_unstale":"2024-04-15T15:14:57.136942+0000","last_undegraded":"2024-04-15T15:14:57.136942+0000","last_fullsized":"2024-04-15T15:14:57.136942+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:08:15.196752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.107969+0000","last_change":"2024-04-15T15:14:51.067941+0000","last_active":"2024-04-15T15:14:57.107969+0000","last_peered":"2024-04-15T15:14:57.107969+0000","last_clean":"2024-04-15T15:14:57.107969+0000","last_became_active":"2024-04-15T15:14:51.067669+0000","last_became_peered":"2024-04-15T15:14:51.067669+0000","last_unstale":"2024-04-15T15:14:57.107969+0000","last_undegraded":"2024-04-15T15:14:57.107969+0000","last_fullsized":"2024-04-15T15:14:57.107969+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:50:01.422527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108366+0000","last_change":"2024-04-15T15:14:56.252233+0000","last_active":"2024-04-15T15:14:57.108366+0000","last_peered":"2024-04-15T15:14:57.108366+0000","last_clean":"2024-04-15T15:14:57.108366+0000","last_became_active":"2024-04-15T15:14:56.251084+0000","last_became_peered":"2024-04-15T15:14:56.251084+0000","last_unstale":"2024-04-15T15:14:57.108366+0000","last_undegraded":"2024-04-15T15:14:57.108366+0000","last_fullsized":"2024-04-15T15:14:57.108366+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:42:46.671056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136568+0000","last_change":"2024-04-15T15:14:52.085987+0000","last_active":"2024-04-15T15:14:57.136568+0000","last_peered":"2024-04-15T15:14:57.136568+0000","last_clean":"2024-04-15T15:14:57.136568+0000","last_became_active":"2024-04-15T15:14:52.085579+0000","last_became_peered":"2024-04-15T15:14:52.085579+0000","last_unstale":"2024-04-15T15:14:57.136568+0000","last_undegraded":"2024-04-15T15:14:57.136568+0000","last_fullsized":"2024-04-15T15:14:57.136568+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:08.475460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136963+0000","last_change":"2024-04-15T15:14:51.066803+0000","last_active":"2024-04-15T15:14:57.136963+0000","last_peered":"2024-04-15T15:14:57.136963+0000","last_clean":"2024-04-15T15:14:57.136963+0000","last_became_active":"2024-04-15T15:14:51.066103+0000","last_became_peered":"2024-04-15T15:14:51.066103+0000","last_unstale":"2024-04-15T15:14:57.136963+0000","last_undegraded":"2024-04-15T15:14:57.136963+0000","last_fullsized":"2024-04-15T15:14:57.136963+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:35:13.836817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135900+0000","last_change":"2024-04-15T15:14:56.098601+0000","last_active":"2024-04-15T15:14:57.135900+0000","last_peered":"2024-04-15T15:14:57.135900+0000","last_clean":"2024-04-15T15:14:57.135900+0000","last_became_active":"2024-04-15T15:14:56.098191+0000","last_became_peered":"2024-04-15T15:14:56.098191+0000","last_unstale":"2024-04-15T15:14:57.135900+0000","last_undegraded":"2024-04-15T15:14:57.135900+0000","last_fullsized":"2024-04-15T15:14:57.135900+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:12:11.797852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135812+0000","last_change":"2024-04-15T15:14:52.086933+0000","last_active":"2024-04-15T15:14:57.135812+0000","last_peered":"2024-04-15T15:14:57.135812+0000","last_clean":"2024-04-15T15:14:57.135812+0000","last_became_active":"2024-04-15T15:14:52.086546+0000","last_became_peered":"2024-04-15T15:14:52.086546+0000","last_unstale":"2024-04-15T15:14:57.135812+0000","last_undegraded":"2024-04-15T15:14:57.135812+0000","last_fullsized":"2024-04-15T15:14:57.135812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:10:19.696254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.107592+0000","last_change":"2024-04-15T15:14:51.071716+0000","last_active":"2024-04-15T15:14:57.107592+0000","last_peered":"2024-04-15T15:14:57.107592+0000","last_clean":"2024-04-15T15:14:57.107592+0000","last_became_active":"2024-04-15T15:14:51.071084+0000","last_became_peered":"2024-04-15T15:14:51.071084+0000","last_unstale":"2024-04-15T15:14:57.107592+0000","last_undegraded":"2024-04-15T15:14:57.107592+0000","last_fullsized":"2024-04-15T15:14:57.107592+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:01:16.772802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138211+0000","last_change":"2024-04-15T15:14:56.102060+0000","last_active":"2024-04-15T15:14:57.138211+0000","last_peered":"2024-04-15T15:14:57.138211+0000","last_clean":"2024-04-15T15:14:57.138211+0000","last_became_active":"2024-04-15T15:14:56.101644+0000","last_became_peered":"2024-04-15T15:14:56.101644+0000","last_unstale":"2024-04-15T15:14:57.138211+0000","last_undegraded":"2024-04-15T15:14:57.138211+0000","last_fullsized":"2024-04-15T15:14:57.138211+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:03:07.876012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109139+0000","last_change":"2024-04-15T15:14:52.088688+0000","last_active":"2024-04-15T15:14:57.109139+0000","last_peered":"2024-04-15T15:14:57.109139+0000","last_clean":"2024-04-15T15:14:57.109139+0000","last_became_active":"2024-04-15T15:14:52.088433+0000","last_became_peered":"2024-04-15T15:14:52.088433+0000","last_unstale":"2024-04-15T15:14:57.109139+0000","last_undegraded":"2024-04-15T15:14:57.109139+0000","last_fullsized":"2024-04-15T15:14:57.109139+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:24:30.714153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.110238+0000","last_change":"2024-04-15T15:14:51.073931+0000","last_active":"2024-04-15T15:14:57.110238+0000","last_peered":"2024-04-15T15:14:57.110238+0000","last_clean":"2024-04-15T15:14:57.110238+0000","last_became_active":"2024-04-15T15:14:51.073571+0000","last_became_peered":"2024-04-15T15:14:51.073571+0000","last_unstale":"2024-04-15T15:14:57.110238+0000","last_undegraded":"2024-04-15T15:14:57.110238+0000","last_fullsized":"2024-04-15T15:14:57.110238+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:44:33.685298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138190+0000","last_change":"2024-04-15T15:14:56.251177+0000","last_active":"2024-04-15T15:14:57.138190+0000","last_peered":"2024-04-15T15:14:57.138190+0000","last_clean":"2024-04-15T15:14:57.138190+0000","last_became_active":"2024-04-15T15:14:56.250803+0000","last_became_peered":"2024-04-15T15:14:56.250803+0000","last_unstale":"2024-04-15T15:14:57.138190+0000","last_undegraded":"2024-04-15T15:14:57.138190+0000","last_fullsized":"2024-04-15T15:14:57.138190+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:39:36.394347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136175+0000","last_change":"2024-04-15T15:14:52.086912+0000","last_active":"2024-04-15T15:14:57.136175+0000","last_peered":"2024-04-15T15:14:57.136175+0000","last_clean":"2024-04-15T15:14:57.136175+0000","last_became_active":"2024-04-15T15:14:52.085621+0000","last_became_peered":"2024-04-15T15:14:52.085621+0000","last_unstale":"2024-04-15T15:14:57.136175+0000","last_undegraded":"2024-04-15T15:14:57.136175+0000","last_fullsized":"2024-04-15T15:14:57.136175+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:38:35.185619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108968+0000","last_change":"2024-04-15T15:14:51.068349+0000","last_active":"2024-04-15T15:14:57.108968+0000","last_peered":"2024-04-15T15:14:57.108968+0000","last_clean":"2024-04-15T15:14:57.108968+0000","last_became_active":"2024-04-15T15:14:51.068078+0000","last_became_peered":"2024-04-15T15:14:51.068078+0000","last_unstale":"2024-04-15T15:14:57.108968+0000","last_undegraded":"2024-04-15T15:14:57.108968+0000","last_fullsized":"2024-04-15T15:14:57.108968+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:47:54.990677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136871+0000","last_change":"2024-04-15T15:14:56.251089+0000","last_active":"2024-04-15T15:14:57.136871+0000","last_peered":"2024-04-15T15:14:57.136871+0000","last_clean":"2024-04-15T15:14:57.136871+0000","last_became_active":"2024-04-15T15:14:56.250735+0000","last_became_peered":"2024-04-15T15:14:56.250735+0000","last_unstale":"2024-04-15T15:14:57.136871+0000","last_undegraded":"2024-04-15T15:14:57.136871+0000","last_fullsized":"2024-04-15T15:14:57.136871+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:39:38.108475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138515+0000","last_change":"2024-04-15T15:14:52.076410+0000","last_active":"2024-04-15T15:14:57.138515+0000","last_peered":"2024-04-15T15:14:57.138515+0000","last_clean":"2024-04-15T15:14:57.138515+0000","last_became_active":"2024-04-15T15:14:52.076115+0000","last_became_peered":"2024-04-15T15:14:52.076115+0000","last_unstale":"2024-04-15T15:14:57.138515+0000","last_undegraded":"2024-04-15T15:14:57.138515+0000","last_fullsized":"2024-04-15T15:14:57.138515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:48:45.790927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137841+0000","last_change":"2024-04-15T15:14:51.072889+0000","last_active":"2024-04-15T15:14:57.137841+0000","last_peered":"2024-04-15T15:14:57.137841+0000","last_clean":"2024-04-15T15:14:57.137841+0000","last_became_active":"2024-04-15T15:14:51.072611+0000","last_became_peered":"2024-04-15T15:14:51.072611+0000","last_unstale":"2024-04-15T15:14:57.137841+0000","last_undegraded":"2024-04-15T15:14:57.137841+0000","last_fullsized":"2024-04-15T15:14:57.137841+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:20:40.577242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.110168+0000","last_change":"2024-04-15T15:14:56.253070+0000","last_active":"2024-04-15T15:14:57.110168+0000","last_peered":"2024-04-15T15:14:57.110168+0000","last_clean":"2024-04-15T15:14:57.110168+0000","last_became_active":"2024-04-15T15:14:56.252652+0000","last_became_peered":"2024-04-15T15:14:56.252652+0000","last_unstale":"2024-04-15T15:14:57.110168+0000","last_undegraded":"2024-04-15T15:14:57.110168+0000","last_fullsized":"2024-04-15T15:14:57.110168+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:34:50.166643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136518+0000","last_change":"2024-04-15T15:14:52.092734+0000","last_active":"2024-04-15T15:14:57.136518+0000","last_peered":"2024-04-15T15:14:57.136518+0000","last_clean":"2024-04-15T15:14:57.136518+0000","last_became_active":"2024-04-15T15:14:52.092105+0000","last_became_peered":"2024-04-15T15:14:52.092105+0000","last_unstale":"2024-04-15T15:14:57.136518+0000","last_undegraded":"2024-04-15T15:14:57.136518+0000","last_fullsized":"2024-04-15T15:14:57.136518+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:01:24.679360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":31,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:15:01.576821+0000","last_change":"2024-04-15T15:14:51.072503+0000","last_active":"2024-04-15T15:15:01.576821+0000","last_peered":"2024-04-15T15:15:01.576821+0000","last_clean":"2024-04-15T15:15:01.576821+0000","last_became_active":"2024-04-15T15:14:51.072212+0000","last_became_peered":"2024-04-15T15:14:51.072212+0000","last_unstale":"2024-04-15T15:15:01.576821+0000","last_undegraded":"2024-04-15T15:15:01.576821+0000","last_fullsized":"2024-04-15T15:15:01.576821+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:18:41.399059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137069+0000","last_change":"2024-04-15T15:14:56.105406+0000","last_active":"2024-04-15T15:14:57.137069+0000","last_peered":"2024-04-15T15:14:57.137069+0000","last_clean":"2024-04-15T15:14:57.137069+0000","last_became_active":"2024-04-15T15:14:56.104951+0000","last_became_peered":"2024-04-15T15:14:56.104951+0000","last_unstale":"2024-04-15T15:14:57.137069+0000","last_undegraded":"2024-04-15T15:14:57.137069+0000","last_fullsized":"2024-04-15T15:14:57.137069+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:58:01.690949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136751+0000","last_change":"2024-04-15T15:14:52.083396+0000","last_active":"2024-04-15T15:14:57.136751+0000","last_peered":"2024-04-15T15:14:57.136751+0000","last_clean":"2024-04-15T15:14:57.136751+0000","last_became_active":"2024-04-15T15:14:52.082661+0000","last_became_peered":"2024-04-15T15:14:52.082661+0000","last_unstale":"2024-04-15T15:14:57.136751+0000","last_undegraded":"2024-04-15T15:14:57.136751+0000","last_fullsized":"2024-04-15T15:14:57.136751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:07:49.420517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137624+0000","last_change":"2024-04-15T15:14:51.074372+0000","last_active":"2024-04-15T15:14:57.137624+0000","last_peered":"2024-04-15T15:14:57.137624+0000","last_clean":"2024-04-15T15:14:57.137624+0000","last_became_active":"2024-04-15T15:14:51.074030+0000","last_became_peered":"2024-04-15T15:14:51.074030+0000","last_unstale":"2024-04-15T15:14:57.137624+0000","last_undegraded":"2024-04-15T15:14:57.137624+0000","last_fullsized":"2024-04-15T15:14:57.137624+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:47:54.990677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136760+0000","last_change":"2024-04-15T15:14:56.099051+0000","last_active":"2024-04-15T15:14:57.136760+0000","last_peered":"2024-04-15T15:14:57.136760+0000","last_clean":"2024-04-15T15:14:57.136760+0000","last_became_active":"2024-04-15T15:14:56.098034+0000","last_became_peered":"2024-04-15T15:14:56.098034+0000","last_unstale":"2024-04-15T15:14:57.136760+0000","last_undegraded":"2024-04-15T15:14:57.136760+0000","last_fullsized":"2024-04-15T15:14:57.136760+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:58:01.690949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136894+0000","last_change":"2024-04-15T15:14:52.084508+0000","last_active":"2024-04-15T15:14:57.136894+0000","last_peered":"2024-04-15T15:14:57.136894+0000","last_clean":"2024-04-15T15:14:57.136894+0000","last_became_active":"2024-04-15T15:14:52.083550+0000","last_became_peered":"2024-04-15T15:14:52.083550+0000","last_unstale":"2024-04-15T15:14:57.136894+0000","last_undegraded":"2024-04-15T15:14:57.136894+0000","last_fullsized":"2024-04-15T15:14:57.136894+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:12:40.420221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108919+0000","last_change":"2024-04-15T15:14:51.071093+0000","last_active":"2024-04-15T15:14:57.108919+0000","last_peered":"2024-04-15T15:14:57.108919+0000","last_clean":"2024-04-15T15:14:57.108919+0000","last_became_active":"2024-04-15T15:14:51.070810+0000","last_became_peered":"2024-04-15T15:14:51.070810+0000","last_unstale":"2024-04-15T15:14:57.108919+0000","last_undegraded":"2024-04-15T15:14:57.108919+0000","last_fullsized":"2024-04-15T15:14:57.108919+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:41:20.990702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136272+0000","last_change":"2024-04-15T15:14:56.092919+0000","last_active":"2024-04-15T15:14:57.136272+0000","last_peered":"2024-04-15T15:14:57.136272+0000","last_clean":"2024-04-15T15:14:57.136272+0000","last_became_active":"2024-04-15T15:14:56.092459+0000","last_became_peered":"2024-04-15T15:14:56.092459+0000","last_unstale":"2024-04-15T15:14:57.136272+0000","last_undegraded":"2024-04-15T15:14:57.136272+0000","last_fullsized":"2024-04-15T15:14:57.136272+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:46:38.024003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108815+0000","last_change":"2024-04-15T15:14:52.079880+0000","last_active":"2024-04-15T15:14:57.108815+0000","last_peered":"2024-04-15T15:14:57.108815+0000","last_clean":"2024-04-15T15:14:57.108815+0000","last_became_active":"2024-04-15T15:14:52.079619+0000","last_became_peered":"2024-04-15T15:14:52.079619+0000","last_unstale":"2024-04-15T15:14:57.108815+0000","last_undegraded":"2024-04-15T15:14:57.108815+0000","last_fullsized":"2024-04-15T15:14:57.108815+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:21:48.470063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109438+0000","last_change":"2024-04-15T15:14:51.075415+0000","last_active":"2024-04-15T15:14:57.109438+0000","last_peered":"2024-04-15T15:14:57.109438+0000","last_clean":"2024-04-15T15:14:57.109438+0000","last_became_active":"2024-04-15T15:14:51.075014+0000","last_became_peered":"2024-04-15T15:14:51.075014+0000","last_unstale":"2024-04-15T15:14:57.109438+0000","last_undegraded":"2024-04-15T15:14:57.109438+0000","last_fullsized":"2024-04-15T15:14:57.109438+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:41:20.990702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135982+0000","last_change":"2024-04-15T15:14:56.253630+0000","last_active":"2024-04-15T15:14:57.135982+0000","last_peered":"2024-04-15T15:14:57.135982+0000","last_clean":"2024-04-15T15:14:57.135982+0000","last_became_active":"2024-04-15T15:14:56.253253+0000","last_became_peered":"2024-04-15T15:14:56.253253+0000","last_unstale":"2024-04-15T15:14:57.135982+0000","last_undegraded":"2024-04-15T15:14:57.135982+0000","last_fullsized":"2024-04-15T15:14:57.135982+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:34:50.166643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138051+0000","last_change":"2024-04-15T15:14:56.253163+0000","last_active":"2024-04-15T15:14:57.138051+0000","last_peered":"2024-04-15T15:14:57.138051+0000","last_clean":"2024-04-15T15:14:57.138051+0000","last_became_active":"2024-04-15T15:14:56.252769+0000","last_became_peered":"2024-04-15T15:14:56.252769+0000","last_unstale":"2024-04-15T15:14:57.138051+0000","last_undegraded":"2024-04-15T15:14:57.138051+0000","last_fullsized":"2024-04-15T15:14:57.138051+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:21:52.495367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135905+0000","last_change":"2024-04-15T15:14:52.086725+0000","last_active":"2024-04-15T15:14:57.135905+0000","last_peered":"2024-04-15T15:14:57.135905+0000","last_clean":"2024-04-15T15:14:57.135905+0000","last_became_active":"2024-04-15T15:14:52.086220+0000","last_became_peered":"2024-04-15T15:14:52.086220+0000","last_unstale":"2024-04-15T15:14:57.135905+0000","last_undegraded":"2024-04-15T15:14:57.135905+0000","last_fullsized":"2024-04-15T15:14:57.135905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:21:50.111549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108433+0000","last_change":"2024-04-15T15:14:51.069916+0000","last_active":"2024-04-15T15:14:57.108433+0000","last_peered":"2024-04-15T15:14:57.108433+0000","last_clean":"2024-04-15T15:14:57.108433+0000","last_became_active":"2024-04-15T15:14:51.069631+0000","last_became_peered":"2024-04-15T15:14:51.069631+0000","last_unstale":"2024-04-15T15:14:57.108433+0000","last_undegraded":"2024-04-15T15:14:57.108433+0000","last_fullsized":"2024-04-15T15:14:57.108433+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:44:33.685298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109306+0000","last_change":"2024-04-15T15:14:56.102205+0000","last_active":"2024-04-15T15:14:57.109306+0000","last_peered":"2024-04-15T15:14:57.109306+0000","last_clean":"2024-04-15T15:14:57.109306+0000","last_became_active":"2024-04-15T15:14:56.101827+0000","last_became_peered":"2024-04-15T15:14:56.101827+0000","last_unstale":"2024-04-15T15:14:57.109306+0000","last_undegraded":"2024-04-15T15:14:57.109306+0000","last_fullsized":"2024-04-15T15:14:57.109306+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:21:52.495367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136314+0000","last_change":"2024-04-15T15:14:52.079898+0000","last_active":"2024-04-15T15:14:57.136314+0000","last_peered":"2024-04-15T15:14:57.136314+0000","last_clean":"2024-04-15T15:14:57.136314+0000","last_became_active":"2024-04-15T15:14:52.079430+0000","last_became_peered":"2024-04-15T15:14:52.079430+0000","last_unstale":"2024-04-15T15:14:57.136314+0000","last_undegraded":"2024-04-15T15:14:57.136314+0000","last_fullsized":"2024-04-15T15:14:57.136314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:12:07.772549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136279+0000","last_change":"2024-04-15T15:14:51.069720+0000","last_active":"2024-04-15T15:14:57.136279+0000","last_peered":"2024-04-15T15:14:57.136279+0000","last_clean":"2024-04-15T15:14:57.136279+0000","last_became_active":"2024-04-15T15:14:51.069416+0000","last_became_peered":"2024-04-15T15:14:51.069416+0000","last_unstale":"2024-04-15T15:14:57.136279+0000","last_undegraded":"2024-04-15T15:14:57.136279+0000","last_fullsized":"2024-04-15T15:14:57.136279+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:20:40.577242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108868+0000","last_change":"2024-04-15T15:14:51.070913+0000","last_active":"2024-04-15T15:14:57.108868+0000","last_peered":"2024-04-15T15:14:57.108868+0000","last_clean":"2024-04-15T15:14:57.108868+0000","last_became_active":"2024-04-15T15:14:51.070640+0000","last_became_peered":"2024-04-15T15:14:51.070640+0000","last_unstale":"2024-04-15T15:14:57.108868+0000","last_undegraded":"2024-04-15T15:14:57.108868+0000","last_fullsized":"2024-04-15T15:14:57.108868+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:19:37.987965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137741+0000","last_change":"2024-04-15T15:14:52.078627+0000","last_active":"2024-04-15T15:14:57.137741+0000","last_peered":"2024-04-15T15:14:57.137741+0000","last_clean":"2024-04-15T15:14:57.137741+0000","last_became_active":"2024-04-15T15:14:52.078333+0000","last_became_peered":"2024-04-15T15:14:52.078333+0000","last_unstale":"2024-04-15T15:14:57.137741+0000","last_undegraded":"2024-04-15T15:14:57.137741+0000","last_fullsized":"2024-04-15T15:14:57.137741+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:50:41.263447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108003+0000","last_change":"2024-04-15T15:14:56.093858+0000","last_active":"2024-04-15T15:14:57.108003+0000","last_peered":"2024-04-15T15:14:57.108003+0000","last_clean":"2024-04-15T15:14:57.108003+0000","last_became_active":"2024-04-15T15:14:56.093481+0000","last_became_peered":"2024-04-15T15:14:56.093481+0000","last_unstale":"2024-04-15T15:14:57.108003+0000","last_undegraded":"2024-04-15T15:14:57.108003+0000","last_fullsized":"2024-04-15T15:14:57.108003+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:48:49.816230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137031+0000","last_change":"2024-04-15T15:14:51.073707+0000","last_active":"2024-04-15T15:14:57.137031+0000","last_peered":"2024-04-15T15:14:57.137031+0000","last_clean":"2024-04-15T15:14:57.137031+0000","last_became_active":"2024-04-15T15:14:51.073451+0000","last_became_peered":"2024-04-15T15:14:51.073451+0000","last_unstale":"2024-04-15T15:14:57.137031+0000","last_undegraded":"2024-04-15T15:14:57.137031+0000","last_fullsized":"2024-04-15T15:14:57.137031+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:50.046113+0000","last_clean_scrub_stamp":"2024-04-15T15:14:50.046113+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:01:16.772802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136421+0000","last_change":"2024-04-15T15:14:52.078644+0000","last_active":"2024-04-15T15:14:57.136421+0000","last_peered":"2024-04-15T15:14:57.136421+0000","last_clean":"2024-04-15T15:14:57.136421+0000","last_became_active":"2024-04-15T15:14:52.078177+0000","last_became_peered":"2024-04-15T15:14:52.078177+0000","last_unstale":"2024-04-15T15:14:57.136421+0000","last_undegraded":"2024-04-15T15:14:57.136421+0000","last_fullsized":"2024-04-15T15:14:57.136421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:18:42.403098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136932+0000","last_change":"2024-04-15T15:14:56.092926+0000","last_active":"2024-04-15T15:14:57.136932+0000","last_peered":"2024-04-15T15:14:57.136932+0000","last_clean":"2024-04-15T15:14:57.136932+0000","last_became_active":"2024-04-15T15:14:56.092506+0000","last_became_peered":"2024-04-15T15:14:56.092506+0000","last_unstale":"2024-04-15T15:14:57.136932+0000","last_undegraded":"2024-04-15T15:14:57.136932+0000","last_fullsized":"2024-04-15T15:14:57.136932+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:55.075455+0000","last_clean_scrub_stamp":"2024-04-15T15:14:55.075455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:55:14.590939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108967+0000","last_change":"2024-04-15T15:14:52.082533+0000","last_active":"2024-04-15T15:14:57.108967+0000","last_peered":"2024-04-15T15:14:57.108967+0000","last_clean":"2024-04-15T15:14:57.108967+0000","last_became_active":"2024-04-15T15:14:52.082257+0000","last_became_peered":"2024-04-15T15:14:52.082257+0000","last_unstale":"2024-04-15T15:14:57.108967+0000","last_undegraded":"2024-04-15T15:14:57.108967+0000","last_fullsized":"2024-04-15T15:14:57.108967+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:34:51.775979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.107671+0000","last_change":"2024-04-15T15:14:52.084944+0000","last_active":"2024-04-15T15:14:57.107671+0000","last_peered":"2024-04-15T15:14:57.107671+0000","last_clean":"2024-04-15T15:14:57.107671+0000","last_became_active":"2024-04-15T15:14:52.084665+0000","last_became_peered":"2024-04-15T15:14:52.084665+0000","last_unstale":"2024-04-15T15:14:57.107671+0000","last_undegraded":"2024-04-15T15:14:57.107671+0000","last_fullsized":"2024-04-15T15:14:57.107671+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:58:41.444122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108873+0000","last_change":"2024-04-15T15:14:52.082741+0000","last_active":"2024-04-15T15:14:57.108873+0000","last_peered":"2024-04-15T15:14:57.108873+0000","last_clean":"2024-04-15T15:14:57.108873+0000","last_became_active":"2024-04-15T15:14:52.082494+0000","last_became_peered":"2024-04-15T15:14:52.082494+0000","last_unstale":"2024-04-15T15:14:57.108873+0000","last_undegraded":"2024-04-15T15:14:57.108873+0000","last_fullsized":"2024-04-15T15:14:57.108873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:38:35.185619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137649+0000","last_change":"2024-04-15T15:14:52.076826+0000","last_active":"2024-04-15T15:14:57.137649+0000","last_peered":"2024-04-15T15:14:57.137649+0000","last_clean":"2024-04-15T15:14:57.137649+0000","last_became_active":"2024-04-15T15:14:52.076541+0000","last_became_peered":"2024-04-15T15:14:52.076541+0000","last_unstale":"2024-04-15T15:14:57.137649+0000","last_undegraded":"2024-04-15T15:14:57.137649+0000","last_fullsized":"2024-04-15T15:14:57.137649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:13:01.642838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137024+0000","last_change":"2024-04-15T15:14:52.075468+0000","last_active":"2024-04-15T15:14:57.137024+0000","last_peered":"2024-04-15T15:14:57.137024+0000","last_clean":"2024-04-15T15:14:57.137024+0000","last_became_active":"2024-04-15T15:14:52.075173+0000","last_became_peered":"2024-04-15T15:14:52.075173+0000","last_unstale":"2024-04-15T15:14:57.137024+0000","last_undegraded":"2024-04-15T15:14:57.137024+0000","last_fullsized":"2024-04-15T15:14:57.137024+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:50:41.263447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137555+0000","last_change":"2024-04-15T15:14:52.086866+0000","last_active":"2024-04-15T15:14:57.137555+0000","last_peered":"2024-04-15T15:14:57.137555+0000","last_clean":"2024-04-15T15:14:57.137555+0000","last_became_active":"2024-04-15T15:14:52.086547+0000","last_became_peered":"2024-04-15T15:14:52.086547+0000","last_unstale":"2024-04-15T15:14:57.137555+0000","last_undegraded":"2024-04-15T15:14:57.137555+0000","last_fullsized":"2024-04-15T15:14:57.137555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:37:31.656434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.110000+0000","last_change":"2024-04-15T15:14:52.074917+0000","last_active":"2024-04-15T15:14:57.110000+0000","last_peered":"2024-04-15T15:14:57.110000+0000","last_clean":"2024-04-15T15:14:57.110000+0000","last_became_active":"2024-04-15T15:14:52.074643+0000","last_became_peered":"2024-04-15T15:14:52.074643+0000","last_unstale":"2024-04-15T15:14:57.110000+0000","last_undegraded":"2024-04-15T15:14:57.110000+0000","last_fullsized":"2024-04-15T15:14:57.110000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:12:07.772549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138133+0000","last_change":"2024-04-15T15:14:52.076263+0000","last_active":"2024-04-15T15:14:57.138133+0000","last_peered":"2024-04-15T15:14:57.138133+0000","last_clean":"2024-04-15T15:14:57.138133+0000","last_became_active":"2024-04-15T15:14:52.075961+0000","last_became_peered":"2024-04-15T15:14:52.075961+0000","last_unstale":"2024-04-15T15:14:57.138133+0000","last_undegraded":"2024-04-15T15:14:57.138133+0000","last_fullsized":"2024-04-15T15:14:57.138133+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:12:40.420221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108421+0000","last_change":"2024-04-15T15:14:52.074353+0000","last_active":"2024-04-15T15:14:57.108421+0000","last_peered":"2024-04-15T15:14:57.108421+0000","last_clean":"2024-04-15T15:14:57.108421+0000","last_became_active":"2024-04-15T15:14:52.074067+0000","last_became_peered":"2024-04-15T15:14:52.074067+0000","last_unstale":"2024-04-15T15:14:57.108421+0000","last_undegraded":"2024-04-15T15:14:57.108421+0000","last_fullsized":"2024-04-15T15:14:57.108421+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:26:45.460787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136083+0000","last_change":"2024-04-15T15:14:52.075541+0000","last_active":"2024-04-15T15:14:57.136083+0000","last_peered":"2024-04-15T15:14:57.136083+0000","last_clean":"2024-04-15T15:14:57.136083+0000","last_became_active":"2024-04-15T15:14:52.075227+0000","last_became_peered":"2024-04-15T15:14:52.075227+0000","last_unstale":"2024-04-15T15:14:57.136083+0000","last_undegraded":"2024-04-15T15:14:57.136083+0000","last_fullsized":"2024-04-15T15:14:57.136083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:24:30.714153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108791+0000","last_change":"2024-04-15T15:14:52.080306+0000","last_active":"2024-04-15T15:14:57.108791+0000","last_peered":"2024-04-15T15:14:57.108791+0000","last_clean":"2024-04-15T15:14:57.108791+0000","last_became_active":"2024-04-15T15:14:52.079784+0000","last_became_peered":"2024-04-15T15:14:52.079784+0000","last_unstale":"2024-04-15T15:14:57.108791+0000","last_undegraded":"2024-04-15T15:14:57.108791+0000","last_fullsized":"2024-04-15T15:14:57.108791+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:10:19.696254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137426+0000","last_change":"2024-04-15T15:14:52.081224+0000","last_active":"2024-04-15T15:14:57.137426+0000","last_peered":"2024-04-15T15:14:57.137426+0000","last_clean":"2024-04-15T15:14:57.137426+0000","last_became_active":"2024-04-15T15:14:52.080976+0000","last_became_peered":"2024-04-15T15:14:52.080976+0000","last_unstale":"2024-04-15T15:14:57.137426+0000","last_undegraded":"2024-04-15T15:14:57.137426+0000","last_fullsized":"2024-04-15T15:14:57.137426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:21:50.111549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109464+0000","last_change":"2024-04-15T15:14:52.085463+0000","last_active":"2024-04-15T15:14:57.109464+0000","last_peered":"2024-04-15T15:14:57.109464+0000","last_clean":"2024-04-15T15:14:57.109464+0000","last_became_active":"2024-04-15T15:14:52.085175+0000","last_became_peered":"2024-04-15T15:14:52.085175+0000","last_unstale":"2024-04-15T15:14:57.109464+0000","last_undegraded":"2024-04-15T15:14:57.109464+0000","last_fullsized":"2024-04-15T15:14:57.109464+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:50:41.263447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136838+0000","last_change":"2024-04-15T15:14:52.092434+0000","last_active":"2024-04-15T15:14:57.136838+0000","last_peered":"2024-04-15T15:14:57.136838+0000","last_clean":"2024-04-15T15:14:57.136838+0000","last_became_active":"2024-04-15T15:14:52.088973+0000","last_became_peered":"2024-04-15T15:14:52.088973+0000","last_unstale":"2024-04-15T15:14:57.136838+0000","last_undegraded":"2024-04-15T15:14:57.136838+0000","last_fullsized":"2024-04-15T15:14:57.136838+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:03:03.850709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.107875+0000","last_change":"2024-04-15T15:14:52.076802+0000","last_active":"2024-04-15T15:14:57.107875+0000","last_peered":"2024-04-15T15:14:57.107875+0000","last_clean":"2024-04-15T15:14:57.107875+0000","last_became_active":"2024-04-15T15:14:52.076523+0000","last_became_peered":"2024-04-15T15:14:52.076523+0000","last_unstale":"2024-04-15T15:14:57.107875+0000","last_undegraded":"2024-04-15T15:14:57.107875+0000","last_fullsized":"2024-04-15T15:14:57.107875+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:07:49.420517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109362+0000","last_change":"2024-04-15T15:14:52.084418+0000","last_active":"2024-04-15T15:14:57.109362+0000","last_peered":"2024-04-15T15:14:57.109362+0000","last_clean":"2024-04-15T15:14:57.109362+0000","last_became_active":"2024-04-15T15:14:52.083428+0000","last_became_peered":"2024-04-15T15:14:52.083428+0000","last_unstale":"2024-04-15T15:14:57.109362+0000","last_undegraded":"2024-04-15T15:14:57.109362+0000","last_fullsized":"2024-04-15T15:14:57.109362+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:12:59.006568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108279+0000","last_change":"2024-04-15T15:14:52.083894+0000","last_active":"2024-04-15T15:14:57.108279+0000","last_peered":"2024-04-15T15:14:57.108279+0000","last_clean":"2024-04-15T15:14:57.108279+0000","last_became_active":"2024-04-15T15:14:52.083636+0000","last_became_peered":"2024-04-15T15:14:52.083636+0000","last_unstale":"2024-04-15T15:14:57.108279+0000","last_undegraded":"2024-04-15T15:14:57.108279+0000","last_fullsized":"2024-04-15T15:14:57.108279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:12:07.772549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136007+0000","last_change":"2024-04-15T15:14:52.087934+0000","last_active":"2024-04-15T15:14:57.136007+0000","last_peered":"2024-04-15T15:14:57.136007+0000","last_clean":"2024-04-15T15:14:57.136007+0000","last_became_active":"2024-04-15T15:14:52.087040+0000","last_became_peered":"2024-04-15T15:14:52.087040+0000","last_unstale":"2024-04-15T15:14:57.136007+0000","last_undegraded":"2024-04-15T15:14:57.136007+0000","last_fullsized":"2024-04-15T15:14:57.136007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:24:30.714153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.138511+0000","last_change":"2024-04-15T15:14:52.079696+0000","last_active":"2024-04-15T15:14:57.138511+0000","last_peered":"2024-04-15T15:14:57.138511+0000","last_clean":"2024-04-15T15:14:57.138511+0000","last_became_active":"2024-04-15T15:14:52.079438+0000","last_became_peered":"2024-04-15T15:14:52.079438+0000","last_unstale":"2024-04-15T15:14:57.138511+0000","last_undegraded":"2024-04-15T15:14:57.138511+0000","last_fullsized":"2024-04-15T15:14:57.138511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:27:02.854265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136589+0000","last_change":"2024-04-15T15:14:52.076841+0000","last_active":"2024-04-15T15:14:57.136589+0000","last_peered":"2024-04-15T15:14:57.136589+0000","last_clean":"2024-04-15T15:14:57.136589+0000","last_became_active":"2024-04-15T15:14:52.076567+0000","last_became_peered":"2024-04-15T15:14:52.076567+0000","last_unstale":"2024-04-15T15:14:57.136589+0000","last_undegraded":"2024-04-15T15:14:57.136589+0000","last_fullsized":"2024-04-15T15:14:57.136589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:51:22.814881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136685+0000","last_change":"2024-04-15T15:14:52.078599+0000","last_active":"2024-04-15T15:14:57.136685+0000","last_peered":"2024-04-15T15:14:57.136685+0000","last_clean":"2024-04-15T15:14:57.136685+0000","last_became_active":"2024-04-15T15:14:52.078231+0000","last_became_peered":"2024-04-15T15:14:52.078231+0000","last_unstale":"2024-04-15T15:14:57.136685+0000","last_undegraded":"2024-04-15T15:14:57.136685+0000","last_fullsized":"2024-04-15T15:14:57.136685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:58:41.444122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109712+0000","last_change":"2024-04-15T15:14:52.080747+0000","last_active":"2024-04-15T15:14:57.109712+0000","last_peered":"2024-04-15T15:14:57.109712+0000","last_clean":"2024-04-15T15:14:57.109712+0000","last_became_active":"2024-04-15T15:14:52.080475+0000","last_became_peered":"2024-04-15T15:14:52.080475+0000","last_unstale":"2024-04-15T15:14:57.109712+0000","last_undegraded":"2024-04-15T15:14:57.109712+0000","last_fullsized":"2024-04-15T15:14:57.109712+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:37:31.656434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109996+0000","last_change":"2024-04-15T15:14:52.081405+0000","last_active":"2024-04-15T15:14:57.109996+0000","last_peered":"2024-04-15T15:14:57.109996+0000","last_clean":"2024-04-15T15:14:57.109996+0000","last_became_active":"2024-04-15T15:14:52.081120+0000","last_became_peered":"2024-04-15T15:14:52.081120+0000","last_unstale":"2024-04-15T15:14:57.109996+0000","last_undegraded":"2024-04-15T15:14:57.109996+0000","last_fullsized":"2024-04-15T15:14:57.109996+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:12:40.420221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136834+0000","last_change":"2024-04-15T15:14:52.092337+0000","last_active":"2024-04-15T15:14:57.136834+0000","last_peered":"2024-04-15T15:14:57.136834+0000","last_clean":"2024-04-15T15:14:57.136834+0000","last_became_active":"2024-04-15T15:14:52.091800+0000","last_became_peered":"2024-04-15T15:14:52.091800+0000","last_unstale":"2024-04-15T15:14:57.136834+0000","last_undegraded":"2024-04-15T15:14:57.136834+0000","last_fullsized":"2024-04-15T15:14:57.136834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:12:59.006568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136083+0000","last_change":"2024-04-15T15:14:52.079078+0000","last_active":"2024-04-15T15:14:57.136083+0000","last_peered":"2024-04-15T15:14:57.136083+0000","last_clean":"2024-04-15T15:14:57.136083+0000","last_became_active":"2024-04-15T15:14:52.078796+0000","last_became_peered":"2024-04-15T15:14:52.078796+0000","last_unstale":"2024-04-15T15:14:57.136083+0000","last_undegraded":"2024-04-15T15:14:57.136083+0000","last_fullsized":"2024-04-15T15:14:57.136083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:34:51.775979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.135766+0000","last_change":"2024-04-15T15:14:52.091742+0000","last_active":"2024-04-15T15:14:57.135766+0000","last_peered":"2024-04-15T15:14:57.135766+0000","last_clean":"2024-04-15T15:14:57.135766+0000","last_became_active":"2024-04-15T15:14:52.087579+0000","last_became_peered":"2024-04-15T15:14:52.087579+0000","last_unstale":"2024-04-15T15:14:57.135766+0000","last_undegraded":"2024-04-15T15:14:57.135766+0000","last_fullsized":"2024-04-15T15:14:57.135766+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:51:22.814881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.108707+0000","last_change":"2024-04-15T15:14:52.084878+0000","last_active":"2024-04-15T15:14:57.108707+0000","last_peered":"2024-04-15T15:14:57.108707+0000","last_clean":"2024-04-15T15:14:57.108707+0000","last_became_active":"2024-04-15T15:14:52.084602+0000","last_became_peered":"2024-04-15T15:14:52.084602+0000","last_unstale":"2024-04-15T15:14:57.108707+0000","last_undegraded":"2024-04-15T15:14:57.108707+0000","last_fullsized":"2024-04-15T15:14:57.108707+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:51:22.814881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136011+0000","last_change":"2024-04-15T15:14:52.085909+0000","last_active":"2024-04-15T15:14:57.136011+0000","last_peered":"2024-04-15T15:14:57.136011+0000","last_clean":"2024-04-15T15:14:57.136011+0000","last_became_active":"2024-04-15T15:14:52.085266+0000","last_became_peered":"2024-04-15T15:14:52.085266+0000","last_unstale":"2024-04-15T15:14:57.136011+0000","last_undegraded":"2024-04-15T15:14:57.136011+0000","last_fullsized":"2024-04-15T15:14:57.136011+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:26:45.460787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.109544+0000","last_change":"2024-04-15T15:14:52.084086+0000","last_active":"2024-04-15T15:14:57.109544+0000","last_peered":"2024-04-15T15:14:57.109544+0000","last_clean":"2024-04-15T15:14:57.109544+0000","last_became_active":"2024-04-15T15:14:52.083193+0000","last_became_peered":"2024-04-15T15:14:52.083193+0000","last_unstale":"2024-04-15T15:14:57.109544+0000","last_undegraded":"2024-04-15T15:14:57.109544+0000","last_fullsized":"2024-04-15T15:14:57.109544+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:12:40.420221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136555+0000","last_change":"2024-04-15T15:14:52.072448+0000","last_active":"2024-04-15T15:14:57.136555+0000","last_peered":"2024-04-15T15:14:57.136555+0000","last_clean":"2024-04-15T15:14:57.136555+0000","last_became_active":"2024-04-15T15:14:52.072162+0000","last_became_peered":"2024-04-15T15:14:52.072162+0000","last_unstale":"2024-04-15T15:14:57.136555+0000","last_undegraded":"2024-04-15T15:14:57.136555+0000","last_fullsized":"2024-04-15T15:14:57.136555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:24:30.714153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.137467+0000","last_change":"2024-04-15T15:14:52.087484+0000","last_active":"2024-04-15T15:14:57.137467+0000","last_peered":"2024-04-15T15:14:57.137467+0000","last_clean":"2024-04-15T15:14:57.137467+0000","last_became_active":"2024-04-15T15:14:52.087195+0000","last_became_peered":"2024-04-15T15:14:52.087195+0000","last_unstale":"2024-04-15T15:14:57.137467+0000","last_undegraded":"2024-04-15T15:14:57.137467+0000","last_fullsized":"2024-04-15T15:14:57.137467+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:50:41.263447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-15T15:14:57.136721+0000","last_change":"2024-04-15T15:14:52.083998+0000","last_active":"2024-04-15T15:14:57.136721+0000","last_peered":"2024-04-15T15:14:57.136721+0000","last_clean":"2024-04-15T15:14:57.136721+0000","last_became_active":"2024-04-15T15:14:52.083702+0000","last_became_peered":"2024-04-15T15:14:52.083702+0000","last_unstale":"2024-04-15T15:14:57.136721+0000","last_undegraded":"2024-04-15T15:14:57.136721+0000","last_fullsized":"2024-04-15T15:14:57.136721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:51.050152+0000","last_clean_scrub_stamp":"2024-04-15T15:14:51.050152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:21:48.470063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:15:13.868 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-15T15:15:13.868 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-15T15:15:13.868 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-15T15:15:13.868 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-15T15:15:14.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.002+0000 7f5c7f237640 1 Processor -- start 2024-04-15T15:15:14.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.002+0000 7f5c7f237640 1 -- start start 2024-04-15T15:15:14.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.003+0000 7f5c7f237640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5c780cd5f0 0x7f5c780ce9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.003+0000 7f5c7f237640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c780afb60 con 0x7f5c780cef30 2024-04-15T15:15:14.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.003+0000 7f5c7f237640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c780af880 con 0x7f5c780cab00 2024-04-15T15:15:14.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.003+0000 7f5c7f237640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f5c780b2e10 con 0x7f5c780cd5f0 2024-04-15T15:15:14.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.003+0000 7f5c7d7d4640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f5c780cef30 0x7f5c780cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37900/0 (socket says 172.21.15.72:37900) 2024-04-15T15:15:14.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.003+0000 7f5c7d7d4640 1 -- 172.21.15.72:0/655360652 learned_addr learned my addr 172.21.15.72:0/655360652 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:14.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.003+0000 7f5c6ffff640 1 --2- 172.21.15.72:0/655360652 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5c780cd5f0 0x7f5c780ce9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.003+0000 7f5c6f7fe640 1 -- 172.21.15.72:0/655360652 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1900989602 0 0) 0x7f5c780afb60 con 0x7f5c780cef30 2024-04-15T15:15:14.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.003+0000 7f5c6f7fe640 1 -- 172.21.15.72:0/655360652 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c60003520 con 0x7f5c780cef30 2024-04-15T15:15:14.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.003+0000 7f5c6f7fe640 1 -- 172.21.15.72:0/655360652 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 243283585 0 0) 0x7f5c780af880 con 0x7f5c780cab00 2024-04-15T15:15:14.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.003+0000 7f5c6f7fe640 1 -- 172.21.15.72:0/655360652 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c780afb60 con 0x7f5c780cab00 2024-04-15T15:15:14.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.003+0000 7f5c6f7fe640 1 -- 172.21.15.72:0/655360652 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2642127506 0 0) 0x7f5c60003520 con 0x7f5c780cef30 2024-04-15T15:15:14.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.003+0000 7f5c6f7fe640 1 -- 172.21.15.72:0/655360652 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5c780cd5f0 msgr2=0x7f5c780ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:14.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.003+0000 7f5c6f7fe640 1 --2- 172.21.15.72:0/655360652 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5c780cd5f0 0x7f5c780ce9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.003+0000 7f5c6f7fe640 1 -- 172.21.15.72:0/655360652 >> v1:172.21.15.175:6789/0 conn(0x7f5c780cab00 legacy=0x7f5c780caee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.003+0000 7f5c6f7fe640 1 -- 172.21.15.72:0/655360652 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c780cdf10 con 0x7f5c780cef30 2024-04-15T15:15:14.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.003+0000 7f5c6f7fe640 1 -- 172.21.15.72:0/655360652 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f5c70001670 con 0x7f5c780cef30 2024-04-15T15:15:14.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.003+0000 7f5c6ffff640 1 --2- 172.21.15.72:0/655360652 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5c780cd5f0 0x7f5c780ce9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:14.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.003+0000 7f5c6f7fe640 1 -- 172.21.15.72:0/655360652 >> v1:172.21.15.72:6789/0 conn(0x7f5c780cef30 legacy=0x7f5c780cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.004+0000 7f5c6f7fe640 1 --2- 172.21.15.72:0/655360652 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c60003b70 0x7f5c60003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.004+0000 7f5c6f7fe640 1 --2- 172.21.15.72:0/655360652 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5c780cd5f0 0x7f5c60004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.004+0000 7f5c6f7fe640 1 --2- 172.21.15.72:0/655360652 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5c60004d40 0x7f5c600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.004+0000 7f5c6f7fe640 1 -- 172.21.15.72:0/655360652 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f5c6000bad0 con 0x7f5c60003b70 2024-04-15T15:15:14.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.004+0000 7f5c6f7fe640 1 -- 172.21.15.72:0/655360652 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f5c6000bc80 con 0x7f5c60004d40 2024-04-15T15:15:14.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.004+0000 7f5c6f7fe640 1 -- 172.21.15.72:0/655360652 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f5c6000be80 con 0x7f5c780cd5f0 2024-04-15T15:15:14.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.004+0000 7f5c7cfd3640 1 --2- 172.21.15.72:0/655360652 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c60003b70 0x7f5c60003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.004+0000 7f5c6ffff640 1 --2- 172.21.15.72:0/655360652 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5c780cd5f0 0x7f5c60004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.004+0000 7f5c7d7d4640 1 --2- 172.21.15.72:0/655360652 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5c60004d40 0x7f5c600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.004+0000 7f5c7cfd3640 1 -- 172.21.15.72:0/655360652 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5c780cd5f0 msgr2=0x7f5c60004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:14.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.004+0000 7f5c7cfd3640 1 --2- 172.21.15.72:0/655360652 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5c780cd5f0 0x7f5c60004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.004+0000 7f5c7cfd3640 1 -- 172.21.15.72:0/655360652 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5c60004d40 msgr2=0x7f5c600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:14.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.004+0000 7f5c7cfd3640 1 --2- 172.21.15.72:0/655360652 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5c60004d40 0x7f5c600051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.004+0000 7f5c7cfd3640 1 -- 172.21.15.72:0/655360652 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c60003520 con 0x7f5c60003b70 2024-04-15T15:15:14.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.005+0000 7f5c7cfd3640 1 --2- 172.21.15.72:0/655360652 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c60003b70 0x7f5c60003f70 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5c74001fd0 tx=0x7f5c60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=261806a14e56f7e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:14.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.005+0000 7f5c6f7fe640 1 -- 172.21.15.72:0/655360652 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c74001bf0 con 0x7f5c60003b70 2024-04-15T15:15:14.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.005+0000 7f5c6f7fe640 1 -- 172.21.15.72:0/655360652 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c74003b70 con 0x7f5c60003b70 2024-04-15T15:15:14.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.005+0000 7f5c6f7fe640 1 -- 172.21.15.72:0/655360652 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c740050d0 con 0x7f5c60003b70 2024-04-15T15:15:14.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.005+0000 7f5c7f237640 1 -- 172.21.15.72:0/655360652 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c60003b70 msgr2=0x7f5c60003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.005+0000 7f5c7f237640 1 --2- 172.21.15.72:0/655360652 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c60003b70 0x7f5c60003f70 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5c74001fd0 tx=0x7f5c60005830 comp rx=0 tx=0).stop 2024-04-15T15:15:14.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.005+0000 7f5c7d7d4640 1 -- 172.21.15.72:0/655360652 reap_dead start 2024-04-15T15:15:14.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.005+0000 7f5c7f237640 1 -- 172.21.15.72:0/655360652 shutdown_connections 2024-04-15T15:15:14.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.005+0000 7f5c7f237640 1 -- 172.21.15.72:0/655360652 >> 172.21.15.72:0/655360652 conn(0x7f5c780b1ee0 msgr2=0x7f5c780b22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:14.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.005+0000 7f5c7f237640 1 -- 172.21.15.72:0/655360652 shutdown_connections 2024-04-15T15:15:14.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.005+0000 7f5c7f237640 1 -- 172.21.15.72:0/655360652 wait complete. 2024-04-15T15:15:14.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.007+0000 7f5c7f237640 1 Processor -- start 2024-04-15T15:15:14.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.007+0000 7f5c7f237640 1 -- start start 2024-04-15T15:15:14.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.007+0000 7f5c7f237640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c60003b70 0x7f5c781579d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.007+0000 7f5c7f237640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5c781f2b20 0x7f5c780aec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.007+0000 7f5c7f237640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5c780af190 0x7f5c7809d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.007+0000 7f5c7f237640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f5c780cfad0 con 0x7f5c60003b70 2024-04-15T15:15:14.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.007+0000 7f5c7f237640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f5c780ce4d0 con 0x7f5c781f2b20 2024-04-15T15:15:14.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.007+0000 7f5c7f237640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f5c781f1e10 con 0x7f5c780af190 2024-04-15T15:15:14.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.008+0000 7f5c7cfd3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c60003b70 0x7f5c781579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.008+0000 7f5c7cfd3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c60003b70 0x7f5c781579d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45490/0 (socket says 172.21.15.72:45490) 2024-04-15T15:15:14.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.008+0000 7f5c7cfd3640 1 -- 172.21.15.72:0/2259912637 learned_addr learned my addr 172.21.15.72:0/2259912637 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:14.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.008+0000 7f5c6ffff640 1 --2- 172.21.15.72:0/2259912637 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5c781f2b20 0x7f5c780aec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.008+0000 7f5c7d7d4640 1 --2- 172.21.15.72:0/2259912637 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5c780af190 0x7f5c7809d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.008+0000 7f5c7cfd3640 1 -- 172.21.15.72:0/2259912637 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5c780af190 msgr2=0x7f5c7809d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.008+0000 7f5c7cfd3640 1 --2- 172.21.15.72:0/2259912637 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5c780af190 0x7f5c7809d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.008+0000 7f5c7cfd3640 1 -- 172.21.15.72:0/2259912637 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5c781f2b20 msgr2=0x7f5c780aec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.008+0000 7f5c7cfd3640 1 --2- 172.21.15.72:0/2259912637 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5c781f2b20 0x7f5c780aec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.008+0000 7f5c7cfd3640 1 -- 172.21.15.72:0/2259912637 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c781f2060 con 0x7f5c60003b70 2024-04-15T15:15:14.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.008+0000 7f5c7d7d4640 1 --2- 172.21.15.72:0/2259912637 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5c780af190 0x7f5c7809d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:14.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.008+0000 7f5c7cfd3640 1 --2- 172.21.15.72:0/2259912637 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c60003b70 0x7f5c781579d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5c74005040 tx=0x7f5c74001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:14.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.008+0000 7f5c6d7fa640 1 -- 172.21.15.72:0/2259912637 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c74039040 con 0x7f5c60003b70 2024-04-15T15:15:14.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.008+0000 7f5c6d7fa640 1 -- 172.21.15.72:0/2259912637 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c74042070 con 0x7f5c60003b70 2024-04-15T15:15:14.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.008+0000 7f5c7f237640 1 -- 172.21.15.72:0/2259912637 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c7809dfa0 con 0x7f5c60003b70 2024-04-15T15:15:14.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.008+0000 7f5c6d7fa640 1 -- 172.21.15.72:0/2259912637 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c74001bf0 con 0x7f5c60003b70 2024-04-15T15:15:14.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.008+0000 7f5c7f237640 1 -- 172.21.15.72:0/2259912637 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c781f2600 con 0x7f5c60003b70 2024-04-15T15:15:14.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.009+0000 7f5c7f237640 1 -- 172.21.15.72:0/2259912637 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c780b4180 con 0x7f5c60003b70 2024-04-15T15:15:14.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.010+0000 7f5c6d7fa640 1 -- 172.21.15.72:0/2259912637 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5c7402c570 con 0x7f5c60003b70 2024-04-15T15:15:14.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.010+0000 7f5c6d7fa640 1 --2- 172.21.15.72:0/2259912637 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f5c6406c8f0 0x7f5c6406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.010+0000 7f5c6d7fa640 1 -- 172.21.15.72:0/2259912637 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f5c740ad7d0 con 0x7f5c60003b70 2024-04-15T15:15:14.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.010+0000 7f5c6ffff640 1 --2- 172.21.15.72:0/2259912637 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f5c6406c8f0 0x7f5c6406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.010+0000 7f5c6ffff640 1 --2- 172.21.15.72:0/2259912637 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f5c6406c8f0 0x7f5c6406edb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5c7815e4c0 tx=0x7f5c68008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:14.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.014+0000 7f5c6d7fa640 1 -- 172.21.15.72:0/2259912637 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5c780b4180 con 0x7f5c60003b70 2024-04-15T15:15:14.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.231+0000 7f5c7f237640 1 -- 172.21.15.72:0/2259912637 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f5c780689e0 con 0x7f5c60003b70 2024-04-15T15:15:14.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.232+0000 7f5c6d7fa640 1 -- 172.21.15.72:0/2259912637 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f5c7403c390 con 0x7f5c60003b70 2024-04-15T15:15:14.233 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:14.233 INFO:teuthology.orchestra.run.smithi072.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-15T15:15:14.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.234+0000 7f5c7f237640 1 -- 172.21.15.72:0/2259912637 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f5c6406c8f0 msgr2=0x7f5c6406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.234+0000 7f5c7f237640 1 --2- 172.21.15.72:0/2259912637 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f5c6406c8f0 0x7f5c6406edb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5c7815e4c0 tx=0x7f5c68008040 comp rx=0 tx=0).stop 2024-04-15T15:15:14.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.234+0000 7f5c7f237640 1 -- 172.21.15.72:0/2259912637 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c60003b70 msgr2=0x7f5c781579d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.234+0000 7f5c7f237640 1 --2- 172.21.15.72:0/2259912637 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c60003b70 0x7f5c781579d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5c74005040 tx=0x7f5c74001650 comp rx=0 tx=0).stop 2024-04-15T15:15:14.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.234+0000 7f5c7f237640 1 -- 172.21.15.72:0/2259912637 shutdown_connections 2024-04-15T15:15:14.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.234+0000 7f5c7f237640 1 --2- 172.21.15.72:0/2259912637 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f5c6406c8f0 0x7f5c6406edb0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.234+0000 7f5c7f237640 1 --2- 172.21.15.72:0/2259912637 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5c780af190 0x7f5c7809d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.234+0000 7f5c7f237640 1 --2- 172.21.15.72:0/2259912637 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5c781f2b20 0x7f5c780aec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.234+0000 7f5c7f237640 1 --2- 172.21.15.72:0/2259912637 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5c60003b70 0x7f5c781579d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.234+0000 7f5c7f237640 1 -- 172.21.15.72:0/2259912637 >> 172.21.15.72:0/2259912637 conn(0x7f5c780b1ee0 msgr2=0x7f5c780b0190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:14.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.234+0000 7f5c7f237640 1 -- 172.21.15.72:0/2259912637 shutdown_connections 2024-04-15T15:15:14.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.234+0000 7f5c7f237640 1 -- 172.21.15.72:0/2259912637 wait complete. 2024-04-15T15:15:14.246 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-15T15:15:14.247 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:14.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.349+0000 7fe04f674640 1 Processor -- start 2024-04-15T15:15:14.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.349+0000 7fe04f674640 1 -- start start 2024-04-15T15:15:14.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.350+0000 7fe04f674640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe0480d1790 0x7fe0480b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.350+0000 7fe04f674640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe04815caf0 con 0x7fe04815c710 2024-04-15T15:15:14.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.350+0000 7fe04f674640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe04806f510 con 0x7fe0480b5790 2024-04-15T15:15:14.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.350+0000 7fe04f674640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fe0480ccb90 con 0x7fe0480d1790 2024-04-15T15:15:14.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.350+0000 7fe04d410640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fe04815c710 0x7fe0480d1080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37912/0 (socket says 172.21.15.72:37912) 2024-04-15T15:15:14.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.350+0000 7fe04d410640 1 -- 172.21.15.72:0/3930345246 learned_addr learned my addr 172.21.15.72:0/3930345246 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:14.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.350+0000 7fe04cc0f640 1 --2- 172.21.15.72:0/3930345246 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe0480d1790 0x7fe0480b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.350+0000 7fe03ffff640 1 -- 172.21.15.72:0/3930345246 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1889311931 0 0) 0x7fe04815caf0 con 0x7fe04815c710 2024-04-15T15:15:14.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.350+0000 7fe03ffff640 1 -- 172.21.15.72:0/3930345246 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe034003520 con 0x7fe04815c710 2024-04-15T15:15:14.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.350+0000 7fe03ffff640 1 -- 172.21.15.72:0/3930345246 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1526880014 0 0) 0x7fe034003520 con 0x7fe04815c710 2024-04-15T15:15:14.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.350+0000 7fe03ffff640 1 -- 172.21.15.72:0/3930345246 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe0480d1790 msgr2=0x7fe0480b66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:14.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.350+0000 7fe03ffff640 1 --2- 172.21.15.72:0/3930345246 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe0480d1790 0x7fe0480b66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.350+0000 7fe03ffff640 1 -- 172.21.15.72:0/3930345246 >> v1:172.21.15.175:6789/0 conn(0x7fe0480b5790 legacy=0x7fe0480cd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.350+0000 7fe03ffff640 1 -- 172.21.15.72:0/3930345246 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0480cce70 con 0x7fe04815c710 2024-04-15T15:15:14.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.350+0000 7fe03ffff640 1 -- 172.21.15.72:0/3930345246 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7fe044001670 con 0x7fe04815c710 2024-04-15T15:15:14.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.350+0000 7fe03ffff640 1 -- 172.21.15.72:0/3930345246 >> v1:172.21.15.72:6789/0 conn(0x7fe04815c710 legacy=0x7fe0480d1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.350+0000 7fe03ffff640 1 --2- 172.21.15.72:0/3930345246 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe0480d1790 0x7fe0340039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.350+0000 7fe03ffff640 1 --2- 172.21.15.72:0/3930345246 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe034003fb0 0x7fe034004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.350+0000 7fe03ffff640 1 --2- 172.21.15.72:0/3930345246 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe034004d40 0x7fe0340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.350+0000 7fe03ffff640 1 -- 172.21.15.72:0/3930345246 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe0480ccb90 con 0x7fe034003fb0 2024-04-15T15:15:14.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.350+0000 7fe03ffff640 1 -- 172.21.15.72:0/3930345246 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fe03400bb90 con 0x7fe034004d40 2024-04-15T15:15:14.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.350+0000 7fe03ffff640 1 -- 172.21.15.72:0/3930345246 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fe03400bd90 con 0x7fe0480d1790 2024-04-15T15:15:14.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.350+0000 7fe04cc0f640 1 --2- 172.21.15.72:0/3930345246 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe034003fb0 0x7fe034004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.351+0000 7fe04d410640 1 --2- 172.21.15.72:0/3930345246 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe0480d1790 0x7fe0340039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.351+0000 7fe04dc11640 1 --2- 172.21.15.72:0/3930345246 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe034004d40 0x7fe0340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.351+0000 7fe04cc0f640 1 -- 172.21.15.72:0/3930345246 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe0480d1790 msgr2=0x7fe0340039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:14.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.351+0000 7fe04cc0f640 1 --2- 172.21.15.72:0/3930345246 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe0480d1790 0x7fe0340039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.351+0000 7fe04cc0f640 1 -- 172.21.15.72:0/3930345246 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe034004d40 msgr2=0x7fe0340051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:14.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.351+0000 7fe04cc0f640 1 --2- 172.21.15.72:0/3930345246 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe034004d40 0x7fe0340051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.351+0000 7fe04cc0f640 1 -- 172.21.15.72:0/3930345246 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe034003520 con 0x7fe034003fb0 2024-04-15T15:15:14.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.352+0000 7fe04cc0f640 1 --2- 172.21.15.72:0/3930345246 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe034003fb0 0x7fe034004800 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe038009ac0 tx=0x7fe034005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f23fd47930312099 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:14.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.352+0000 7fe03ffff640 1 -- 172.21.15.72:0/3930345246 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe038039070 con 0x7fe034003fb0 2024-04-15T15:15:14.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.352+0000 7fe03ffff640 1 -- 172.21.15.72:0/3930345246 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe03802c600 con 0x7fe034003fb0 2024-04-15T15:15:14.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.352+0000 7fe03ffff640 1 -- 172.21.15.72:0/3930345246 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe038009690 con 0x7fe034003fb0 2024-04-15T15:15:14.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.352+0000 7fe04f674640 1 -- 172.21.15.72:0/3930345246 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe034003fb0 msgr2=0x7fe034004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.352+0000 7fe04f674640 1 --2- 172.21.15.72:0/3930345246 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe034003fb0 0x7fe034004800 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe038009ac0 tx=0x7fe034005830 comp rx=0 tx=0).stop 2024-04-15T15:15:14.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.352+0000 7fe04dc11640 1 -- 172.21.15.72:0/3930345246 reap_dead start 2024-04-15T15:15:14.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.352+0000 7fe04f674640 1 -- 172.21.15.72:0/3930345246 shutdown_connections 2024-04-15T15:15:14.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.352+0000 7fe04f674640 1 -- 172.21.15.72:0/3930345246 >> 172.21.15.72:0/3930345246 conn(0x7fe0480b3f90 msgr2=0x7fe0480b43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:14.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.352+0000 7fe04f674640 1 -- 172.21.15.72:0/3930345246 shutdown_connections 2024-04-15T15:15:14.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.352+0000 7fe04f674640 1 -- 172.21.15.72:0/3930345246 wait complete. 2024-04-15T15:15:14.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.353+0000 7fe04f674640 1 Processor -- start 2024-04-15T15:15:14.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.353+0000 7fe04f674640 1 -- start start 2024-04-15T15:15:14.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.354+0000 7fe04f674640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe034003fb0 0x7fe048156430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.354+0000 7fe04f674640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe0481f86d0 0x7fe0480c7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.354+0000 7fe04f674640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0480b0f30 0x7fe0480caf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.354+0000 7fe04f674640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe0480d1cd0 con 0x7fe0480b0f30 2024-04-15T15:15:14.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.354+0000 7fe04f674640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fe0480cd430 con 0x7fe034003fb0 2024-04-15T15:15:14.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.354+0000 7fe04f674640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fe0480b6cd0 con 0x7fe0481f86d0 2024-04-15T15:15:14.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.354+0000 7fe04dc11640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0480b0f30 0x7fe0480caf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.354+0000 7fe04dc11640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0480b0f30 0x7fe0480caf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45498/0 (socket says 172.21.15.72:45498) 2024-04-15T15:15:14.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.354+0000 7fe04dc11640 1 -- 172.21.15.72:0/1829359674 learned_addr learned my addr 172.21.15.72:0/1829359674 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:14.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.354+0000 7fe04d410640 1 --2- 172.21.15.72:0/1829359674 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe034003fb0 0x7fe048156430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.354+0000 7fe04cc0f640 1 --2- 172.21.15.72:0/1829359674 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe0481f86d0 0x7fe0480c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.355+0000 7fe04dc11640 1 -- 172.21.15.72:0/1829359674 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe0481f86d0 msgr2=0x7fe0480c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.355+0000 7fe04dc11640 1 --2- 172.21.15.72:0/1829359674 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe0481f86d0 0x7fe0480c7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.355+0000 7fe04dc11640 1 -- 172.21.15.72:0/1829359674 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe034003fb0 msgr2=0x7fe048156430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.355+0000 7fe04dc11640 1 --2- 172.21.15.72:0/1829359674 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe034003fb0 0x7fe048156430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.355+0000 7fe04dc11640 1 -- 172.21.15.72:0/1829359674 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0480c79d0 con 0x7fe0480b0f30 2024-04-15T15:15:14.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.355+0000 7fe04cc0f640 1 --2- 172.21.15.72:0/1829359674 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe0481f86d0 0x7fe0480c7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:14.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.355+0000 7fe04dc11640 1 --2- 172.21.15.72:0/1829359674 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0480b0f30 0x7fe0480caf40 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe0480ceef0 tx=0x7fe04000fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:14.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.355+0000 7fe03dffb640 1 -- 172.21.15.72:0/1829359674 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe04001a070 con 0x7fe0480b0f30 2024-04-15T15:15:14.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.355+0000 7fe03dffb640 1 -- 172.21.15.72:0/1829359674 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe040006d60 con 0x7fe0480b0f30 2024-04-15T15:15:14.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.355+0000 7fe03dffb640 1 -- 172.21.15.72:0/1829359674 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe040003b90 con 0x7fe0480b0f30 2024-04-15T15:15:14.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.355+0000 7fe04f674640 1 -- 172.21.15.72:0/1829359674 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe04806d550 con 0x7fe0480b0f30 2024-04-15T15:15:14.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.355+0000 7fe04f674640 1 -- 172.21.15.72:0/1829359674 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0480c75c0 con 0x7fe0480b0f30 2024-04-15T15:15:14.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.356+0000 7fe04f674640 1 -- 172.21.15.72:0/1829359674 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0480689e0 con 0x7fe0480b0f30 2024-04-15T15:15:14.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.356+0000 7fe03dffb640 1 -- 172.21.15.72:0/1829359674 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe04001d070 con 0x7fe0480b0f30 2024-04-15T15:15:14.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.357+0000 7fe03dffb640 1 --2- 172.21.15.72:0/1829359674 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe02c06c8f0 0x7fe02c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.357+0000 7fe04d410640 1 --2- 172.21.15.72:0/1829359674 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe02c06c8f0 0x7fe02c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.357+0000 7fe03dffb640 1 -- 172.21.15.72:0/1829359674 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fe04008b420 con 0x7fe0480b0f30 2024-04-15T15:15:14.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.357+0000 7fe04d410640 1 --2- 172.21.15.72:0/1829359674 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe02c06c8f0 0x7fe02c06edb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe044001050 tx=0x7fe044000fa0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:14.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.362+0000 7fe03dffb640 1 -- 172.21.15.72:0/1829359674 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe040058f50 con 0x7fe0480b0f30 2024-04-15T15:15:14.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.538+0000 7fe04f674640 1 -- 172.21.15.72:0/1829359674 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe0480b48a0 con 0x7fe0480b0f30 2024-04-15T15:15:14.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.539+0000 7fe03dffb640 1 -- 172.21.15.72:0/1829359674 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7fe04005d250 con 0x7fe0480b0f30 2024-04-15T15:15:14.541 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:14.541 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":24,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:14:56.451113+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:14:49.438366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:14:50.368379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:14:54.081157+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:14.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.541+0000 7fe04f674640 1 -- 172.21.15.72:0/1829359674 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe02c06c8f0 msgr2=0x7fe02c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.541+0000 7fe04f674640 1 --2- 172.21.15.72:0/1829359674 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe02c06c8f0 0x7fe02c06edb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe044001050 tx=0x7fe044000fa0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.541+0000 7fe04f674640 1 -- 172.21.15.72:0/1829359674 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0480b0f30 msgr2=0x7fe0480caf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.541+0000 7fe04f674640 1 --2- 172.21.15.72:0/1829359674 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0480b0f30 0x7fe0480caf40 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe0480ceef0 tx=0x7fe04000fd40 comp rx=0 tx=0).stop 2024-04-15T15:15:14.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.541+0000 7fe04f674640 1 -- 172.21.15.72:0/1829359674 shutdown_connections 2024-04-15T15:15:14.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.541+0000 7fe04f674640 1 --2- 172.21.15.72:0/1829359674 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe02c06c8f0 0x7fe02c06edb0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.541+0000 7fe04f674640 1 --2- 172.21.15.72:0/1829359674 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe0480b0f30 0x7fe0480caf40 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.541+0000 7fe04f674640 1 --2- 172.21.15.72:0/1829359674 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe0481f86d0 0x7fe0480c7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.541+0000 7fe04f674640 1 --2- 172.21.15.72:0/1829359674 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe034003fb0 0x7fe048156430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.541+0000 7fe04f674640 1 -- 172.21.15.72:0/1829359674 >> 172.21.15.72:0/1829359674 conn(0x7fe0480b3f90 msgr2=0x7fe0480b28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:14.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.541+0000 7fe04f674640 1 -- 172.21.15.72:0/1829359674 shutdown_connections 2024-04-15T15:15:14.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.541+0000 7fe04f674640 1 -- 172.21.15.72:0/1829359674 wait complete. 2024-04-15T15:15:14.554 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:39.359102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:14:49.438366+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:14:50.368379+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:14:54.081157+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:15:14.555 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:14.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.659+0000 7fa57097c640 1 Processor -- start 2024-04-15T15:15:14.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.659+0000 7fa57097c640 1 -- start start 2024-04-15T15:15:14.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.660+0000 7fa57097c640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa56c15c250 0x7fa56c06d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.660+0000 7fa57097c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa56c06df30 con 0x7fa56c0d0010 2024-04-15T15:15:14.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.660+0000 7fa57097c640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa56c06e210 con 0x7fa56c15f9f0 2024-04-15T15:15:14.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.660+0000 7fa57097c640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fa56c0cd1a0 con 0x7fa56c15c250 2024-04-15T15:15:14.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.660+0000 7fa56ad9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fa56c0d0010 0x7fa56c0b3e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37920/0 (socket says 172.21.15.72:37920) 2024-04-15T15:15:14.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.660+0000 7fa56ad9d640 1 -- 172.21.15.72:0/3713439757 learned_addr learned my addr 172.21.15.72:0/3713439757 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:14.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.660+0000 7fa56a59c640 1 --2- 172.21.15.72:0/3713439757 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa56c15c250 0x7fa56c06d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.660+0000 7fa569d9b640 1 -- 172.21.15.72:0/3713439757 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1446266826 0 0) 0x7fa56c06df30 con 0x7fa56c0d0010 2024-04-15T15:15:14.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.660+0000 7fa569d9b640 1 -- 172.21.15.72:0/3713439757 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa558003520 con 0x7fa56c0d0010 2024-04-15T15:15:14.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.660+0000 7fa569d9b640 1 -- 172.21.15.72:0/3713439757 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3596152952 0 0) 0x7fa558003520 con 0x7fa56c0d0010 2024-04-15T15:15:14.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.660+0000 7fa569d9b640 1 -- 172.21.15.72:0/3713439757 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa56c15c250 msgr2=0x7fa56c06d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:14.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.660+0000 7fa569d9b640 1 --2- 172.21.15.72:0/3713439757 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa56c15c250 0x7fa56c06d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.660+0000 7fa569d9b640 1 -- 172.21.15.72:0/3713439757 >> v1:172.21.15.175:6789/0 conn(0x7fa56c15f9f0 legacy=0x7fa56c0cca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.660+0000 7fa569d9b640 1 -- 172.21.15.72:0/3713439757 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa56c06e4f0 con 0x7fa56c0d0010 2024-04-15T15:15:14.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.660+0000 7fa569d9b640 1 -- 172.21.15.72:0/3713439757 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7fa5640019f0 con 0x7fa56c0d0010 2024-04-15T15:15:14.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.660+0000 7fa569d9b640 1 -- 172.21.15.72:0/3713439757 >> v1:172.21.15.72:6789/0 conn(0x7fa56c0d0010 legacy=0x7fa56c0b3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.660+0000 7fa569d9b640 1 --2- 172.21.15.72:0/3713439757 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5580039e0 0x7fa558003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.660+0000 7fa569d9b640 1 --2- 172.21.15.72:0/3713439757 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa558004390 0x7fa558004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.660+0000 7fa569d9b640 1 --2- 172.21.15.72:0/3713439757 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa56c15c250 0x7fa558005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.660+0000 7fa569d9b640 1 -- 172.21.15.72:0/3713439757 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fa56c0cd1a0 con 0x7fa5580039e0 2024-04-15T15:15:14.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.660+0000 7fa569d9b640 1 -- 172.21.15.72:0/3713439757 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fa55800ba70 con 0x7fa558004390 2024-04-15T15:15:14.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.660+0000 7fa569d9b640 1 -- 172.21.15.72:0/3713439757 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fa55800bc70 con 0x7fa56c15c250 2024-04-15T15:15:14.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.660+0000 7fa56a59c640 1 --2- 172.21.15.72:0/3713439757 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5580039e0 0x7fa558003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.661+0000 7fa561d9b640 1 --2- 172.21.15.72:0/3713439757 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa558004390 0x7fa558004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.661+0000 7fa56ad9d640 1 --2- 172.21.15.72:0/3713439757 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa56c15c250 0x7fa558005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.661+0000 7fa56a59c640 1 -- 172.21.15.72:0/3713439757 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa56c15c250 msgr2=0x7fa558005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:14.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.661+0000 7fa56a59c640 1 --2- 172.21.15.72:0/3713439757 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa56c15c250 0x7fa558005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.661+0000 7fa56a59c640 1 -- 172.21.15.72:0/3713439757 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa558004390 msgr2=0x7fa558004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:14.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.661+0000 7fa56a59c640 1 --2- 172.21.15.72:0/3713439757 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa558004390 0x7fa558004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.661+0000 7fa56a59c640 1 -- 172.21.15.72:0/3713439757 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa558003520 con 0x7fa5580039e0 2024-04-15T15:15:14.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.661+0000 7fa561d9b640 1 --2- 172.21.15.72:0/3713439757 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa558004390 0x7fa558004860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:14.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.661+0000 7fa56a59c640 1 --2- 172.21.15.72:0/3713439757 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5580039e0 0x7fa558003dc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa55c009ac0 tx=0x7fa558005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f895b65677ac46f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:14.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.662+0000 7fa569d9b640 1 -- 172.21.15.72:0/3713439757 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa55c039070 con 0x7fa5580039e0 2024-04-15T15:15:14.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.662+0000 7fa569d9b640 1 -- 172.21.15.72:0/3713439757 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa55c02c6b0 con 0x7fa5580039e0 2024-04-15T15:15:14.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.662+0000 7fa569d9b640 1 -- 172.21.15.72:0/3713439757 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa55c009690 con 0x7fa5580039e0 2024-04-15T15:15:14.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.662+0000 7fa57097c640 1 -- 172.21.15.72:0/3713439757 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5580039e0 msgr2=0x7fa558003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.662+0000 7fa57097c640 1 --2- 172.21.15.72:0/3713439757 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5580039e0 0x7fa558003dc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa55c009ac0 tx=0x7fa558005830 comp rx=0 tx=0).stop 2024-04-15T15:15:14.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.662+0000 7fa56ad9d640 1 -- 172.21.15.72:0/3713439757 reap_dead start 2024-04-15T15:15:14.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.662+0000 7fa57097c640 1 -- 172.21.15.72:0/3713439757 shutdown_connections 2024-04-15T15:15:14.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.662+0000 7fa57097c640 1 -- 172.21.15.72:0/3713439757 >> 172.21.15.72:0/3713439757 conn(0x7fa56c0ceec0 msgr2=0x7fa56c0cf2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:14.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.662+0000 7fa57097c640 1 -- 172.21.15.72:0/3713439757 shutdown_connections 2024-04-15T15:15:14.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.662+0000 7fa57097c640 1 -- 172.21.15.72:0/3713439757 wait complete. 2024-04-15T15:15:14.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.664+0000 7fa57097c640 1 Processor -- start 2024-04-15T15:15:14.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.664+0000 7fa57097c640 1 -- start start 2024-04-15T15:15:14.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.664+0000 7fa57097c640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa5580039e0 0x7fa56c1e93f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.664+0000 7fa57097c640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa56c1e9930 0x7fa56c15da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.664+0000 7fa57097c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa56c15df40 0x7fa56c1e8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.664+0000 7fa57097c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fa56c15c630 con 0x7fa56c15df40 2024-04-15T15:15:14.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.664+0000 7fa57097c640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fa56c0b38f0 con 0x7fa5580039e0 2024-04-15T15:15:14.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.664+0000 7fa57097c640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fa56c15c7b0 con 0x7fa56c1e9930 2024-04-15T15:15:14.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.664+0000 7fa56ad9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa56c15df40 0x7fa56c1e8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.665+0000 7fa56ad9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa56c15df40 0x7fa56c1e8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45516/0 (socket says 172.21.15.72:45516) 2024-04-15T15:15:14.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.665+0000 7fa56ad9d640 1 -- 172.21.15.72:0/1563738218 learned_addr learned my addr 172.21.15.72:0/1563738218 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:14.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.665+0000 7fa56a59c640 1 --2- 172.21.15.72:0/1563738218 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa5580039e0 0x7fa56c1e93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.665+0000 7fa561d9b640 1 --2- 172.21.15.72:0/1563738218 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa56c1e9930 0x7fa56c15da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.665+0000 7fa56ad9d640 1 -- 172.21.15.72:0/1563738218 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa56c1e9930 msgr2=0x7fa56c15da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.665+0000 7fa56ad9d640 1 --2- 172.21.15.72:0/1563738218 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa56c1e9930 0x7fa56c15da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.665+0000 7fa56ad9d640 1 -- 172.21.15.72:0/1563738218 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa5580039e0 msgr2=0x7fa56c1e93f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.665+0000 7fa56ad9d640 1 --2- 172.21.15.72:0/1563738218 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa5580039e0 0x7fa56c1e93f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.665+0000 7fa56ad9d640 1 -- 172.21.15.72:0/1563738218 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa56c14f5d0 con 0x7fa56c15df40 2024-04-15T15:15:14.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.665+0000 7fa56a59c640 1 --2- 172.21.15.72:0/1563738218 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa5580039e0 0x7fa56c1e93f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:14.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.665+0000 7fa56ad9d640 1 --2- 172.21.15.72:0/1563738218 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa56c15df40 0x7fa56c1e8170 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa56c0cfc70 tx=0x7fa56400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:14.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.665+0000 7fa5637fe640 1 -- 172.21.15.72:0/1563738218 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa56400eda0 con 0x7fa56c15df40 2024-04-15T15:15:14.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.665+0000 7fa5637fe640 1 -- 172.21.15.72:0/1563738218 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa564018070 con 0x7fa56c15df40 2024-04-15T15:15:14.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.665+0000 7fa57097c640 1 -- 172.21.15.72:0/1563738218 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa56c14f8b0 con 0x7fa56c15df40 2024-04-15T15:15:14.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.665+0000 7fa5637fe640 1 -- 172.21.15.72:0/1563738218 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa564013660 con 0x7fa56c15df40 2024-04-15T15:15:14.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.665+0000 7fa57097c640 1 -- 172.21.15.72:0/1563738218 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa56c14fe70 con 0x7fa56c15df40 2024-04-15T15:15:14.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.666+0000 7fa57097c640 1 -- 172.21.15.72:0/1563738218 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa56c075c50 con 0x7fa56c15df40 2024-04-15T15:15:14.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.667+0000 7fa5637fe640 1 -- 172.21.15.72:0/1563738218 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa564013800 con 0x7fa56c15df40 2024-04-15T15:15:14.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.667+0000 7fa5637fe640 1 --2- 172.21.15.72:0/1563738218 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa54c06c920 0x7fa54c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.667+0000 7fa5637fe640 1 -- 172.21.15.72:0/1563738218 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa56408c0c0 con 0x7fa56c15df40 2024-04-15T15:15:14.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.667+0000 7fa56a59c640 1 --2- 172.21.15.72:0/1563738218 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa54c06c920 0x7fa54c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.668+0000 7fa56a59c640 1 --2- 172.21.15.72:0/1563738218 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa54c06c920 0x7fa54c06ede0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa55c036000 tx=0x7fa55c030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:14.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.671+0000 7fa5637fe640 1 -- 172.21.15.72:0/1563738218 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa56405a1b0 con 0x7fa56c15df40 2024-04-15T15:15:14.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.843+0000 7fa57097c640 1 -- 172.21.15.72:0/1563738218 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa56c1e9d10 con 0x7fa56c15df40 2024-04-15T15:15:14.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.843+0000 7fa5637fe640 1 -- 172.21.15.72:0/1563738218 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7fa564059ab0 con 0x7fa56c15df40 2024-04-15T15:15:14.845 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-15T15:15:14.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.845+0000 7fa57097c640 1 -- 172.21.15.72:0/1563738218 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa54c06c920 msgr2=0x7fa54c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.845+0000 7fa57097c640 1 --2- 172.21.15.72:0/1563738218 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa54c06c920 0x7fa54c06ede0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa55c036000 tx=0x7fa55c030000 comp rx=0 tx=0).stop 2024-04-15T15:15:14.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.845+0000 7fa57097c640 1 -- 172.21.15.72:0/1563738218 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa56c15df40 msgr2=0x7fa56c1e8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.845+0000 7fa57097c640 1 --2- 172.21.15.72:0/1563738218 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa56c15df40 0x7fa56c1e8170 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa56c0cfc70 tx=0x7fa56400e530 comp rx=0 tx=0).stop 2024-04-15T15:15:14.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.845+0000 7fa57097c640 1 -- 172.21.15.72:0/1563738218 shutdown_connections 2024-04-15T15:15:14.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.845+0000 7fa57097c640 1 --2- 172.21.15.72:0/1563738218 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa54c06c920 0x7fa54c06ede0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.845+0000 7fa57097c640 1 --2- 172.21.15.72:0/1563738218 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa56c15df40 0x7fa56c1e8170 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.845+0000 7fa57097c640 1 --2- 172.21.15.72:0/1563738218 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa56c1e9930 0x7fa56c15da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.845+0000 7fa57097c640 1 --2- 172.21.15.72:0/1563738218 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa5580039e0 0x7fa56c1e93f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.845+0000 7fa57097c640 1 -- 172.21.15.72:0/1563738218 >> 172.21.15.72:0/1563738218 conn(0x7fa56c0ceec0 msgr2=0x7fa56c0b3c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:14.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.846+0000 7fa57097c640 1 -- 172.21.15.72:0/1563738218 shutdown_connections 2024-04-15T15:15:14.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.846+0000 7fa57097c640 1 -- 172.21.15.72:0/1563738218 wait complete. 2024-04-15T15:15:14.858 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:15:14.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.962+0000 7f31a7a2a640 1 Processor -- start 2024-04-15T15:15:14.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.962+0000 7f31a7a2a640 1 -- start start 2024-04-15T15:15:14.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.963+0000 7f31a7a2a640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f31a00b12f0 0x7f31a00d2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.963+0000 7f31a7a2a640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f31a006ef70 0x7f31a00d3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.963+0000 7f31a7a2a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31a00cfd20 con 0x7f31a015e950 2024-04-15T15:15:14.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.963+0000 7f31a7a2a640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f31a00d0bd0 con 0x7f31a006ef70 2024-04-15T15:15:14.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.963+0000 7f31a7a2a640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f31a00d54d0 con 0x7f31a00b12f0 2024-04-15T15:15:14.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.963+0000 7f31a57c6640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f31a015e950 0x7f31a00d2210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37922/0 (socket says 172.21.15.72:37922) 2024-04-15T15:15:14.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.963+0000 7f31a57c6640 1 -- 172.21.15.72:0/2543305016 learned_addr learned my addr 172.21.15.72:0/2543305016 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:14.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.964+0000 7f31a5fc7640 1 --2- 172.21.15.72:0/2543305016 >> v2:172.21.15.175:3300/0 conn(0x7f31a006ef70 0x7f31a00d3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.964+0000 7f3197fff640 1 -- 172.21.15.72:0/2543305016 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1132960310 0 0) 0x7f31a00cfd20 con 0x7f31a015e950 2024-04-15T15:15:14.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.964+0000 7f31a4fc5640 1 --2- 172.21.15.72:0/2543305016 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f31a00b12f0 0x7f31a00d2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.964+0000 7f3197fff640 1 -- 172.21.15.72:0/2543305016 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f318c003520 con 0x7f31a015e950 2024-04-15T15:15:14.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.964+0000 7f3197fff640 1 -- 172.21.15.72:0/2543305016 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2741605495 0 0) 0x7f318c003520 con 0x7f31a015e950 2024-04-15T15:15:14.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.964+0000 7f3197fff640 1 -- 172.21.15.72:0/2543305016 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f31a00b12f0 msgr2=0x7f31a00d2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:14.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.964+0000 7f3197fff640 1 --2- 172.21.15.72:0/2543305016 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f31a00b12f0 0x7f31a00d2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.964+0000 7f3197fff640 1 -- 172.21.15.72:0/2543305016 >> v2:172.21.15.175:3300/0 conn(0x7f31a006ef70 msgr2=0x7f31a00d3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:14.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.964+0000 7f3197fff640 1 --2- 172.21.15.72:0/2543305016 >> v2:172.21.15.175:3300/0 conn(0x7f31a006ef70 0x7f31a00d3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.964+0000 7f3197fff640 1 -- 172.21.15.72:0/2543305016 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31a00cffc0 con 0x7f31a015e950 2024-04-15T15:15:14.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.964+0000 7f3197fff640 1 -- 172.21.15.72:0/2543305016 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f319c0019f0 con 0x7f31a015e950 2024-04-15T15:15:14.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.964+0000 7f3197fff640 1 -- 172.21.15.72:0/2543305016 >> v1:172.21.15.72:6789/0 conn(0x7f31a015e950 legacy=0x7f31a00d2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.964+0000 7f3197fff640 1 --2- 172.21.15.72:0/2543305016 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f31a00b12f0 0x7f318c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.964+0000 7f3197fff640 1 --2- 172.21.15.72:0/2543305016 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f318c003f30 0x7f318c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.964+0000 7f3197fff640 1 --2- 172.21.15.72:0/2543305016 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f318c004d40 0x7f318c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.964+0000 7f3197fff640 1 -- 172.21.15.72:0/2543305016 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f31a00d0bd0 con 0x7f318c004d40 2024-04-15T15:15:14.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.964+0000 7f3197fff640 1 -- 172.21.15.72:0/2543305016 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f31a00d54d0 con 0x7f318c003f30 2024-04-15T15:15:14.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.964+0000 7f3197fff640 1 -- 172.21.15.72:0/2543305016 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f318c00b9c0 con 0x7f31a00b12f0 2024-04-15T15:15:14.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.964+0000 7f31a5fc7640 1 --2- 172.21.15.72:0/2543305016 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f318c004d40 0x7f318c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.964+0000 7f31a57c6640 1 --2- 172.21.15.72:0/2543305016 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f31a00b12f0 0x7f318c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.964+0000 7f31a4fc5640 1 --2- 172.21.15.72:0/2543305016 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f318c003f30 0x7f318c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.964+0000 7f31a5fc7640 1 -- 172.21.15.72:0/2543305016 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f31a00b12f0 msgr2=0x7f318c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:14.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.964+0000 7f31a5fc7640 1 --2- 172.21.15.72:0/2543305016 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f31a00b12f0 0x7f318c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.964+0000 7f31a5fc7640 1 -- 172.21.15.72:0/2543305016 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f318c003f30 msgr2=0x7f318c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:14.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.964+0000 7f31a5fc7640 1 --2- 172.21.15.72:0/2543305016 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f318c003f30 0x7f318c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.964+0000 7f31a5fc7640 1 -- 172.21.15.72:0/2543305016 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f318c003520 con 0x7f318c004d40 2024-04-15T15:15:14.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.965+0000 7f31a5fc7640 1 --2- 172.21.15.72:0/2543305016 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f318c004d40 0x7f318c0051c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3198009ac0 tx=0x7f318c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40664cbcf6d7eca4 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:14.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.965+0000 7f3197fff640 1 -- 172.21.15.72:0/2543305016 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3198039070 con 0x7f318c004d40 2024-04-15T15:15:14.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.965+0000 7f3197fff640 1 -- 172.21.15.72:0/2543305016 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f319802c6d0 con 0x7f318c004d40 2024-04-15T15:15:14.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.965+0000 7f3197fff640 1 -- 172.21.15.72:0/2543305016 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3198009690 con 0x7f318c004d40 2024-04-15T15:15:14.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.966+0000 7f31a7a2a640 1 -- 172.21.15.72:0/2543305016 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f318c004d40 msgr2=0x7f318c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.966+0000 7f31a7a2a640 1 --2- 172.21.15.72:0/2543305016 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f318c004d40 0x7f318c0051c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3198009ac0 tx=0x7f318c005830 comp rx=0 tx=0).stop 2024-04-15T15:15:14.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.966+0000 7f31a5fc7640 1 -- 172.21.15.72:0/2543305016 reap_dead start 2024-04-15T15:15:14.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.966+0000 7f31a7a2a640 1 -- 172.21.15.72:0/2543305016 shutdown_connections 2024-04-15T15:15:14.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.966+0000 7f31a7a2a640 1 -- 172.21.15.72:0/2543305016 >> 172.21.15.72:0/2543305016 conn(0x7f31a00b9a20 msgr2=0x7f31a00b7590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:14.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.966+0000 7f31a7a2a640 1 -- 172.21.15.72:0/2543305016 shutdown_connections 2024-04-15T15:15:14.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.966+0000 7f31a7a2a640 1 -- 172.21.15.72:0/2543305016 wait complete. 2024-04-15T15:15:14.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.967+0000 7f31a7a2a640 1 Processor -- start 2024-04-15T15:15:14.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.967+0000 7f31a7a2a640 1 -- start start 2024-04-15T15:15:14.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.968+0000 7f31a7a2a640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f318c004d40 0x7f31a015acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.968+0000 7f31a7a2a640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31a01e9b50 0x7f31a01e9f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.968+0000 7f31a7a2a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31a01e5760 0x7f31a01e5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.968+0000 7f31a7a2a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f31a00d0860 con 0x7f31a01e5760 2024-04-15T15:15:14.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.968+0000 7f31a7a2a640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f31a00d0580 con 0x7f31a01e9b50 2024-04-15T15:15:14.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.968+0000 7f31a7a2a640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f31a015ed30 con 0x7f318c004d40 2024-04-15T15:15:14.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.968+0000 7f31a5fc7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31a01e5760 0x7f31a01e5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.968+0000 7f31a5fc7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31a01e5760 0x7f31a01e5be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45526/0 (socket says 172.21.15.72:45526) 2024-04-15T15:15:14.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.968+0000 7f31a5fc7640 1 -- 172.21.15.72:0/4220318580 learned_addr learned my addr 172.21.15.72:0/4220318580 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:14.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.968+0000 7f31a4fc5640 1 --2- 172.21.15.72:0/4220318580 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31a01e9b50 0x7f31a01e9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.968+0000 7f31a57c6640 1 --2- 172.21.15.72:0/4220318580 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f318c004d40 0x7f31a015acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.968+0000 7f31a5fc7640 1 -- 172.21.15.72:0/4220318580 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f318c004d40 msgr2=0x7f31a015acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.968+0000 7f31a5fc7640 1 --2- 172.21.15.72:0/4220318580 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f318c004d40 0x7f31a015acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.968+0000 7f31a5fc7640 1 -- 172.21.15.72:0/4220318580 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31a01e9b50 msgr2=0x7f31a01e9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.968+0000 7f31a5fc7640 1 --2- 172.21.15.72:0/4220318580 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31a01e9b50 0x7f31a01e9f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.968+0000 7f31a5fc7640 1 -- 172.21.15.72:0/4220318580 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31a01ea4b0 con 0x7f31a01e5760 2024-04-15T15:15:14.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.968+0000 7f31a57c6640 1 --2- 172.21.15.72:0/4220318580 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f318c004d40 0x7f31a015acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:14.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.968+0000 7f31a5fc7640 1 --2- 172.21.15.72:0/4220318580 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31a01e5760 0x7f31a01e5be0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3198037040 tx=0x7f3198004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:14.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.969+0000 7f3195ffb640 1 -- 172.21.15.72:0/4220318580 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3198039070 con 0x7f31a01e5760 2024-04-15T15:15:14.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.969+0000 7f3195ffb640 1 -- 172.21.15.72:0/4220318580 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3198043070 con 0x7f31a01e5760 2024-04-15T15:15:14.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.969+0000 7f31a7a2a640 1 -- 172.21.15.72:0/4220318580 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31a01e5350 con 0x7f31a01e5760 2024-04-15T15:15:14.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.969+0000 7f3195ffb640 1 -- 172.21.15.72:0/4220318580 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3198009690 con 0x7f31a01e5760 2024-04-15T15:15:14.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.969+0000 7f31a7a2a640 1 -- 172.21.15.72:0/4220318580 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31a01e4ab0 con 0x7f31a01e5760 2024-04-15T15:15:14.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.970+0000 7f31a7a2a640 1 -- 172.21.15.72:0/4220318580 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31a0075c50 con 0x7f31a01e5760 2024-04-15T15:15:14.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.970+0000 7f3195ffb640 1 -- 172.21.15.72:0/4220318580 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3198003830 con 0x7f31a01e5760 2024-04-15T15:15:14.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.971+0000 7f3195ffb640 1 --2- 172.21.15.72:0/4220318580 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f318406c8f0 0x7f318406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.971+0000 7f3195ffb640 1 -- 172.21.15.72:0/4220318580 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f31980acf60 con 0x7f31a01e5760 2024-04-15T15:15:14.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.971+0000 7f31a57c6640 1 --2- 172.21.15.72:0/4220318580 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f318406c8f0 0x7f318406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.971+0000 7f31a57c6640 1 --2- 172.21.15.72:0/4220318580 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f318406c8f0 0x7f318406edb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f319c002230 tx=0x7f319c006040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:14.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:14.975+0000 7f3195ffb640 1 -- 172.21.15.72:0/4220318580 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f319804c020 con 0x7f31a01e5760 2024-04-15T15:15:15.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.151+0000 7f31a7a2a640 1 -- 172.21.15.72:0/4220318580 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f31a00b7fd0 con 0x7f31a01e5760 2024-04-15T15:15:15.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.151+0000 7f3195ffb640 1 -- 172.21.15.72:0/4220318580 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7f319807b0e0 con 0x7f31a01e5760 2024-04-15T15:15:15.153 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:15.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.153+0000 7f31a7a2a640 1 -- 172.21.15.72:0/4220318580 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f318406c8f0 msgr2=0x7f318406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.153+0000 7f31a7a2a640 1 --2- 172.21.15.72:0/4220318580 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f318406c8f0 0x7f318406edb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f319c002230 tx=0x7f319c006040 comp rx=0 tx=0).stop 2024-04-15T15:15:15.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.153+0000 7f31a7a2a640 1 -- 172.21.15.72:0/4220318580 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31a01e5760 msgr2=0x7f31a01e5be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.153+0000 7f31a7a2a640 1 --2- 172.21.15.72:0/4220318580 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31a01e5760 0x7f31a01e5be0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3198037040 tx=0x7f3198004730 comp rx=0 tx=0).stop 2024-04-15T15:15:15.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.153+0000 7f31a7a2a640 1 -- 172.21.15.72:0/4220318580 shutdown_connections 2024-04-15T15:15:15.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.153+0000 7f31a7a2a640 1 --2- 172.21.15.72:0/4220318580 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f318406c8f0 0x7f318406edb0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.153+0000 7f31a7a2a640 1 --2- 172.21.15.72:0/4220318580 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f31a01e5760 0x7f31a01e5be0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.153+0000 7f31a7a2a640 1 --2- 172.21.15.72:0/4220318580 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f31a01e9b50 0x7f31a01e9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.153+0000 7f31a7a2a640 1 --2- 172.21.15.72:0/4220318580 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f318c004d40 0x7f31a015acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.153+0000 7f31a7a2a640 1 -- 172.21.15.72:0/4220318580 >> 172.21.15.72:0/4220318580 conn(0x7f31a00b9a20 msgr2=0x7f31a00b69a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:15.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.153+0000 7f31a7a2a640 1 -- 172.21.15.72:0/4220318580 shutdown_connections 2024-04-15T15:15:15.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.153+0000 7f31a7a2a640 1 -- 172.21.15.72:0/4220318580 wait complete. 2024-04-15T15:15:15.166 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:15:15.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.270+0000 7f9890d72640 1 Processor -- start 2024-04-15T15:15:15.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.270+0000 7f9890d72640 1 -- start start 2024-04-15T15:15:15.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.271+0000 7f9890d72640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f988c15e9f0 0x7f988c0d3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.271+0000 7f9890d72640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f988c0d2320 con 0x7f988c15e610 2024-04-15T15:15:15.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.271+0000 7f9890d72640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f988c0d25c0 con 0x7f988c06c400 2024-04-15T15:15:15.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.271+0000 7f9890d72640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f988c0d31d0 con 0x7f988c15e9f0 2024-04-15T15:15:15.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.271+0000 7f988a59c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f988c15e610 0x7f988c0d0690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37936/0 (socket says 172.21.15.72:37936) 2024-04-15T15:15:15.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.271+0000 7f988a59c640 1 -- 172.21.15.72:0/558063481 learned_addr learned my addr 172.21.15.72:0/558063481 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:15.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.272+0000 7f988959a640 1 -- 172.21.15.72:0/558063481 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 620360700 0 0) 0x7f988c0d2320 con 0x7f988c15e610 2024-04-15T15:15:15.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.272+0000 7f9889d9b640 1 --2- 172.21.15.72:0/558063481 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f988c15e9f0 0x7f988c0d3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.272+0000 7f988959a640 1 -- 172.21.15.72:0/558063481 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9874003520 con 0x7f988c15e610 2024-04-15T15:15:15.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.272+0000 7f988959a640 1 -- 172.21.15.72:0/558063481 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 349992524 0 0) 0x7f9874003520 con 0x7f988c15e610 2024-04-15T15:15:15.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.272+0000 7f988959a640 1 -- 172.21.15.72:0/558063481 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f988c15e9f0 msgr2=0x7f988c0d3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:15.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.272+0000 7f988959a640 1 --2- 172.21.15.72:0/558063481 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f988c15e9f0 0x7f988c0d3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.272+0000 7f988959a640 1 -- 172.21.15.72:0/558063481 >> v1:172.21.15.175:6789/0 conn(0x7f988c06c400 legacy=0x7f988c0b7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.272+0000 7f988959a640 1 -- 172.21.15.72:0/558063481 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f988c0d28a0 con 0x7f988c15e610 2024-04-15T15:15:15.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.272+0000 7f988959a640 1 -- 172.21.15.72:0/558063481 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f987c0019f0 con 0x7f988c15e610 2024-04-15T15:15:15.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.272+0000 7f988959a640 1 -- 172.21.15.72:0/558063481 >> v1:172.21.15.72:6789/0 conn(0x7f988c15e610 legacy=0x7f988c0d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.272+0000 7f988959a640 1 --2- 172.21.15.72:0/558063481 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f98740039e0 0x7f9874003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.272+0000 7f988959a640 1 --2- 172.21.15.72:0/558063481 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f988c15e9f0 0x7f9874004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.272+0000 7f988959a640 1 --2- 172.21.15.72:0/558063481 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9874004d40 0x7f98740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.272+0000 7f988959a640 1 -- 172.21.15.72:0/558063481 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f988c0d31d0 con 0x7f9874004d40 2024-04-15T15:15:15.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.272+0000 7f988959a640 1 -- 172.21.15.72:0/558063481 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f987400ba70 con 0x7f98740039e0 2024-04-15T15:15:15.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.272+0000 7f988959a640 1 -- 172.21.15.72:0/558063481 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f987400bbf0 con 0x7f988c15e9f0 2024-04-15T15:15:15.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.272+0000 7f988ad9d640 1 --2- 172.21.15.72:0/558063481 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9874004d40 0x7f98740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.272+0000 7f988a59c640 1 --2- 172.21.15.72:0/558063481 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f98740039e0 0x7f9874003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.272+0000 7f9889d9b640 1 --2- 172.21.15.72:0/558063481 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f988c15e9f0 0x7f9874004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.272+0000 7f988ad9d640 1 -- 172.21.15.72:0/558063481 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f988c15e9f0 msgr2=0x7f9874004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:15.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.272+0000 7f988ad9d640 1 --2- 172.21.15.72:0/558063481 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f988c15e9f0 0x7f9874004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.272+0000 7f988ad9d640 1 -- 172.21.15.72:0/558063481 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f98740039e0 msgr2=0x7f9874003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:15.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.272+0000 7f988ad9d640 1 --2- 172.21.15.72:0/558063481 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f98740039e0 0x7f9874003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.272+0000 7f988ad9d640 1 -- 172.21.15.72:0/558063481 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9874003520 con 0x7f9874004d40 2024-04-15T15:15:15.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.273+0000 7f988ad9d640 1 --2- 172.21.15.72:0/558063481 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9874004d40 0x7f98740051c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9884001860 tx=0x7f9874005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5524ccaf7162572a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:15.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.273+0000 7f988959a640 1 -- 172.21.15.72:0/558063481 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9884038070 con 0x7f9874004d40 2024-04-15T15:15:15.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.273+0000 7f988959a640 1 -- 172.21.15.72:0/558063481 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9884033410 con 0x7f9874004d40 2024-04-15T15:15:15.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.273+0000 7f988959a640 1 -- 172.21.15.72:0/558063481 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9884001430 con 0x7f9874004d40 2024-04-15T15:15:15.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.274+0000 7f9890d72640 1 -- 172.21.15.72:0/558063481 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9874004d40 msgr2=0x7f98740051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.274+0000 7f9890d72640 1 --2- 172.21.15.72:0/558063481 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9874004d40 0x7f98740051c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9884001860 tx=0x7f9874005830 comp rx=0 tx=0).stop 2024-04-15T15:15:15.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.274+0000 7f988ad9d640 1 -- 172.21.15.72:0/558063481 reap_dead start 2024-04-15T15:15:15.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.274+0000 7f9890d72640 1 -- 172.21.15.72:0/558063481 shutdown_connections 2024-04-15T15:15:15.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.274+0000 7f9890d72640 1 -- 172.21.15.72:0/558063481 >> 172.21.15.72:0/558063481 conn(0x7f988c0d4750 msgr2=0x7f988c0d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:15.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.274+0000 7f9890d72640 1 -- 172.21.15.72:0/558063481 shutdown_connections 2024-04-15T15:15:15.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.274+0000 7f9890d72640 1 -- 172.21.15.72:0/558063481 wait complete. 2024-04-15T15:15:15.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.275+0000 7f9890d72640 1 Processor -- start 2024-04-15T15:15:15.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.275+0000 7f9890d72640 1 -- start start 2024-04-15T15:15:15.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.276+0000 7f9890d72640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9874004d40 0x7f988c153a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.276+0000 7f9890d72640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f988c150ff0 0x7f988c1552a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.276+0000 7f9890d72640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f988c1557e0 0x7f988c155c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.276+0000 7f9890d72640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f988c06d130 con 0x7f988c1557e0 2024-04-15T15:15:15.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.276+0000 7f9890d72640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f988c0d2e60 con 0x7f9874004d40 2024-04-15T15:15:15.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.276+0000 7f9890d72640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f988c0b8050 con 0x7f988c150ff0 2024-04-15T15:15:15.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.276+0000 7f988ad9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f988c1557e0 0x7f988c155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.276+0000 7f988ad9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f988c1557e0 0x7f988c155c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45536/0 (socket says 172.21.15.72:45536) 2024-04-15T15:15:15.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.276+0000 7f988ad9d640 1 -- 172.21.15.72:0/1263279154 learned_addr learned my addr 172.21.15.72:0/1263279154 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:15.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.276+0000 7f988a59c640 1 --2- 172.21.15.72:0/1263279154 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9874004d40 0x7f988c153a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.276+0000 7f9889d9b640 1 --2- 172.21.15.72:0/1263279154 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f988c150ff0 0x7f988c1552a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.276+0000 7f988ad9d640 1 -- 172.21.15.72:0/1263279154 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f988c150ff0 msgr2=0x7f988c1552a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.276+0000 7f988ad9d640 1 --2- 172.21.15.72:0/1263279154 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f988c150ff0 0x7f988c1552a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.276+0000 7f988ad9d640 1 -- 172.21.15.72:0/1263279154 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9874004d40 msgr2=0x7f988c153a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.276+0000 7f988ad9d640 1 --2- 172.21.15.72:0/1263279154 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9874004d40 0x7f988c153a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.276+0000 7f988ad9d640 1 -- 172.21.15.72:0/1263279154 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f988c15f330 con 0x7f988c1557e0 2024-04-15T15:15:15.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.277+0000 7f988ad9d640 1 --2- 172.21.15.72:0/1263279154 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f988c1557e0 0x7f988c155c60 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9884001980 tx=0x7f9884001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:15.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.277+0000 7f987affd640 1 -- 172.21.15.72:0/1263279154 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9884038040 con 0x7f988c1557e0 2024-04-15T15:15:15.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.277+0000 7f987affd640 1 -- 172.21.15.72:0/1263279154 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9884043070 con 0x7f988c1557e0 2024-04-15T15:15:15.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.277+0000 7f987affd640 1 -- 172.21.15.72:0/1263279154 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9884001430 con 0x7f988c1557e0 2024-04-15T15:15:15.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.277+0000 7f9890d72640 1 -- 172.21.15.72:0/1263279154 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f988c1561a0 con 0x7f988c1557e0 2024-04-15T15:15:15.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.277+0000 7f9890d72640 1 -- 172.21.15.72:0/1263279154 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f988c151b20 con 0x7f988c1557e0 2024-04-15T15:15:15.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.278+0000 7f9890d72640 1 -- 172.21.15.72:0/1263279154 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f988c075c50 con 0x7f988c1557e0 2024-04-15T15:15:15.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.278+0000 7f987affd640 1 -- 172.21.15.72:0/1263279154 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9884005070 con 0x7f988c1557e0 2024-04-15T15:15:15.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.279+0000 7f987affd640 1 --2- 172.21.15.72:0/1263279154 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f986c06c920 0x7f986c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.279+0000 7f987affd640 1 -- 172.21.15.72:0/1263279154 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f98840ace10 con 0x7f988c1557e0 2024-04-15T15:15:15.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.279+0000 7f988a59c640 1 --2- 172.21.15.72:0/1263279154 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f986c06c920 0x7f986c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.279+0000 7f988a59c640 1 --2- 172.21.15.72:0/1263279154 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f986c06c920 0x7f986c06ede0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f987c000f80 tx=0x7f987c006040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:15.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.283+0000 7f987affd640 1 -- 172.21.15.72:0/1263279154 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f988407af00 con 0x7f988c1557e0 2024-04-15T15:15:15.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.460+0000 7f9890d72640 1 -- 172.21.15.72:0/1263279154 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f988c0cf740 con 0x7f988c1557e0 2024-04-15T15:15:15.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.461+0000 7f987affd640 1 -- 172.21.15.72:0/1263279154 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7f98840810d0 con 0x7f988c1557e0 2024-04-15T15:15:15.463 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-04-15T15:15:15.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.463+0000 7f9890d72640 1 -- 172.21.15.72:0/1263279154 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f986c06c920 msgr2=0x7f986c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.463+0000 7f9890d72640 1 --2- 172.21.15.72:0/1263279154 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f986c06c920 0x7f986c06ede0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f987c000f80 tx=0x7f987c006040 comp rx=0 tx=0).stop 2024-04-15T15:15:15.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.463+0000 7f9890d72640 1 -- 172.21.15.72:0/1263279154 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f988c1557e0 msgr2=0x7f988c155c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.463+0000 7f9890d72640 1 --2- 172.21.15.72:0/1263279154 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f988c1557e0 0x7f988c155c60 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f9884001980 tx=0x7f9884001230 comp rx=0 tx=0).stop 2024-04-15T15:15:15.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.463+0000 7f9890d72640 1 -- 172.21.15.72:0/1263279154 shutdown_connections 2024-04-15T15:15:15.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.463+0000 7f9890d72640 1 --2- 172.21.15.72:0/1263279154 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f986c06c920 0x7f986c06ede0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.463+0000 7f9890d72640 1 --2- 172.21.15.72:0/1263279154 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f988c1557e0 0x7f988c155c60 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.463+0000 7f9890d72640 1 --2- 172.21.15.72:0/1263279154 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f988c150ff0 0x7f988c1552a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.463+0000 7f9890d72640 1 --2- 172.21.15.72:0/1263279154 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9874004d40 0x7f988c153a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.463+0000 7f9890d72640 1 -- 172.21.15.72:0/1263279154 >> 172.21.15.72:0/1263279154 conn(0x7f988c0d4750 msgr2=0x7f988c1621f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:15.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.463+0000 7f9890d72640 1 -- 172.21.15.72:0/1263279154 shutdown_connections 2024-04-15T15:15:15.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.463+0000 7f9890d72640 1 -- 172.21.15.72:0/1263279154 wait complete. 2024-04-15T15:15:15.476 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:15:15.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.581+0000 7f2a6c614640 1 Processor -- start 2024-04-15T15:15:15.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.581+0000 7f2a6c614640 1 -- start start 2024-04-15T15:15:15.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.582+0000 7f2a6c614640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2a640ce2f0 0x7f2a6415c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.582+0000 7f2a6c614640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2a640cc7c0 0x7f2a64155600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.582+0000 7f2a6c614640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a6406bbb0 con 0x7f2a640d08f0 2024-04-15T15:15:15.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.582+0000 7f2a6c614640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f2a6406b960 con 0x7f2a640cc7c0 2024-04-15T15:15:15.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.582+0000 7f2a6c614640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f2a6406be90 con 0x7f2a640ce2f0 2024-04-15T15:15:15.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.582+0000 7f2a6abb1640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2a640cc7c0 0x7f2a64155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.582+0000 7f2a6abb1640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2a640cc7c0 0x7f2a64155600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45538/0 (socket says 172.21.15.72:45538) 2024-04-15T15:15:15.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.582+0000 7f2a6abb1640 1 -- 172.21.15.72:0/4216271583 learned_addr learned my addr 172.21.15.72:0/4216271583 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:15.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.582+0000 7f2a6abb1640 1 -- 172.21.15.72:0/4216271583 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2a640ce2f0 msgr2=0x7f2a6415c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:15.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.582+0000 7f2a6a3b0640 1 --2- 172.21.15.72:0/4216271583 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2a640ce2f0 0x7f2a6415c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.582+0000 7f2a6abb1640 1 --2- 172.21.15.72:0/4216271583 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2a640ce2f0 0x7f2a6415c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.582+0000 7f2a6abb1640 1 -- 172.21.15.72:0/4216271583 >> v1:172.21.15.175:6789/0 conn(0x7f2a640d08f0 legacy=0x7f2a640cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:15.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.582+0000 7f2a6abb1640 1 -- 172.21.15.72:0/4216271583 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a6406c170 con 0x7f2a640cc7c0 2024-04-15T15:15:15.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.583+0000 7f2a6abb1640 1 --2- 172.21.15.72:0/4216271583 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a640cc7c0 0x7f2a64155600 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c009870 tx=0x7f2a5c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad58842de73c7e4c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:15.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.583+0000 7f2a693ae640 1 -- 172.21.15.72:0/4216271583 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c035020 con 0x7f2a640cc7c0 2024-04-15T15:15:15.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.583+0000 7f2a693ae640 1 -- 172.21.15.72:0/4216271583 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a5c031640 con 0x7f2a640cc7c0 2024-04-15T15:15:15.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.583+0000 7f2a693ae640 1 -- 172.21.15.72:0/4216271583 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c02d040 con 0x7f2a640cc7c0 2024-04-15T15:15:15.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.584+0000 7f2a6c614640 1 -- 172.21.15.72:0/4216271583 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a640cc7c0 msgr2=0x7f2a64155600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.584+0000 7f2a6c614640 1 --2- 172.21.15.72:0/4216271583 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a640cc7c0 0x7f2a64155600 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c009870 tx=0x7f2a5c02b8f0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.584+0000 7f2a6c614640 1 -- 172.21.15.72:0/4216271583 shutdown_connections 2024-04-15T15:15:15.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.584+0000 7f2a6c614640 1 --2- 172.21.15.72:0/4216271583 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a640cc7c0 0x7f2a64155600 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.584+0000 7f2a6c614640 1 --2- 172.21.15.72:0/4216271583 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2a640ce2f0 0x7f2a6415c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.584+0000 7f2a6c614640 1 -- 172.21.15.72:0/4216271583 >> 172.21.15.72:0/4216271583 conn(0x7f2a640b31d0 msgr2=0x7f2a640b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:15.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.584+0000 7f2a6c614640 1 -- 172.21.15.72:0/4216271583 shutdown_connections 2024-04-15T15:15:15.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.584+0000 7f2a6c614640 1 -- 172.21.15.72:0/4216271583 wait complete. 2024-04-15T15:15:15.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.585+0000 7f2a6c614640 1 Processor -- start 2024-04-15T15:15:15.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.585+0000 7f2a6c614640 1 -- start start 2024-04-15T15:15:15.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.586+0000 7f2a6c614640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2a640cc7c0 0x7f2a641535a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.586+0000 7f2a6c614640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2a640ce2f0 0x7f2a64151360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.586+0000 7f2a6c614640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a640d08f0 0x7f2a64154fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.586+0000 7f2a6c614640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2a640ce6d0 con 0x7f2a640d08f0 2024-04-15T15:15:15.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.586+0000 7f2a6c614640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f2a6415ca70 con 0x7f2a640cc7c0 2024-04-15T15:15:15.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.586+0000 7f2a6c614640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f2a640ce850 con 0x7f2a640ce2f0 2024-04-15T15:15:15.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.586+0000 7f2a6abb1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a640d08f0 0x7f2a64154fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.586+0000 7f2a6abb1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a640d08f0 0x7f2a64154fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45554/0 (socket says 172.21.15.72:45554) 2024-04-15T15:15:15.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.586+0000 7f2a6abb1640 1 -- 172.21.15.72:0/1910477050 learned_addr learned my addr 172.21.15.72:0/1910477050 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:15.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.586+0000 7f2a6a3b0640 1 --2- 172.21.15.72:0/1910477050 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2a640cc7c0 0x7f2a641535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.586+0000 7f2a69baf640 1 --2- 172.21.15.72:0/1910477050 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2a640ce2f0 0x7f2a64151360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.586+0000 7f2a6abb1640 1 -- 172.21.15.72:0/1910477050 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2a640ce2f0 msgr2=0x7f2a64151360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.586+0000 7f2a6abb1640 1 --2- 172.21.15.72:0/1910477050 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2a640ce2f0 0x7f2a64151360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.586+0000 7f2a6abb1640 1 -- 172.21.15.72:0/1910477050 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2a640cc7c0 msgr2=0x7f2a641535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.586+0000 7f2a6abb1640 1 --2- 172.21.15.72:0/1910477050 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2a640cc7c0 0x7f2a641535a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.586+0000 7f2a6abb1640 1 -- 172.21.15.72:0/1910477050 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a64157a80 con 0x7f2a640d08f0 2024-04-15T15:15:15.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.586+0000 7f2a69baf640 1 --2- 172.21.15.72:0/1910477050 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2a640ce2f0 0x7f2a64151360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:15.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.586+0000 7f2a6abb1640 1 --2- 172.21.15.72:0/1910477050 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a640d08f0 0x7f2a64154fa0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c02b930 tx=0x7f2a5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:15.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.586+0000 7f2a5affd640 1 -- 172.21.15.72:0/1910477050 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c040070 con 0x7f2a640d08f0 2024-04-15T15:15:15.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.586+0000 7f2a5affd640 1 -- 172.21.15.72:0/1910477050 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a5c039cd0 con 0x7f2a640d08f0 2024-04-15T15:15:15.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.586+0000 7f2a6c614640 1 -- 172.21.15.72:0/1910477050 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a64156e00 con 0x7f2a640d08f0 2024-04-15T15:15:15.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.586+0000 7f2a5affd640 1 -- 172.21.15.72:0/1910477050 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c035040 con 0x7f2a640d08f0 2024-04-15T15:15:15.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.586+0000 7f2a6c614640 1 -- 172.21.15.72:0/1910477050 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a641573c0 con 0x7f2a640d08f0 2024-04-15T15:15:15.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.588+0000 7f2a6c614640 1 -- 172.21.15.72:0/1910477050 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a64075c50 con 0x7f2a640d08f0 2024-04-15T15:15:15.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.588+0000 7f2a5affd640 1 -- 172.21.15.72:0/1910477050 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2a5c038020 con 0x7f2a640d08f0 2024-04-15T15:15:15.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.588+0000 7f2a5affd640 1 --2- 172.21.15.72:0/1910477050 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2a4806c8f0 0x7f2a4806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.588+0000 7f2a6a3b0640 1 --2- 172.21.15.72:0/1910477050 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2a4806c8f0 0x7f2a4806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.588+0000 7f2a5affd640 1 -- 172.21.15.72:0/1910477050 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f2a5c0ad880 con 0x7f2a640d08f0 2024-04-15T15:15:15.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.589+0000 7f2a6a3b0640 1 --2- 172.21.15.72:0/1910477050 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2a4806c8f0 0x7f2a4806edb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2a6000f510 tx=0x7f2a600077b0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:15.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.593+0000 7f2a5affd640 1 -- 172.21.15.72:0/1910477050 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2a5c07b320 con 0x7f2a640d08f0 2024-04-15T15:15:15.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.766+0000 7f2a6c614640 1 -- 172.21.15.72:0/1910477050 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f2a6406cab0 con 0x7f2a640d08f0 2024-04-15T15:15:15.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.767+0000 7f2a5affd640 1 -- 172.21.15.72:0/1910477050 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7f2a5c07f620 con 0x7f2a640d08f0 2024-04-15T15:15:15.769 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:15.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.769+0000 7f2a6c614640 1 -- 172.21.15.72:0/1910477050 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2a4806c8f0 msgr2=0x7f2a4806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.769+0000 7f2a6c614640 1 --2- 172.21.15.72:0/1910477050 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2a4806c8f0 0x7f2a4806edb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2a6000f510 tx=0x7f2a600077b0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.769+0000 7f2a6c614640 1 -- 172.21.15.72:0/1910477050 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a640d08f0 msgr2=0x7f2a64154fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.769+0000 7f2a6c614640 1 --2- 172.21.15.72:0/1910477050 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a640d08f0 0x7f2a64154fa0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c02b930 tx=0x7f2a5c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.769+0000 7f2a6c614640 1 -- 172.21.15.72:0/1910477050 shutdown_connections 2024-04-15T15:15:15.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.769+0000 7f2a6c614640 1 --2- 172.21.15.72:0/1910477050 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2a4806c8f0 0x7f2a4806edb0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.769+0000 7f2a6c614640 1 --2- 172.21.15.72:0/1910477050 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a640d08f0 0x7f2a64154fa0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.769+0000 7f2a6c614640 1 --2- 172.21.15.72:0/1910477050 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2a640ce2f0 0x7f2a64151360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.769+0000 7f2a6c614640 1 --2- 172.21.15.72:0/1910477050 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2a640cc7c0 0x7f2a641535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.769+0000 7f2a6c614640 1 -- 172.21.15.72:0/1910477050 >> 172.21.15.72:0/1910477050 conn(0x7f2a640b31d0 msgr2=0x7f2a64159f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:15.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.769+0000 7f2a6c614640 1 -- 172.21.15.72:0/1910477050 shutdown_connections 2024-04-15T15:15:15.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.769+0000 7f2a6c614640 1 -- 172.21.15.72:0/1910477050 wait complete. 2024-04-15T15:15:15.782 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:15:15.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.882+0000 7f4e82fba640 1 Processor -- start 2024-04-15T15:15:15.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.882+0000 7f4e82fba640 1 -- start start 2024-04-15T15:15:15.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.883+0000 7f4e82fba640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4e7c156d30 0x7f4e7c0d11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.883+0000 7f4e82fba640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4e7c0cfa70 0x7f4e7c15dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.883+0000 7f4e82fba640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e7c0d0100 con 0x7f4e7c0cd160 2024-04-15T15:15:15.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.883+0000 7f4e82fba640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f4e7c0d03e0 con 0x7f4e7c0cfa70 2024-04-15T15:15:15.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.883+0000 7f4e82fba640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f4e7c0d06c0 con 0x7f4e7c156d30 2024-04-15T15:15:15.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.883+0000 7f4e81557640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4e7c0cfa70 0x7f4e7c15dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.884+0000 7f4e7bfff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f4e7c0cd160 0x7f4e7c0cf360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37950/0 (socket says 172.21.15.72:37950) 2024-04-15T15:15:15.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.884+0000 7f4e7bfff640 1 -- 172.21.15.72:0/1546743416 learned_addr learned my addr 172.21.15.72:0/1546743416 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:15.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.884+0000 7f4e80d56640 1 --2- 172.21.15.72:0/1546743416 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4e7c156d30 0x7f4e7c0d11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.884+0000 7f4e81557640 1 -- 172.21.15.72:0/1546743416 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4e7c156d30 msgr2=0x7f4e7c0d11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:15.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.884+0000 7f4e81557640 1 --2- 172.21.15.72:0/1546743416 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4e7c156d30 0x7f4e7c0d11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.884+0000 7f4e81557640 1 -- 172.21.15.72:0/1546743416 >> v1:172.21.15.72:6789/0 conn(0x7f4e7c0cd160 legacy=0x7f4e7c0cf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.884+0000 7f4e81557640 1 -- 172.21.15.72:0/1546743416 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e7c0d09a0 con 0x7f4e7c0cfa70 2024-04-15T15:15:15.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.884+0000 7f4e7b7fe640 1 -- 172.21.15.72:0/1546743416 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 668358203 0 0) 0x7f4e7c0d0100 con 0x7f4e7c0cd160 2024-04-15T15:15:15.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.884+0000 7f4e80d56640 1 --2- 172.21.15.72:0/1546743416 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4e7c156d30 0x7f4e7c0d11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:15.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.885+0000 7f4e81557640 1 --2- 172.21.15.72:0/1546743416 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e7c0cfa70 0x7f4e7c15dc30 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4e74009d90 tx=0x7f4e7402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33d29b84c076bbe7 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:15.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.885+0000 7f4e7b7fe640 1 -- 172.21.15.72:0/1546743416 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e74039070 con 0x7f4e7c0cfa70 2024-04-15T15:15:15.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.885+0000 7f4e7b7fe640 1 -- 172.21.15.72:0/1546743416 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e7402cc50 con 0x7f4e7c0cfa70 2024-04-15T15:15:15.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.885+0000 7f4e7b7fe640 1 -- 172.21.15.72:0/1546743416 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e74034440 con 0x7f4e7c0cfa70 2024-04-15T15:15:15.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.885+0000 7f4e82fba640 1 -- 172.21.15.72:0/1546743416 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e7c0cfa70 msgr2=0x7f4e7c15dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.885+0000 7f4e82fba640 1 --2- 172.21.15.72:0/1546743416 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e7c0cfa70 0x7f4e7c15dc30 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4e74009d90 tx=0x7f4e7402bea0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.885+0000 7f4e82fba640 1 -- 172.21.15.72:0/1546743416 shutdown_connections 2024-04-15T15:15:15.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.885+0000 7f4e82fba640 1 --2- 172.21.15.72:0/1546743416 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e7c0cfa70 0x7f4e7c15dc30 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.885+0000 7f4e82fba640 1 --2- 172.21.15.72:0/1546743416 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4e7c156d30 0x7f4e7c0d11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.885+0000 7f4e82fba640 1 -- 172.21.15.72:0/1546743416 >> 172.21.15.72:0/1546743416 conn(0x7f4e7c0b43c0 msgr2=0x7f4e7c0b47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:15.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.885+0000 7f4e82fba640 1 -- 172.21.15.72:0/1546743416 shutdown_connections 2024-04-15T15:15:15.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.885+0000 7f4e82fba640 1 -- 172.21.15.72:0/1546743416 wait complete. 2024-04-15T15:15:15.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.886+0000 7f4e82fba640 1 Processor -- start 2024-04-15T15:15:15.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.887+0000 7f4e82fba640 1 -- start start 2024-04-15T15:15:15.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.887+0000 7f4e82fba640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4e7c0cd160 0x7f4e7c14df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.887+0000 7f4e82fba640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e7c0cfa70 0x7f4e7c06e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.887+0000 7f4e82fba640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4e7c156d30 0x7f4e7c150840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.887+0000 7f4e82fba640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7c0cce90 con 0x7f4e7c0cfa70 2024-04-15T15:15:15.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.887+0000 7f4e82fba640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7c0ccd10 con 0x7f4e7c0cd160 2024-04-15T15:15:15.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.887+0000 7f4e82fba640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f4e7c0cd5a0 con 0x7f4e7c156d30 2024-04-15T15:15:15.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.887+0000 7f4e7bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e7c0cfa70 0x7f4e7c06e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.888+0000 7f4e7bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e7c0cfa70 0x7f4e7c06e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45566/0 (socket says 172.21.15.72:45566) 2024-04-15T15:15:15.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.888+0000 7f4e7bfff640 1 -- 172.21.15.72:0/3529761093 learned_addr learned my addr 172.21.15.72:0/3529761093 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:15.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.888+0000 7f4e81557640 1 --2- 172.21.15.72:0/3529761093 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4e7c156d30 0x7f4e7c150840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.888+0000 7f4e80d56640 1 --2- 172.21.15.72:0/3529761093 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4e7c0cd160 0x7f4e7c14df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.888+0000 7f4e7bfff640 1 -- 172.21.15.72:0/3529761093 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4e7c156d30 msgr2=0x7f4e7c150840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.888+0000 7f4e7bfff640 1 --2- 172.21.15.72:0/3529761093 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4e7c156d30 0x7f4e7c150840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.888+0000 7f4e7bfff640 1 -- 172.21.15.72:0/3529761093 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4e7c0cd160 msgr2=0x7f4e7c14df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.888+0000 7f4e7bfff640 1 --2- 172.21.15.72:0/3529761093 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4e7c0cd160 0x7f4e7c14df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.888+0000 7f4e7bfff640 1 -- 172.21.15.72:0/3529761093 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e7c0b32e0 con 0x7f4e7c0cfa70 2024-04-15T15:15:15.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.888+0000 7f4e80d56640 1 --2- 172.21.15.72:0/3529761093 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4e7c0cd160 0x7f4e7c14df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:15.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.888+0000 7f4e7bfff640 1 --2- 172.21.15.72:0/3529761093 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e7c0cfa70 0x7f4e7c06e720 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4e7000c4a0 tx=0x7f4e7000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:15.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.888+0000 7f4e797fa640 1 -- 172.21.15.72:0/3529761093 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e70018070 con 0x7f4e7c0cfa70 2024-04-15T15:15:15.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.888+0000 7f4e797fa640 1 -- 172.21.15.72:0/3529761093 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e700074e0 con 0x7f4e7c0cfa70 2024-04-15T15:15:15.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.888+0000 7f4e82fba640 1 -- 172.21.15.72:0/3529761093 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e7c0b35c0 con 0x7f4e7c0cfa70 2024-04-15T15:15:15.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.888+0000 7f4e797fa640 1 -- 172.21.15.72:0/3529761093 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e70004a20 con 0x7f4e7c0cfa70 2024-04-15T15:15:15.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.888+0000 7f4e82fba640 1 -- 172.21.15.72:0/3529761093 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e7c06e200 con 0x7f4e7c0cfa70 2024-04-15T15:15:15.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.889+0000 7f4e82fba640 1 -- 172.21.15.72:0/3529761093 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e7c0689e0 con 0x7f4e7c0cfa70 2024-04-15T15:15:15.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.890+0000 7f4e797fa640 1 -- 172.21.15.72:0/3529761093 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4e7001b070 con 0x7f4e7c0cfa70 2024-04-15T15:15:15.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.890+0000 7f4e797fa640 1 --2- 172.21.15.72:0/3529761093 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4e6806c8f0 0x7f4e6806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.890+0000 7f4e797fa640 1 -- 172.21.15.72:0/3529761093 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f4e7008b180 con 0x7f4e7c0cfa70 2024-04-15T15:15:15.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.890+0000 7f4e80d56640 1 --2- 172.21.15.72:0/3529761093 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4e6806c8f0 0x7f4e6806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.890+0000 7f4e80d56640 1 --2- 172.21.15.72:0/3529761093 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4e6806c8f0 0x7f4e6806edb0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c006fd0 tx=0x7f4e6c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:15.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:15.894+0000 7f4e797fa640 1 -- 172.21.15.72:0/3529761093 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4e70059270 con 0x7f4e7c0cfa70 2024-04-15T15:15:16.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.109+0000 7f4e82fba640 1 -- 172.21.15.72:0/3529761093 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e7c0d1240 con 0x7f4e7c0cfa70 2024-04-15T15:15:16.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.110+0000 7f4e797fa640 1 -- 172.21.15.72:0/3529761093 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5183 (secure 0 0 0) 0x7f4e70058b70 con 0x7f4e7c0cfa70 2024-04-15T15:15:16.111 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:16.111 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4267,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6833/1659343919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1659343919},{"type":"v1","addr":"172.21.15.175:6833","nonce":1659343919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6835/4113588141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6834","nonce":4113588141},{"type":"v1","addr":"172.21.15.175:6835","nonce":4113588141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4296,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/4151548261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4151548261},{"type":"v1","addr":"172.21.15.72:6836","nonce":4151548261}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:14:51.411091+0000","modified":"2024-04-15T15:15:07.485436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/3458067804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3458067804},{"type":"v1","addr":"172.21.15.72:6837","nonce":3458067804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4299,"qdb_cluster":[4299]},"id":1}]} 2024-04-15T15:15:16.112 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 9 2024-04-15T15:15:16.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.112+0000 7f4e82fba640 1 -- 172.21.15.72:0/3529761093 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4e6806c8f0 msgr2=0x7f4e6806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.112+0000 7f4e82fba640 1 --2- 172.21.15.72:0/3529761093 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4e6806c8f0 0x7f4e6806edb0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c006fd0 tx=0x7f4e6c008040 comp rx=0 tx=0).stop 2024-04-15T15:15:16.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.112+0000 7f4e82fba640 1 -- 172.21.15.72:0/3529761093 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e7c0cfa70 msgr2=0x7f4e7c06e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.112+0000 7f4e82fba640 1 --2- 172.21.15.72:0/3529761093 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e7c0cfa70 0x7f4e7c06e720 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f4e7000c4a0 tx=0x7f4e7000c960 comp rx=0 tx=0).stop 2024-04-15T15:15:16.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.112+0000 7f4e82fba640 1 -- 172.21.15.72:0/3529761093 shutdown_connections 2024-04-15T15:15:16.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.112+0000 7f4e82fba640 1 --2- 172.21.15.72:0/3529761093 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4e6806c8f0 0x7f4e6806edb0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.112+0000 7f4e82fba640 1 --2- 172.21.15.72:0/3529761093 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4e7c156d30 0x7f4e7c150840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.112+0000 7f4e82fba640 1 --2- 172.21.15.72:0/3529761093 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e7c0cfa70 0x7f4e7c06e720 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.112+0000 7f4e82fba640 1 --2- 172.21.15.72:0/3529761093 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4e7c0cd160 0x7f4e7c14df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.112+0000 7f4e82fba640 1 -- 172.21.15.72:0/3529761093 >> 172.21.15.72:0/3529761093 conn(0x7f4e7c0b43c0 msgr2=0x7f4e7c15e6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:16.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.112+0000 7f4e82fba640 1 -- 172.21.15.72:0/3529761093 shutdown_connections 2024-04-15T15:15:16.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.112+0000 7f4e82fba640 1 -- 172.21.15.72:0/3529761093 wait complete. 2024-04-15T15:15:16.125 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:16.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.226+0000 7f624953d640 1 Processor -- start 2024-04-15T15:15:16.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.226+0000 7f624953d640 1 -- start start 2024-04-15T15:15:16.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.227+0000 7f624953d640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f62440af400 0x7f62440ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.227+0000 7f624953d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f62440af860 0x7f62440cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.227+0000 7f624953d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f624406f8f0 con 0x7f624415eba0 2024-04-15T15:15:16.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.227+0000 7f624953d640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f624415ef80 con 0x7f62440af860 2024-04-15T15:15:16.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.227+0000 7f624953d640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f62440ce2b0 con 0x7f62440af400 2024-04-15T15:15:16.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.227+0000 7f62437fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f62440af860 0x7f62440cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.227+0000 7f62437fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f62440af860 0x7f62440cd0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45568/0 (socket says 172.21.15.72:45568) 2024-04-15T15:15:16.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.227+0000 7f62437fe640 1 -- 172.21.15.72:0/638127660 learned_addr learned my addr 172.21.15.72:0/638127660 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:16.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.227+0000 7f6242ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f624415eba0 0x7f624406f1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37960/0 (socket says 172.21.15.72:37960) 2024-04-15T15:15:16.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.227+0000 7f623a7fc640 1 --2- 172.21.15.72:0/638127660 >> v2:172.21.15.175:3300/0 conn(0x7f62440af400 0x7f62440ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.227+0000 7f62437fe640 1 -- 172.21.15.72:0/638127660 >> v2:172.21.15.175:3300/0 conn(0x7f62440af400 msgr2=0x7f62440ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:16.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.227+0000 7f62437fe640 1 --2- 172.21.15.72:0/638127660 >> v2:172.21.15.175:3300/0 conn(0x7f62440af400 0x7f62440ccb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.227+0000 7f62437fe640 1 -- 172.21.15.72:0/638127660 >> v1:172.21.15.72:6789/0 conn(0x7f624415eba0 legacy=0x7f624406f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.227+0000 7f62437fe640 1 -- 172.21.15.72:0/638127660 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62440ce590 con 0x7f62440af860 2024-04-15T15:15:16.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.227+0000 7f62427fc640 1 -- 172.21.15.72:0/638127660 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 291684945 0 0) 0x7f624406f8f0 con 0x7f624415eba0 2024-04-15T15:15:16.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.228+0000 7f62437fe640 1 --2- 172.21.15.72:0/638127660 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62440af860 0x7f62440cd0d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6244075490 tx=0x7f623c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54d9751bcdca8bba server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:16.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.228+0000 7f62427fc640 1 -- 172.21.15.72:0/638127660 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f623c02ca80 con 0x7f62440af860 2024-04-15T15:15:16.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.228+0000 7f62427fc640 1 -- 172.21.15.72:0/638127660 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f623c02cc20 con 0x7f62440af860 2024-04-15T15:15:16.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.228+0000 7f62427fc640 1 -- 172.21.15.72:0/638127660 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f623c0343f0 con 0x7f62440af860 2024-04-15T15:15:16.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.229+0000 7f624953d640 1 -- 172.21.15.72:0/638127660 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62440af860 msgr2=0x7f62440cd0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.229+0000 7f624953d640 1 --2- 172.21.15.72:0/638127660 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62440af860 0x7f62440cd0d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6244075490 tx=0x7f623c02bee0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.229+0000 7f624953d640 1 -- 172.21.15.72:0/638127660 shutdown_connections 2024-04-15T15:15:16.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.229+0000 7f624953d640 1 --2- 172.21.15.72:0/638127660 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62440af860 0x7f62440cd0d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.229+0000 7f624953d640 1 --2- 172.21.15.72:0/638127660 >> v2:172.21.15.175:3300/0 conn(0x7f62440af400 0x7f62440ccb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.229+0000 7f624953d640 1 -- 172.21.15.72:0/638127660 >> 172.21.15.72:0/638127660 conn(0x7f62440b1dd0 msgr2=0x7f62440b21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:16.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.229+0000 7f624953d640 1 -- 172.21.15.72:0/638127660 shutdown_connections 2024-04-15T15:15:16.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.229+0000 7f624953d640 1 -- 172.21.15.72:0/638127660 wait complete. 2024-04-15T15:15:16.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.230+0000 7f624953d640 1 Processor -- start 2024-04-15T15:15:16.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.230+0000 7f624953d640 1 -- start start 2024-04-15T15:15:16.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.231+0000 7f624953d640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f62440af400 0x7f62441470e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.231+0000 7f624953d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62440af860 0x7f624409dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.231+0000 7f624953d640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f624415eba0 0x7f6244157010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.231+0000 7f624953d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f62440b2b80 con 0x7f62440af860 2024-04-15T15:15:16.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.231+0000 7f624953d640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f624406ecc0 con 0x7f624415eba0 2024-04-15T15:15:16.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.231+0000 7f624953d640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f62440ceb50 con 0x7f62440af400 2024-04-15T15:15:16.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.231+0000 7f623a7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62440af860 0x7f624409dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.231+0000 7f623a7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62440af860 0x7f624409dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45570/0 (socket says 172.21.15.72:45570) 2024-04-15T15:15:16.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.231+0000 7f623a7fc640 1 -- 172.21.15.72:0/2964079971 learned_addr learned my addr 172.21.15.72:0/2964079971 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:16.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.231+0000 7f62437fe640 1 --2- 172.21.15.72:0/2964079971 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f624415eba0 0x7f6244157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.231+0000 7f6242ffd640 1 --2- 172.21.15.72:0/2964079971 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f62440af400 0x7f62441470e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.231+0000 7f623a7fc640 1 -- 172.21.15.72:0/2964079971 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f62440af400 msgr2=0x7f62441470e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.231+0000 7f623a7fc640 1 --2- 172.21.15.72:0/2964079971 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f62440af400 0x7f62441470e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.231+0000 7f623a7fc640 1 -- 172.21.15.72:0/2964079971 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f624415eba0 msgr2=0x7f6244157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.231+0000 7f623a7fc640 1 --2- 172.21.15.72:0/2964079971 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f624415eba0 0x7f6244157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.231+0000 7f623a7fc640 1 -- 172.21.15.72:0/2964079971 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6244153870 con 0x7f62440af860 2024-04-15T15:15:16.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.231+0000 7f6242ffd640 1 --2- 172.21.15.72:0/2964079971 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f62440af400 0x7f62441470e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:16.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.231+0000 7f62437fe640 1 --2- 172.21.15.72:0/2964079971 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f624415eba0 0x7f6244157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:16.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.231+0000 7f623a7fc640 1 --2- 172.21.15.72:0/2964079971 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62440af860 0x7f624409dc70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f622c00d5f0 tx=0x7f622c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:16.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.232+0000 7f623bfff640 1 -- 172.21.15.72:0/2964079971 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f622c0125f0 con 0x7f62440af860 2024-04-15T15:15:16.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.232+0000 7f623bfff640 1 -- 172.21.15.72:0/2964079971 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f622c019070 con 0x7f62440af860 2024-04-15T15:15:16.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.232+0000 7f624953d640 1 -- 172.21.15.72:0/2964079971 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6244153b10 con 0x7f62440af860 2024-04-15T15:15:16.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.232+0000 7f623bfff640 1 -- 172.21.15.72:0/2964079971 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f622c0129d0 con 0x7f62440af860 2024-04-15T15:15:16.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.232+0000 7f624953d640 1 -- 172.21.15.72:0/2964079971 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62441540d0 con 0x7f62440af860 2024-04-15T15:15:16.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.233+0000 7f624953d640 1 -- 172.21.15.72:0/2964079971 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62440689e0 con 0x7f62440af860 2024-04-15T15:15:16.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.233+0000 7f623bfff640 1 -- 172.21.15.72:0/2964079971 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f622c012b70 con 0x7f62440af860 2024-04-15T15:15:16.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.233+0000 7f623bfff640 1 --2- 172.21.15.72:0/2964079971 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f622406c8f0 0x7f622406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.234+0000 7f623bfff640 1 -- 172.21.15.72:0/2964079971 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f622c08c250 con 0x7f62440af860 2024-04-15T15:15:16.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.234+0000 7f6242ffd640 1 --2- 172.21.15.72:0/2964079971 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f622406c8f0 0x7f622406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.234+0000 7f6242ffd640 1 --2- 172.21.15.72:0/2964079971 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f622406c8f0 0x7f622406edb0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6234000f60 tx=0x7f6234009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:16.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.238+0000 7f623bfff640 1 -- 172.21.15.72:0/2964079971 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f622c05a3d0 con 0x7f62440af860 2024-04-15T15:15:16.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.413+0000 7f624953d640 1 -- 172.21.15.72:0/2964079971 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6244155fb0 con 0x7f62440af860 2024-04-15T15:15:16.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.414+0000 7f623bfff640 1 -- 172.21.15.72:0/2964079971 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7f622c059cd0 con 0x7f62440af860 2024-04-15T15:15:16.415 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:16.416 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":24,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:14:56.451113+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:14:49.438366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:14:50.368379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:14:54.081157+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:16.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.416+0000 7f624953d640 1 -- 172.21.15.72:0/2964079971 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f622406c8f0 msgr2=0x7f622406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.416+0000 7f624953d640 1 --2- 172.21.15.72:0/2964079971 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f622406c8f0 0x7f622406edb0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6234000f60 tx=0x7f6234009040 comp rx=0 tx=0).stop 2024-04-15T15:15:16.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.416+0000 7f624953d640 1 -- 172.21.15.72:0/2964079971 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62440af860 msgr2=0x7f624409dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.416+0000 7f624953d640 1 --2- 172.21.15.72:0/2964079971 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62440af860 0x7f624409dc70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f622c00d5f0 tx=0x7f622c00dab0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.416+0000 7f624953d640 1 -- 172.21.15.72:0/2964079971 shutdown_connections 2024-04-15T15:15:16.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.416+0000 7f624953d640 1 --2- 172.21.15.72:0/2964079971 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f622406c8f0 0x7f622406edb0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.416+0000 7f624953d640 1 --2- 172.21.15.72:0/2964079971 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f624415eba0 0x7f6244157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.416+0000 7f624953d640 1 --2- 172.21.15.72:0/2964079971 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62440af860 0x7f624409dc70 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.416+0000 7f624953d640 1 --2- 172.21.15.72:0/2964079971 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f62440af400 0x7f62441470e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.416+0000 7f624953d640 1 -- 172.21.15.72:0/2964079971 >> 172.21.15.72:0/2964079971 conn(0x7f62440b1dd0 msgr2=0x7f62440cc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:16.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.416+0000 7f624953d640 1 -- 172.21.15.72:0/2964079971 shutdown_connections 2024-04-15T15:15:16.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.416+0000 7f624953d640 1 -- 172.21.15.72:0/2964079971 wait complete. 2024-04-15T15:15:16.429 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:39.359102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:14:49.438366+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:14:50.368379+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:14:54.081157+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:15:16.429 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:16.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.533+0000 7f2a10c4e640 1 Processor -- start 2024-04-15T15:15:16.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.533+0000 7f2a10c4e640 1 -- start start 2024-04-15T15:15:16.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.534+0000 7f2a10c4e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2a0c0ce2f0 0x7f2a0c15c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.534+0000 7f2a10c4e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a0c06bbb0 con 0x7f2a0c0d08f0 2024-04-15T15:15:16.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.534+0000 7f2a10c4e640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a0c06be90 con 0x7f2a0c0cc7c0 2024-04-15T15:15:16.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.534+0000 7f2a10c4e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f2a0c06b960 con 0x7f2a0c0ce2f0 2024-04-15T15:15:16.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.534+0000 7f2a0a59c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2a0c0ce2f0 0x7f2a0c15c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.534+0000 7f2a09d9b640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f2a0c0d08f0 0x7f2a0c0cc0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37974/0 (socket says 172.21.15.72:37974) 2024-04-15T15:15:16.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.534+0000 7f2a09d9b640 1 -- 172.21.15.72:0/1638142732 learned_addr learned my addr 172.21.15.72:0/1638142732 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:16.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.535+0000 7f2a0a59c640 1 -- 172.21.15.72:0/1638142732 >> v1:172.21.15.175:6789/0 conn(0x7f2a0c0cc7c0 legacy=0x7f2a0c155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.535+0000 7f2a0a59c640 1 -- 172.21.15.72:0/1638142732 >> v1:172.21.15.72:6789/0 conn(0x7f2a0c0d08f0 legacy=0x7f2a0c0cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.535+0000 7f2a0a59c640 1 -- 172.21.15.72:0/1638142732 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a0c06c170 con 0x7f2a0c0ce2f0 2024-04-15T15:15:16.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.535+0000 7f2a0a59c640 1 --2- 172.21.15.72:0/1638142732 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a0c0ce2f0 0x7f2a0c15c530 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c06bae0 tx=0x7f29fc02bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46f797943623dc2c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:16.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.536+0000 7f2a0959a640 1 -- 172.21.15.72:0/1638142732 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29fc039070 con 0x7f2a0c0ce2f0 2024-04-15T15:15:16.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.536+0000 7f2a0959a640 1 -- 172.21.15.72:0/1638142732 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29fc02cce0 con 0x7f2a0c0ce2f0 2024-04-15T15:15:16.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.536+0000 7f2a0959a640 1 -- 172.21.15.72:0/1638142732 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29fc034660 con 0x7f2a0c0ce2f0 2024-04-15T15:15:16.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.536+0000 7f2a10c4e640 1 -- 172.21.15.72:0/1638142732 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a0c0ce2f0 msgr2=0x7f2a0c15c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.536+0000 7f2a10c4e640 1 --2- 172.21.15.72:0/1638142732 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a0c0ce2f0 0x7f2a0c15c530 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c06bae0 tx=0x7f29fc02bf30 comp rx=0 tx=0).stop 2024-04-15T15:15:16.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.536+0000 7f2a10c4e640 1 -- 172.21.15.72:0/1638142732 shutdown_connections 2024-04-15T15:15:16.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.536+0000 7f2a10c4e640 1 --2- 172.21.15.72:0/1638142732 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a0c0ce2f0 0x7f2a0c15c530 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.536+0000 7f2a10c4e640 1 -- 172.21.15.72:0/1638142732 >> 172.21.15.72:0/1638142732 conn(0x7f2a0c0b31d0 msgr2=0x7f2a0c0b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:16.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.536+0000 7f2a10c4e640 1 -- 172.21.15.72:0/1638142732 shutdown_connections 2024-04-15T15:15:16.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.536+0000 7f2a10c4e640 1 -- 172.21.15.72:0/1638142732 wait complete. 2024-04-15T15:15:16.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.537+0000 7f2a10c4e640 1 Processor -- start 2024-04-15T15:15:16.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.538+0000 7f2a10c4e640 1 -- start start 2024-04-15T15:15:16.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.538+0000 7f2a10c4e640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2a0c0cc7c0 0x7f2a0c1559e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.538+0000 7f2a10c4e640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2a0c0ce2f0 0x7f2a0c1f82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.538+0000 7f2a10c4e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a0c0d08f0 0x7f2a0c1e6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.538+0000 7f2a10c4e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2a0c06edd0 con 0x7f2a0c0d08f0 2024-04-15T15:15:16.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.538+0000 7f2a10c4e640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f2a0c15ca70 con 0x7f2a0c0cc7c0 2024-04-15T15:15:16.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.538+0000 7f2a10c4e640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f2a0c06ef50 con 0x7f2a0c0ce2f0 2024-04-15T15:15:16.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.538+0000 7f2a0ad9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a0c0d08f0 0x7f2a0c1e6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.538+0000 7f2a0ad9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a0c0d08f0 0x7f2a0c1e6690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45584/0 (socket says 172.21.15.72:45584) 2024-04-15T15:15:16.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.538+0000 7f2a0ad9d640 1 -- 172.21.15.72:0/2798388785 learned_addr learned my addr 172.21.15.72:0/2798388785 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:16.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.538+0000 7f2a0a59c640 1 --2- 172.21.15.72:0/2798388785 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2a0c0cc7c0 0x7f2a0c1559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.538+0000 7f2a09d9b640 1 --2- 172.21.15.72:0/2798388785 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2a0c0ce2f0 0x7f2a0c1f82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.539+0000 7f2a0ad9d640 1 -- 172.21.15.72:0/2798388785 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2a0c0ce2f0 msgr2=0x7f2a0c1f82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.539+0000 7f2a0ad9d640 1 --2- 172.21.15.72:0/2798388785 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2a0c0ce2f0 0x7f2a0c1f82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.539+0000 7f2a0ad9d640 1 -- 172.21.15.72:0/2798388785 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2a0c0cc7c0 msgr2=0x7f2a0c1559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.539+0000 7f2a0ad9d640 1 --2- 172.21.15.72:0/2798388785 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2a0c0cc7c0 0x7f2a0c1559e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.539+0000 7f2a0ad9d640 1 -- 172.21.15.72:0/2798388785 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a0c1f8830 con 0x7f2a0c0d08f0 2024-04-15T15:15:16.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.539+0000 7f2a09d9b640 1 --2- 172.21.15.72:0/2798388785 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2a0c0ce2f0 0x7f2a0c1f82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:16.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.539+0000 7f2a0ad9d640 1 --2- 172.21.15.72:0/2798388785 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a0c0d08f0 0x7f2a0c1e6690 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2a0400c3a0 tx=0x7f2a0400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:16.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.539+0000 7f29faffd640 1 -- 172.21.15.72:0/2798388785 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a04017070 con 0x7f2a0c0d08f0 2024-04-15T15:15:16.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.539+0000 7f29faffd640 1 -- 172.21.15.72:0/2798388785 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a040052d0 con 0x7f2a0c0d08f0 2024-04-15T15:15:16.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.539+0000 7f2a10c4e640 1 -- 172.21.15.72:0/2798388785 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a0c1e6bd0 con 0x7f2a0c0d08f0 2024-04-15T15:15:16.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.539+0000 7f29faffd640 1 -- 172.21.15.72:0/2798388785 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a04003830 con 0x7f2a0c0d08f0 2024-04-15T15:15:16.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.539+0000 7f2a10c4e640 1 -- 172.21.15.72:0/2798388785 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a0c0cd210 con 0x7f2a0c0d08f0 2024-04-15T15:15:16.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.540+0000 7f2a10c4e640 1 -- 172.21.15.72:0/2798388785 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a0c075c50 con 0x7f2a0c0d08f0 2024-04-15T15:15:16.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.541+0000 7f29faffd640 1 -- 172.21.15.72:0/2798388785 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2a0401a070 con 0x7f2a0c0d08f0 2024-04-15T15:15:16.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.541+0000 7f29faffd640 1 --2- 172.21.15.72:0/2798388785 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f29ec06c920 0x7f29ec06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.541+0000 7f2a0a59c640 1 --2- 172.21.15.72:0/2798388785 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f29ec06c920 0x7f29ec06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.541+0000 7f29faffd640 1 -- 172.21.15.72:0/2798388785 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f2a0408b050 con 0x7f2a0c0d08f0 2024-04-15T15:15:16.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.541+0000 7f2a0a59c640 1 --2- 172.21.15.72:0/2798388785 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f29ec06c920 0x7f29ec06ede0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f29fc036000 tx=0x7f29fc030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:16.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.545+0000 7f29faffd640 1 -- 172.21.15.72:0/2798388785 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2a04058c40 con 0x7f2a0c0d08f0 2024-04-15T15:15:16.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.719+0000 7f2a10c4e640 1 -- 172.21.15.72:0/2798388785 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2a0c06cab0 con 0x7f2a0c0d08f0 2024-04-15T15:15:16.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.719+0000 7f29faffd640 1 -- 172.21.15.72:0/2798388785 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7f2a0405cf40 con 0x7f2a0c0d08f0 2024-04-15T15:15:16.721 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-15T15:15:16.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.721+0000 7f2a10c4e640 1 -- 172.21.15.72:0/2798388785 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f29ec06c920 msgr2=0x7f29ec06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.721+0000 7f2a10c4e640 1 --2- 172.21.15.72:0/2798388785 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f29ec06c920 0x7f29ec06ede0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f29fc036000 tx=0x7f29fc030000 comp rx=0 tx=0).stop 2024-04-15T15:15:16.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.721+0000 7f2a10c4e640 1 -- 172.21.15.72:0/2798388785 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a0c0d08f0 msgr2=0x7f2a0c1e6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.721+0000 7f2a10c4e640 1 --2- 172.21.15.72:0/2798388785 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a0c0d08f0 0x7f2a0c1e6690 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2a0400c3a0 tx=0x7f2a0400c860 comp rx=0 tx=0).stop 2024-04-15T15:15:16.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.721+0000 7f2a10c4e640 1 -- 172.21.15.72:0/2798388785 shutdown_connections 2024-04-15T15:15:16.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.721+0000 7f2a10c4e640 1 --2- 172.21.15.72:0/2798388785 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f29ec06c920 0x7f29ec06ede0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.721+0000 7f2a10c4e640 1 --2- 172.21.15.72:0/2798388785 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2a0c0d08f0 0x7f2a0c1e6690 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.721+0000 7f2a10c4e640 1 --2- 172.21.15.72:0/2798388785 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2a0c0ce2f0 0x7f2a0c1f82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.721+0000 7f2a10c4e640 1 --2- 172.21.15.72:0/2798388785 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2a0c0cc7c0 0x7f2a0c1559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.721+0000 7f2a10c4e640 1 -- 172.21.15.72:0/2798388785 >> 172.21.15.72:0/2798388785 conn(0x7f2a0c0b31d0 msgr2=0x7f2a0c06f490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:16.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.721+0000 7f2a10c4e640 1 -- 172.21.15.72:0/2798388785 shutdown_connections 2024-04-15T15:15:16.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.721+0000 7f2a10c4e640 1 -- 172.21.15.72:0/2798388785 wait complete. 2024-04-15T15:15:16.735 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:15:16.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.839+0000 7f1836e60640 1 Processor -- start 2024-04-15T15:15:16.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.839+0000 7f1836e60640 1 -- start start 2024-04-15T15:15:16.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.840+0000 7f1836e60640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f18300ccb50 0x7f183006dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.840+0000 7f1836e60640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f183006bf20 0x7f183006c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.840+0000 7f1836e60640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f183015c110 con 0x7f183015af20 2024-04-15T15:15:16.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.840+0000 7f1836e60640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f183006ba90 con 0x7f18300ccb50 2024-04-15T15:15:16.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.840+0000 7f1836e60640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f18300cfae0 con 0x7f183006bf20 2024-04-15T15:15:16.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.840+0000 7f1834bfc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f18300ccb50 0x7f183006dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.840+0000 7f1834bfc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f18300ccb50 0x7f183006dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45586/0 (socket says 172.21.15.72:45586) 2024-04-15T15:15:16.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.840+0000 7f1834bfc640 1 -- 172.21.15.72:0/2337444464 learned_addr learned my addr 172.21.15.72:0/2337444464 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:16.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.840+0000 7f18353fd640 1 --2- 172.21.15.72:0/2337444464 >> v2:172.21.15.175:3300/0 conn(0x7f183006bf20 0x7f183006c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.840+0000 7f1834bfc640 1 -- 172.21.15.72:0/2337444464 >> v2:172.21.15.175:3300/0 conn(0x7f183006bf20 msgr2=0x7f183006c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:16.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.840+0000 7f1834bfc640 1 --2- 172.21.15.72:0/2337444464 >> v2:172.21.15.175:3300/0 conn(0x7f183006bf20 0x7f183006c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.840+0000 7f1834bfc640 1 -- 172.21.15.72:0/2337444464 >> v1:172.21.15.175:6789/0 conn(0x7f183015af20 legacy=0x7f18300ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:16.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.841+0000 7f1834bfc640 1 -- 172.21.15.72:0/2337444464 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18300cfdc0 con 0x7f18300ccb50 2024-04-15T15:15:16.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.841+0000 7f18353fd640 1 --2- 172.21.15.72:0/2337444464 >> v2:172.21.15.175:3300/0 conn(0x7f183006bf20 0x7f183006c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:16.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.841+0000 7f1834bfc640 1 --2- 172.21.15.72:0/2337444464 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18300ccb50 0x7f183006dc10 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f18300b56b0 tx=0x7f182002cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba93854bebdd388a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:16.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.841+0000 7f182f7fe640 1 -- 172.21.15.72:0/2337444464 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182003a070 con 0x7f18300ccb50 2024-04-15T15:15:16.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.841+0000 7f182f7fe640 1 -- 172.21.15.72:0/2337444464 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f182002db00 con 0x7f18300ccb50 2024-04-15T15:15:16.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.841+0000 7f182f7fe640 1 -- 172.21.15.72:0/2337444464 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182000b040 con 0x7f18300ccb50 2024-04-15T15:15:16.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.842+0000 7f1836e60640 1 -- 172.21.15.72:0/2337444464 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18300ccb50 msgr2=0x7f183006dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.842+0000 7f1836e60640 1 --2- 172.21.15.72:0/2337444464 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18300ccb50 0x7f183006dc10 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f18300b56b0 tx=0x7f182002cd50 comp rx=0 tx=0).stop 2024-04-15T15:15:16.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.842+0000 7f1836e60640 1 -- 172.21.15.72:0/2337444464 shutdown_connections 2024-04-15T15:15:16.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.842+0000 7f1836e60640 1 --2- 172.21.15.72:0/2337444464 >> v2:172.21.15.175:3300/0 conn(0x7f183006bf20 0x7f183006c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.842+0000 7f1836e60640 1 --2- 172.21.15.72:0/2337444464 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18300ccb50 0x7f183006dc10 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.842+0000 7f1836e60640 1 -- 172.21.15.72:0/2337444464 >> 172.21.15.72:0/2337444464 conn(0x7f18300b31d0 msgr2=0x7f18300b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:16.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.842+0000 7f1836e60640 1 -- 172.21.15.72:0/2337444464 shutdown_connections 2024-04-15T15:15:16.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.842+0000 7f1836e60640 1 -- 172.21.15.72:0/2337444464 wait complete. 2024-04-15T15:15:16.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.843+0000 7f1836e60640 1 Processor -- start 2024-04-15T15:15:16.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.843+0000 7f1836e60640 1 -- start start 2024-04-15T15:15:16.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.844+0000 7f1836e60640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f183006bf20 0x7f1830145b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.844+0000 7f1836e60640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f18300ccb50 0x7f183015f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.844+0000 7f1836e60640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f183015af20 0x7f18301471d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.844+0000 7f1836e60640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f18300cc8f0 con 0x7f183006bf20 2024-04-15T15:15:16.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.844+0000 7f1836e60640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f18300cc770 con 0x7f183015af20 2024-04-15T15:15:16.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.844+0000 7f1836e60640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f183015bcf0 con 0x7f18300ccb50 2024-04-15T15:15:16.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.844+0000 7f1834bfc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f183006bf20 0x7f1830145b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.844+0000 7f1834bfc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f183006bf20 0x7f1830145b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45600/0 (socket says 172.21.15.72:45600) 2024-04-15T15:15:16.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.844+0000 7f1834bfc640 1 -- 172.21.15.72:0/3913802059 learned_addr learned my addr 172.21.15.72:0/3913802059 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:16.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.844+0000 7f18353fd640 1 --2- 172.21.15.72:0/3913802059 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f183015af20 0x7f18301471d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.844+0000 7f182ffff640 1 --2- 172.21.15.72:0/3913802059 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f18300ccb50 0x7f183015f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.844+0000 7f1834bfc640 1 -- 172.21.15.72:0/3913802059 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f18300ccb50 msgr2=0x7f183015f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.844+0000 7f1834bfc640 1 --2- 172.21.15.72:0/3913802059 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f18300ccb50 0x7f183015f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.844+0000 7f1834bfc640 1 -- 172.21.15.72:0/3913802059 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f183015af20 msgr2=0x7f18301471d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.844+0000 7f1834bfc640 1 --2- 172.21.15.72:0/3913802059 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f183015af20 0x7f18301471d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.844+0000 7f1834bfc640 1 -- 172.21.15.72:0/3913802059 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f183015f6b0 con 0x7f183006bf20 2024-04-15T15:15:16.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.845+0000 7f1834bfc640 1 --2- 172.21.15.72:0/3913802059 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f183006bf20 0x7f1830145b50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1820000c00 tx=0x7f1820004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:16.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.845+0000 7f182d7fa640 1 -- 172.21.15.72:0/3913802059 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1820042070 con 0x7f183006bf20 2024-04-15T15:15:16.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.845+0000 7f182d7fa640 1 -- 172.21.15.72:0/3913802059 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f182003c830 con 0x7f183006bf20 2024-04-15T15:15:16.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.845+0000 7f182d7fa640 1 -- 172.21.15.72:0/3913802059 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182003cc00 con 0x7f183006bf20 2024-04-15T15:15:16.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.845+0000 7f1836e60640 1 -- 172.21.15.72:0/3913802059 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1830146130 con 0x7f183006bf20 2024-04-15T15:15:16.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.845+0000 7f1836e60640 1 -- 172.21.15.72:0/3913802059 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18301469d0 con 0x7f183006bf20 2024-04-15T15:15:16.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.846+0000 7f1836e60640 1 -- 172.21.15.72:0/3913802059 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18300ce3c0 con 0x7f183006bf20 2024-04-15T15:15:16.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.846+0000 7f182d7fa640 1 -- 172.21.15.72:0/3913802059 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f182002d980 con 0x7f183006bf20 2024-04-15T15:15:16.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.847+0000 7f182d7fa640 1 --2- 172.21.15.72:0/3913802059 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f181c06c8f0 0x7f181c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.847+0000 7f182d7fa640 1 -- 172.21.15.72:0/3913802059 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f18200af980 con 0x7f183006bf20 2024-04-15T15:15:16.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.847+0000 7f182ffff640 1 --2- 172.21.15.72:0/3913802059 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f181c06c8f0 0x7f181c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.847+0000 7f182ffff640 1 --2- 172.21.15.72:0/3913802059 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f181c06c8f0 0x7f181c06edb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1824006fd0 tx=0x7f1824008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:16.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:16.851+0000 7f182d7fa640 1 -- 172.21.15.72:0/3913802059 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f18300ce3c0 con 0x7f183006bf20 2024-04-15T15:15:17.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.025+0000 7f1836e60640 1 -- 172.21.15.72:0/3913802059 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f183015a2a0 con 0x7f183006bf20 2024-04-15T15:15:17.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.025+0000 7f182d7fa640 1 -- 172.21.15.72:0/3913802059 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7f182007da70 con 0x7f183006bf20 2024-04-15T15:15:17.027 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:17.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.028+0000 7f1836e60640 1 -- 172.21.15.72:0/3913802059 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f181c06c8f0 msgr2=0x7f181c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.028+0000 7f1836e60640 1 --2- 172.21.15.72:0/3913802059 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f181c06c8f0 0x7f181c06edb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1824006fd0 tx=0x7f1824008040 comp rx=0 tx=0).stop 2024-04-15T15:15:17.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.028+0000 7f1836e60640 1 -- 172.21.15.72:0/3913802059 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f183006bf20 msgr2=0x7f1830145b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.028+0000 7f1836e60640 1 --2- 172.21.15.72:0/3913802059 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f183006bf20 0x7f1830145b50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f1820000c00 tx=0x7f1820004730 comp rx=0 tx=0).stop 2024-04-15T15:15:17.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.028+0000 7f1836e60640 1 -- 172.21.15.72:0/3913802059 shutdown_connections 2024-04-15T15:15:17.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.028+0000 7f1836e60640 1 --2- 172.21.15.72:0/3913802059 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f181c06c8f0 0x7f181c06edb0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.028+0000 7f1836e60640 1 --2- 172.21.15.72:0/3913802059 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f183015af20 0x7f18301471d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.028+0000 7f1836e60640 1 --2- 172.21.15.72:0/3913802059 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f18300ccb50 0x7f183015f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.028+0000 7f1836e60640 1 --2- 172.21.15.72:0/3913802059 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f183006bf20 0x7f1830145b50 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.028+0000 7f1836e60640 1 -- 172.21.15.72:0/3913802059 >> 172.21.15.72:0/3913802059 conn(0x7f18300b31d0 msgr2=0x7f18300afd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:17.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.028+0000 7f1836e60640 1 -- 172.21.15.72:0/3913802059 shutdown_connections 2024-04-15T15:15:17.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.028+0000 7f1836e60640 1 -- 172.21.15.72:0/3913802059 wait complete. 2024-04-15T15:15:17.041 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:15:17.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.145+0000 7ff20bb29640 1 Processor -- start 2024-04-15T15:15:17.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.146+0000 7ff20bb29640 1 -- start start 2024-04-15T15:15:17.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.147+0000 7ff20bb29640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff2040b5410 0x7ff2040b57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.147+0000 7ff20bb29640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff2040afae0 0x7ff2040cf960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.147+0000 7ff20bb29640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2040d07c0 con 0x7ff2040b0050 2024-04-15T15:15:17.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.147+0000 7ff20bb29640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7ff2040d0330 con 0x7ff2040b5410 2024-04-15T15:15:17.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.147+0000 7ff20bb29640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7ff2040b5d30 con 0x7ff2040afae0 2024-04-15T15:15:17.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.147+0000 7ff2098c5640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff2040b5410 0x7ff2040b57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.147+0000 7ff2098c5640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff2040b5410 0x7ff2040b57f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45608/0 (socket says 172.21.15.72:45608) 2024-04-15T15:15:17.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.147+0000 7ff2098c5640 1 -- 172.21.15.72:0/3418218216 learned_addr learned my addr 172.21.15.72:0/3418218216 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:17.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.147+0000 7ff20a0c6640 1 --2- 172.21.15.72:0/3418218216 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff2040afae0 0x7ff2040cf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.147+0000 7ff2098c5640 1 -- 172.21.15.72:0/3418218216 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff2040afae0 msgr2=0x7ff2040cf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:17.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.147+0000 7ff2098c5640 1 --2- 172.21.15.72:0/3418218216 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff2040afae0 0x7ff2040cf960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.147+0000 7ff2098c5640 1 -- 172.21.15.72:0/3418218216 >> v1:172.21.15.175:6789/0 conn(0x7ff2040b0050 legacy=0x7ff20406c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.147+0000 7ff2098c5640 1 -- 172.21.15.72:0/3418218216 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2040b3aa0 con 0x7ff2040b5410 2024-04-15T15:15:17.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.147+0000 7ff20a0c6640 1 --2- 172.21.15.72:0/3418218216 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff2040afae0 0x7ff2040cf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:17.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.148+0000 7ff2098c5640 1 --2- 172.21.15.72:0/3418218216 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2040b5410 0x7ff2040b57f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc009870 tx=0x7ff1fc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9d70970a2010e04 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:17.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.148+0000 7ff2088c3640 1 -- 172.21.15.72:0/3418218216 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc035020 con 0x7ff2040b5410 2024-04-15T15:15:17.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.148+0000 7ff2088c3640 1 -- 172.21.15.72:0/3418218216 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1fc031640 con 0x7ff2040b5410 2024-04-15T15:15:17.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.148+0000 7ff2088c3640 1 -- 172.21.15.72:0/3418218216 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc02d040 con 0x7ff2040b5410 2024-04-15T15:15:17.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.148+0000 7ff20bb29640 1 -- 172.21.15.72:0/3418218216 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2040b5410 msgr2=0x7ff2040b57f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.148+0000 7ff20bb29640 1 --2- 172.21.15.72:0/3418218216 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2040b5410 0x7ff2040b57f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc009870 tx=0x7ff1fc02b8f0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.149+0000 7ff20bb29640 1 -- 172.21.15.72:0/3418218216 shutdown_connections 2024-04-15T15:15:17.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.149+0000 7ff20bb29640 1 --2- 172.21.15.72:0/3418218216 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff2040afae0 0x7ff2040cf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.149+0000 7ff20bb29640 1 --2- 172.21.15.72:0/3418218216 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2040b5410 0x7ff2040b57f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.149+0000 7ff20bb29640 1 -- 172.21.15.72:0/3418218216 >> 172.21.15.72:0/3418218216 conn(0x7ff2040d0a50 msgr2=0x7ff2040d0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:17.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.149+0000 7ff20bb29640 1 -- 172.21.15.72:0/3418218216 shutdown_connections 2024-04-15T15:15:17.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.149+0000 7ff20bb29640 1 -- 172.21.15.72:0/3418218216 wait complete. 2024-04-15T15:15:17.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.150+0000 7ff20bb29640 1 Processor -- start 2024-04-15T15:15:17.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.150+0000 7ff20bb29640 1 -- start start 2024-04-15T15:15:17.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.150+0000 7ff20bb29640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff2040afae0 0x7ff20415ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.150+0000 7ff20bb29640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff2040b0050 0x7ff20414b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.150+0000 7ff20bb29640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2040b5410 0x7ff2040ae3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.150+0000 7ff20bb29640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ff2040d18f0 con 0x7ff2040b5410 2024-04-15T15:15:17.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.150+0000 7ff20bb29640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7ff2040d1610 con 0x7ff2040b0050 2024-04-15T15:15:17.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.150+0000 7ff20bb29640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7ff2040cfea0 con 0x7ff2040afae0 2024-04-15T15:15:17.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.151+0000 7ff20a0c6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2040b5410 0x7ff2040ae3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.151+0000 7ff20a0c6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2040b5410 0x7ff2040ae3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45624/0 (socket says 172.21.15.72:45624) 2024-04-15T15:15:17.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.151+0000 7ff20a0c6640 1 -- 172.21.15.72:0/2505674733 learned_addr learned my addr 172.21.15.72:0/2505674733 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:17.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.151+0000 7ff2090c4640 1 --2- 172.21.15.72:0/2505674733 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff2040b0050 0x7ff20414b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.151+0000 7ff2098c5640 1 --2- 172.21.15.72:0/2505674733 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff2040afae0 0x7ff20415ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.151+0000 7ff20a0c6640 1 -- 172.21.15.72:0/2505674733 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff2040afae0 msgr2=0x7ff20415ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.151+0000 7ff20a0c6640 1 --2- 172.21.15.72:0/2505674733 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff2040afae0 0x7ff20415ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.151+0000 7ff20a0c6640 1 -- 172.21.15.72:0/2505674733 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff2040b0050 msgr2=0x7ff20414b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.151+0000 7ff20a0c6640 1 --2- 172.21.15.72:0/2505674733 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff2040b0050 0x7ff20414b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.151+0000 7ff20a0c6640 1 -- 172.21.15.72:0/2505674733 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2041f9080 con 0x7ff2040b5410 2024-04-15T15:15:17.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.151+0000 7ff20a0c6640 1 --2- 172.21.15.72:0/2505674733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2040b5410 0x7ff2040ae3b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff1f400cd50 tx=0x7ff1f400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:17.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.151+0000 7ff2027fc640 1 -- 172.21.15.72:0/2505674733 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1f400f680 con 0x7ff2040b5410 2024-04-15T15:15:17.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.151+0000 7ff2027fc640 1 -- 172.21.15.72:0/2505674733 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1f400f840 con 0x7ff2040b5410 2024-04-15T15:15:17.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.151+0000 7ff20bb29640 1 -- 172.21.15.72:0/2505674733 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2041f8da0 con 0x7ff2040b5410 2024-04-15T15:15:17.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.151+0000 7ff2027fc640 1 -- 172.21.15.72:0/2505674733 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1f400fc30 con 0x7ff2040b5410 2024-04-15T15:15:17.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.151+0000 7ff20bb29640 1 -- 172.21.15.72:0/2505674733 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff20415f740 con 0x7ff2040b5410 2024-04-15T15:15:17.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.152+0000 7ff20bb29640 1 -- 172.21.15.72:0/2505674733 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2040d0e30 con 0x7ff2040b5410 2024-04-15T15:15:17.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.153+0000 7ff2027fc640 1 -- 172.21.15.72:0/2505674733 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff1f40213f0 con 0x7ff2040b5410 2024-04-15T15:15:17.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.153+0000 7ff2027fc640 1 --2- 172.21.15.72:0/2505674733 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff1f006c8f0 0x7ff1f006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.153+0000 7ff2098c5640 1 --2- 172.21.15.72:0/2505674733 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff1f006c8f0 0x7ff1f006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.153+0000 7ff2027fc640 1 -- 172.21.15.72:0/2505674733 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ff1f408cc40 con 0x7ff2040b5410 2024-04-15T15:15:17.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.154+0000 7ff2098c5640 1 --2- 172.21.15.72:0/2505674733 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff1f006c8f0 0x7ff1f006edb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc036000 tx=0x7ff1fc03c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:17.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.158+0000 7ff2027fc640 1 -- 172.21.15.72:0/2505674733 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff2040d0e30 con 0x7ff2040b5410 2024-04-15T15:15:17.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.331+0000 7ff20bb29640 1 -- 172.21.15.72:0/2505674733 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff20415cbe0 con 0x7ff2040b5410 2024-04-15T15:15:17.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.331+0000 7ff2027fc640 1 -- 172.21.15.72:0/2505674733 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7ff20415cbe0 con 0x7ff2040b5410 2024-04-15T15:15:17.333 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-04-15T15:15:17.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.333+0000 7ff20bb29640 1 -- 172.21.15.72:0/2505674733 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff1f006c8f0 msgr2=0x7ff1f006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.333+0000 7ff20bb29640 1 --2- 172.21.15.72:0/2505674733 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff1f006c8f0 0x7ff1f006edb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc036000 tx=0x7ff1fc03c040 comp rx=0 tx=0).stop 2024-04-15T15:15:17.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.333+0000 7ff20bb29640 1 -- 172.21.15.72:0/2505674733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2040b5410 msgr2=0x7ff2040ae3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.333+0000 7ff20bb29640 1 --2- 172.21.15.72:0/2505674733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2040b5410 0x7ff2040ae3b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff1f400cd50 tx=0x7ff1f400a650 comp rx=0 tx=0).stop 2024-04-15T15:15:17.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.333+0000 7ff20bb29640 1 -- 172.21.15.72:0/2505674733 shutdown_connections 2024-04-15T15:15:17.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.333+0000 7ff20bb29640 1 --2- 172.21.15.72:0/2505674733 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff1f006c8f0 0x7ff1f006edb0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.333+0000 7ff20bb29640 1 --2- 172.21.15.72:0/2505674733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2040b5410 0x7ff2040ae3b0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.333+0000 7ff20bb29640 1 --2- 172.21.15.72:0/2505674733 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff2040b0050 0x7ff20414b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.333+0000 7ff20bb29640 1 --2- 172.21.15.72:0/2505674733 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff2040afae0 0x7ff20415ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.333+0000 7ff20bb29640 1 -- 172.21.15.72:0/2505674733 >> 172.21.15.72:0/2505674733 conn(0x7ff2040d0a50 msgr2=0x7ff2040b07a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:17.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.333+0000 7ff20bb29640 1 -- 172.21.15.72:0/2505674733 shutdown_connections 2024-04-15T15:15:17.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.333+0000 7ff20bb29640 1 -- 172.21.15.72:0/2505674733 wait complete. 2024-04-15T15:15:17.353 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:15:17.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.458+0000 7f64a1860640 1 Processor -- start 2024-04-15T15:15:17.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.458+0000 7f64a1860640 1 -- start start 2024-04-15T15:15:17.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.459+0000 7f64a1860640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f649c15e610 0x7f649c0d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.459+0000 7f64a1860640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f649c15e9f0 0x7f649c0d3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.459+0000 7f64a1860640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f649c06c400 0x7f649c06c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.459+0000 7f64a1860640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f649c0d31d0 con 0x7f649c15e610 2024-04-15T15:15:17.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.459+0000 7f64a1860640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f649c0d2320 con 0x7f649c06c400 2024-04-15T15:15:17.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.459+0000 7f64a1860640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f649c0d25c0 con 0x7f649c15e9f0 2024-04-15T15:15:17.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.459+0000 7f649affd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f649c15e610 0x7f649c0d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.459+0000 7f649affd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f649c15e610 0x7f649c0d0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45634/0 (socket says 172.21.15.72:45634) 2024-04-15T15:15:17.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.459+0000 7f649affd640 1 -- 172.21.15.72:0/46472690 learned_addr learned my addr 172.21.15.72:0/46472690 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:17.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.459+0000 7f649affd640 1 -- 172.21.15.72:0/46472690 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f649c15e9f0 msgr2=0x7f649c0d3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:17.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.459+0000 7f649b7fe640 1 --2- 172.21.15.72:0/46472690 >> v2:172.21.15.175:3300/0 conn(0x7f649c06c400 0x7f649c06c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.459+0000 7f649affd640 1 --2- 172.21.15.72:0/46472690 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f649c15e9f0 0x7f649c0d3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.459+0000 7f649affd640 1 -- 172.21.15.72:0/46472690 >> v2:172.21.15.175:3300/0 conn(0x7f649c06c400 msgr2=0x7f649c06c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:17.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.459+0000 7f649affd640 1 --2- 172.21.15.72:0/46472690 >> v2:172.21.15.175:3300/0 conn(0x7f649c06c400 0x7f649c06c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.459+0000 7f649affd640 1 -- 172.21.15.72:0/46472690 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f649c0d28a0 con 0x7f649c15e610 2024-04-15T15:15:17.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.460+0000 7f649affd640 1 --2- 172.21.15.72:0/46472690 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f649c15e610 0x7f649c0d0690 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f649c15ee30 tx=0x7f648c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f9560cf51b8ef00 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:17.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.460+0000 7f6499ffb640 1 -- 172.21.15.72:0/46472690 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f648c034070 con 0x7f649c15e610 2024-04-15T15:15:17.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.460+0000 7f6499ffb640 1 -- 172.21.15.72:0/46472690 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f648c030860 con 0x7f649c15e610 2024-04-15T15:15:17.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.460+0000 7f6499ffb640 1 -- 172.21.15.72:0/46472690 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f648c030c30 con 0x7f649c15e610 2024-04-15T15:15:17.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.461+0000 7f64a1860640 1 -- 172.21.15.72:0/46472690 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f649c15e610 msgr2=0x7f649c0d0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.461+0000 7f64a1860640 1 --2- 172.21.15.72:0/46472690 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f649c15e610 0x7f649c0d0690 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f649c15ee30 tx=0x7f648c02b9a0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.461+0000 7f64a1860640 1 -- 172.21.15.72:0/46472690 shutdown_connections 2024-04-15T15:15:17.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.461+0000 7f64a1860640 1 --2- 172.21.15.72:0/46472690 >> v2:172.21.15.175:3300/0 conn(0x7f649c06c400 0x7f649c06c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.461+0000 7f64a1860640 1 --2- 172.21.15.72:0/46472690 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f649c15e9f0 0x7f649c0d3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.461+0000 7f64a1860640 1 --2- 172.21.15.72:0/46472690 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f649c15e610 0x7f649c0d0690 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.461+0000 7f64a1860640 1 -- 172.21.15.72:0/46472690 >> 172.21.15.72:0/46472690 conn(0x7f649c0d4750 msgr2=0x7f649c0d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:17.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.461+0000 7f64a1860640 1 -- 172.21.15.72:0/46472690 shutdown_connections 2024-04-15T15:15:17.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.461+0000 7f64a1860640 1 -- 172.21.15.72:0/46472690 wait complete. 2024-04-15T15:15:17.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.462+0000 7f64a1860640 1 Processor -- start 2024-04-15T15:15:17.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.462+0000 7f64a1860640 1 -- start start 2024-04-15T15:15:17.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.463+0000 7f64a1860640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f649c06c400 0x7f649c1f7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.463+0000 7f64a1860640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f649c15e610 0x7f649c160d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.463+0000 7f64a1860640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f649c15e9f0 0x7f649c15b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.463+0000 7f64a1860640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f649c06d130 con 0x7f649c15e610 2024-04-15T15:15:17.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.463+0000 7f64a1860640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f649c0d2e60 con 0x7f649c15e9f0 2024-04-15T15:15:17.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.463+0000 7f64a1860640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f649c0d0bd0 con 0x7f649c06c400 2024-04-15T15:15:17.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.463+0000 7f649a7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f649c15e610 0x7f649c160d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.463+0000 7f649a7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f649c15e610 0x7f649c160d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45638/0 (socket says 172.21.15.72:45638) 2024-04-15T15:15:17.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.463+0000 7f649a7fc640 1 -- 172.21.15.72:0/788315409 learned_addr learned my addr 172.21.15.72:0/788315409 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:17.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.463+0000 7f649b7fe640 1 --2- 172.21.15.72:0/788315409 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f649c15e9f0 0x7f649c15b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.463+0000 7f649affd640 1 --2- 172.21.15.72:0/788315409 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f649c06c400 0x7f649c1f7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.463+0000 7f649a7fc640 1 -- 172.21.15.72:0/788315409 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f649c06c400 msgr2=0x7f649c1f7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.463+0000 7f649a7fc640 1 --2- 172.21.15.72:0/788315409 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f649c06c400 0x7f649c1f7230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.463+0000 7f649a7fc640 1 -- 172.21.15.72:0/788315409 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f649c15e9f0 msgr2=0x7f649c15b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.463+0000 7f649a7fc640 1 --2- 172.21.15.72:0/788315409 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f649c15e9f0 0x7f649c15b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.463+0000 7f649a7fc640 1 -- 172.21.15.72:0/788315409 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f649c1612a0 con 0x7f649c15e610 2024-04-15T15:15:17.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.463+0000 7f649a7fc640 1 --2- 172.21.15.72:0/788315409 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f649c15e610 0x7f649c160d60 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f649000b410 tx=0x7f649000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:17.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.464+0000 7f6483fff640 1 -- 172.21.15.72:0/788315409 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6490015020 con 0x7f649c15e610 2024-04-15T15:15:17.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.464+0000 7f6483fff640 1 -- 172.21.15.72:0/788315409 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6490011850 con 0x7f649c15e610 2024-04-15T15:15:17.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.464+0000 7f64a1860640 1 -- 172.21.15.72:0/788315409 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f649c1f43c0 con 0x7f649c15e610 2024-04-15T15:15:17.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.464+0000 7f6483fff640 1 -- 172.21.15.72:0/788315409 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6490011c40 con 0x7f649c15e610 2024-04-15T15:15:17.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.464+0000 7f64a1860640 1 -- 172.21.15.72:0/788315409 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f649c1f4980 con 0x7f649c15e610 2024-04-15T15:15:17.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.465+0000 7f64a1860640 1 -- 172.21.15.72:0/788315409 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f649c075c50 con 0x7f649c15e610 2024-04-15T15:15:17.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.465+0000 7f6483fff640 1 -- 172.21.15.72:0/788315409 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6490018020 con 0x7f649c15e610 2024-04-15T15:15:17.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.465+0000 7f6483fff640 1 --2- 172.21.15.72:0/788315409 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f648806c8f0 0x7f648806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.465+0000 7f6483fff640 1 -- 172.21.15.72:0/788315409 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f649008bbd0 con 0x7f649c15e610 2024-04-15T15:15:17.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.466+0000 7f649affd640 1 --2- 172.21.15.72:0/788315409 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f648806c8f0 0x7f648806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.466+0000 7f649affd640 1 --2- 172.21.15.72:0/788315409 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f648806c8f0 0x7f648806edb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f648c036000 tx=0x7f648c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:17.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.470+0000 7f6483fff640 1 -- 172.21.15.72:0/788315409 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f649001f020 con 0x7f649c15e610 2024-04-15T15:15:17.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.643+0000 7f64a1860640 1 -- 172.21.15.72:0/788315409 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f649c0cf740 con 0x7f649c15e610 2024-04-15T15:15:17.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.644+0000 7f6483fff640 1 -- 172.21.15.72:0/788315409 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7f6490059700 con 0x7f649c15e610 2024-04-15T15:15:17.645 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:17.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.645+0000 7f64a1860640 1 -- 172.21.15.72:0/788315409 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f648806c8f0 msgr2=0x7f648806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.645+0000 7f64a1860640 1 --2- 172.21.15.72:0/788315409 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f648806c8f0 0x7f648806edb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f648c036000 tx=0x7f648c008030 comp rx=0 tx=0).stop 2024-04-15T15:15:17.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.645+0000 7f64a1860640 1 -- 172.21.15.72:0/788315409 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f649c15e610 msgr2=0x7f649c160d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.645+0000 7f64a1860640 1 --2- 172.21.15.72:0/788315409 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f649c15e610 0x7f649c160d60 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f649000b410 tx=0x7f649000b8d0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.645+0000 7f64a1860640 1 -- 172.21.15.72:0/788315409 shutdown_connections 2024-04-15T15:15:17.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.645+0000 7f64a1860640 1 --2- 172.21.15.72:0/788315409 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f648806c8f0 0x7f648806edb0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.645+0000 7f64a1860640 1 --2- 172.21.15.72:0/788315409 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f649c15e9f0 0x7f649c15b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.645+0000 7f64a1860640 1 --2- 172.21.15.72:0/788315409 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f649c15e610 0x7f649c160d60 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.645+0000 7f64a1860640 1 --2- 172.21.15.72:0/788315409 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f649c06c400 0x7f649c1f7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.645+0000 7f64a1860640 1 -- 172.21.15.72:0/788315409 >> 172.21.15.72:0/788315409 conn(0x7f649c0d4750 msgr2=0x7f649c06bb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:17.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.646+0000 7f64a1860640 1 -- 172.21.15.72:0/788315409 shutdown_connections 2024-04-15T15:15:17.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.646+0000 7f64a1860640 1 -- 172.21.15.72:0/788315409 wait complete. 2024-04-15T15:15:17.658 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:15:17.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.760+0000 7ff49091c640 1 Processor -- start 2024-04-15T15:15:17.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.760+0000 7ff49091c640 1 -- start start 2024-04-15T15:15:17.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.761+0000 7ff49091c640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff48c156d30 0x7ff48c0d11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.761+0000 7ff49091c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff48c0cd160 0x7ff48c0cf360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.761+0000 7ff49091c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff48c0d0100 con 0x7ff48c0cd5a0 2024-04-15T15:15:17.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.761+0000 7ff49091c640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7ff48c0d03e0 con 0x7ff48c0cd160 2024-04-15T15:15:17.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.761+0000 7ff49091c640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7ff48c0d06c0 con 0x7ff48c156d30 2024-04-15T15:15:17.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.761+0000 7ff489d9b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff48c0cd160 0x7ff48c0cf360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.761+0000 7ff48ad9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ff48c0cd5a0 0x7ff48c15dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:37990/0 (socket says 172.21.15.72:37990) 2024-04-15T15:15:17.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.761+0000 7ff48ad9d640 1 -- 172.21.15.72:0/1889673383 learned_addr learned my addr 172.21.15.72:0/1889673383 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:17.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.761+0000 7ff48a59c640 1 --2- 172.21.15.72:0/1889673383 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff48c156d30 0x7ff48c0d11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.761+0000 7ff489d9b640 1 -- 172.21.15.72:0/1889673383 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff48c156d30 msgr2=0x7ff48c0d11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:17.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.761+0000 7ff489d9b640 1 --2- 172.21.15.72:0/1889673383 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff48c156d30 0x7ff48c0d11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.761+0000 7ff489d9b640 1 -- 172.21.15.72:0/1889673383 >> v1:172.21.15.72:6789/0 conn(0x7ff48c0cd5a0 legacy=0x7ff48c15dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.761+0000 7ff489d9b640 1 -- 172.21.15.72:0/1889673383 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff48c0d09a0 con 0x7ff48c0cd160 2024-04-15T15:15:17.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.761+0000 7ff48959a640 1 -- 172.21.15.72:0/1889673383 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 489593247 0 0) 0x7ff48c0d0100 con 0x7ff48c0cd5a0 2024-04-15T15:15:17.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.761+0000 7ff48a59c640 1 --2- 172.21.15.72:0/1889673383 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff48c156d30 0x7ff48c0d11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:17.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.762+0000 7ff489d9b640 1 --2- 172.21.15.72:0/1889673383 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff48c0cd160 0x7ff48c0cf360 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff4800097a0 tx=0x7ff48002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23006d1655f2718f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:17.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.762+0000 7ff48959a640 1 -- 172.21.15.72:0/1889673383 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff480035020 con 0x7ff48c0cd160 2024-04-15T15:15:17.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.762+0000 7ff48959a640 1 -- 172.21.15.72:0/1889673383 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff480031830 con 0x7ff48c0cd160 2024-04-15T15:15:17.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.762+0000 7ff48959a640 1 -- 172.21.15.72:0/1889673383 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff480031c00 con 0x7ff48c0cd160 2024-04-15T15:15:17.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.763+0000 7ff49091c640 1 -- 172.21.15.72:0/1889673383 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff48c0cd160 msgr2=0x7ff48c0cf360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.763+0000 7ff49091c640 1 --2- 172.21.15.72:0/1889673383 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff48c0cd160 0x7ff48c0cf360 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff4800097a0 tx=0x7ff48002b8d0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.763+0000 7ff49091c640 1 -- 172.21.15.72:0/1889673383 shutdown_connections 2024-04-15T15:15:17.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.763+0000 7ff49091c640 1 --2- 172.21.15.72:0/1889673383 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff48c0cd160 0x7ff48c0cf360 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.763+0000 7ff49091c640 1 --2- 172.21.15.72:0/1889673383 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff48c156d30 0x7ff48c0d11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.763+0000 7ff49091c640 1 -- 172.21.15.72:0/1889673383 >> 172.21.15.72:0/1889673383 conn(0x7ff48c0b43c0 msgr2=0x7ff48c0b47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:17.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.763+0000 7ff49091c640 1 -- 172.21.15.72:0/1889673383 shutdown_connections 2024-04-15T15:15:17.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.763+0000 7ff49091c640 1 -- 172.21.15.72:0/1889673383 wait complete. 2024-04-15T15:15:17.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.764+0000 7ff49091c640 1 Processor -- start 2024-04-15T15:15:17.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.764+0000 7ff49091c640 1 -- start start 2024-04-15T15:15:17.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.765+0000 7ff49091c640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff48c0cd160 0x7ff48c149230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.765+0000 7ff49091c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff48c0cd5a0 0x7ff48c15cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.765+0000 7ff49091c640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff48c156d30 0x7ff48c15d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.765+0000 7ff49091c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ff48c0cce90 con 0x7ff48c0cd5a0 2024-04-15T15:15:17.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.765+0000 7ff49091c640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7ff48c0ccd10 con 0x7ff48c0cd160 2024-04-15T15:15:17.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.765+0000 7ff49091c640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7ff48c0cf8a0 con 0x7ff48c156d30 2024-04-15T15:15:17.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.765+0000 7ff489d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff48c0cd5a0 0x7ff48c15cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.765+0000 7ff489d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff48c0cd5a0 0x7ff48c15cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45654/0 (socket says 172.21.15.72:45654) 2024-04-15T15:15:17.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.765+0000 7ff489d9b640 1 -- 172.21.15.72:0/1919598655 learned_addr learned my addr 172.21.15.72:0/1919598655 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:17.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.765+0000 7ff48ad9d640 1 --2- 172.21.15.72:0/1919598655 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff48c156d30 0x7ff48c15d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.765+0000 7ff48a59c640 1 --2- 172.21.15.72:0/1919598655 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff48c0cd160 0x7ff48c149230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.765+0000 7ff489d9b640 1 -- 172.21.15.72:0/1919598655 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff48c156d30 msgr2=0x7ff48c15d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.765+0000 7ff489d9b640 1 --2- 172.21.15.72:0/1919598655 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff48c156d30 0x7ff48c15d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.765+0000 7ff489d9b640 1 -- 172.21.15.72:0/1919598655 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff48c0cd160 msgr2=0x7ff48c149230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.765+0000 7ff489d9b640 1 --2- 172.21.15.72:0/1919598655 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff48c0cd160 0x7ff48c149230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.765+0000 7ff489d9b640 1 -- 172.21.15.72:0/1919598655 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff48c0b7230 con 0x7ff48c0cd5a0 2024-04-15T15:15:17.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.765+0000 7ff48ad9d640 1 --2- 172.21.15.72:0/1919598655 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff48c156d30 0x7ff48c15d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:17.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.765+0000 7ff48a59c640 1 --2- 172.21.15.72:0/1919598655 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff48c0cd160 0x7ff48c149230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:17.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.765+0000 7ff489d9b640 1 --2- 172.21.15.72:0/1919598655 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff48c0cd5a0 0x7ff48c15cae0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff480009890 tx=0x7ff4800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:17.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.765+0000 7ff47affd640 1 -- 172.21.15.72:0/1919598655 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff480041070 con 0x7ff48c0cd5a0 2024-04-15T15:15:17.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.766+0000 7ff47affd640 1 -- 172.21.15.72:0/1919598655 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff48003aa30 con 0x7ff48c0cd5a0 2024-04-15T15:15:17.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.766+0000 7ff47affd640 1 -- 172.21.15.72:0/1919598655 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff480035040 con 0x7ff48c0cd5a0 2024-04-15T15:15:17.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.766+0000 7ff49091c640 1 -- 172.21.15.72:0/1919598655 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff48c0b7510 con 0x7ff48c0cd5a0 2024-04-15T15:15:17.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.766+0000 7ff49091c640 1 -- 172.21.15.72:0/1919598655 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff48c15ab10 con 0x7ff48c0cd5a0 2024-04-15T15:15:17.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.767+0000 7ff49091c640 1 -- 172.21.15.72:0/1919598655 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff48c0689e0 con 0x7ff48c0cd5a0 2024-04-15T15:15:17.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.767+0000 7ff47affd640 1 -- 172.21.15.72:0/1919598655 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff48004c060 con 0x7ff48c0cd5a0 2024-04-15T15:15:17.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.767+0000 7ff47affd640 1 --2- 172.21.15.72:0/1919598655 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff46c06c920 0x7ff46c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.768+0000 7ff47affd640 1 -- 172.21.15.72:0/1919598655 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ff4800ad8c0 con 0x7ff48c0cd5a0 2024-04-15T15:15:17.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.768+0000 7ff48a59c640 1 --2- 172.21.15.72:0/1919598655 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff46c06c920 0x7ff46c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.768+0000 7ff48a59c640 1 --2- 172.21.15.72:0/1919598655 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff46c06c920 0x7ff46c06ede0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff47c006fd0 tx=0x7ff47c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:17.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.772+0000 7ff47affd640 1 -- 172.21.15.72:0/1919598655 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff48007b9b0 con 0x7ff48c0cd5a0 2024-04-15T15:15:17.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.988+0000 7ff49091c640 1 -- 172.21.15.72:0/1919598655 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff48c0b76e0 con 0x7ff48c0cd5a0 2024-04-15T15:15:17.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.989+0000 7ff47affd640 1 -- 172.21.15.72:0/1919598655 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5183 (secure 0 0 0) 0x7ff48007b2b0 con 0x7ff48c0cd5a0 2024-04-15T15:15:17.991 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:17.991 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4267,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6833/1659343919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1659343919},{"type":"v1","addr":"172.21.15.175:6833","nonce":1659343919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6835/4113588141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6834","nonce":4113588141},{"type":"v1","addr":"172.21.15.175:6835","nonce":4113588141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4296,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/4151548261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4151548261},{"type":"v1","addr":"172.21.15.72:6836","nonce":4151548261}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:14:51.411091+0000","modified":"2024-04-15T15:15:07.485436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/3458067804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3458067804},{"type":"v1","addr":"172.21.15.72:6837","nonce":3458067804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4299,"qdb_cluster":[4299]},"id":1}]} 2024-04-15T15:15:17.991 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 9 2024-04-15T15:15:17.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.991+0000 7ff49091c640 1 -- 172.21.15.72:0/1919598655 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff46c06c920 msgr2=0x7ff46c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.991+0000 7ff49091c640 1 --2- 172.21.15.72:0/1919598655 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff46c06c920 0x7ff46c06ede0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff47c006fd0 tx=0x7ff47c008040 comp rx=0 tx=0).stop 2024-04-15T15:15:17.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.991+0000 7ff49091c640 1 -- 172.21.15.72:0/1919598655 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff48c0cd5a0 msgr2=0x7ff48c15cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.991+0000 7ff49091c640 1 --2- 172.21.15.72:0/1919598655 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff48c0cd5a0 0x7ff48c15cae0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff480009890 tx=0x7ff4800046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.991+0000 7ff49091c640 1 -- 172.21.15.72:0/1919598655 shutdown_connections 2024-04-15T15:15:17.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.991+0000 7ff49091c640 1 --2- 172.21.15.72:0/1919598655 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff46c06c920 0x7ff46c06ede0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.991+0000 7ff49091c640 1 --2- 172.21.15.72:0/1919598655 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff48c156d30 0x7ff48c15d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.991+0000 7ff49091c640 1 --2- 172.21.15.72:0/1919598655 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff48c0cd5a0 0x7ff48c15cae0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.991+0000 7ff49091c640 1 --2- 172.21.15.72:0/1919598655 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff48c0cd160 0x7ff48c149230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.991+0000 7ff49091c640 1 -- 172.21.15.72:0/1919598655 >> 172.21.15.72:0/1919598655 conn(0x7ff48c0b43c0 msgr2=0x7ff48c15dfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:17.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.991+0000 7ff49091c640 1 -- 172.21.15.72:0/1919598655 shutdown_connections 2024-04-15T15:15:17.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:17.991+0000 7ff49091c640 1 -- 172.21.15.72:0/1919598655 wait complete. 2024-04-15T15:15:18.004 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:18.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.106+0000 7f8657ac5640 1 Processor -- start 2024-04-15T15:15:18.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.106+0000 7f8657ac5640 1 -- start start 2024-04-15T15:15:18.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.107+0000 7f8657ac5640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f865006eba0 0x7f86500d4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:18.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.107+0000 7f8657ac5640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86500d1d40 0x7f86500d5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:18.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.107+0000 7f8657ac5640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86500b6160 con 0x7f865015e9e0 2024-04-15T15:15:18.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.107+0000 7f8657ac5640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f86500b7030 con 0x7f865006eba0 2024-04-15T15:15:18.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.107+0000 7f8657ac5640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f86500d1590 con 0x7f86500d1d40 2024-04-15T15:15:18.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.107+0000 7f8655060640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f865006eba0 0x7f86500d4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:18.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.107+0000 7f8655861640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f865015e9e0 0x7f86500b6920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38004/0 (socket says 172.21.15.72:38004) 2024-04-15T15:15:18.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.107+0000 7f8655861640 1 -- 172.21.15.72:0/400727991 learned_addr learned my addr 172.21.15.72:0/400727991 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:18.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.107+0000 7f8656062640 1 --2- 172.21.15.72:0/400727991 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86500d1d40 0x7f86500d5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:18.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.107+0000 7f8655060640 1 -- 172.21.15.72:0/400727991 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86500d1d40 msgr2=0x7f86500d5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:18.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.107+0000 7f8655060640 1 --2- 172.21.15.72:0/400727991 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86500d1d40 0x7f86500d5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:18.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.107+0000 7f8655060640 1 -- 172.21.15.72:0/400727991 >> v1:172.21.15.72:6789/0 conn(0x7f865015e9e0 legacy=0x7f86500b6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:18.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.107+0000 7f8655060640 1 -- 172.21.15.72:0/400727991 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86500d17e0 con 0x7f865006eba0 2024-04-15T15:15:18.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.107+0000 7f865485f640 1 -- 172.21.15.72:0/400727991 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 785833339 0 0) 0x7f86500b6160 con 0x7f865015e9e0 2024-04-15T15:15:18.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.108+0000 7f8655060640 1 --2- 172.21.15.72:0/400727991 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f865006eba0 0x7f86500d4b50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f86400097a0 tx=0x7f864002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65e8fcf72ecd352d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:18.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.108+0000 7f865485f640 1 -- 172.21.15.72:0/400727991 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86400315a0 con 0x7f865006eba0 2024-04-15T15:15:18.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.108+0000 7f865485f640 1 -- 172.21.15.72:0/400727991 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8640037070 con 0x7f865006eba0 2024-04-15T15:15:18.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.108+0000 7f865485f640 1 -- 172.21.15.72:0/400727991 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8640031960 con 0x7f865006eba0 2024-04-15T15:15:18.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.109+0000 7f8657ac5640 1 -- 172.21.15.72:0/400727991 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f865006eba0 msgr2=0x7f86500d4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:18.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.109+0000 7f8657ac5640 1 --2- 172.21.15.72:0/400727991 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f865006eba0 0x7f86500d4b50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f86400097a0 tx=0x7f864002b840 comp rx=0 tx=0).stop 2024-04-15T15:15:18.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.109+0000 7f8657ac5640 1 -- 172.21.15.72:0/400727991 shutdown_connections 2024-04-15T15:15:18.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.109+0000 7f8657ac5640 1 --2- 172.21.15.72:0/400727991 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86500d1d40 0x7f86500d5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:18.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.109+0000 7f8657ac5640 1 --2- 172.21.15.72:0/400727991 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f865006eba0 0x7f86500d4b50 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:18.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.109+0000 7f8657ac5640 1 -- 172.21.15.72:0/400727991 >> 172.21.15.72:0/400727991 conn(0x7f865009db70 msgr2=0x7f86500b9600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:18.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.109+0000 7f8657ac5640 1 -- 172.21.15.72:0/400727991 shutdown_connections 2024-04-15T15:15:18.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.109+0000 7f8657ac5640 1 -- 172.21.15.72:0/400727991 wait complete. 2024-04-15T15:15:18.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.110+0000 7f8657ac5640 1 Processor -- start 2024-04-15T15:15:18.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.110+0000 7f8657ac5640 1 -- start start 2024-04-15T15:15:18.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.110+0000 7f8657ac5640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f86500d1d40 0x7f865009f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:18.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.111+0000 7f8657ac5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f865015e9e0 0x7f86500b1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:18.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.111+0000 7f8657ac5640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86500b18c0 0x7f865009e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:18.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.111+0000 7f8657ac5640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f865015edc0 con 0x7f865015e9e0 2024-04-15T15:15:18.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.111+0000 7f8657ac5640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f86500d1ac0 con 0x7f86500d1d40 2024-04-15T15:15:18.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.111+0000 7f8657ac5640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f86500b3c20 con 0x7f86500b18c0 2024-04-15T15:15:18.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.111+0000 7f8655060640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f865015e9e0 0x7f86500b1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:18.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.111+0000 7f8655060640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f865015e9e0 0x7f86500b1380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45666/0 (socket says 172.21.15.72:45666) 2024-04-15T15:15:18.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.111+0000 7f8655060640 1 -- 172.21.15.72:0/2568541102 learned_addr learned my addr 172.21.15.72:0/2568541102 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:18.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.111+0000 7f8655861640 1 --2- 172.21.15.72:0/2568541102 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f86500d1d40 0x7f865009f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:18.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.111+0000 7f8656062640 1 --2- 172.21.15.72:0/2568541102 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86500b18c0 0x7f865009e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:18.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.111+0000 7f8655060640 1 -- 172.21.15.72:0/2568541102 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86500b18c0 msgr2=0x7f865009e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:18.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.111+0000 7f8655060640 1 --2- 172.21.15.72:0/2568541102 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86500b18c0 0x7f865009e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:18.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.111+0000 7f8655060640 1 -- 172.21.15.72:0/2568541102 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f86500d1d40 msgr2=0x7f865009f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:18.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.111+0000 7f8655060640 1 --2- 172.21.15.72:0/2568541102 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f86500d1d40 0x7f865009f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:18.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.111+0000 7f8655060640 1 -- 172.21.15.72:0/2568541102 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f865009feb0 con 0x7f865015e9e0 2024-04-15T15:15:18.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.111+0000 7f8655861640 1 --2- 172.21.15.72:0/2568541102 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f86500d1d40 0x7f865009f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:18.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.111+0000 7f8655060640 1 --2- 172.21.15.72:0/2568541102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f865015e9e0 0x7f86500b1380 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8640000c00 tx=0x7f86400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:18.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.111+0000 7f86467fc640 1 -- 172.21.15.72:0/2568541102 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8640032630 con 0x7f865015e9e0 2024-04-15T15:15:18.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.111+0000 7f86467fc640 1 -- 172.21.15.72:0/2568541102 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86400327d0 con 0x7f865015e9e0 2024-04-15T15:15:18.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.111+0000 7f86467fc640 1 -- 172.21.15.72:0/2568541102 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8640032bc0 con 0x7f865015e9e0 2024-04-15T15:15:18.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.111+0000 7f8657ac5640 1 -- 172.21.15.72:0/2568541102 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8650152370 con 0x7f865015e9e0 2024-04-15T15:15:18.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.111+0000 7f8657ac5640 1 -- 172.21.15.72:0/2568541102 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8650152930 con 0x7f865015e9e0 2024-04-15T15:15:18.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.112+0000 7f8657ac5640 1 -- 172.21.15.72:0/2568541102 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8650068b90 con 0x7f865015e9e0 2024-04-15T15:15:18.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.113+0000 7f86467fc640 1 -- 172.21.15.72:0/2568541102 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f864004d3f0 con 0x7f865015e9e0 2024-04-15T15:15:18.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.113+0000 7f86467fc640 1 --2- 172.21.15.72:0/2568541102 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f863406c920 0x7f863406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:18.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.113+0000 7f86467fc640 1 -- 172.21.15.72:0/2568541102 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f864002f030 con 0x7f865015e9e0 2024-04-15T15:15:18.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.113+0000 7f8655861640 1 --2- 172.21.15.72:0/2568541102 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f863406c920 0x7f863406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:18.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.114+0000 7f8655861640 1 --2- 172.21.15.72:0/2568541102 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f863406c920 0x7f863406ede0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f864c000f60 tx=0x7f864c007ca0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:18.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.117+0000 7f86467fc640 1 -- 172.21.15.72:0/2568541102 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8650068b90 con 0x7f865015e9e0 2024-04-15T15:15:18.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.294+0000 7f8657ac5640 1 -- 172.21.15.72:0/2568541102 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f86500d12d0 con 0x7f865015e9e0 2024-04-15T15:15:18.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.295+0000 7f86467fc640 1 -- 172.21.15.72:0/2568541102 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7f86500d12d0 con 0x7f865015e9e0 2024-04-15T15:15:18.297 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:18.297 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":24,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:14:56.451113+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:14:49.438366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:14:50.368379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:14:54.081157+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:18.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.297+0000 7f8657ac5640 1 -- 172.21.15.72:0/2568541102 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f863406c920 msgr2=0x7f863406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:18.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.297+0000 7f8657ac5640 1 --2- 172.21.15.72:0/2568541102 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f863406c920 0x7f863406ede0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f864c000f60 tx=0x7f864c007ca0 comp rx=0 tx=0).stop 2024-04-15T15:15:18.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.297+0000 7f8657ac5640 1 -- 172.21.15.72:0/2568541102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f865015e9e0 msgr2=0x7f86500b1380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:18.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.297+0000 7f8657ac5640 1 --2- 172.21.15.72:0/2568541102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f865015e9e0 0x7f86500b1380 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8640000c00 tx=0x7f86400046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:18.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.297+0000 7f8657ac5640 1 -- 172.21.15.72:0/2568541102 shutdown_connections 2024-04-15T15:15:18.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.297+0000 7f8657ac5640 1 --2- 172.21.15.72:0/2568541102 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f863406c920 0x7f863406ede0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:18.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.297+0000 7f8657ac5640 1 --2- 172.21.15.72:0/2568541102 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86500b18c0 0x7f865009e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:18.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.297+0000 7f8657ac5640 1 --2- 172.21.15.72:0/2568541102 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f865015e9e0 0x7f86500b1380 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:18.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.297+0000 7f8657ac5640 1 --2- 172.21.15.72:0/2568541102 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f86500d1d40 0x7f865009f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:18.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.297+0000 7f8657ac5640 1 -- 172.21.15.72:0/2568541102 >> 172.21.15.72:0/2568541102 conn(0x7f865009db70 msgr2=0x7f86500d4b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:18.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.297+0000 7f8657ac5640 1 -- 172.21.15.72:0/2568541102 shutdown_connections 2024-04-15T15:15:18.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.298+0000 7f8657ac5640 1 -- 172.21.15.72:0/2568541102 wait complete. 2024-04-15T15:15:18.311 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:15:18.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.411+0000 7f02eec14640 1 Processor -- start 2024-04-15T15:15:18.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.412+0000 7f02eec14640 1 -- start start 2024-04-15T15:15:18.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.412+0000 7f02eec14640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f02e815c480 0x7f02e806ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:18.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.412+0000 7f02eec14640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f02e80ce670 0x7f02e80cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:18.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.412+0000 7f02eec14640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02e80d1d80 con 0x7f02e80cf300 2024-04-15T15:15:18.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.412+0000 7f02eec14640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f02e80d1b30 con 0x7f02e80ce670 2024-04-15T15:15:18.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.412+0000 7f02eec14640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f02e80cca00 con 0x7f02e815c480 2024-04-15T15:15:18.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.413+0000 7f02e7fff640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f02e80ce670 0x7f02e80cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:18.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.413+0000 7f02ed1b1640 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f02e80cf300 0x7f02e80cff80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.72:34150/0 (socket says 172.21.15.72:34150) 2024-04-15T15:15:18.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.413+0000 7f02ed1b1640 1 -- 172.21.15.72:0/2812672170 learned_addr learned my addr 172.21.15.72:0/2812672170 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:18.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.413+0000 7f02ec9b0640 1 --2- 172.21.15.72:0/2812672170 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f02e815c480 0x7f02e806ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:18.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.413+0000 7f02ec9b0640 1 -- 172.21.15.72:0/2812672170 >> v2:172.21.15.175:3300/0 conn(0x7f02e80ce670 msgr2=0x7f02e80cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:18.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.413+0000 7f02ec9b0640 1 --2- 172.21.15.72:0/2812672170 >> v2:172.21.15.175:3300/0 conn(0x7f02e80ce670 0x7f02e80cedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:18.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.413+0000 7f02ec9b0640 1 -- 172.21.15.72:0/2812672170 >> v1:172.21.15.175:6789/0 conn(0x7f02e80cf300 legacy=0x7f02e80cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:18.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.413+0000 7f02ec9b0640 1 -- 172.21.15.72:0/2812672170 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02e80d2060 con 0x7f02e815c480 2024-04-15T15:15:18.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.413+0000 7f02e7fff640 1 --2- 172.21.15.72:0/2812672170 >> v2:172.21.15.175:3300/0 conn(0x7f02e80ce670 0x7f02e80cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:18.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.414+0000 7f02ec9b0640 1 --2- 172.21.15.72:0/2812672170 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f02e815c480 0x7f02e806ff20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f02d8009d90 tx=0x7f02d802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1df3f05a7cacbbb server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:18.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.414+0000 7f02e77fe640 1 -- 172.21.15.72:0/2812672170 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02d8039070 con 0x7f02e815c480 2024-04-15T15:15:18.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.414+0000 7f02e77fe640 1 -- 172.21.15.72:0/2812672170 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02d802cc50 con 0x7f02e815c480 2024-04-15T15:15:18.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.414+0000 7f02e77fe640 1 -- 172.21.15.72:0/2812672170 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02d8034440 con 0x7f02e815c480 2024-04-15T15:15:18.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.415+0000 7f02eec14640 1 -- 172.21.15.72:0/2812672170 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f02e815c480 msgr2=0x7f02e806ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:18.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.415+0000 7f02eec14640 1 --2- 172.21.15.72:0/2812672170 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f02e815c480 0x7f02e806ff20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f02d8009d90 tx=0x7f02d802bea0 comp rx=0 tx=0).stop 2024-04-15T15:15:18.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.415+0000 7f02eec14640 1 -- 172.21.15.72:0/2812672170 shutdown_connections 2024-04-15T15:15:18.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.415+0000 7f02eec14640 1 --2- 172.21.15.72:0/2812672170 >> v2:172.21.15.175:3300/0 conn(0x7f02e80ce670 0x7f02e80cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:18.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.415+0000 7f02eec14640 1 --2- 172.21.15.72:0/2812672170 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f02e815c480 0x7f02e806ff20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:18.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.415+0000 7f02eec14640 1 -- 172.21.15.72:0/2812672170 >> 172.21.15.72:0/2812672170 conn(0x7f02e80b3fa0 msgr2=0x7f02e80b43b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:18.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.415+0000 7f02eec14640 1 -- 172.21.15.72:0/2812672170 shutdown_connections 2024-04-15T15:15:18.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.415+0000 7f02eec14640 1 -- 172.21.15.72:0/2812672170 wait complete. 2024-04-15T15:15:18.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.416+0000 7f02eec14640 1 Processor -- start 2024-04-15T15:15:18.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.416+0000 7f02eec14640 1 -- start start 2024-04-15T15:15:18.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.417+0000 7f02eec14640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f02e80ce670 0x7f02e80c8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:18.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.417+0000 7f02eec14640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f02e80cf300 0x7f02e81e2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:18.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.417+0000 7f02eec14640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02e815c480 0x7f02e8148ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:18.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.417+0000 7f02eec14640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f02e80cd170 con 0x7f02e815c480 2024-04-15T15:15:18.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.417+0000 7f02eec14640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f02e80cead0 con 0x7f02e80cf300 2024-04-15T15:15:18.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.417+0000 7f02eec14640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f02e815c860 con 0x7f02e80ce670 2024-04-15T15:15:18.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.417+0000 7f02ed1b1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02e815c480 0x7f02e8148ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:18.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.417+0000 7f02ed1b1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02e815c480 0x7f02e8148ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45672/0 (socket says 172.21.15.72:45672) 2024-04-15T15:15:18.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.417+0000 7f02ed1b1640 1 -- 172.21.15.72:0/326778542 learned_addr learned my addr 172.21.15.72:0/326778542 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:18.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.417+0000 7f02e7fff640 1 --2- 172.21.15.72:0/326778542 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f02e80cf300 0x7f02e81e2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:18.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.417+0000 7f02ec9b0640 1 --2- 172.21.15.72:0/326778542 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f02e80ce670 0x7f02e80c8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:18.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.417+0000 7f02ed1b1640 1 -- 172.21.15.72:0/326778542 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f02e80ce670 msgr2=0x7f02e80c8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:18.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.417+0000 7f02ed1b1640 1 --2- 172.21.15.72:0/326778542 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f02e80ce670 0x7f02e80c8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:18.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.417+0000 7f02ed1b1640 1 -- 172.21.15.72:0/326778542 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f02e80cf300 msgr2=0x7f02e81e2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:18.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.417+0000 7f02ed1b1640 1 --2- 172.21.15.72:0/326778542 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f02e80cf300 0x7f02e81e2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:18.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.417+0000 7f02ed1b1640 1 -- 172.21.15.72:0/326778542 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02e80ca740 con 0x7f02e815c480 2024-04-15T15:15:18.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.417+0000 7f02ed1b1640 1 --2- 172.21.15.72:0/326778542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02e815c480 0x7f02e8148ac0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f02e000c370 tx=0x7f02e000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:18.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.418+0000 7f02e57fa640 1 -- 172.21.15.72:0/326778542 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e0017070 con 0x7f02e815c480 2024-04-15T15:15:18.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.418+0000 7f02e57fa640 1 -- 172.21.15.72:0/326778542 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02e0005ce0 con 0x7f02e815c480 2024-04-15T15:15:18.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.418+0000 7f02eec14640 1 -- 172.21.15.72:0/326778542 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02e80ca9e0 con 0x7f02e815c480 2024-04-15T15:15:18.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.418+0000 7f02e57fa640 1 -- 172.21.15.72:0/326778542 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e00072d0 con 0x7f02e815c480 2024-04-15T15:15:18.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.418+0000 7f02eec14640 1 -- 172.21.15.72:0/326778542 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02e80cafa0 con 0x7f02e815c480 2024-04-15T15:15:18.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.419+0000 7f02eec14640 1 -- 172.21.15.72:0/326778542 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02e80690f0 con 0x7f02e815c480 2024-04-15T15:15:18.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.419+0000 7f02e57fa640 1 -- 172.21.15.72:0/326778542 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f02e001a070 con 0x7f02e815c480 2024-04-15T15:15:18.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.420+0000 7f02e57fa640 1 --2- 172.21.15.72:0/326778542 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f02c806c8f0 0x7f02c806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:18.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.420+0000 7f02ec9b0640 1 --2- 172.21.15.72:0/326778542 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f02c806c8f0 0x7f02c806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:18.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.420+0000 7f02e57fa640 1 -- 172.21.15.72:0/326778542 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f02e008b110 con 0x7f02e815c480 2024-04-15T15:15:18.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.420+0000 7f02ec9b0640 1 --2- 172.21.15.72:0/326778542 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f02c806c8f0 0x7f02c806edb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f02d8036000 tx=0x7f02d8030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:18.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.424+0000 7f02e57fa640 1 -- 172.21.15.72:0/326778542 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f02e0058b30 con 0x7f02e815c480 2024-04-15T15:15:18.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.634+0000 7f02eec14640 1 -- 172.21.15.72:0/326778542 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f02e81e2e90 con 0x7f02e815c480 2024-04-15T15:15:18.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.634+0000 7f02e57fa640 1 -- 172.21.15.72:0/326778542 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5183 (secure 0 0 0) 0x7f02e005ce30 con 0x7f02e815c480 2024-04-15T15:15:18.636 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:18.637 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4267,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6833/1659343919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1659343919},{"type":"v1","addr":"172.21.15.175:6833","nonce":1659343919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6835/4113588141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6834","nonce":4113588141},{"type":"v1","addr":"172.21.15.175:6835","nonce":4113588141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4296,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/4151548261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4151548261},{"type":"v1","addr":"172.21.15.72:6836","nonce":4151548261}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:14:51.411091+0000","modified":"2024-04-15T15:15:07.485436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/3458067804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3458067804},{"type":"v1","addr":"172.21.15.72:6837","nonce":3458067804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4299,"qdb_cluster":[4299]},"id":1}]} 2024-04-15T15:15:18.637 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 9 2024-04-15T15:15:18.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.636+0000 7f02eec14640 1 -- 172.21.15.72:0/326778542 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f02c806c8f0 msgr2=0x7f02c806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:18.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.636+0000 7f02eec14640 1 --2- 172.21.15.72:0/326778542 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f02c806c8f0 0x7f02c806edb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f02d8036000 tx=0x7f02d8030000 comp rx=0 tx=0).stop 2024-04-15T15:15:18.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.636+0000 7f02eec14640 1 -- 172.21.15.72:0/326778542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02e815c480 msgr2=0x7f02e8148ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:18.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.636+0000 7f02eec14640 1 --2- 172.21.15.72:0/326778542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02e815c480 0x7f02e8148ac0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f02e000c370 tx=0x7f02e000c830 comp rx=0 tx=0).stop 2024-04-15T15:15:18.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.636+0000 7f02eec14640 1 -- 172.21.15.72:0/326778542 shutdown_connections 2024-04-15T15:15:18.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.636+0000 7f02eec14640 1 --2- 172.21.15.72:0/326778542 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f02c806c8f0 0x7f02c806edb0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:18.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.636+0000 7f02eec14640 1 --2- 172.21.15.72:0/326778542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f02e815c480 0x7f02e8148ac0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:18.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.636+0000 7f02eec14640 1 --2- 172.21.15.72:0/326778542 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f02e80cf300 0x7f02e81e2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:18.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.636+0000 7f02eec14640 1 --2- 172.21.15.72:0/326778542 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f02e80ce670 0x7f02e80c8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:18.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.636+0000 7f02eec14640 1 -- 172.21.15.72:0/326778542 >> 172.21.15.72:0/326778542 conn(0x7f02e80b3fa0 msgr2=0x7f02e80b5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:18.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.636+0000 7f02eec14640 1 -- 172.21.15.72:0/326778542 shutdown_connections 2024-04-15T15:15:18.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:18.636+0000 7f02eec14640 1 -- 172.21.15.72:0/326778542 wait complete. 2024-04-15T15:15:19.052 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-15T15:14:51.411091+0000', 'modified': '2024-04-15T15:15:07.485436+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4299}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4299': {'gid': 4299, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6837/3458067804', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6835', 'nonce': 3458067804}, {'type': 'v1', 'addr': '172.21.15.72:6837', 'nonce': 3458067804}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4299, 'qdb_cluster': [4299]} 2024-04-15T15:15:19.052 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-15T15:15:19.052 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-15T15:15:19.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.156+0000 7fb7b66e4640 1 Processor -- start 2024-04-15T15:15:19.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.156+0000 7fb7b66e4640 1 -- start start 2024-04-15T15:15:19.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.157+0000 7fb7b66e4640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb7b015c250 0x7fb7b006d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:19.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.157+0000 7fb7b66e4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb7b015f9f0 0x7fb7b00cca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:19.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.157+0000 7fb7b66e4640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb7b00d0010 0x7fb7b00d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:19.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.157+0000 7fb7b66e4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fb7b00d09b0 con 0x7fb7b015f9f0 2024-04-15T15:15:19.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.157+0000 7fb7b66e4640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fb7b006df30 con 0x7fb7b00d0010 2024-04-15T15:15:19.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.157+0000 7fb7b66e4640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fb7b006e210 con 0x7fb7b015c250 2024-04-15T15:15:19.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.157+0000 7fb7a7fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb7b015f9f0 0x7fb7b00cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:19.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.157+0000 7fb7a7fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb7b015f9f0 0x7fb7b00cca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45680/0 (socket says 172.21.15.72:45680) 2024-04-15T15:15:19.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.157+0000 7fb7a7fff640 1 -- 172.21.15.72:0/2218083775 learned_addr learned my addr 172.21.15.72:0/2218083775 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:19.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.157+0000 7fb7b4c81640 1 --2- 172.21.15.72:0/2218083775 >> v2:172.21.15.175:3300/0 conn(0x7fb7b00d0010 0x7fb7b00d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:19.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.157+0000 7fb7affff640 1 --2- 172.21.15.72:0/2218083775 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb7b015c250 0x7fb7b006d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:19.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.157+0000 7fb7a7fff640 1 -- 172.21.15.72:0/2218083775 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb7b015c250 msgr2=0x7fb7b006d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:19.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.157+0000 7fb7a7fff640 1 --2- 172.21.15.72:0/2218083775 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb7b015c250 0x7fb7b006d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.157+0000 7fb7a7fff640 1 -- 172.21.15.72:0/2218083775 >> v2:172.21.15.175:3300/0 conn(0x7fb7b00d0010 msgr2=0x7fb7b00d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:19.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.157+0000 7fb7a7fff640 1 --2- 172.21.15.72:0/2218083775 >> v2:172.21.15.175:3300/0 conn(0x7fb7b00d0010 0x7fb7b00d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.157+0000 7fb7a7fff640 1 -- 172.21.15.72:0/2218083775 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7b006e4f0 con 0x7fb7b015f9f0 2024-04-15T15:15:19.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.157+0000 7fb7b4c81640 1 --2- 172.21.15.72:0/2218083775 >> v2:172.21.15.175:3300/0 conn(0x7fb7b00d0010 0x7fb7b00d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:19.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.157+0000 7fb7affff640 1 --2- 172.21.15.72:0/2218083775 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb7b015c250 0x7fb7b006d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:19.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.158+0000 7fb7a7fff640 1 --2- 172.21.15.72:0/2218083775 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7b015f9f0 0x7fb7b00cca90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb7b006e140 tx=0x7fb79802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9279fb6942ec0aca server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:19.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.158+0000 7fb7af7fe640 1 -- 172.21.15.72:0/2218083775 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7980317c0 con 0x7fb7b015f9f0 2024-04-15T15:15:19.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.158+0000 7fb7af7fe640 1 -- 172.21.15.72:0/2218083775 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb798037070 con 0x7fb7b015f9f0 2024-04-15T15:15:19.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.158+0000 7fb7af7fe640 1 -- 172.21.15.72:0/2218083775 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb798031b80 con 0x7fb7b015f9f0 2024-04-15T15:15:19.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.159+0000 7fb7b66e4640 1 -- 172.21.15.72:0/2218083775 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7b015f9f0 msgr2=0x7fb7b00cca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:19.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.159+0000 7fb7b66e4640 1 --2- 172.21.15.72:0/2218083775 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7b015f9f0 0x7fb7b00cca90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb7b006e140 tx=0x7fb79802b860 comp rx=0 tx=0).stop 2024-04-15T15:15:19.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.159+0000 7fb7b66e4640 1 -- 172.21.15.72:0/2218083775 shutdown_connections 2024-04-15T15:15:19.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.159+0000 7fb7b66e4640 1 --2- 172.21.15.72:0/2218083775 >> v2:172.21.15.175:3300/0 conn(0x7fb7b00d0010 0x7fb7b00d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.159+0000 7fb7b66e4640 1 --2- 172.21.15.72:0/2218083775 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7b015f9f0 0x7fb7b00cca90 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.159+0000 7fb7b66e4640 1 --2- 172.21.15.72:0/2218083775 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb7b015c250 0x7fb7b006d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.159+0000 7fb7b66e4640 1 -- 172.21.15.72:0/2218083775 >> 172.21.15.72:0/2218083775 conn(0x7fb7b00ceec0 msgr2=0x7fb7b00cf2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:19.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.159+0000 7fb7b66e4640 1 -- 172.21.15.72:0/2218083775 shutdown_connections 2024-04-15T15:15:19.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.159+0000 7fb7b66e4640 1 -- 172.21.15.72:0/2218083775 wait complete. 2024-04-15T15:15:19.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.160+0000 7fb7b66e4640 1 Processor -- start 2024-04-15T15:15:19.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.160+0000 7fb7b66e4640 1 -- start start 2024-04-15T15:15:19.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.161+0000 7fb7b66e4640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7b00d0010 0x7fb7b015e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:19.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.161+0000 7fb7b66e4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7b015c250 0x7fb7b00c6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:19.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.161+0000 7fb7b66e4640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb7b015f9f0 0x7fb7b0154ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:19.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.161+0000 7fb7b66e4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b006dc20 con 0x7fb7b015c250 2024-04-15T15:15:19.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.161+0000 7fb7b66e4640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b00b38f0 con 0x7fb7b00d0010 2024-04-15T15:15:19.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.161+0000 7fb7b66e4640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fb7b015c630 con 0x7fb7b015f9f0 2024-04-15T15:15:19.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.161+0000 7fb7a7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7b015c250 0x7fb7b00c6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:19.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.161+0000 7fb7a7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7b015c250 0x7fb7b00c6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45692/0 (socket says 172.21.15.72:45692) 2024-04-15T15:15:19.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.161+0000 7fb7a7fff640 1 -- 172.21.15.72:0/279969072 learned_addr learned my addr 172.21.15.72:0/279969072 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:19.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.161+0000 7fb7affff640 1 --2- 172.21.15.72:0/279969072 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7b00d0010 0x7fb7b015e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:19.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.161+0000 7fb7b4c81640 1 --2- 172.21.15.72:0/279969072 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb7b015f9f0 0x7fb7b0154ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:19.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.161+0000 7fb7a7fff640 1 -- 172.21.15.72:0/279969072 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb7b015f9f0 msgr2=0x7fb7b0154ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:19.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.161+0000 7fb7a7fff640 1 --2- 172.21.15.72:0/279969072 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb7b015f9f0 0x7fb7b0154ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.161+0000 7fb7a7fff640 1 -- 172.21.15.72:0/279969072 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7b00d0010 msgr2=0x7fb7b015e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:19.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.161+0000 7fb7a7fff640 1 --2- 172.21.15.72:0/279969072 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7b00d0010 0x7fb7b015e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.161+0000 7fb7a7fff640 1 -- 172.21.15.72:0/279969072 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7b00c7d00 con 0x7fb7b015c250 2024-04-15T15:15:19.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.161+0000 7fb7a7fff640 1 --2- 172.21.15.72:0/279969072 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7b015c250 0x7fb7b00c6980 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb79802b8a0 tx=0x7fb7980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:19.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.161+0000 7fb7ad7fa640 1 -- 172.21.15.72:0/279969072 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb798032860 con 0x7fb7b015c250 2024-04-15T15:15:19.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.161+0000 7fb7ad7fa640 1 -- 172.21.15.72:0/279969072 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb798032a00 con 0x7fb7b015c250 2024-04-15T15:15:19.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.161+0000 7fb7b66e4640 1 -- 172.21.15.72:0/279969072 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7b00c70f0 con 0x7fb7b015c250 2024-04-15T15:15:19.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.161+0000 7fb7ad7fa640 1 -- 172.21.15.72:0/279969072 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb798032df0 con 0x7fb7b015c250 2024-04-15T15:15:19.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.161+0000 7fb7b66e4640 1 -- 172.21.15.72:0/279969072 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7b00c76b0 con 0x7fb7b015c250 2024-04-15T15:15:19.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.163+0000 7fb7ad7fa640 1 -- 172.21.15.72:0/279969072 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb7980453f0 con 0x7fb7b015c250 2024-04-15T15:15:19.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.164+0000 7fb7ad7fa640 1 --2- 172.21.15.72:0/279969072 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb79006c8f0 0x7fb79006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:19.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.164+0000 7fb7affff640 1 --2- 172.21.15.72:0/279969072 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb79006c8f0 0x7fb79006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:19.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.164+0000 7fb7ad7fa640 1 -- 172.21.15.72:0/279969072 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb7980adff0 con 0x7fb7b015c250 2024-04-15T15:15:19.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.164+0000 7fb7affff640 1 --2- 172.21.15.72:0/279969072 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb79006c8f0 0x7fb79006edb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb7a000a990 tx=0x7fb7a0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:19.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.165+0000 7fb7b93ac740 1 Processor -- start 2024-04-15T15:15:19.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.165+0000 7fb7b93ac740 1 -- start start 2024-04-15T15:15:19.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.165+0000 7fb7b93ac740 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5648db67ae10 0x5648db64ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:19.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.165+0000 7fb7b93ac740 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x5648db67b1f0 0x5648db67dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:19.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.165+0000 7fb7b93ac740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5648db64e2f0 0x5648db67e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:19.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.165+0000 7fb7b93ac740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x5648db5b89c0 con 0x5648db64e2f0 2024-04-15T15:15:19.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.165+0000 7fb7b93ac740 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x5648db55f650 con 0x5648db67ae10 2024-04-15T15:15:19.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.165+0000 7fb7b93ac740 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x5648db5b4980 con 0x5648db67b1f0 2024-04-15T15:15:19.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.165+0000 7fb7b4c81640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5648db64e2f0 0x5648db67e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:19.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.165+0000 7fb7b4c81640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5648db64e2f0 0x5648db67e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45694/0 (socket says 172.21.15.72:45694) 2024-04-15T15:15:19.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.165+0000 7fb7b4c81640 1 -- 172.21.15.72:0/3027985623 learned_addr learned my addr 172.21.15.72:0/3027985623 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:19.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.166+0000 7fb7affff640 1 --2- 172.21.15.72:0/3027985623 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5648db67ae10 0x5648db64ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:19.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.166+0000 7fb7a7fff640 1 --2- 172.21.15.72:0/3027985623 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x5648db67b1f0 0x5648db67dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:19.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.166+0000 7fb7b4c81640 1 -- 172.21.15.72:0/3027985623 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x5648db67b1f0 msgr2=0x5648db67dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:19.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.166+0000 7fb7b4c81640 1 --2- 172.21.15.72:0/3027985623 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x5648db67b1f0 0x5648db67dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.166+0000 7fb7b4c81640 1 -- 172.21.15.72:0/3027985623 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5648db67ae10 msgr2=0x5648db64ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:19.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.166+0000 7fb7b4c81640 1 --2- 172.21.15.72:0/3027985623 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5648db67ae10 0x5648db64ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.166+0000 7fb7b4c81640 1 -- 172.21.15.72:0/3027985623 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648db2cb2d0 con 0x5648db64e2f0 2024-04-15T15:15:19.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.166+0000 7fb7affff640 1 --2- 172.21.15.72:0/3027985623 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5648db67ae10 0x5648db64ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:19.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.166+0000 7fb7b4c81640 1 --2- 172.21.15.72:0/3027985623 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5648db64e2f0 0x5648db67e0e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb7a800e520 tx=0x7fb7a8007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac38bbfb509ac70c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:19.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.166+0000 7fb7a5ffb640 1 -- 172.21.15.72:0/3027985623 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a800c970 con 0x5648db64e2f0 2024-04-15T15:15:19.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.166+0000 7fb7a5ffb640 1 -- 172.21.15.72:0/3027985623 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7a800cb10 con 0x5648db64e2f0 2024-04-15T15:15:19.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.166+0000 7fb7a5ffb640 1 -- 172.21.15.72:0/3027985623 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a800a410 con 0x5648db64e2f0 2024-04-15T15:15:19.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.167+0000 7fb7b93ac740 1 -- 172.21.15.72:0/3027985623 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5648db64e2f0 msgr2=0x5648db67e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:19.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.167+0000 7fb7b93ac740 1 --2- 172.21.15.72:0/3027985623 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5648db64e2f0 0x5648db67e0e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb7a800e520 tx=0x7fb7a8007f90 comp rx=0 tx=0).stop 2024-04-15T15:15:19.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.167+0000 7fb7b93ac740 1 -- 172.21.15.72:0/3027985623 shutdown_connections 2024-04-15T15:15:19.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.167+0000 7fb7b93ac740 1 --2- 172.21.15.72:0/3027985623 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5648db64e2f0 0x5648db67e0e0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.167+0000 7fb7b93ac740 1 --2- 172.21.15.72:0/3027985623 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x5648db67b1f0 0x5648db67dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.167+0000 7fb7b93ac740 1 --2- 172.21.15.72:0/3027985623 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5648db67ae10 0x5648db64ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.167+0000 7fb7b93ac740 1 -- 172.21.15.72:0/3027985623 >> 172.21.15.72:0/3027985623 conn(0x5648db650020 msgr2=0x5648db698ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:19.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.167+0000 7fb7b93ac740 1 -- 172.21.15.72:0/3027985623 shutdown_connections 2024-04-15T15:15:19.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.167+0000 7fb7b93ac740 1 -- 172.21.15.72:0/3027985623 wait complete. 2024-04-15T15:15:19.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.167+0000 7fb7b93ac740 1 -- start start 2024-04-15T15:15:19.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.168+0000 7fb7b93ac740 1 Processor -- start 2024-04-15T15:15:19.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.168+0000 7fb7b93ac740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5648db67ae10 0x5648db6722f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:19.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.168+0000 7fb7b93ac740 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x5648db67b1f0 0x5648db6abed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:19.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.168+0000 7fb7b93ac740 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5648db6ac410 0x5648db6ac870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:19.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.168+0000 7fb7b93ac740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x5648db6525c0 con 0x5648db67ae10 2024-04-15T15:15:19.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.168+0000 7fb7b93ac740 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x5648db572a10 con 0x5648db6ac410 2024-04-15T15:15:19.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.168+0000 7fb7b93ac740 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x5648db6519b0 con 0x5648db67b1f0 2024-04-15T15:15:19.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.168+0000 7fb7b93ac740 10 client.0 _ll_register_callbacks cb 0x5648db5ae250 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:15:19.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.168+0000 7fb7affff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5648db67ae10 0x5648db6722f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:19.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.168+0000 7fb7affff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5648db67ae10 0x5648db6722f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45704/0 (socket says 172.21.15.72:45704) 2024-04-15T15:15:19.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.168+0000 7fb7affff640 1 -- 172.21.15.72:0/481038147 learned_addr learned my addr 172.21.15.72:0/481038147 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:19.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.169+0000 7fb7b4c81640 1 --2- 172.21.15.72:0/481038147 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5648db6ac410 0x5648db6ac870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:19.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.169+0000 7fb7a7fff640 1 --2- 172.21.15.72:0/481038147 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x5648db67b1f0 0x5648db6abed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:19.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.169+0000 7fb7affff640 1 -- 172.21.15.72:0/481038147 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x5648db67b1f0 msgr2=0x5648db6abed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:19.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.169+0000 7fb7affff640 1 --2- 172.21.15.72:0/481038147 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x5648db67b1f0 0x5648db6abed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.169+0000 7fb7affff640 1 -- 172.21.15.72:0/481038147 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5648db6ac410 msgr2=0x5648db6ac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:19.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.169+0000 7fb7affff640 1 --2- 172.21.15.72:0/481038147 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5648db6ac410 0x5648db6ac870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.169+0000 7fb7affff640 1 -- 172.21.15.72:0/481038147 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648db3956b0 con 0x5648db67ae10 2024-04-15T15:15:19.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.169+0000 7fb7affff640 1 --2- 172.21.15.72:0/481038147 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5648db67ae10 0x5648db6722f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb7a00144e0 tx=0x7fb7a0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e51dc309756bfa66 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:19.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.169+0000 7fb78effd640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-04-15T15:15:19.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.169+0000 7fb78effd640 1 -- 172.21.15.72:0/481038147 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a0025020 con 0x5648db67ae10 2024-04-15T15:15:19.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.169+0000 7fb78effd640 1 -- 172.21.15.72:0/481038147 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb7a0021830 con 0x5648db67ae10 2024-04-15T15:15:19.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.169+0000 7fb7b93ac740 1 -- 172.21.15.72:0/481038147 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7a800cb10 con 0x5648db67ae10 2024-04-15T15:15:19.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.169+0000 7fb78effd640 1 -- 172.21.15.72:0/481038147 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) ==== 24+0+0 (secure 0 0 0) 0x7fb7a002f050 con 0x5648db67ae10 2024-04-15T15:15:19.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.169+0000 7fb7b93ac740 10 client.4893 fetch_fsmap learned FSMap version 9 2024-04-15T15:15:19.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.169+0000 7fb7b93ac740 1 -- 172.21.15.72:0/481038147 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) -- 0x5648db67cc10 con 0x5648db67ae10 2024-04-15T15:15:19.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.169+0000 7fb78effd640 1 -- 172.21.15.72:0/481038147 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7a800cb10 con 0x5648db67ae10 2024-04-15T15:15:19.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.170+0000 7fb78effd640 1 -- 172.21.15.72:0/481038147 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 9) ==== 3502+0+0 (secure 0 0 0) 0x7fb7a0032020 con 0x5648db67ae10 2024-04-15T15:15:19.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.170+0000 7fb7b93ac740 10 client.4893 fetch_fsmap finished waiting for FSMap version 9 2024-04-15T15:15:19.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.170+0000 7fb7b93ac740 10 client.4893 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-15T15:15:19.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.170+0000 7fb7b93ac740 20 client.4893 populate_metadata read hostname 'smithi072' 2024-04-15T15:15:19.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.170+0000 7fb7b93ac740 1 --2- 172.21.15.72:0/481038147 >> [v2:172.21.15.72:6835/3458067804,v1:172.21.15.72:6837/3458067804] conn(0x5648db67f4b0 0x5648db6b6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:19.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.170+0000 7fb7b93ac740 4 client.4893 mds_command: new command op to 4299 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-15T15:15:19.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.170+0000 7fb7b93ac740 1 -- 172.21.15.72:0/481038147 --> [v2:172.21.15.72:6835/3458067804,v1:172.21.15.72:6837/3458067804] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5648db5c63e0 con 0x5648db67f4b0 2024-04-15T15:15:19.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.170+0000 7fb7a7fff640 1 --2- 172.21.15.72:0/481038147 >> [v2:172.21.15.72:6835/3458067804,v1:172.21.15.72:6837/3458067804] conn(0x5648db67f4b0 0x5648db6b6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:19.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.170+0000 7fb78effd640 10 client.4893 ms_handle_connect on v2:172.21.15.72:6835/3458067804 2024-04-15T15:15:19.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.170+0000 7fb7a7fff640 1 --2- 172.21.15.72:0/481038147 >> [v2:172.21.15.72:6835/3458067804,v1:172.21.15.72:6837/3458067804] conn(0x5648db67f4b0 0x5648db6b6920 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0b3afa666d6709b server_cookie=33428beeded877f7 in_seq=0 out_seq=0 2024-04-15T15:15:19.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.171+0000 7fb78effd640 1 -- 172.21.15.72:0/481038147 <== mds.0 v2:172.21.15.72:6835/3458067804 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5648db5c63e0 con 0x5648db67f4b0 2024-04-15T15:15:19.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.171+0000 7fb78effd640 10 client.4893 handle_command_reply: tid=0 2024-04-15T15:15:19.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.171+0000 7fb7b93ac740 1 client.4893 shutdown 2024-04-15T15:15:19.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.171+0000 7fb7b93ac740 1 -- 172.21.15.72:0/481038147 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5648db67ae10 msgr2=0x5648db6722f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:19.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.171+0000 7fb7b93ac740 1 --2- 172.21.15.72:0/481038147 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5648db67ae10 0x5648db6722f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb7a00144e0 tx=0x7fb7a0004730 comp rx=0 tx=0).stop 2024-04-15T15:15:19.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.171+0000 7fb7b93ac740 1 -- 172.21.15.72:0/481038147 shutdown_connections 2024-04-15T15:15:19.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.171+0000 7fb7b93ac740 1 --2- 172.21.15.72:0/481038147 >> [v2:172.21.15.72:6835/3458067804,v1:172.21.15.72:6837/3458067804] conn(0x5648db67f4b0 0x5648db6b6920 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.171+0000 7fb7b93ac740 1 --2- 172.21.15.72:0/481038147 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5648db6ac410 0x5648db6ac870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.171+0000 7fb78effd640 0 client.4893 ms_handle_reset on v2:172.21.15.72:6835/3458067804 2024-04-15T15:15:19.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.171+0000 7fb7b93ac740 1 --2- 172.21.15.72:0/481038147 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x5648db67b1f0 0x5648db6abed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.171+0000 7fb7b93ac740 1 --2- 172.21.15.72:0/481038147 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5648db67ae10 0x5648db6722f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.171+0000 7fb7b93ac740 1 -- 172.21.15.72:0/481038147 >> 172.21.15.72:0/481038147 conn(0x5648db64e2f0 msgr2=0x5648db69c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:19.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.171+0000 7fb7b93ac740 1 -- 172.21.15.72:0/481038147 shutdown_connections 2024-04-15T15:15:19.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.171+0000 7fb7b93ac740 1 -- 172.21.15.72:0/481038147 wait complete. 2024-04-15T15:15:19.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.171+0000 7fb7b93ac740 20 client.4893 trim_cache size 0 max 16384 2024-04-15T15:15:19.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.189+0000 7fb7b93ac740 1 Processor -- start 2024-04-15T15:15:19.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.189+0000 7fb7b93ac740 1 -- start start 2024-04-15T15:15:19.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.190+0000 7fb7b93ac740 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x5648db67ae10 0x5648db66eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:19.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.190+0000 7fb7b93ac740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5648db67b1f0 0x5648db66f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:19.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.190+0000 7fb7b93ac740 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5648db6ac410 0x5648db675950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:19.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.190+0000 7fb7b93ac740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b0201070 con 0x5648db67b1f0 2024-04-15T15:15:19.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.190+0000 7fb7b93ac740 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fb7b02003d0 con 0x5648db6ac410 2024-04-15T15:15:19.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.190+0000 7fb7b93ac740 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fb7b0200a20 con 0x5648db67ae10 2024-04-15T15:15:19.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.190+0000 7fb7a7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5648db67b1f0 0x5648db66f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:19.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.190+0000 7fb7a7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5648db67b1f0 0x5648db66f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45716/0 (socket says 172.21.15.72:45716) 2024-04-15T15:15:19.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.190+0000 7fb7a7fff640 1 -- 172.21.15.72:0/2594842361 learned_addr learned my addr 172.21.15.72:0/2594842361 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:19.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.190+0000 7fb7b4c81640 1 --2- 172.21.15.72:0/2594842361 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5648db6ac410 0x5648db675950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:19.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.190+0000 7fb7affff640 1 --2- 172.21.15.72:0/2594842361 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x5648db67ae10 0x5648db66eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:19.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.190+0000 7fb7a7fff640 1 -- 172.21.15.72:0/2594842361 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x5648db67ae10 msgr2=0x5648db66eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:19.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.190+0000 7fb7a7fff640 1 --2- 172.21.15.72:0/2594842361 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x5648db67ae10 0x5648db66eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.190+0000 7fb7a7fff640 1 -- 172.21.15.72:0/2594842361 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5648db6ac410 msgr2=0x5648db675950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:19.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.190+0000 7fb7a7fff640 1 --2- 172.21.15.72:0/2594842361 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5648db6ac410 0x5648db675950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.190+0000 7fb7a7fff640 1 -- 172.21.15.72:0/2594842361 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648db64dd40 con 0x5648db67b1f0 2024-04-15T15:15:19.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.190+0000 7fb7b4c81640 1 --2- 172.21.15.72:0/2594842361 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5648db6ac410 0x5648db675950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:19.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.190+0000 7fb7a7fff640 1 --2- 172.21.15.72:0/2594842361 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5648db67b1f0 0x5648db66f3e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb79807ba00 tx=0x7fb798047040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d69a0b30dc4ae36 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:19.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.190+0000 7fb7a57fa640 1 -- 172.21.15.72:0/2594842361 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7980aa070 con 0x5648db67b1f0 2024-04-15T15:15:19.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.190+0000 7fb7a57fa640 1 -- 172.21.15.72:0/2594842361 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb798084650 con 0x5648db67b1f0 2024-04-15T15:15:19.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.190+0000 7fb7a57fa640 1 -- 172.21.15.72:0/2594842361 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb798094030 con 0x5648db67b1f0 2024-04-15T15:15:19.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.191+0000 7fb7b93ac740 1 -- 172.21.15.72:0/2594842361 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5648db67b1f0 msgr2=0x5648db66f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:19.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.191+0000 7fb7b93ac740 1 --2- 172.21.15.72:0/2594842361 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5648db67b1f0 0x5648db66f3e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb79807ba00 tx=0x7fb798047040 comp rx=0 tx=0).stop 2024-04-15T15:15:19.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.191+0000 7fb7b93ac740 1 -- 172.21.15.72:0/2594842361 shutdown_connections 2024-04-15T15:15:19.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.191+0000 7fb7b93ac740 1 --2- 172.21.15.72:0/2594842361 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5648db6ac410 0x5648db675950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.191+0000 7fb7b93ac740 1 --2- 172.21.15.72:0/2594842361 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5648db67b1f0 0x5648db66f3e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.191+0000 7fb7b93ac740 1 --2- 172.21.15.72:0/2594842361 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x5648db67ae10 0x5648db66eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.191+0000 7fb7b93ac740 1 -- 172.21.15.72:0/2594842361 >> 172.21.15.72:0/2594842361 conn(0x5648db64e2f0 msgr2=0x5648db6abcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:19.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.191+0000 7fb7b93ac740 1 -- 172.21.15.72:0/2594842361 shutdown_connections 2024-04-15T15:15:19.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.191+0000 7fb7b93ac740 1 -- 172.21.15.72:0/2594842361 wait complete. 2024-04-15T15:15:19.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.191+0000 7fb7b93ac740 1 -- start start 2024-04-15T15:15:19.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.192+0000 7fb7b93ac740 1 Processor -- start 2024-04-15T15:15:19.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.192+0000 7fb7b93ac740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5648db67ae10 0x5648db66eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:19.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.192+0000 7fb7b93ac740 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5648db67b1f0 0x5648db66f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:19.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.192+0000 7fb7b93ac740 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x5648db6ac410 0x5648db6aa950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:19.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.192+0000 7fb7b93ac740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x5648db67f330 con 0x5648db67ae10 2024-04-15T15:15:19.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.192+0000 7fb7b93ac740 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x5648db67cf00 con 0x5648db67b1f0 2024-04-15T15:15:19.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.192+0000 7fb7b93ac740 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x5648db67f950 con 0x5648db6ac410 2024-04-15T15:15:19.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.192+0000 7fb7b93ac740 10 client.0 _ll_register_callbacks cb 0x7fb7a000a740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:15:19.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.192+0000 7fb7affff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5648db67ae10 0x5648db66eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:19.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.192+0000 7fb7affff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5648db67ae10 0x5648db66eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45732/0 (socket says 172.21.15.72:45732) 2024-04-15T15:15:19.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.192+0000 7fb7affff640 1 -- 172.21.15.72:0/4213169124 learned_addr learned my addr 172.21.15.72:0/4213169124 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:19.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.193+0000 7fb7affff640 1 -- 172.21.15.72:0/4213169124 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x5648db6ac410 msgr2=0x5648db6aa950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:19.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.193+0000 7fb7a7fff640 1 --2- 172.21.15.72:0/4213169124 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5648db67b1f0 0x5648db66f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:19.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.193+0000 7fb7b4c81640 1 --2- 172.21.15.72:0/4213169124 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x5648db6ac410 0x5648db6aa950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:19.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.193+0000 7fb7affff640 1 --2- 172.21.15.72:0/4213169124 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x5648db6ac410 0x5648db6aa950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.193+0000 7fb7affff640 1 -- 172.21.15.72:0/4213169124 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5648db67b1f0 msgr2=0x5648db66f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:19.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.193+0000 7fb7affff640 1 --2- 172.21.15.72:0/4213169124 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5648db67b1f0 0x5648db66f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.193+0000 7fb7affff640 1 -- 172.21.15.72:0/4213169124 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648db671a40 con 0x5648db67ae10 2024-04-15T15:15:19.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.193+0000 7fb7a7fff640 1 --2- 172.21.15.72:0/4213169124 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5648db67b1f0 0x5648db66f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:19.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.193+0000 7fb7affff640 1 --2- 172.21.15.72:0/4213169124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5648db67ae10 0x5648db66eea0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb7a000fce0 tx=0x7fb7a0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffdff15d8f42b7c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:19.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.193+0000 7fb78effd640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-04-15T15:15:19.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.193+0000 7fb78effd640 1 -- 172.21.15.72:0/4213169124 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7a0037070 con 0x5648db67ae10 2024-04-15T15:15:19.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.193+0000 7fb78effd640 1 -- 172.21.15.72:0/4213169124 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb7a0005b90 con 0x5648db67ae10 2024-04-15T15:15:19.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.193+0000 7fb7b93ac740 1 -- 172.21.15.72:0/4213169124 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb798084650 con 0x5648db67ae10 2024-04-15T15:15:19.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.193+0000 7fb78effd640 1 -- 172.21.15.72:0/4213169124 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) ==== 24+0+0 (secure 0 0 0) 0x7fb7a0042050 con 0x5648db67ae10 2024-04-15T15:15:19.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.193+0000 7fb7b93ac740 10 client.4899 fetch_fsmap learned FSMap version 9 2024-04-15T15:15:19.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.193+0000 7fb7b93ac740 1 -- 172.21.15.72:0/4213169124 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) -- 0x5648db653630 con 0x5648db67ae10 2024-04-15T15:15:19.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.193+0000 7fb78effd640 1 -- 172.21.15.72:0/4213169124 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb798084650 con 0x5648db67ae10 2024-04-15T15:15:19.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.193+0000 7fb78effd640 1 -- 172.21.15.72:0/4213169124 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 9) ==== 3502+0+0 (secure 0 0 0) 0x7fb7a0045020 con 0x5648db67ae10 2024-04-15T15:15:19.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.193+0000 7fb7b93ac740 10 client.4899 fetch_fsmap finished waiting for FSMap version 9 2024-04-15T15:15:19.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.193+0000 7fb7b93ac740 10 client.4899 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-15T15:15:19.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.193+0000 7fb7b93ac740 20 client.4899 populate_metadata read hostname 'smithi072' 2024-04-15T15:15:19.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.193+0000 7fb7b93ac740 1 --2- 172.21.15.72:0/4213169124 >> [v2:172.21.15.72:6835/3458067804,v1:172.21.15.72:6837/3458067804] conn(0x5648db679300 0x5648db6796e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:19.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.193+0000 7fb7b93ac740 4 client.4899 mds_command: new command op to 4299 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-15T15:15:19.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.193+0000 7fb7b93ac740 1 -- 172.21.15.72:0/4213169124 --> [v2:172.21.15.72:6835/3458067804,v1:172.21.15.72:6837/3458067804] -- command(tid 0: {"prefix": "status"}) -- 0x5648db5c0250 con 0x5648db679300 2024-04-15T15:15:19.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.194+0000 7fb7a7fff640 1 --2- 172.21.15.72:0/4213169124 >> [v2:172.21.15.72:6835/3458067804,v1:172.21.15.72:6837/3458067804] conn(0x5648db679300 0x5648db6796e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:19.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.194+0000 7fb78effd640 10 client.4899 ms_handle_connect on v2:172.21.15.72:6835/3458067804 2024-04-15T15:15:19.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.194+0000 7fb7a7fff640 1 --2- 172.21.15.72:0/4213169124 >> [v2:172.21.15.72:6835/3458067804,v1:172.21.15.72:6837/3458067804] conn(0x5648db679300 0x5648db6796e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c529aa555038164c server_cookie=58b955bdfc746487 in_seq=0 out_seq=0 2024-04-15T15:15:19.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.194+0000 7fb78effd640 1 -- 172.21.15.72:0/4213169124 <== mds.0 v2:172.21.15.72:6835/3458067804 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x5648db5c0250 con 0x5648db679300 2024-04-15T15:15:19.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.194+0000 7fb78effd640 10 client.4899 handle_command_reply: tid=0 2024-04-15T15:15:19.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.194+0000 7fb7b93ac740 1 client.4899 shutdown 2024-04-15T15:15:19.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.194+0000 7fb7b93ac740 1 -- 172.21.15.72:0/4213169124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5648db67ae10 msgr2=0x5648db66eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:19.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.194+0000 7fb7b93ac740 1 --2- 172.21.15.72:0/4213169124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5648db67ae10 0x5648db66eea0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb7a000fce0 tx=0x7fb7a0014000 comp rx=0 tx=0).stop 2024-04-15T15:15:19.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.194+0000 7fb7b93ac740 1 -- 172.21.15.72:0/4213169124 shutdown_connections 2024-04-15T15:15:19.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.194+0000 7fb7b93ac740 1 --2- 172.21.15.72:0/4213169124 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x5648db6ac410 0x5648db6aa950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.194+0000 7fb7b93ac740 1 --2- 172.21.15.72:0/4213169124 >> [v2:172.21.15.72:6835/3458067804,v1:172.21.15.72:6837/3458067804] conn(0x5648db679300 0x5648db6796e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.194+0000 7fb7b93ac740 1 --2- 172.21.15.72:0/4213169124 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x5648db67b1f0 0x5648db66f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.194+0000 7fb7b93ac740 1 --2- 172.21.15.72:0/4213169124 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x5648db67ae10 0x5648db66eea0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.194+0000 7fb78effd640 0 client.4899 ms_handle_reset on v2:172.21.15.72:6835/3458067804 2024-04-15T15:15:19.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.194+0000 7fb7b93ac740 1 -- 172.21.15.72:0/4213169124 >> 172.21.15.72:0/4213169124 conn(0x5648db64e2f0 msgr2=0x5648db6bc0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:19.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.194+0000 7fb7b93ac740 1 -- 172.21.15.72:0/4213169124 shutdown_connections 2024-04-15T15:15:19.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.194+0000 7fb7b93ac740 1 -- 172.21.15.72:0/4213169124 wait complete. 2024-04-15T15:15:19.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.195+0000 7fb7b93ac740 20 client.4899 trim_cache size 0 max 16384 2024-04-15T15:15:19.212 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-04-15T15:15:19.212 INFO:teuthology.orchestra.run.smithi072.stdout: "cluster_fsid": "05091e39-b18a-4545-8b68-c0ae4699e116", 2024-04-15T15:15:19.212 INFO:teuthology.orchestra.run.smithi072.stdout: "whoami": 0, 2024-04-15T15:15:19.213 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 4299, 2024-04-15T15:15:19.213 INFO:teuthology.orchestra.run.smithi072.stdout: "want_state": "up:active", 2024-04-15T15:15:19.213 INFO:teuthology.orchestra.run.smithi072.stdout: "state": "up:active", 2024-04-15T15:15:19.213 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs", 2024-04-15T15:15:19.213 INFO:teuthology.orchestra.run.smithi072.stdout: "rank_uptime": 27.124477061, 2024-04-15T15:15:19.213 INFO:teuthology.orchestra.run.smithi072.stdout: "mdsmap_epoch": 9, 2024-04-15T15:15:19.213 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch": 24, 2024-04-15T15:15:19.213 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch_barrier": 24, 2024-04-15T15:15:19.213 INFO:teuthology.orchestra.run.smithi072.stdout: "uptime": 34.214194548999998 2024-04-15T15:15:19.213 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-04-15T15:15:19.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.195+0000 7fb7b66e4640 1 -- 172.21.15.72:0/279969072 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb79006c8f0 msgr2=0x7fb79006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:19.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.195+0000 7fb7b66e4640 1 --2- 172.21.15.72:0/279969072 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb79006c8f0 0x7fb79006edb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb7a000a990 tx=0x7fb7a0008040 comp rx=0 tx=0).stop 2024-04-15T15:15:19.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.195+0000 7fb7b66e4640 1 -- 172.21.15.72:0/279969072 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7b015c250 msgr2=0x7fb7b00c6980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:19.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.195+0000 7fb7b66e4640 1 --2- 172.21.15.72:0/279969072 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7b015c250 0x7fb7b00c6980 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb79802b8a0 tx=0x7fb7980046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.195+0000 7fb7b66e4640 1 -- 172.21.15.72:0/279969072 shutdown_connections 2024-04-15T15:15:19.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.195+0000 7fb7b66e4640 1 --2- 172.21.15.72:0/279969072 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb79006c8f0 0x7fb79006edb0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.195+0000 7fb7b66e4640 1 --2- 172.21.15.72:0/279969072 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb7b015f9f0 0x7fb7b0154ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.195+0000 7fb7b66e4640 1 --2- 172.21.15.72:0/279969072 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb7b015c250 0x7fb7b00c6980 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.195+0000 7fb7b66e4640 1 --2- 172.21.15.72:0/279969072 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb7b00d0010 0x7fb7b015e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.195+0000 7fb7b66e4640 1 -- 172.21.15.72:0/279969072 >> 172.21.15.72:0/279969072 conn(0x7fb7b00ceec0 msgr2=0x7fb7b00cf2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:19.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.195+0000 7fb7b66e4640 1 -- 172.21.15.72:0/279969072 shutdown_connections 2024-04-15T15:15:19.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.195+0000 7fb7b66e4640 1 -- 172.21.15.72:0/279969072 wait complete. 2024-04-15T15:15:19.216 INFO:teuthology.run_tasks:Running task sequential... 2024-04-15T15:15:19.231 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-15T15:15:19.232 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-15T15:15:19.232 INFO:teuthology.task.print:Enabling mgr modules 2024-04-15T15:15:19.232 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-04-15T15:15:19.246 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-04-15T15:15:19.246 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-04-15T15:15:19.246 INFO:tasks.ceph_fuse:client.0 config is {} 2024-04-15T15:15:19.247 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-15T15:15:19.247 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-15T15:15:19.247 INFO:tasks.ceph_fuse:client.2 config is {} 2024-04-15T15:15:19.247 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-15T15:15:19.247 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-15T15:15:19.248 INFO:tasks.ceph_fuse:client.1 config is {} 2024-04-15T15:15:19.248 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-15T15:15:19.248 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-15T15:15:19.248 INFO:tasks.ceph_fuse:client.3 config is {} 2024-04-15T15:15:19.248 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-15T15:15:19.248 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-15T15:15:19.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:19.249 DEBUG:teuthology.orchestra.run.smithi175:> ip netns list 2024-04-15T15:15:19.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:19.270 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip link delete ceph-brx 2024-04-15T15:15:19.345 INFO:teuthology.orchestra.run.smithi175.stderr:Cannot find device "ceph-brx" 2024-04-15T15:15:19.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:15:19.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:19.348 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-04-15T15:15:19.364 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-15T15:15:19.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:19.364 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-15T15:15:19.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:19.435 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link delete ceph-brx 2024-04-15T15:15:19.529 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-04-15T15:15:19.529 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:15:19.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.631+0000 7f69b489a640 1 Processor -- start 2024-04-15T15:15:19.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.631+0000 7f69b489a640 1 -- start start 2024-04-15T15:15:19.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.632+0000 7f69b489a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f69b015eba0 0x7f69b006f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:19.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.632+0000 7f69b489a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69b00afb10 con 0x7f69b00ccb90 2024-04-15T15:15:19.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.632+0000 7f69b489a640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69b015ef80 con 0x7f69b006f720 2024-04-15T15:15:19.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.632+0000 7f69b489a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f69b00ce2b0 con 0x7f69b015eba0 2024-04-15T15:15:19.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.632+0000 7f69ae59c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f69b015eba0 0x7f69b006f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:19.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.632+0000 7f69ae59c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f69b015eba0 0x7f69b006f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45738/0 (socket says 172.21.15.72:45738) 2024-04-15T15:15:19.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.632+0000 7f69aed9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f69b00ccb90 0x7f69b00ad0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:38008/0 (socket says 172.21.15.72:38008) 2024-04-15T15:15:19.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.632+0000 7f69ae59c640 1 -- 172.21.15.72:0/202770827 learned_addr learned my addr 172.21.15.72:0/202770827 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:19.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.632+0000 7f69ae59c640 1 -- 172.21.15.72:0/202770827 >> v1:172.21.15.175:6789/0 conn(0x7f69b006f720 legacy=0x7f69b00af400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:19.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.632+0000 7f69ae59c640 1 -- 172.21.15.72:0/202770827 >> v1:172.21.15.72:6789/0 conn(0x7f69b00ccb90 legacy=0x7f69b00ad0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:19.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.632+0000 7f69ad59a640 1 -- 172.21.15.72:0/202770827 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4051928671 0 0) 0x7f69b00afb10 con 0x7f69b00ccb90 2024-04-15T15:15:19.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.632+0000 7f69ae59c640 1 -- 172.21.15.72:0/202770827 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69b00ce590 con 0x7f69b015eba0 2024-04-15T15:15:19.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.633+0000 7f69ae59c640 1 --2- 172.21.15.72:0/202770827 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69b015eba0 0x7f69b006f1e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f69a8009e50 tx=0x7f69a802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a899e0ad7a9161e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:19.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.633+0000 7f69ad59a640 1 -- 172.21.15.72:0/202770827 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69a802cb20 con 0x7f69b015eba0 2024-04-15T15:15:19.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.633+0000 7f69ad59a640 1 -- 172.21.15.72:0/202770827 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69a802ccc0 con 0x7f69b015eba0 2024-04-15T15:15:19.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.633+0000 7f69ad59a640 1 -- 172.21.15.72:0/202770827 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69a8034610 con 0x7f69b015eba0 2024-04-15T15:15:19.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.633+0000 7f69b489a640 1 -- 172.21.15.72:0/202770827 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69b015eba0 msgr2=0x7f69b006f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:19.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.633+0000 7f69b489a640 1 --2- 172.21.15.72:0/202770827 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69b015eba0 0x7f69b006f1e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f69a8009e50 tx=0x7f69a802bf80 comp rx=0 tx=0).stop 2024-04-15T15:15:19.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.634+0000 7f69b489a640 1 -- 172.21.15.72:0/202770827 shutdown_connections 2024-04-15T15:15:19.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.634+0000 7f69b489a640 1 --2- 172.21.15.72:0/202770827 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69b015eba0 0x7f69b006f1e0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.634+0000 7f69b489a640 1 -- 172.21.15.72:0/202770827 >> 172.21.15.72:0/202770827 conn(0x7f69b00b1dd0 msgr2=0x7f69b00b21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:19.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.634+0000 7f69b489a640 1 -- 172.21.15.72:0/202770827 shutdown_connections 2024-04-15T15:15:19.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.634+0000 7f69b489a640 1 -- 172.21.15.72:0/202770827 wait complete. 2024-04-15T15:15:19.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.635+0000 7f69b489a640 1 Processor -- start 2024-04-15T15:15:19.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.635+0000 7f69b489a640 1 -- start start 2024-04-15T15:15:19.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.636+0000 7f69b489a640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69b00ccb90 0x7f69b01f1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:19.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.636+0000 7f69b489a640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f69b015eba0 0x7f69b014d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:19.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.636+0000 7f69b489a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69b01530f0 0x7f69b00ae610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:19.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.636+0000 7f69b489a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f69b00b2b80 con 0x7f69b01530f0 2024-04-15T15:15:19.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.636+0000 7f69b489a640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f69b006ecc0 con 0x7f69b00ccb90 2024-04-15T15:15:19.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.636+0000 7f69b489a640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f69b00ceb50 con 0x7f69b015eba0 2024-04-15T15:15:19.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.636+0000 7f69aed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69b01530f0 0x7f69b00ae610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:19.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.636+0000 7f69aed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69b01530f0 0x7f69b00ae610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45750/0 (socket says 172.21.15.72:45750) 2024-04-15T15:15:19.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.636+0000 7f69aed9d640 1 -- 172.21.15.72:0/2768239005 learned_addr learned my addr 172.21.15.72:0/2768239005 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:19.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.636+0000 7f69aed9d640 1 -- 172.21.15.72:0/2768239005 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f69b015eba0 msgr2=0x7f69b014d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:19.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.636+0000 7f69aed9d640 1 --2- 172.21.15.72:0/2768239005 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f69b015eba0 0x7f69b014d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.636+0000 7f69aed9d640 1 -- 172.21.15.72:0/2768239005 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69b00ccb90 msgr2=0x7f69b01f1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:19.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.636+0000 7f69ae59c640 1 --2- 172.21.15.72:0/2768239005 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69b00ccb90 0x7f69b01f1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:19.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.636+0000 7f69aed9d640 1 --2- 172.21.15.72:0/2768239005 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69b00ccb90 0x7f69b01f1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.636+0000 7f69aed9d640 1 -- 172.21.15.72:0/2768239005 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69b01536f0 con 0x7f69b01530f0 2024-04-15T15:15:19.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.636+0000 7f69ae59c640 1 --2- 172.21.15.72:0/2768239005 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69b00ccb90 0x7f69b01f1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:19.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.636+0000 7f69aed9d640 1 --2- 172.21.15.72:0/2768239005 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69b01530f0 0x7f69b00ae610 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f69a000c450 tx=0x7f69a000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:19.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.636+0000 7f69a6ffd640 1 -- 172.21.15.72:0/2768239005 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69a0018070 con 0x7f69b01530f0 2024-04-15T15:15:19.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.636+0000 7f69a6ffd640 1 -- 172.21.15.72:0/2768239005 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69a0007500 con 0x7f69b01530f0 2024-04-15T15:15:19.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.636+0000 7f69b489a640 1 -- 172.21.15.72:0/2768239005 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69b01539d0 con 0x7f69b01530f0 2024-04-15T15:15:19.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.636+0000 7f69a6ffd640 1 -- 172.21.15.72:0/2768239005 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69a0004a20 con 0x7f69b01530f0 2024-04-15T15:15:19.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.636+0000 7f69b489a640 1 -- 172.21.15.72:0/2768239005 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69b0153f90 con 0x7f69b01530f0 2024-04-15T15:15:19.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.637+0000 7f69b489a640 1 -- 172.21.15.72:0/2768239005 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69b00689e0 con 0x7f69b01530f0 2024-04-15T15:15:19.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.638+0000 7f69a6ffd640 1 -- 172.21.15.72:0/2768239005 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f69a001b070 con 0x7f69b01530f0 2024-04-15T15:15:19.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.638+0000 7f69a6ffd640 1 --2- 172.21.15.72:0/2768239005 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f699006c920 0x7f699006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:19.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.638+0000 7f69a6ffd640 1 -- 172.21.15.72:0/2768239005 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f69a008b080 con 0x7f69b01530f0 2024-04-15T15:15:19.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.638+0000 7f69ae59c640 1 --2- 172.21.15.72:0/2768239005 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f699006c920 0x7f699006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:19.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.639+0000 7f69ae59c640 1 --2- 172.21.15.72:0/2768239005 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f699006c920 0x7f699006ede0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f69a8035000 tx=0x7f69a8030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:19.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.643+0000 7f69a6ffd640 1 -- 172.21.15.72:0/2768239005 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f69a0059200 con 0x7f69b01530f0 2024-04-15T15:15:19.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.852+0000 7f69b489a640 1 -- 172.21.15.72:0/2768239005 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f69b0141010 con 0x7f69b01530f0 2024-04-15T15:15:19.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.853+0000 7f69a6ffd640 1 -- 172.21.15.72:0/2768239005 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7f69b0141010 con 0x7f69b01530f0 2024-04-15T15:15:19.855 INFO:teuthology.orchestra.run.smithi072.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-15T15:15:19.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.854+0000 7f69b489a640 1 -- 172.21.15.72:0/2768239005 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f699006c920 msgr2=0x7f699006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:19.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.854+0000 7f69b489a640 1 --2- 172.21.15.72:0/2768239005 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f699006c920 0x7f699006ede0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f69a8035000 tx=0x7f69a8030000 comp rx=0 tx=0).stop 2024-04-15T15:15:19.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.854+0000 7f69b489a640 1 -- 172.21.15.72:0/2768239005 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69b01530f0 msgr2=0x7f69b00ae610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:19.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.854+0000 7f69b489a640 1 --2- 172.21.15.72:0/2768239005 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69b01530f0 0x7f69b00ae610 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f69a000c450 tx=0x7f69a000c910 comp rx=0 tx=0).stop 2024-04-15T15:15:19.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.854+0000 7f69b489a640 1 -- 172.21.15.72:0/2768239005 shutdown_connections 2024-04-15T15:15:19.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.854+0000 7f69b489a640 1 --2- 172.21.15.72:0/2768239005 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f699006c920 0x7f699006ede0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.854+0000 7f69b489a640 1 --2- 172.21.15.72:0/2768239005 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69b01530f0 0x7f69b00ae610 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.854+0000 7f69b489a640 1 --2- 172.21.15.72:0/2768239005 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f69b015eba0 0x7f69b014d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.854+0000 7f69b489a640 1 --2- 172.21.15.72:0/2768239005 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69b00ccb90 0x7f69b01f1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.855+0000 7f69b489a640 1 -- 172.21.15.72:0/2768239005 >> 172.21.15.72:0/2768239005 conn(0x7f69b00b1dd0 msgr2=0x7f69b015ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:19.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.855+0000 7f69b489a640 1 -- 172.21.15.72:0/2768239005 shutdown_connections 2024-04-15T15:15:19.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19.855+0000 7f69b489a640 1 -- 172.21.15.72:0/2768239005 wait complete. 2024-04-15T15:15:19.868 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:15:19.868 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:15:19.868 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi072.front.sepia.ceph.com 2024-04-15T15:15:19.868 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-15T15:15:19.868 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-15T15:15:19.868 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:15:19.868 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-15T15:15:19.869 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:15:19.869 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-15T15:15:19.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:19.869 DEBUG:teuthology.orchestra.run.smithi072:> ip addr 2024-04-15T15:15:19.885 INFO:teuthology.orchestra.run.smithi072.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:15:19.885 INFO:teuthology.orchestra.run.smithi072.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:15:19.885 INFO:teuthology.orchestra.run.smithi072.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:15:19.885 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:19.885 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 ::1/128 scope host 2024-04-15T15:15:19.885 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:19.886 INFO:teuthology.orchestra.run.smithi072.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:15:19.886 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f4 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:19.886 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f0 2024-04-15T15:15:19.886 INFO:teuthology.orchestra.run.smithi072.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:15:19.886 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f5 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:19.886 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f1 2024-04-15T15:15:19.886 INFO:teuthology.orchestra.run.smithi072.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:15:19.886 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:64 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:19.886 INFO:teuthology.orchestra.run.smithi072.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:15:19.886 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:65 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:19.886 INFO:teuthology.orchestra.run.smithi072.stdout: inet 172.21.15.72/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-15T15:15:19.886 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft 42683sec preferred_lft 42683sec 2024-04-15T15:15:19.886 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::ec4:7aff:fe8f:b765/64 scope link 2024-04-15T15:15:19.886 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:19.887 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-15T15:15:19.887 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:19.887 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-04-15T15:15:19.887 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link add name ceph-brx type bridge 2024-04-15T15:15:19.887 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip addr flush dev ceph-brx 2024-04-15T15:15:19.887 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set ceph-brx up 2024-04-15T15:15:19.888 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-15T15:15:19.888 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-04-15T15:15:19.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:19 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:20.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:20.037 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:20.037 DEBUG:teuthology.orchestra.run.smithi072:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:15:20.065 INFO:teuthology.orchestra.run.smithi072.stdout:1 2024-04-15T15:15:20.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:20.067 DEBUG:teuthology.orchestra.run.smithi072:> route 2024-04-15T15:15:20.124 INFO:teuthology.orchestra.run.smithi072.stdout:Kernel IP routing table 2024-04-15T15:15:20.124 INFO:teuthology.orchestra.run.smithi072.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:15:20.124 INFO:teuthology.orchestra.run.smithi072.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-15T15:15:20.124 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-15T15:15:20.124 INFO:teuthology.orchestra.run.smithi072.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:15:20.125 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:20.125 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-04-15T15:15:20.125 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-15T15:15:20.125 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-15T15:15:20.125 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-15T15:15:20.125 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-04-15T15:15:20.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:20.258 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:20.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:20.264 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-04-15T15:15:20.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:20.280 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list-id 2024-04-15T15:15:20.336 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:20.336 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-04-15T15:15:20.336 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:15:20.336 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-15T15:15:20.336 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-04-15T15:15:20.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:20.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:20.462 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-15T15:15:20.463 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:20.463 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-04-15T15:15:20.463 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-15T15:15:20.463 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-15T15:15:20.463 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-15T15:15:20.463 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-15T15:15:20.463 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-15T15:15:20.463 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-04-15T15:15:20.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:20.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:20.592 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:20.592 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-04-15T15:15:20.592 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set brx.0 up 2024-04-15T15:15:20.592 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set dev brx.0 master ceph-brx 2024-04-15T15:15:20.592 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-04-15T15:15:20.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:20 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:20.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:20 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:20.675 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-15T15:15:20.675 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:15:20.675 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-15T15:15:20.690 INFO:teuthology.orchestra.run.smithi072.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-15T15:15:20.691 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:15:20.691 DEBUG:teuthology.orchestra.run.smithi072:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-15T15:15:20.747 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-04-15T15:15:20.817 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:15:20.854 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:15:20.863 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:15:20.874 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-04-15T15:15:20.874 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-04-15T15:15:20.874 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-04-15T15:15:20.874 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-04-15T15:15:20.874 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-04-15T15:15:20.874 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-04-15T15:15:20.874 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-04-15T15:15:20.874 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-04-15T15:15:20.874 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-04-15T15:15:20.874 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-04-15T15:15:20.874 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-04-15T15:15:20.875 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-04-15T15:15:20.875 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:15:20.875 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-04-15T15:15:20.875 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-04-15T15:15:20.875 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-04-15T15:15:20.875 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-04-15T15:15:20.875 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:15:20.875 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-04-15T15:15:20.875 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-04-15T15:15:20.875 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:15:20.875 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:15:20.875 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:15:20.875 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-04-15T15:15:20.875 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-04-15T15:15:20.875 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:15:20.876 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:15:20.876 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:15:20.876 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:15:20.876 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-04-15T15:15:20.876 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:15:20.876 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:15:20.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:20.876 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-04-15T15:15:20.930 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-15T15:15:20.930 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-15T15:15:20.972 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-04-15T15:15:21.000 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:15:21.058 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-04-15T15:15:21.058 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-04-15T15:15:21.058 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-04-15T15:15:21.058 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-04-15T15:15:21.058 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-04-15T15:15:21.058 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-04-15T15:15:21.058 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-04-15T15:15:21.059 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-04-15T15:15:21.059 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-04-15T15:15:21.059 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-04-15T15:15:21.059 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-04-15T15:15:21.059 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-04-15T15:15:21.059 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:15:21.059 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-04-15T15:15:21.059 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-04-15T15:15:21.059 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-04-15T15:15:21.059 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-04-15T15:15:21.059 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:15:21.059 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-04-15T15:15:21.059 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-04-15T15:15:21.059 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:15:21.060 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:15:21.060 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:15:21.060 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-04-15T15:15:21.060 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-04-15T15:15:21.060 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:15:21.060 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:15:21.060 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:15:21.060 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:15:21.060 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-04-15T15:15:21.060 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:15:21.060 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:15:21.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:21.060 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-04-15T15:15:21.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:2024-04-15T15:15:21.059+0000 7f8ba9779480 -1 init, newargv = 0x564cfb0720f0 newargc=15 2024-04-15T15:15:21.061 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[48052]: starting ceph client 2024-04-15T15:15:21.067 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[48052]: starting fuse 2024-04-15T15:15:21.077 INFO:teuthology.orchestra.run.smithi072.stdout:43 2024-04-15T15:15:21.077 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-04-15T15:15:21.078 DEBUG:teuthology.orchestra.run.smithi072:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:15:21.078 DEBUG:teuthology.orchestra.run.smithi072:> import glob 2024-04-15T15:15:21.078 DEBUG:teuthology.orchestra.run.smithi072:> import re 2024-04-15T15:15:21.078 DEBUG:teuthology.orchestra.run.smithi072:> import os 2024-04-15T15:15:21.078 DEBUG:teuthology.orchestra.run.smithi072:> import subprocess 2024-04-15T15:15:21.078 DEBUG:teuthology.orchestra.run.smithi072:> 2024-04-15T15:15:21.078 DEBUG:teuthology.orchestra.run.smithi072:> def _find_admin_socket(client_name): 2024-04-15T15:15:21.078 DEBUG:teuthology.orchestra.run.smithi072:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-15T15:15:21.078 DEBUG:teuthology.orchestra.run.smithi072:> files = glob.glob(asok_path) 2024-04-15T15:15:21.078 DEBUG:teuthology.orchestra.run.smithi072:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-15T15:15:21.078 DEBUG:teuthology.orchestra.run.smithi072:> 2024-04-15T15:15:21.078 DEBUG:teuthology.orchestra.run.smithi072:> # Given a non-glob path, it better be there 2024-04-15T15:15:21.078 DEBUG:teuthology.orchestra.run.smithi072:> if "*" not in asok_path: 2024-04-15T15:15:21.078 DEBUG:teuthology.orchestra.run.smithi072:> assert(len(files) == 1) 2024-04-15T15:15:21.081 DEBUG:teuthology.orchestra.run.smithi072:> return files[0] 2024-04-15T15:15:21.082 DEBUG:teuthology.orchestra.run.smithi072:> 2024-04-15T15:15:21.082 DEBUG:teuthology.orchestra.run.smithi072:> for f in files: 2024-04-15T15:15:21.082 DEBUG:teuthology.orchestra.run.smithi072:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:15:21.082 DEBUG:teuthology.orchestra.run.smithi072:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:15:21.082 DEBUG:teuthology.orchestra.run.smithi072:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:15:21.082 DEBUG:teuthology.orchestra.run.smithi072:> contents = proc_f.read() 2024-04-15T15:15:21.082 DEBUG:teuthology.orchestra.run.smithi072:> if mountpoint in contents: 2024-04-15T15:15:21.082 DEBUG:teuthology.orchestra.run.smithi072:> return f 2024-04-15T15:15:21.083 DEBUG:teuthology.orchestra.run.smithi072:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:15:21.083 DEBUG:teuthology.orchestra.run.smithi072:> 2024-04-15T15:15:21.083 DEBUG:teuthology.orchestra.run.smithi072:> print(_find_admin_socket("client.0")) 2024-04-15T15:15:21.083 DEBUG:teuthology.orchestra.run.smithi072:> ' 2024-04-15T15:15:21.202 INFO:teuthology.orchestra.run.smithi072.stdout:/var/run/ceph/ceph-client.0.48052.asok 2024-04-15T15:15:21.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:21.213 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.48052.asok 2024-04-15T15:15:21.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:21.213 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.48052.asok status 2024-04-15T15:15:21.353 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-04-15T15:15:21.354 INFO:teuthology.orchestra.run.smithi072.stdout: "metadata": { 2024-04-15T15:15:21.354 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:15:21.354 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:15:21.354 INFO:teuthology.orchestra.run.smithi072.stdout: "entity_id": "0", 2024-04-15T15:15:21.354 INFO:teuthology.orchestra.run.smithi072.stdout: "hostname": "smithi072", 2024-04-15T15:15:21.354 INFO:teuthology.orchestra.run.smithi072.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-15T15:15:21.354 INFO:teuthology.orchestra.run.smithi072.stdout: "pid": "48052", 2024-04-15T15:15:21.354 INFO:teuthology.orchestra.run.smithi072.stdout: "root": "/" 2024-04-15T15:15:21.354 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:15:21.354 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_count": 0, 2024-04-15T15:15:21.354 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_pinned_count": 0, 2024-04-15T15:15:21.354 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 4911, 2024-04-15T15:15:21.355 INFO:teuthology.orchestra.run.smithi072.stdout: "inst": { 2024-04-15T15:15:21.355 INFO:teuthology.orchestra.run.smithi072.stdout: "name": { 2024-04-15T15:15:21.355 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "client", 2024-04-15T15:15:21.355 INFO:teuthology.orchestra.run.smithi072.stdout: "num": 4911 2024-04-15T15:15:21.355 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:15:21.355 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-04-15T15:15:21.355 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-04-15T15:15:21.355 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "172.21.15.72:0", 2024-04-15T15:15:21.355 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 2819150670 2024-04-15T15:15:21.355 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:15:21.355 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:15:21.355 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-04-15T15:15:21.355 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-04-15T15:15:21.355 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "172.21.15.72:0", 2024-04-15T15:15:21.355 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 2819150670 2024-04-15T15:15:21.356 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:15:21.356 INFO:teuthology.orchestra.run.smithi072.stdout: "inst_str": "client.4911 172.21.15.72:0/2819150670", 2024-04-15T15:15:21.356 INFO:teuthology.orchestra.run.smithi072.stdout: "addr_str": "172.21.15.72:0/2819150670", 2024-04-15T15:15:21.356 INFO:teuthology.orchestra.run.smithi072.stdout: "inode_count": 1, 2024-04-15T15:15:21.356 INFO:teuthology.orchestra.run.smithi072.stdout: "mds_epoch": 9, 2024-04-15T15:15:21.356 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch": 24, 2024-04-15T15:15:21.356 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:15:21.356 INFO:teuthology.orchestra.run.smithi072.stdout: "blocklisted": false, 2024-04-15T15:15:21.356 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs" 2024-04-15T15:15:21.356 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-04-15T15:15:21.363 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:15:21.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.467+0000 7f59f8782640 1 Processor -- start 2024-04-15T15:15:21.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.467+0000 7f59f8782640 1 -- start start 2024-04-15T15:15:21.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.467+0000 7f59f8782640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f59f015eba0 0x7f59f006f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:21.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.467+0000 7f59f8782640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f59f006f720 0x7f59f00af400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:21.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.467+0000 7f59f8782640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f59f00af940 0x7f59f00ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:21.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.468+0000 7f59f8782640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f59f015ef80 con 0x7f59f006f720 2024-04-15T15:15:21.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.468+0000 7f59f8782640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f59f00ce2b0 con 0x7f59f00af940 2024-04-15T15:15:21.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.468+0000 7f59f8782640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f59f00ce590 con 0x7f59f015eba0 2024-04-15T15:15:21.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.468+0000 7f59f5d1d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f59f006f720 0x7f59f00af400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:21.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.468+0000 7f59f5d1d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f59f006f720 0x7f59f00af400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45760/0 (socket says 172.21.15.72:45760) 2024-04-15T15:15:21.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.468+0000 7f59f5d1d640 1 -- 172.21.15.72:0/2106544346 learned_addr learned my addr 172.21.15.72:0/2106544346 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:21.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.468+0000 7f59f6d1f640 1 --2- 172.21.15.72:0/2106544346 >> v2:172.21.15.175:3300/0 conn(0x7f59f00af940 0x7f59f00ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:21.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.468+0000 7f59f651e640 1 --2- 172.21.15.72:0/2106544346 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f59f015eba0 0x7f59f006f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:21.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.468+0000 7f59f5d1d640 1 -- 172.21.15.72:0/2106544346 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f59f015eba0 msgr2=0x7f59f006f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:21.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.468+0000 7f59f5d1d640 1 --2- 172.21.15.72:0/2106544346 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f59f015eba0 0x7f59f006f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.468+0000 7f59f5d1d640 1 -- 172.21.15.72:0/2106544346 >> v2:172.21.15.175:3300/0 conn(0x7f59f00af940 msgr2=0x7f59f00ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:21.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.468+0000 7f59f5d1d640 1 --2- 172.21.15.72:0/2106544346 >> v2:172.21.15.175:3300/0 conn(0x7f59f00af940 0x7f59f00ccb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.468+0000 7f59f5d1d640 1 -- 172.21.15.72:0/2106544346 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59f00ce870 con 0x7f59f006f720 2024-04-15T15:15:21.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.469+0000 7f59f5d1d640 1 --2- 172.21.15.72:0/2106544346 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59f006f720 0x7f59f00af400 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f59f00cc5b0 tx=0x7f59e002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb3c8424971feef2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:21.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.469+0000 7f59f551c640 1 -- 172.21.15.72:0/2106544346 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e002e070 con 0x7f59f006f720 2024-04-15T15:15:21.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.469+0000 7f59f551c640 1 -- 172.21.15.72:0/2106544346 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59e0037070 con 0x7f59f006f720 2024-04-15T15:15:21.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.469+0000 7f59f551c640 1 -- 172.21.15.72:0/2106544346 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e0031a00 con 0x7f59f006f720 2024-04-15T15:15:21.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.470+0000 7f59f8782640 1 -- 172.21.15.72:0/2106544346 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59f006f720 msgr2=0x7f59f00af400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:21.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.470+0000 7f59f8782640 1 --2- 172.21.15.72:0/2106544346 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59f006f720 0x7f59f00af400 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f59f00cc5b0 tx=0x7f59e002b880 comp rx=0 tx=0).stop 2024-04-15T15:15:21.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.470+0000 7f59f8782640 1 -- 172.21.15.72:0/2106544346 shutdown_connections 2024-04-15T15:15:21.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.470+0000 7f59f8782640 1 --2- 172.21.15.72:0/2106544346 >> v2:172.21.15.175:3300/0 conn(0x7f59f00af940 0x7f59f00ccb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.470+0000 7f59f8782640 1 --2- 172.21.15.72:0/2106544346 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59f006f720 0x7f59f00af400 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.470+0000 7f59f8782640 1 --2- 172.21.15.72:0/2106544346 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f59f015eba0 0x7f59f006f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.470+0000 7f59f8782640 1 -- 172.21.15.72:0/2106544346 >> 172.21.15.72:0/2106544346 conn(0x7f59f00b1dd0 msgr2=0x7f59f00b21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:21.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.470+0000 7f59f8782640 1 -- 172.21.15.72:0/2106544346 shutdown_connections 2024-04-15T15:15:21.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.470+0000 7f59f8782640 1 -- 172.21.15.72:0/2106544346 wait complete. 2024-04-15T15:15:21.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.471+0000 7f59f8782640 1 Processor -- start 2024-04-15T15:15:21.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.471+0000 7f59f8782640 1 -- start start 2024-04-15T15:15:21.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.472+0000 7f59f8782640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59f00af940 0x7f59f014d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:21.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.472+0000 7f59f8782640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59f015eba0 0x7f59f0151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:21.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.472+0000 7f59f8782640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f59f0073530 0x7f59f00b0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:21.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.472+0000 7f59f8782640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f59f00ceb50 con 0x7f59f015eba0 2024-04-15T15:15:21.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.472+0000 7f59f8782640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f59f00b2b80 con 0x7f59f00af940 2024-04-15T15:15:21.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.472+0000 7f59f8782640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f59f00cd0d0 con 0x7f59f0073530 2024-04-15T15:15:21.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.472+0000 7f59f5d1d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59f015eba0 0x7f59f0151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:21.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.472+0000 7f59f5d1d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59f015eba0 0x7f59f0151fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45774/0 (socket says 172.21.15.72:45774) 2024-04-15T15:15:21.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.472+0000 7f59f5d1d640 1 -- 172.21.15.72:0/1812916726 learned_addr learned my addr 172.21.15.72:0/1812916726 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:21.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.472+0000 7f59f651e640 1 --2- 172.21.15.72:0/1812916726 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59f00af940 0x7f59f014d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:21.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.472+0000 7f59f6d1f640 1 --2- 172.21.15.72:0/1812916726 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f59f0073530 0x7f59f00b0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:21.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.472+0000 7f59f5d1d640 1 -- 172.21.15.72:0/1812916726 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f59f0073530 msgr2=0x7f59f00b0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:21.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.472+0000 7f59f5d1d640 1 --2- 172.21.15.72:0/1812916726 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f59f0073530 0x7f59f00b0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.472+0000 7f59f5d1d640 1 -- 172.21.15.72:0/1812916726 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59f00af940 msgr2=0x7f59f014d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:21.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.472+0000 7f59f5d1d640 1 --2- 172.21.15.72:0/1812916726 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59f00af940 0x7f59f014d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.472+0000 7f59f5d1d640 1 -- 172.21.15.72:0/1812916726 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59f00ae1a0 con 0x7f59f015eba0 2024-04-15T15:15:21.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.472+0000 7f59f651e640 1 --2- 172.21.15.72:0/1812916726 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59f00af940 0x7f59f014d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:21.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.472+0000 7f59f5d1d640 1 --2- 172.21.15.72:0/1812916726 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59f015eba0 0x7f59f0151fb0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f59e002b8c0 tx=0x7f59e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:21.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.473+0000 7f59e6ffd640 1 -- 172.21.15.72:0/1812916726 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e0031ed0 con 0x7f59f015eba0 2024-04-15T15:15:21.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.473+0000 7f59e6ffd640 1 -- 172.21.15.72:0/1812916726 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59e0032880 con 0x7f59f015eba0 2024-04-15T15:15:21.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.473+0000 7f59e6ffd640 1 -- 172.21.15.72:0/1812916726 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59e0032c50 con 0x7f59f015eba0 2024-04-15T15:15:21.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.473+0000 7f59f8782640 1 -- 172.21.15.72:0/1812916726 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59f0071f70 con 0x7f59f015eba0 2024-04-15T15:15:21.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.473+0000 7f59f8782640 1 -- 172.21.15.72:0/1812916726 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59f00ae410 con 0x7f59f015eba0 2024-04-15T15:15:21.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.474+0000 7f59f8782640 1 -- 172.21.15.72:0/1812916726 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59f00689e0 con 0x7f59f015eba0 2024-04-15T15:15:21.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.474+0000 7f59e6ffd640 1 -- 172.21.15.72:0/1812916726 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f59e004d3f0 con 0x7f59f015eba0 2024-04-15T15:15:21.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.475+0000 7f59e6ffd640 1 --2- 172.21.15.72:0/1812916726 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f59d406c920 0x7f59d406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:21.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.475+0000 7f59f651e640 1 --2- 172.21.15.72:0/1812916726 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f59d406c920 0x7f59d406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:21.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.475+0000 7f59e6ffd640 1 -- 172.21.15.72:0/1812916726 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f59e00ad980 con 0x7f59f015eba0 2024-04-15T15:15:21.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.475+0000 7f59f651e640 1 --2- 172.21.15.72:0/1812916726 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f59d406c920 0x7f59d406ede0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f59f015e280 tx=0x7f59ec008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:21.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.483+0000 7f59e6ffd640 1 -- 172.21.15.72:0/1812916726 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f59f0071f70 con 0x7f59f015eba0 2024-04-15T15:15:21.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.702+0000 7f59f8782640 1 -- 172.21.15.72:0/1812916726 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f59f0074e30 con 0x7f59f015eba0 2024-04-15T15:15:21.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.703+0000 7f59e6ffd640 1 -- 172.21.15.72:0/1812916726 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7f59e0040070 con 0x7f59f015eba0 2024-04-15T15:15:21.705 INFO:teuthology.orchestra.run.smithi072.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-15T15:15:21.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.705+0000 7f59f8782640 1 -- 172.21.15.72:0/1812916726 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f59d406c920 msgr2=0x7f59d406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:21.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.705+0000 7f59f8782640 1 --2- 172.21.15.72:0/1812916726 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f59d406c920 0x7f59d406ede0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f59f015e280 tx=0x7f59ec008040 comp rx=0 tx=0).stop 2024-04-15T15:15:21.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.705+0000 7f59f8782640 1 -- 172.21.15.72:0/1812916726 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59f015eba0 msgr2=0x7f59f0151fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:21.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.705+0000 7f59f8782640 1 --2- 172.21.15.72:0/1812916726 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59f015eba0 0x7f59f0151fb0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f59e002b8c0 tx=0x7f59e00046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.705+0000 7f59f8782640 1 -- 172.21.15.72:0/1812916726 shutdown_connections 2024-04-15T15:15:21.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.705+0000 7f59f8782640 1 --2- 172.21.15.72:0/1812916726 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f59d406c920 0x7f59d406ede0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.705+0000 7f59f8782640 1 --2- 172.21.15.72:0/1812916726 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f59f0073530 0x7f59f00b0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.705+0000 7f59f8782640 1 --2- 172.21.15.72:0/1812916726 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59f015eba0 0x7f59f0151fb0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.705+0000 7f59f8782640 1 --2- 172.21.15.72:0/1812916726 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59f00af940 0x7f59f014d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.705+0000 7f59f8782640 1 -- 172.21.15.72:0/1812916726 >> 172.21.15.72:0/1812916726 conn(0x7f59f00b1dd0 msgr2=0x7f59f015d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:21.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.705+0000 7f59f8782640 1 -- 172.21.15.72:0/1812916726 shutdown_connections 2024-04-15T15:15:21.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21.706+0000 7f59f8782640 1 -- 172.21.15.72:0/1812916726 wait complete. 2024-04-15T15:15:21.722 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:15:21.722 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:15:21.722 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi072.front.sepia.ceph.com 2024-04-15T15:15:21.722 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-04-15T15:15:21.722 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-15T15:15:21.722 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:15:21.723 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-15T15:15:21.723 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:15:21.723 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-04-15T15:15:21.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:21.723 DEBUG:teuthology.orchestra.run.smithi072:> ip addr 2024-04-15T15:15:21.741 INFO:teuthology.orchestra.run.smithi072.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:15:21.741 INFO:teuthology.orchestra.run.smithi072.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:15:21.741 INFO:teuthology.orchestra.run.smithi072.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:15:21.741 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:21.742 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 ::1/128 scope host 2024-04-15T15:15:21.742 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:21.742 INFO:teuthology.orchestra.run.smithi072.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:15:21.742 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f4 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:21.742 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f0 2024-04-15T15:15:21.742 INFO:teuthology.orchestra.run.smithi072.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:15:21.742 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f5 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:21.742 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f1 2024-04-15T15:15:21.742 INFO:teuthology.orchestra.run.smithi072.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:15:21.742 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:64 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:21.742 INFO:teuthology.orchestra.run.smithi072.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:15:21.742 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:65 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:21.742 INFO:teuthology.orchestra.run.smithi072.stdout: inet 172.21.15.72/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-15T15:15:21.742 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft 42682sec preferred_lft 42682sec 2024-04-15T15:15:21.743 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::ec4:7aff:fe8f:b765/64 scope link 2024-04-15T15:15:21.743 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:21.743 INFO:teuthology.orchestra.run.smithi072.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:15:21.743 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether b6:ae:7a:2e:38:96 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:21.743 INFO:teuthology.orchestra.run.smithi072.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-15T15:15:21.743 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:21.743 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::6871:2cff:fe2a:4823/64 scope link 2024-04-15T15:15:21.743 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:21.743 INFO:teuthology.orchestra.run.smithi072.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:15:21.743 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether b6:ae:7a:2e:38:96 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:15:21.743 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::b4ae:7aff:fe2e:3896/64 scope link tentative 2024-04-15T15:15:21.743 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:21.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:21.744 DEBUG:teuthology.orchestra.run.smithi072:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:15:21.810 INFO:teuthology.orchestra.run.smithi072.stdout:1 2024-04-15T15:15:21.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:21.811 DEBUG:teuthology.orchestra.run.smithi072:> route 2024-04-15T15:15:21.867 INFO:teuthology.orchestra.run.smithi072.stdout:Kernel IP routing table 2024-04-15T15:15:21.868 INFO:teuthology.orchestra.run.smithi072.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:15:21.868 INFO:teuthology.orchestra.run.smithi072.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-15T15:15:21.868 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-15T15:15:21.868 INFO:teuthology.orchestra.run.smithi072.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:15:21.868 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:21.868 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-04-15T15:15:21.868 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-15T15:15:21.868 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-15T15:15:21.869 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-15T15:15:21.869 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-04-15T15:15:21.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:21 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:22.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:22.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:22.008 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-04-15T15:15:22.023 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-15T15:15:22.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:22.024 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list-id 2024-04-15T15:15:22.079 INFO:teuthology.orchestra.run.smithi072.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-04-15T15:15:22.079 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:22.079 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-04-15T15:15:22.079 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:15:22.080 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-04-15T15:15:22.080 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-04-15T15:15:22.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:22 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:22.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:22.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:22.206 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-15T15:15:22.234 INFO:teuthology.orchestra.run.smithi072.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:15:22.234 INFO:teuthology.orchestra.run.smithi072.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:15:22.234 INFO:teuthology.orchestra.run.smithi072.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:15:22.234 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:22.234 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 ::1/128 scope host 2024-04-15T15:15:22.235 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:22.235 INFO:teuthology.orchestra.run.smithi072.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:15:22.235 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 2a:90:f9:55:31:bc brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-15T15:15:22.235 INFO:teuthology.orchestra.run.smithi072.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-15T15:15:22.235 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:22.235 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::2890:f9ff:fe55:31bc/64 scope link 2024-04-15T15:15:22.235 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:22.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:22.236 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-15T15:15:22.305 INFO:teuthology.orchestra.run.smithi072.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:15:22.305 INFO:teuthology.orchestra.run.smithi072.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:15:22.305 INFO:teuthology.orchestra.run.smithi072.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:15:22.305 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:22.305 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 ::1/128 scope host 2024-04-15T15:15:22.305 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:22.305 INFO:teuthology.orchestra.run.smithi072.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:15:22.305 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 2a:90:f9:55:31:bc brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-15T15:15:22.305 INFO:teuthology.orchestra.run.smithi072.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-15T15:15:22.305 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:22.305 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::2890:f9ff:fe55:31bc/64 scope link 2024-04-15T15:15:22.305 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:22.307 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-04-15T15:15:22.307 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:22.307 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-04-15T15:15:22.307 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-04-15T15:15:22.307 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-15T15:15:22.307 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-04-15T15:15:22.307 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-04-15T15:15:22.307 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-04-15T15:15:22.307 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-04-15T15:15:22.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:22 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:22.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:22.477 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:22.477 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-04-15T15:15:22.478 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set brx.1 up 2024-04-15T15:15:22.478 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set dev brx.1 master ceph-brx 2024-04-15T15:15:22.478 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-04-15T15:15:22.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:22 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:22.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:22.561 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-04-15T15:15:22.561 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:15:22.561 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-04-15T15:15:22.576 INFO:teuthology.orchestra.run.smithi072.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-04-15T15:15:22.576 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:15:22.576 DEBUG:teuthology.orchestra.run.smithi072:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-04-15T15:15:22.632 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-04-15T15:15:22.701 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:15:22.758 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-04-15T15:15:22.758 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-04-15T15:15:22.758 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-04-15T15:15:22.758 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-04-15T15:15:22.758 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-04-15T15:15:22.758 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-04-15T15:15:22.758 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-04-15T15:15:22.758 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-04-15T15:15:22.758 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-04-15T15:15:22.758 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-04-15T15:15:22.758 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-04-15T15:15:22.759 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-04-15T15:15:22.759 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:15:22.759 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-04-15T15:15:22.759 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-04-15T15:15:22.759 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-04-15T15:15:22.759 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-04-15T15:15:22.759 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:15:22.759 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-04-15T15:15:22.759 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-04-15T15:15:22.759 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:15:22.759 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:15:22.759 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:15:22.759 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-04-15T15:15:22.759 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-04-15T15:15:22.759 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:15:22.760 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:15:22.760 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:15:22.760 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:15:22.760 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-04-15T15:15:22.760 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:15:22.760 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:15:22.760 INFO:teuthology.orchestra.run.smithi072.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-15T15:15:22.760 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:15:22.760 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:15:22.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:22.760 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-04-15T15:15:22.816 INFO:teuthology.orchestra.run.smithi072.stdout:43 2024-04-15T15:15:22.817 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2024-04-15T15:15:22.817 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-04-15T15:15:22.860 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-04-15T15:15:22.888 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:15:22.945 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-04-15T15:15:22.945 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-04-15T15:15:22.945 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-04-15T15:15:22.945 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-04-15T15:15:22.945 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-04-15T15:15:22.945 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-04-15T15:15:22.945 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-04-15T15:15:22.945 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-04-15T15:15:22.946 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-04-15T15:15:22.946 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-04-15T15:15:22.946 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-04-15T15:15:22.946 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-04-15T15:15:22.946 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:15:22.946 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-04-15T15:15:22.946 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-04-15T15:15:22.946 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-04-15T15:15:22.946 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-04-15T15:15:22.946 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:15:22.946 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-04-15T15:15:22.946 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-04-15T15:15:22.946 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:15:22.947 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:15:22.947 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:15:22.947 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-04-15T15:15:22.947 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-04-15T15:15:22.947 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:15:22.947 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:15:22.947 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:15:22.947 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:15:22.948 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-04-15T15:15:22.948 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:15:22.948 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:15:22.948 INFO:teuthology.orchestra.run.smithi072.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-15T15:15:22.948 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:15:22.948 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:15:22.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:22.948 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-04-15T15:15:22.949 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi072.stderr:2024-04-15T15:15:22.946+0000 7fa00e436480 -1 init, newargv = 0x560981c6bf80 newargc=15 2024-04-15T15:15:22.949 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi072.stderr:ceph-fuse[48562]: starting ceph client 2024-04-15T15:15:22.953 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi072.stderr:ceph-fuse[48562]: starting fuse 2024-04-15T15:15:22.964 INFO:teuthology.orchestra.run.smithi072.stdout:43 2024-04-15T15:15:22.964 INFO:teuthology.orchestra.run.smithi072.stdout:44 2024-04-15T15:15:22.964 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2024-04-15T15:15:22.965 DEBUG:teuthology.orchestra.run.smithi072:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:15:22.965 DEBUG:teuthology.orchestra.run.smithi072:> import glob 2024-04-15T15:15:22.965 DEBUG:teuthology.orchestra.run.smithi072:> import re 2024-04-15T15:15:22.965 DEBUG:teuthology.orchestra.run.smithi072:> import os 2024-04-15T15:15:22.965 DEBUG:teuthology.orchestra.run.smithi072:> import subprocess 2024-04-15T15:15:22.965 DEBUG:teuthology.orchestra.run.smithi072:> 2024-04-15T15:15:22.965 DEBUG:teuthology.orchestra.run.smithi072:> def _find_admin_socket(client_name): 2024-04-15T15:15:22.965 DEBUG:teuthology.orchestra.run.smithi072:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-04-15T15:15:22.965 DEBUG:teuthology.orchestra.run.smithi072:> files = glob.glob(asok_path) 2024-04-15T15:15:22.965 DEBUG:teuthology.orchestra.run.smithi072:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-04-15T15:15:22.965 DEBUG:teuthology.orchestra.run.smithi072:> 2024-04-15T15:15:22.965 DEBUG:teuthology.orchestra.run.smithi072:> # Given a non-glob path, it better be there 2024-04-15T15:15:22.965 DEBUG:teuthology.orchestra.run.smithi072:> if "*" not in asok_path: 2024-04-15T15:15:22.966 DEBUG:teuthology.orchestra.run.smithi072:> assert(len(files) == 1) 2024-04-15T15:15:22.966 DEBUG:teuthology.orchestra.run.smithi072:> return files[0] 2024-04-15T15:15:22.966 DEBUG:teuthology.orchestra.run.smithi072:> 2024-04-15T15:15:22.966 DEBUG:teuthology.orchestra.run.smithi072:> for f in files: 2024-04-15T15:15:22.966 DEBUG:teuthology.orchestra.run.smithi072:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:15:22.966 DEBUG:teuthology.orchestra.run.smithi072:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:15:22.966 DEBUG:teuthology.orchestra.run.smithi072:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:15:22.966 DEBUG:teuthology.orchestra.run.smithi072:> contents = proc_f.read() 2024-04-15T15:15:22.966 DEBUG:teuthology.orchestra.run.smithi072:> if mountpoint in contents: 2024-04-15T15:15:22.966 DEBUG:teuthology.orchestra.run.smithi072:> return f 2024-04-15T15:15:22.966 DEBUG:teuthology.orchestra.run.smithi072:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:15:22.967 DEBUG:teuthology.orchestra.run.smithi072:> 2024-04-15T15:15:22.967 DEBUG:teuthology.orchestra.run.smithi072:> print(_find_admin_socket("client.2")) 2024-04-15T15:15:22.967 DEBUG:teuthology.orchestra.run.smithi072:> ' 2024-04-15T15:15:23.088 INFO:teuthology.orchestra.run.smithi072.stdout:/var/run/ceph/ceph-client.2.48562.asok 2024-04-15T15:15:23.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:23.098 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.48562.asok 2024-04-15T15:15:23.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:23.098 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.48562.asok status 2024-04-15T15:15:23.236 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-04-15T15:15:23.236 INFO:teuthology.orchestra.run.smithi072.stdout: "metadata": { 2024-04-15T15:15:23.236 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:15:23.236 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:15:23.236 INFO:teuthology.orchestra.run.smithi072.stdout: "entity_id": "2", 2024-04-15T15:15:23.236 INFO:teuthology.orchestra.run.smithi072.stdout: "hostname": "smithi072", 2024-04-15T15:15:23.236 INFO:teuthology.orchestra.run.smithi072.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-04-15T15:15:23.236 INFO:teuthology.orchestra.run.smithi072.stdout: "pid": "48562", 2024-04-15T15:15:23.237 INFO:teuthology.orchestra.run.smithi072.stdout: "root": "/" 2024-04-15T15:15:23.237 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:15:23.237 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_count": 0, 2024-04-15T15:15:23.237 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_pinned_count": 0, 2024-04-15T15:15:23.237 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 4923, 2024-04-15T15:15:23.237 INFO:teuthology.orchestra.run.smithi072.stdout: "inst": { 2024-04-15T15:15:23.237 INFO:teuthology.orchestra.run.smithi072.stdout: "name": { 2024-04-15T15:15:23.237 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "client", 2024-04-15T15:15:23.237 INFO:teuthology.orchestra.run.smithi072.stdout: "num": 4923 2024-04-15T15:15:23.237 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:15:23.237 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-04-15T15:15:23.237 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-04-15T15:15:23.237 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.2:0", 2024-04-15T15:15:23.238 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 1997944176 2024-04-15T15:15:23.238 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:15:23.238 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:15:23.238 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-04-15T15:15:23.238 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-04-15T15:15:23.238 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.2:0", 2024-04-15T15:15:23.238 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 1997944176 2024-04-15T15:15:23.238 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:15:23.238 INFO:teuthology.orchestra.run.smithi072.stdout: "inst_str": "client.4923 192.168.0.2:0/1997944176", 2024-04-15T15:15:23.238 INFO:teuthology.orchestra.run.smithi072.stdout: "addr_str": "192.168.0.2:0/1997944176", 2024-04-15T15:15:23.238 INFO:teuthology.orchestra.run.smithi072.stdout: "inode_count": 1, 2024-04-15T15:15:23.238 INFO:teuthology.orchestra.run.smithi072.stdout: "mds_epoch": 9, 2024-04-15T15:15:23.238 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch": 24, 2024-04-15T15:15:23.238 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:15:23.239 INFO:teuthology.orchestra.run.smithi072.stdout: "blocklisted": false, 2024-04-15T15:15:23.239 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs" 2024-04-15T15:15:23.239 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-04-15T15:15:23.245 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:15:23.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.345+0000 7fe2d2175640 1 Processor -- start 2024-04-15T15:15:23.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.345+0000 7fe2d2175640 1 -- start start 2024-04-15T15:15:23.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.346+0000 7fe2d2175640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe2cc15c120 0x7fe2cc15f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:23.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.346+0000 7fe2d2175640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe2cc0cc0b0 0x7fe2cc0c68d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:23.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.346+0000 7fe2d2175640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2cc1624e0 con 0x7fe2cc15c500 2024-04-15T15:15:23.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.346+0000 7fe2d2175640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fe2cc1627c0 con 0x7fe2cc15c120 2024-04-15T15:15:23.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.346+0000 7fe2d2175640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fe2cc162aa0 con 0x7fe2cc0cc0b0 2024-04-15T15:15:23.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.346+0000 7fe2cb7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe2cc15c120 0x7fe2cc15f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:23.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.346+0000 7fe2caffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fe2cc15c500 0x7fe2cc0cb9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:48050/0 (socket says 172.21.15.72:48050) 2024-04-15T15:15:23.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.346+0000 7fe2caffd640 1 -- 172.21.15.72:0/1189403936 learned_addr learned my addr 172.21.15.72:0/1189403936 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:23.350 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.346+0000 7fe2cbfff640 1 --2- 172.21.15.72:0/1189403936 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe2cc0cc0b0 0x7fe2cc0c68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:23.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.346+0000 7fe2cb7fe640 1 -- 172.21.15.72:0/1189403936 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe2cc0cc0b0 msgr2=0x7fe2cc0c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:23.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.346+0000 7fe2cb7fe640 1 --2- 172.21.15.72:0/1189403936 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe2cc0cc0b0 0x7fe2cc0c68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:23.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.347+0000 7fe2cb7fe640 1 -- 172.21.15.72:0/1189403936 >> v1:172.21.15.72:6789/0 conn(0x7fe2cc15c500 legacy=0x7fe2cc0cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:23.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.347+0000 7fe2cb7fe640 1 -- 172.21.15.72:0/1189403936 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2cc162d80 con 0x7fe2cc15c120 2024-04-15T15:15:23.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.347+0000 7fe2cb7fe640 1 --2- 172.21.15.72:0/1189403936 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2cc15c120 0x7fe2cc15f7f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc009d90 tx=0x7fe2bc02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=826cfa84f7b4cac6 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:23.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.347+0000 7fe2ca7fc640 1 -- 172.21.15.72:0/1189403936 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc039070 con 0x7fe2cc15c120 2024-04-15T15:15:23.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.347+0000 7fe2ca7fc640 1 -- 172.21.15.72:0/1189403936 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2bc02cc50 con 0x7fe2cc15c120 2024-04-15T15:15:23.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.347+0000 7fe2ca7fc640 1 -- 172.21.15.72:0/1189403936 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc034440 con 0x7fe2cc15c120 2024-04-15T15:15:23.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.348+0000 7fe2d2175640 1 -- 172.21.15.72:0/1189403936 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2cc15c120 msgr2=0x7fe2cc15f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:23.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.348+0000 7fe2d2175640 1 --2- 172.21.15.72:0/1189403936 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2cc15c120 0x7fe2cc15f7f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc009d90 tx=0x7fe2bc02bea0 comp rx=0 tx=0).stop 2024-04-15T15:15:23.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.348+0000 7fe2d2175640 1 -- 172.21.15.72:0/1189403936 shutdown_connections 2024-04-15T15:15:23.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.348+0000 7fe2d2175640 1 --2- 172.21.15.72:0/1189403936 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe2cc0cc0b0 0x7fe2cc0c68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:23.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.348+0000 7fe2d2175640 1 --2- 172.21.15.72:0/1189403936 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2cc15c120 0x7fe2cc15f7f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:23.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.348+0000 7fe2d2175640 1 -- 172.21.15.72:0/1189403936 >> 172.21.15.72:0/1189403936 conn(0x7fe2cc0cd4a0 msgr2=0x7fe2cc0cd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:23.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.348+0000 7fe2d2175640 1 -- 172.21.15.72:0/1189403936 shutdown_connections 2024-04-15T15:15:23.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.348+0000 7fe2d2175640 1 -- 172.21.15.72:0/1189403936 wait complete. 2024-04-15T15:15:23.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.349+0000 7fe2d2175640 1 Processor -- start 2024-04-15T15:15:23.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.350+0000 7fe2d2175640 1 -- start start 2024-04-15T15:15:23.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.350+0000 7fe2d2175640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe2cc0cc0b0 0x7fe2cc14e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:23.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.350+0000 7fe2d2175640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe2cc15c120 0x7fe2cc0ca290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:23.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.350+0000 7fe2d2175640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2cc15c500 0x7fe2cc0c8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:23.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.350+0000 7fe2d2175640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe2cc0708e0 con 0x7fe2cc15c500 2024-04-15T15:15:23.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.350+0000 7fe2d2175640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fe2cc070760 con 0x7fe2cc0cc0b0 2024-04-15T15:15:23.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.350+0000 7fe2d2175640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fe2cc070a60 con 0x7fe2cc15c120 2024-04-15T15:15:23.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.350+0000 7fe2cbfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2cc15c500 0x7fe2cc0c8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:23.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.350+0000 7fe2cbfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2cc15c500 0x7fe2cc0c8400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42676/0 (socket says 172.21.15.72:42676) 2024-04-15T15:15:23.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.350+0000 7fe2cbfff640 1 -- 172.21.15.72:0/3631693795 learned_addr learned my addr 172.21.15.72:0/3631693795 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:23.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.351+0000 7fe2cb7fe640 1 --2- 172.21.15.72:0/3631693795 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe2cc0cc0b0 0x7fe2cc14e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:23.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.351+0000 7fe2caffd640 1 --2- 172.21.15.72:0/3631693795 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe2cc15c120 0x7fe2cc0ca290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:23.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.351+0000 7fe2cbfff640 1 -- 172.21.15.72:0/3631693795 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe2cc15c120 msgr2=0x7fe2cc0ca290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:23.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.351+0000 7fe2cbfff640 1 --2- 172.21.15.72:0/3631693795 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe2cc15c120 0x7fe2cc0ca290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:23.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.351+0000 7fe2cbfff640 1 -- 172.21.15.72:0/3631693795 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe2cc0cc0b0 msgr2=0x7fe2cc14e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:23.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.351+0000 7fe2cbfff640 1 --2- 172.21.15.72:0/3631693795 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe2cc0cc0b0 0x7fe2cc14e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:23.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.351+0000 7fe2cbfff640 1 -- 172.21.15.72:0/3631693795 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2cc15fe50 con 0x7fe2cc15c500 2024-04-15T15:15:23.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.351+0000 7fe2cbfff640 1 --2- 172.21.15.72:0/3631693795 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2cc15c500 0x7fe2cc0c8400 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe2c400d5a0 tx=0x7fe2c400da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:23.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.351+0000 7fe2b3fff640 1 -- 172.21.15.72:0/3631693795 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c40125f0 con 0x7fe2cc15c500 2024-04-15T15:15:23.357 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.351+0000 7fe2b3fff640 1 -- 172.21.15.72:0/3631693795 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2c4019070 con 0x7fe2cc15c500 2024-04-15T15:15:23.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.351+0000 7fe2d2175640 1 -- 172.21.15.72:0/3631693795 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2cc160130 con 0x7fe2cc15c500 2024-04-15T15:15:23.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.351+0000 7fe2b3fff640 1 -- 172.21.15.72:0/3631693795 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2c40129f0 con 0x7fe2cc15c500 2024-04-15T15:15:23.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.351+0000 7fe2d2175640 1 -- 172.21.15.72:0/3631693795 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2cc14e9f0 con 0x7fe2cc15c500 2024-04-15T15:15:23.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.352+0000 7fe2d2175640 1 -- 172.21.15.72:0/3631693795 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2cc0689e0 con 0x7fe2cc15c500 2024-04-15T15:15:23.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.353+0000 7fe2b3fff640 1 -- 172.21.15.72:0/3631693795 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe2c4012b90 con 0x7fe2cc15c500 2024-04-15T15:15:23.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.353+0000 7fe2b3fff640 1 --2- 172.21.15.72:0/3631693795 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe2b806c8f0 0x7fe2b806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:23.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.353+0000 7fe2cb7fe640 1 --2- 172.21.15.72:0/3631693795 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe2b806c8f0 0x7fe2b806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:23.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.353+0000 7fe2b3fff640 1 -- 172.21.15.72:0/3631693795 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fe2c408c0d0 con 0x7fe2cc15c500 2024-04-15T15:15:23.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.354+0000 7fe2cb7fe640 1 --2- 172.21.15.72:0/3631693795 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe2b806c8f0 0x7fe2b806edb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc036000 tx=0x7fe2bc030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:23.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.357+0000 7fe2b3fff640 1 -- 172.21.15.72:0/3631693795 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe2c4059b80 con 0x7fe2cc15c500 2024-04-15T15:15:23.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.571+0000 7fe2d2175640 1 -- 172.21.15.72:0/3631693795 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fe2cc0cec90 con 0x7fe2cc15c500 2024-04-15T15:15:23.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.572+0000 7fe2b3fff640 1 -- 172.21.15.72:0/3631693795 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7fe2c405de80 con 0x7fe2cc15c500 2024-04-15T15:15:23.574 INFO:teuthology.orchestra.run.smithi072.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-15T15:15:23.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.574+0000 7fe2d2175640 1 -- 172.21.15.72:0/3631693795 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe2b806c8f0 msgr2=0x7fe2b806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:23.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.574+0000 7fe2d2175640 1 --2- 172.21.15.72:0/3631693795 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe2b806c8f0 0x7fe2b806edb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc036000 tx=0x7fe2bc030000 comp rx=0 tx=0).stop 2024-04-15T15:15:23.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.574+0000 7fe2d2175640 1 -- 172.21.15.72:0/3631693795 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2cc15c500 msgr2=0x7fe2cc0c8400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:23.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.574+0000 7fe2d2175640 1 --2- 172.21.15.72:0/3631693795 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2cc15c500 0x7fe2cc0c8400 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe2c400d5a0 tx=0x7fe2c400da60 comp rx=0 tx=0).stop 2024-04-15T15:15:23.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.574+0000 7fe2d2175640 1 -- 172.21.15.72:0/3631693795 shutdown_connections 2024-04-15T15:15:23.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.574+0000 7fe2d2175640 1 --2- 172.21.15.72:0/3631693795 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe2b806c8f0 0x7fe2b806edb0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:23.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.574+0000 7fe2d2175640 1 --2- 172.21.15.72:0/3631693795 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2cc15c500 0x7fe2cc0c8400 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:23.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.574+0000 7fe2d2175640 1 --2- 172.21.15.72:0/3631693795 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe2cc15c120 0x7fe2cc0ca290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:23.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.574+0000 7fe2d2175640 1 --2- 172.21.15.72:0/3631693795 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe2cc0cc0b0 0x7fe2cc14e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:23.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.574+0000 7fe2d2175640 1 -- 172.21.15.72:0/3631693795 >> 172.21.15.72:0/3631693795 conn(0x7fe2cc0cd4a0 msgr2=0x7fe2cc0b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:23.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.574+0000 7fe2d2175640 1 -- 172.21.15.72:0/3631693795 shutdown_connections 2024-04-15T15:15:23.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:23.574+0000 7fe2d2175640 1 -- 172.21.15.72:0/3631693795 wait complete. 2024-04-15T15:15:23.587 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:15:23.587 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:15:23.587 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi175.front.sepia.ceph.com 2024-04-15T15:15:23.587 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-15T15:15:23.587 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-15T15:15:23.587 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:15:23.587 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-15T15:15:23.588 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:15:23.588 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-15T15:15:23.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:23.588 DEBUG:teuthology.orchestra.run.smithi175:> ip addr 2024-04-15T15:15:23.608 INFO:teuthology.orchestra.run.smithi175.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:15:23.608 INFO:teuthology.orchestra.run.smithi175.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:15:23.608 INFO:teuthology.orchestra.run.smithi175.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:15:23.608 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:23.608 INFO:teuthology.orchestra.run.smithi175.stdout: inet6 ::1/128 scope host 2024-04-15T15:15:23.608 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:23.608 INFO:teuthology.orchestra.run.smithi175.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:15:23.608 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether 0c:c4:7a:d9:39:ee brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:23.608 INFO:teuthology.orchestra.run.smithi175.stdout: altname enp7s0f0 2024-04-15T15:15:23.608 INFO:teuthology.orchestra.run.smithi175.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:15:23.609 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether 0c:c4:7a:d9:39:ef brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:23.609 INFO:teuthology.orchestra.run.smithi175.stdout: altname enp7s0f1 2024-04-15T15:15:23.609 INFO:teuthology.orchestra.run.smithi175.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:15:23.609 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether 0c:c4:7a:88:71:f0 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:23.609 INFO:teuthology.orchestra.run.smithi175.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:15:23.609 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether 0c:c4:7a:88:71:f1 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:23.609 INFO:teuthology.orchestra.run.smithi175.stdout: inet 172.21.15.175/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-15T15:15:23.609 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft 42674sec preferred_lft 42674sec 2024-04-15T15:15:23.609 INFO:teuthology.orchestra.run.smithi175.stdout: inet6 fe80::ec4:7aff:fe88:71f1/64 scope link 2024-04-15T15:15:23.609 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:23.609 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-15T15:15:23.610 DEBUG:teuthology.orchestra.run.smithi175:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:23.610 DEBUG:teuthology.orchestra.run.smithi175:> set -e 2024-04-15T15:15:23.610 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip link add name ceph-brx type bridge 2024-04-15T15:15:23.610 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip addr flush dev ceph-brx 2024-04-15T15:15:23.610 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip link set ceph-brx up 2024-04-15T15:15:23.610 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-15T15:15:23.610 DEBUG:teuthology.orchestra.run.smithi175:> ') 2024-04-15T15:15:23.704 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:15:23 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:23.882 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:15:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:23.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:23.890 DEBUG:teuthology.orchestra.run.smithi175:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:15:23.918 INFO:teuthology.orchestra.run.smithi175.stdout:1 2024-04-15T15:15:23.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:23.920 DEBUG:teuthology.orchestra.run.smithi175:> route 2024-04-15T15:15:23.986 INFO:teuthology.orchestra.run.smithi175.stdout:Kernel IP routing table 2024-04-15T15:15:23.987 INFO:teuthology.orchestra.run.smithi175.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:15:23.987 INFO:teuthology.orchestra.run.smithi175.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-15T15:15:23.987 INFO:teuthology.orchestra.run.smithi175.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-15T15:15:23.987 INFO:teuthology.orchestra.run.smithi175.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:15:23.987 DEBUG:teuthology.orchestra.run.smithi175:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:23.987 DEBUG:teuthology.orchestra.run.smithi175:> set -e 2024-04-15T15:15:23.987 DEBUG:teuthology.orchestra.run.smithi175:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-15T15:15:23.987 DEBUG:teuthology.orchestra.run.smithi175:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-15T15:15:23.987 DEBUG:teuthology.orchestra.run.smithi175:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-15T15:15:23.987 DEBUG:teuthology.orchestra.run.smithi175:> ') 2024-04-15T15:15:24.076 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:15:24 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:24.326 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:15:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:24.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:24.333 DEBUG:teuthology.orchestra.run.smithi175:> ip netns list 2024-04-15T15:15:24.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:24.349 DEBUG:teuthology.orchestra.run.smithi175:> ip netns list-id 2024-04-15T15:15:24.405 DEBUG:teuthology.orchestra.run.smithi175:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:24.406 DEBUG:teuthology.orchestra.run.smithi175:> set -e 2024-04-15T15:15:24.406 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:15:24.406 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-15T15:15:24.406 DEBUG:teuthology.orchestra.run.smithi175:> ') 2024-04-15T15:15:24.495 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:15:24 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:24.525 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:15:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:24.532 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-15T15:15:24.532 DEBUG:teuthology.orchestra.run.smithi175:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:24.532 DEBUG:teuthology.orchestra.run.smithi175:> set -e 2024-04-15T15:15:24.532 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-15T15:15:24.532 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-15T15:15:24.532 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-15T15:15:24.532 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-15T15:15:24.532 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-15T15:15:24.532 DEBUG:teuthology.orchestra.run.smithi175:> ') 2024-04-15T15:15:24.581 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:15:24 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:24.697 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:15:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:24.702 DEBUG:teuthology.orchestra.run.smithi175:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:24.702 DEBUG:teuthology.orchestra.run.smithi175:> set -e 2024-04-15T15:15:24.703 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip link set brx.0 up 2024-04-15T15:15:24.703 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip link set dev brx.0 master ceph-brx 2024-04-15T15:15:24.703 DEBUG:teuthology.orchestra.run.smithi175:> ') 2024-04-15T15:15:24.751 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:15:24 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:24.781 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:15:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:24.787 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-15T15:15:24.787 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:15:24.787 DEBUG:teuthology.orchestra.run.smithi175:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-15T15:15:24.802 INFO:teuthology.orchestra.run.smithi175.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-15T15:15:24.803 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:15:24.803 DEBUG:teuthology.orchestra.run.smithi175:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-15T15:15:24.857 DEBUG:teuthology.orchestra.run.smithi175:> sudo modprobe fuse 2024-04-15T15:15:24.927 DEBUG:teuthology.orchestra.run.smithi175:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:15:24.983 INFO:teuthology.orchestra.run.smithi175.stdout:/proc 2024-04-15T15:15:24.983 INFO:teuthology.orchestra.run.smithi175.stdout:/sys 2024-04-15T15:15:24.983 INFO:teuthology.orchestra.run.smithi175.stdout:/dev 2024-04-15T15:15:24.983 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/security 2024-04-15T15:15:24.983 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/shm 2024-04-15T15:15:24.983 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/pts 2024-04-15T15:15:24.983 INFO:teuthology.orchestra.run.smithi175.stdout:/run 2024-04-15T15:15:24.983 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/cgroup 2024-04-15T15:15:24.983 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/pstore 2024-04-15T15:15:24.983 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/bpf 2024-04-15T15:15:24.983 INFO:teuthology.orchestra.run.smithi175.stdout:/ 2024-04-15T15:15:24.984 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/selinux 2024-04-15T15:15:24.984 INFO:teuthology.orchestra.run.smithi175.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:15:24.984 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/mqueue 2024-04-15T15:15:24.984 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/hugepages 2024-04-15T15:15:24.984 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/debug 2024-04-15T15:15:24.984 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/tracing 2024-04-15T15:15:24.984 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/fuse/connections 2024-04-15T15:15:24.984 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/config 2024-04-15T15:15:24.984 INFO:teuthology.orchestra.run.smithi175.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:15:24.984 INFO:teuthology.orchestra.run.smithi175.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:15:24.984 INFO:teuthology.orchestra.run.smithi175.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:15:24.984 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:15:24.984 INFO:teuthology.orchestra.run.smithi175.stdout:/run/user/1000 2024-04-15T15:15:24.984 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph 2024-04-15T15:15:24.984 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:15:24.985 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:15:24.985 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:15:24.985 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:15:24.985 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns 2024-04-15T15:15:24.985 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:15:24.985 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:15:24.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:24.985 DEBUG:teuthology.orchestra.run.smithi175:> ls /sys/fs/fuse/connections 2024-04-15T15:15:25.039 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-15T15:15:25.039 DEBUG:teuthology.orchestra.run.smithi175:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-04-15T15:15:25.082 DEBUG:teuthology.orchestra.run.smithi175:> sudo modprobe fuse 2024-04-15T15:15:25.111 DEBUG:teuthology.orchestra.run.smithi175:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:15:25.168 INFO:teuthology.orchestra.run.smithi175.stdout:/proc 2024-04-15T15:15:25.168 INFO:teuthology.orchestra.run.smithi175.stdout:/sys 2024-04-15T15:15:25.168 INFO:teuthology.orchestra.run.smithi175.stdout:/dev 2024-04-15T15:15:25.168 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/security 2024-04-15T15:15:25.168 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/shm 2024-04-15T15:15:25.168 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/pts 2024-04-15T15:15:25.168 INFO:teuthology.orchestra.run.smithi175.stdout:/run 2024-04-15T15:15:25.168 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/cgroup 2024-04-15T15:15:25.168 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/pstore 2024-04-15T15:15:25.169 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/bpf 2024-04-15T15:15:25.169 INFO:teuthology.orchestra.run.smithi175.stdout:/ 2024-04-15T15:15:25.169 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/selinux 2024-04-15T15:15:25.169 INFO:teuthology.orchestra.run.smithi175.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:15:25.169 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/mqueue 2024-04-15T15:15:25.169 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/hugepages 2024-04-15T15:15:25.169 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/debug 2024-04-15T15:15:25.169 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/tracing 2024-04-15T15:15:25.169 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/fuse/connections 2024-04-15T15:15:25.169 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/config 2024-04-15T15:15:25.169 INFO:teuthology.orchestra.run.smithi175.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:15:25.169 INFO:teuthology.orchestra.run.smithi175.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:15:25.169 INFO:teuthology.orchestra.run.smithi175.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:15:25.170 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:15:25.170 INFO:teuthology.orchestra.run.smithi175.stdout:/run/user/1000 2024-04-15T15:15:25.170 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph 2024-04-15T15:15:25.170 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:15:25.170 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:15:25.170 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:15:25.170 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:15:25.170 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns 2024-04-15T15:15:25.170 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:15:25.170 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:15:25.171 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:25.171 DEBUG:teuthology.orchestra.run.smithi175:> ls /sys/fs/fuse/connections 2024-04-15T15:15:25.172 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi175.stderr:2024-04-15T15:15:25.170+0000 7f77d724a480 -1 init, newargv = 0x55964f4f9f00 newargc=15 2024-04-15T15:15:25.172 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi175.stderr:ceph-fuse[43843]: starting ceph client 2024-04-15T15:15:25.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi175.stderr:ceph-fuse[43843]: starting fuse 2024-04-15T15:15:25.188 INFO:teuthology.orchestra.run.smithi175.stdout:43 2024-04-15T15:15:25.188 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-04-15T15:15:25.188 DEBUG:teuthology.orchestra.run.smithi175:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:15:25.188 DEBUG:teuthology.orchestra.run.smithi175:> import glob 2024-04-15T15:15:25.188 DEBUG:teuthology.orchestra.run.smithi175:> import re 2024-04-15T15:15:25.188 DEBUG:teuthology.orchestra.run.smithi175:> import os 2024-04-15T15:15:25.188 DEBUG:teuthology.orchestra.run.smithi175:> import subprocess 2024-04-15T15:15:25.188 DEBUG:teuthology.orchestra.run.smithi175:> 2024-04-15T15:15:25.188 DEBUG:teuthology.orchestra.run.smithi175:> def _find_admin_socket(client_name): 2024-04-15T15:15:25.189 DEBUG:teuthology.orchestra.run.smithi175:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-15T15:15:25.189 DEBUG:teuthology.orchestra.run.smithi175:> files = glob.glob(asok_path) 2024-04-15T15:15:25.189 DEBUG:teuthology.orchestra.run.smithi175:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-15T15:15:25.189 DEBUG:teuthology.orchestra.run.smithi175:> 2024-04-15T15:15:25.189 DEBUG:teuthology.orchestra.run.smithi175:> # Given a non-glob path, it better be there 2024-04-15T15:15:25.189 DEBUG:teuthology.orchestra.run.smithi175:> if "*" not in asok_path: 2024-04-15T15:15:25.189 DEBUG:teuthology.orchestra.run.smithi175:> assert(len(files) == 1) 2024-04-15T15:15:25.189 DEBUG:teuthology.orchestra.run.smithi175:> return files[0] 2024-04-15T15:15:25.189 DEBUG:teuthology.orchestra.run.smithi175:> 2024-04-15T15:15:25.189 DEBUG:teuthology.orchestra.run.smithi175:> for f in files: 2024-04-15T15:15:25.189 DEBUG:teuthology.orchestra.run.smithi175:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:15:25.189 DEBUG:teuthology.orchestra.run.smithi175:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:15:25.189 DEBUG:teuthology.orchestra.run.smithi175:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:15:25.189 DEBUG:teuthology.orchestra.run.smithi175:> contents = proc_f.read() 2024-04-15T15:15:25.190 DEBUG:teuthology.orchestra.run.smithi175:> if mountpoint in contents: 2024-04-15T15:15:25.190 DEBUG:teuthology.orchestra.run.smithi175:> return f 2024-04-15T15:15:25.190 DEBUG:teuthology.orchestra.run.smithi175:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:15:25.190 DEBUG:teuthology.orchestra.run.smithi175:> 2024-04-15T15:15:25.190 DEBUG:teuthology.orchestra.run.smithi175:> print(_find_admin_socket("client.1")) 2024-04-15T15:15:25.190 DEBUG:teuthology.orchestra.run.smithi175:> ' 2024-04-15T15:15:25.312 INFO:teuthology.orchestra.run.smithi175.stdout:/var/run/ceph/ceph-client.1.43843.asok 2024-04-15T15:15:25.314 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:15:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:25.322 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.43843.asok 2024-04-15T15:15:25.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:25.322 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.43843.asok status 2024-04-15T15:15:25.462 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-04-15T15:15:25.462 INFO:teuthology.orchestra.run.smithi175.stdout: "metadata": { 2024-04-15T15:15:25.462 INFO:teuthology.orchestra.run.smithi175.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:15:25.463 INFO:teuthology.orchestra.run.smithi175.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:15:25.463 INFO:teuthology.orchestra.run.smithi175.stdout: "entity_id": "1", 2024-04-15T15:15:25.463 INFO:teuthology.orchestra.run.smithi175.stdout: "hostname": "smithi175", 2024-04-15T15:15:25.463 INFO:teuthology.orchestra.run.smithi175.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-15T15:15:25.463 INFO:teuthology.orchestra.run.smithi175.stdout: "pid": "43843", 2024-04-15T15:15:25.463 INFO:teuthology.orchestra.run.smithi175.stdout: "root": "/" 2024-04-15T15:15:25.463 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-15T15:15:25.463 INFO:teuthology.orchestra.run.smithi175.stdout: "dentry_count": 0, 2024-04-15T15:15:25.463 INFO:teuthology.orchestra.run.smithi175.stdout: "dentry_pinned_count": 0, 2024-04-15T15:15:25.463 INFO:teuthology.orchestra.run.smithi175.stdout: "id": 4708, 2024-04-15T15:15:25.463 INFO:teuthology.orchestra.run.smithi175.stdout: "inst": { 2024-04-15T15:15:25.463 INFO:teuthology.orchestra.run.smithi175.stdout: "name": { 2024-04-15T15:15:25.463 INFO:teuthology.orchestra.run.smithi175.stdout: "type": "client", 2024-04-15T15:15:25.463 INFO:teuthology.orchestra.run.smithi175.stdout: "num": 4708 2024-04-15T15:15:25.463 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-15T15:15:25.464 INFO:teuthology.orchestra.run.smithi175.stdout: "addr": { 2024-04-15T15:15:25.464 INFO:teuthology.orchestra.run.smithi175.stdout: "type": "v1", 2024-04-15T15:15:25.464 INFO:teuthology.orchestra.run.smithi175.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:15:25.464 INFO:teuthology.orchestra.run.smithi175.stdout: "nonce": 985028357 2024-04-15T15:15:25.464 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-15T15:15:25.464 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-15T15:15:25.464 INFO:teuthology.orchestra.run.smithi175.stdout: "addr": { 2024-04-15T15:15:25.464 INFO:teuthology.orchestra.run.smithi175.stdout: "type": "v1", 2024-04-15T15:15:25.464 INFO:teuthology.orchestra.run.smithi175.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:15:25.464 INFO:teuthology.orchestra.run.smithi175.stdout: "nonce": 985028357 2024-04-15T15:15:25.464 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-15T15:15:25.464 INFO:teuthology.orchestra.run.smithi175.stdout: "inst_str": "client.4708 192.168.0.1:0/985028357", 2024-04-15T15:15:25.464 INFO:teuthology.orchestra.run.smithi175.stdout: "addr_str": "192.168.0.1:0/985028357", 2024-04-15T15:15:25.464 INFO:teuthology.orchestra.run.smithi175.stdout: "inode_count": 1, 2024-04-15T15:15:25.464 INFO:teuthology.orchestra.run.smithi175.stdout: "mds_epoch": 9, 2024-04-15T15:15:25.465 INFO:teuthology.orchestra.run.smithi175.stdout: "osd_epoch": 24, 2024-04-15T15:15:25.465 INFO:teuthology.orchestra.run.smithi175.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:15:25.465 INFO:teuthology.orchestra.run.smithi175.stdout: "blocklisted": false, 2024-04-15T15:15:25.465 INFO:teuthology.orchestra.run.smithi175.stdout: "fs_name": "cephfs" 2024-04-15T15:15:25.465 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-04-15T15:15:25.471 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:15:25.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.582+0000 7fa9b6ec8640 1 Processor -- start 2024-04-15T15:15:25.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.582+0000 7fa9b6ec8640 1 -- start start 2024-04-15T15:15:25.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.583+0000 7fa9b6ec8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa9b01569c0 0x7fa9b015d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:25.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.583+0000 7fa9b6ec8640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa9b0156da0 0x7fa9b00ccb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:25.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.583+0000 7fa9b6ec8640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9b015da80 con 0x7fa9b00cd0b0 2024-04-15T15:15:25.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.583+0000 7fa9b6ec8640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fa9b00cbad0 con 0x7fa9b01569c0 2024-04-15T15:15:25.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.583+0000 7fa9b6ec8640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fa9b00cbdb0 con 0x7fa9b0156da0 2024-04-15T15:15:25.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.583+0000 7fa9b4c64640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa9b01569c0 0x7fa9b015d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:25.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.583+0000 7fa9b4c64640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa9b01569c0 0x7fa9b015d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42680/0 (socket says 172.21.15.72:42680) 2024-04-15T15:15:25.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.583+0000 7fa9b4c64640 1 -- 172.21.15.72:0/3493566552 learned_addr learned my addr 172.21.15.72:0/3493566552 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:25.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.583+0000 7fa9affff640 1 --2- 172.21.15.72:0/3493566552 >> v2:172.21.15.175:3300/0 conn(0x7fa9b0156da0 0x7fa9b00ccb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:25.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.583+0000 7fa9b4c64640 1 -- 172.21.15.72:0/3493566552 >> v2:172.21.15.175:3300/0 conn(0x7fa9b0156da0 msgr2=0x7fa9b00ccb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:25.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.584+0000 7fa9b4c64640 1 --2- 172.21.15.72:0/3493566552 >> v2:172.21.15.175:3300/0 conn(0x7fa9b0156da0 0x7fa9b00ccb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:25.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.584+0000 7fa9b4c64640 1 -- 172.21.15.72:0/3493566552 >> v1:172.21.15.175:6789/0 conn(0x7fa9b00cd0b0 legacy=0x7fa9b006c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:25.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.584+0000 7fa9b4c64640 1 -- 172.21.15.72:0/3493566552 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9b00cc090 con 0x7fa9b01569c0 2024-04-15T15:15:25.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.584+0000 7fa9b4c64640 1 --2- 172.21.15.72:0/3493566552 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa9b01569c0 0x7fa9b015d540 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa9b00ce070 tx=0x7fa9a402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7698df52827b113 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:25.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.584+0000 7fa9af7fe640 1 -- 172.21.15.72:0/3493566552 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a4035020 con 0x7fa9b01569c0 2024-04-15T15:15:25.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.584+0000 7fa9af7fe640 1 -- 172.21.15.72:0/3493566552 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9a4031870 con 0x7fa9b01569c0 2024-04-15T15:15:25.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.584+0000 7fa9af7fe640 1 -- 172.21.15.72:0/3493566552 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a4031c40 con 0x7fa9b01569c0 2024-04-15T15:15:25.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.585+0000 7fa9b6ec8640 1 -- 172.21.15.72:0/3493566552 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa9b01569c0 msgr2=0x7fa9b015d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:25.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.585+0000 7fa9b6ec8640 1 --2- 172.21.15.72:0/3493566552 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa9b01569c0 0x7fa9b015d540 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa9b00ce070 tx=0x7fa9a402b910 comp rx=0 tx=0).stop 2024-04-15T15:15:25.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.585+0000 7fa9b6ec8640 1 -- 172.21.15.72:0/3493566552 shutdown_connections 2024-04-15T15:15:25.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.585+0000 7fa9b6ec8640 1 --2- 172.21.15.72:0/3493566552 >> v2:172.21.15.175:3300/0 conn(0x7fa9b0156da0 0x7fa9b00ccb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:25.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.585+0000 7fa9b6ec8640 1 --2- 172.21.15.72:0/3493566552 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa9b01569c0 0x7fa9b015d540 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:25.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.585+0000 7fa9b6ec8640 1 -- 172.21.15.72:0/3493566552 >> 172.21.15.72:0/3493566552 conn(0x7fa9b00cd4a0 msgr2=0x7fa9b00cd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:25.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.585+0000 7fa9b6ec8640 1 -- 172.21.15.72:0/3493566552 shutdown_connections 2024-04-15T15:15:25.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.585+0000 7fa9b6ec8640 1 -- 172.21.15.72:0/3493566552 wait complete. 2024-04-15T15:15:25.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.586+0000 7fa9b6ec8640 1 Processor -- start 2024-04-15T15:15:25.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.586+0000 7fa9b6ec8640 1 -- start start 2024-04-15T15:15:25.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.587+0000 7fa9b6ec8640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa9b01569c0 0x7fa9b014db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:25.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.587+0000 7fa9b6ec8640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa9b0156da0 0x7fa9b00c9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:25.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.587+0000 7fa9b6ec8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa9b00ca1c0 0x7fa9b014eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:25.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.587+0000 7fa9b6ec8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b00b01a0 con 0x7fa9b00ca1c0 2024-04-15T15:15:25.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.587+0000 7fa9b6ec8640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b00cc650 con 0x7fa9b0156da0 2024-04-15T15:15:25.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.587+0000 7fa9b6ec8640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fa9b00cb750 con 0x7fa9b01569c0 2024-04-15T15:15:25.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.587+0000 7fa9b5465640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa9b00ca1c0 0x7fa9b014eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:25.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.587+0000 7fa9b5465640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa9b00ca1c0 0x7fa9b014eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42694/0 (socket says 172.21.15.72:42694) 2024-04-15T15:15:25.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.587+0000 7fa9b5465640 1 -- 172.21.15.72:0/2386563047 learned_addr learned my addr 172.21.15.72:0/2386563047 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:25.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.588+0000 7fa9b4c64640 1 --2- 172.21.15.72:0/2386563047 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa9b01569c0 0x7fa9b014db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:25.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.588+0000 7fa9affff640 1 --2- 172.21.15.72:0/2386563047 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa9b0156da0 0x7fa9b00c9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:25.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.588+0000 7fa9b5465640 1 -- 172.21.15.72:0/2386563047 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa9b01569c0 msgr2=0x7fa9b014db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:25.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.588+0000 7fa9b5465640 1 --2- 172.21.15.72:0/2386563047 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa9b01569c0 0x7fa9b014db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:25.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.588+0000 7fa9b5465640 1 -- 172.21.15.72:0/2386563047 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa9b0156da0 msgr2=0x7fa9b00c9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:25.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.588+0000 7fa9b5465640 1 --2- 172.21.15.72:0/2386563047 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa9b0156da0 0x7fa9b00c9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:25.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.588+0000 7fa9b5465640 1 -- 172.21.15.72:0/2386563047 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9b00c8be0 con 0x7fa9b00ca1c0 2024-04-15T15:15:25.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.588+0000 7fa9affff640 1 --2- 172.21.15.72:0/2386563047 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa9b0156da0 0x7fa9b00c9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:25.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.588+0000 7fa9b5465640 1 --2- 172.21.15.72:0/2386563047 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa9b00ca1c0 0x7fa9b014eff0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa9a800c780 tx=0x7fa9a800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:25.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.588+0000 7fa9ad7fa640 1 -- 172.21.15.72:0/2386563047 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a8017070 con 0x7fa9b00ca1c0 2024-04-15T15:15:25.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.588+0000 7fa9ad7fa640 1 -- 172.21.15.72:0/2386563047 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9a8003da0 con 0x7fa9b00ca1c0 2024-04-15T15:15:25.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.588+0000 7fa9ad7fa640 1 -- 172.21.15.72:0/2386563047 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a80052b0 con 0x7fa9b00ca1c0 2024-04-15T15:15:25.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.588+0000 7fa9b6ec8640 1 -- 172.21.15.72:0/2386563047 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9b00c8ec0 con 0x7fa9b00ca1c0 2024-04-15T15:15:25.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.588+0000 7fa9b6ec8640 1 -- 172.21.15.72:0/2386563047 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9b00c9480 con 0x7fa9b00ca1c0 2024-04-15T15:15:25.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.589+0000 7fa9b6ec8640 1 -- 172.21.15.72:0/2386563047 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9b00689e0 con 0x7fa9b00ca1c0 2024-04-15T15:15:25.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.590+0000 7fa9ad7fa640 1 -- 172.21.15.72:0/2386563047 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa9a8005450 con 0x7fa9b00ca1c0 2024-04-15T15:15:25.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.590+0000 7fa9ad7fa640 1 --2- 172.21.15.72:0/2386563047 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa99406c920 0x7fa99406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:25.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.590+0000 7fa9ad7fa640 1 -- 172.21.15.72:0/2386563047 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa9a808b230 con 0x7fa9b00ca1c0 2024-04-15T15:15:25.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.590+0000 7fa9b4c64640 1 --2- 172.21.15.72:0/2386563047 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa99406c920 0x7fa99406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:25.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.591+0000 7fa9b4c64640 1 --2- 172.21.15.72:0/2386563047 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa99406c920 0x7fa99406ede0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4036000 tx=0x7fa9a403c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:25.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.595+0000 7fa9ad7fa640 1 -- 172.21.15.72:0/2386563047 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa9a8058cd0 con 0x7fa9b00ca1c0 2024-04-15T15:15:25.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.807+0000 7fa9b6ec8640 1 -- 172.21.15.72:0/2386563047 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fa9b00cecd0 con 0x7fa9b00ca1c0 2024-04-15T15:15:25.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.808+0000 7fa9ad7fa640 1 -- 172.21.15.72:0/2386563047 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) ==== 53+0+88 (secure 0 0 0) 0x7fa9a805cfd0 con 0x7fa9b00ca1c0 2024-04-15T15:15:25.810 INFO:teuthology.orchestra.run.smithi072.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-15T15:15:25.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.809+0000 7fa9b6ec8640 1 -- 172.21.15.72:0/2386563047 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa99406c920 msgr2=0x7fa99406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:25.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.809+0000 7fa9b6ec8640 1 --2- 172.21.15.72:0/2386563047 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa99406c920 0x7fa99406ede0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4036000 tx=0x7fa9a403c040 comp rx=0 tx=0).stop 2024-04-15T15:15:25.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.809+0000 7fa9b6ec8640 1 -- 172.21.15.72:0/2386563047 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa9b00ca1c0 msgr2=0x7fa9b014eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:25.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.809+0000 7fa9b6ec8640 1 --2- 172.21.15.72:0/2386563047 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa9b00ca1c0 0x7fa9b014eff0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa9a800c780 tx=0x7fa9a800cc40 comp rx=0 tx=0).stop 2024-04-15T15:15:25.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.810+0000 7fa9b6ec8640 1 -- 172.21.15.72:0/2386563047 shutdown_connections 2024-04-15T15:15:25.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.810+0000 7fa9b6ec8640 1 --2- 172.21.15.72:0/2386563047 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa99406c920 0x7fa99406ede0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:25.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.810+0000 7fa9b6ec8640 1 --2- 172.21.15.72:0/2386563047 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa9b00ca1c0 0x7fa9b014eff0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:25.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.810+0000 7fa9b6ec8640 1 --2- 172.21.15.72:0/2386563047 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa9b0156da0 0x7fa9b00c9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:25.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.810+0000 7fa9b6ec8640 1 --2- 172.21.15.72:0/2386563047 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa9b01569c0 0x7fa9b014db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:25.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.810+0000 7fa9b6ec8640 1 -- 172.21.15.72:0/2386563047 >> 172.21.15.72:0/2386563047 conn(0x7fa9b00cd4a0 msgr2=0x7fa9b00b1580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:25.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.810+0000 7fa9b6ec8640 1 -- 172.21.15.72:0/2386563047 shutdown_connections 2024-04-15T15:15:25.813 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:25.810+0000 7fa9b6ec8640 1 -- 172.21.15.72:0/2386563047 wait complete. 2024-04-15T15:15:25.823 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:15:25.823 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:15:25.823 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi175.front.sepia.ceph.com 2024-04-15T15:15:25.823 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-04-15T15:15:25.823 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-15T15:15:25.824 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:15:25.824 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-15T15:15:25.824 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:15:25.824 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-04-15T15:15:25.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:25.824 DEBUG:teuthology.orchestra.run.smithi175:> ip addr 2024-04-15T15:15:25.841 INFO:teuthology.orchestra.run.smithi175.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:15:25.841 INFO:teuthology.orchestra.run.smithi175.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:15:25.841 INFO:teuthology.orchestra.run.smithi175.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:15:25.842 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:25.842 INFO:teuthology.orchestra.run.smithi175.stdout: inet6 ::1/128 scope host 2024-04-15T15:15:25.842 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:25.842 INFO:teuthology.orchestra.run.smithi175.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:15:25.842 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether 0c:c4:7a:d9:39:ee brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:25.842 INFO:teuthology.orchestra.run.smithi175.stdout: altname enp7s0f0 2024-04-15T15:15:25.842 INFO:teuthology.orchestra.run.smithi175.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:15:25.842 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether 0c:c4:7a:d9:39:ef brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:25.842 INFO:teuthology.orchestra.run.smithi175.stdout: altname enp7s0f1 2024-04-15T15:15:25.842 INFO:teuthology.orchestra.run.smithi175.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:15:25.842 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether 0c:c4:7a:88:71:f0 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:25.842 INFO:teuthology.orchestra.run.smithi175.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:15:25.842 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether 0c:c4:7a:88:71:f1 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:25.842 INFO:teuthology.orchestra.run.smithi175.stdout: inet 172.21.15.175/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-15T15:15:25.842 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft 42672sec preferred_lft 42672sec 2024-04-15T15:15:25.843 INFO:teuthology.orchestra.run.smithi175.stdout: inet6 fe80::ec4:7aff:fe88:71f1/64 scope link 2024-04-15T15:15:25.843 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:25.843 INFO:teuthology.orchestra.run.smithi175.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:15:25.843 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether f2:c6:d9:f6:5e:3e brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:25.843 INFO:teuthology.orchestra.run.smithi175.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-15T15:15:25.843 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:25.843 INFO:teuthology.orchestra.run.smithi175.stdout: inet6 fe80::d4be:d9ff:fe0d:518c/64 scope link 2024-04-15T15:15:25.843 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:25.843 INFO:teuthology.orchestra.run.smithi175.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:15:25.843 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether f2:c6:d9:f6:5e:3e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:15:25.843 INFO:teuthology.orchestra.run.smithi175.stdout: inet6 fe80::f0c6:d9ff:fef6:5e3e/64 scope link tentative 2024-04-15T15:15:25.843 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:25.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:25.844 DEBUG:teuthology.orchestra.run.smithi175:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:15:25.910 INFO:teuthology.orchestra.run.smithi175.stdout:1 2024-04-15T15:15:25.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:25.912 DEBUG:teuthology.orchestra.run.smithi175:> route 2024-04-15T15:15:25.969 INFO:teuthology.orchestra.run.smithi175.stdout:Kernel IP routing table 2024-04-15T15:15:25.969 INFO:teuthology.orchestra.run.smithi175.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:15:25.969 INFO:teuthology.orchestra.run.smithi175.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-15T15:15:25.969 INFO:teuthology.orchestra.run.smithi175.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-15T15:15:25.969 INFO:teuthology.orchestra.run.smithi175.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:15:25.970 DEBUG:teuthology.orchestra.run.smithi175:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:25.970 DEBUG:teuthology.orchestra.run.smithi175:> set -e 2024-04-15T15:15:25.970 DEBUG:teuthology.orchestra.run.smithi175:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-15T15:15:25.970 DEBUG:teuthology.orchestra.run.smithi175:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-15T15:15:25.970 DEBUG:teuthology.orchestra.run.smithi175:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-15T15:15:25.970 DEBUG:teuthology.orchestra.run.smithi175:> ') 2024-04-15T15:15:26.060 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:15:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:26.104 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:15:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:26.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:26.111 DEBUG:teuthology.orchestra.run.smithi175:> ip netns list 2024-04-15T15:15:26.127 INFO:teuthology.orchestra.run.smithi175.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-15T15:15:26.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:26.127 DEBUG:teuthology.orchestra.run.smithi175:> ip netns list-id 2024-04-15T15:15:26.182 INFO:teuthology.orchestra.run.smithi175.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-04-15T15:15:26.183 DEBUG:teuthology.orchestra.run.smithi175:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:26.183 DEBUG:teuthology.orchestra.run.smithi175:> set -e 2024-04-15T15:15:26.183 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:15:26.183 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-04-15T15:15:26.183 DEBUG:teuthology.orchestra.run.smithi175:> ') 2024-04-15T15:15:26.271 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:15:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:26.300 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:15:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:26.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:26.307 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-15T15:15:26.335 INFO:teuthology.orchestra.run.smithi175.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:15:26.335 INFO:teuthology.orchestra.run.smithi175.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:15:26.335 INFO:teuthology.orchestra.run.smithi175.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:15:26.335 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:26.335 INFO:teuthology.orchestra.run.smithi175.stdout: inet6 ::1/128 scope host 2024-04-15T15:15:26.335 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:26.335 INFO:teuthology.orchestra.run.smithi175.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:15:26.335 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether b2:e4:c8:02:9b:ca brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-15T15:15:26.335 INFO:teuthology.orchestra.run.smithi175.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-15T15:15:26.336 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:26.336 INFO:teuthology.orchestra.run.smithi175.stdout: inet6 fe80::b0e4:c8ff:fe02:9bca/64 scope link 2024-04-15T15:15:26.336 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:26.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:26.337 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-15T15:15:26.405 INFO:teuthology.orchestra.run.smithi175.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:15:26.406 INFO:teuthology.orchestra.run.smithi175.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:15:26.406 INFO:teuthology.orchestra.run.smithi175.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:15:26.406 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:26.406 INFO:teuthology.orchestra.run.smithi175.stdout: inet6 ::1/128 scope host 2024-04-15T15:15:26.406 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:26.406 INFO:teuthology.orchestra.run.smithi175.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:15:26.406 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether b2:e4:c8:02:9b:ca brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-15T15:15:26.406 INFO:teuthology.orchestra.run.smithi175.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-15T15:15:26.406 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:26.406 INFO:teuthology.orchestra.run.smithi175.stdout: inet6 fe80::b0e4:c8ff:fe02:9bca/64 scope link 2024-04-15T15:15:26.406 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:26.410 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-04-15T15:15:26.410 DEBUG:teuthology.orchestra.run.smithi175:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:26.411 DEBUG:teuthology.orchestra.run.smithi175:> set -e 2024-04-15T15:15:26.411 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-04-15T15:15:26.411 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-15T15:15:26.411 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-04-15T15:15:26.411 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-04-15T15:15:26.411 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-04-15T15:15:26.411 DEBUG:teuthology.orchestra.run.smithi175:> ') 2024-04-15T15:15:26.499 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:15:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:26.574 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:15:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:26.580 DEBUG:teuthology.orchestra.run.smithi175:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:26.581 DEBUG:teuthology.orchestra.run.smithi175:> set -e 2024-04-15T15:15:26.581 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip link set brx.1 up 2024-04-15T15:15:26.581 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip link set dev brx.1 master ceph-brx 2024-04-15T15:15:26.581 DEBUG:teuthology.orchestra.run.smithi175:> ') 2024-04-15T15:15:26.629 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:15:26 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:26.658 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:15:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:26.664 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-04-15T15:15:26.664 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:15:26.664 DEBUG:teuthology.orchestra.run.smithi175:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-04-15T15:15:26.692 INFO:teuthology.orchestra.run.smithi175.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-04-15T15:15:26.693 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:15:26.693 DEBUG:teuthology.orchestra.run.smithi175:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-04-15T15:15:26.747 DEBUG:teuthology.orchestra.run.smithi175:> sudo modprobe fuse 2024-04-15T15:15:26.816 DEBUG:teuthology.orchestra.run.smithi175:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:15:26.873 INFO:teuthology.orchestra.run.smithi175.stdout:/proc 2024-04-15T15:15:26.873 INFO:teuthology.orchestra.run.smithi175.stdout:/sys 2024-04-15T15:15:26.873 INFO:teuthology.orchestra.run.smithi175.stdout:/dev 2024-04-15T15:15:26.873 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/security 2024-04-15T15:15:26.873 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/shm 2024-04-15T15:15:26.873 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/pts 2024-04-15T15:15:26.873 INFO:teuthology.orchestra.run.smithi175.stdout:/run 2024-04-15T15:15:26.873 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/cgroup 2024-04-15T15:15:26.874 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/pstore 2024-04-15T15:15:26.874 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/bpf 2024-04-15T15:15:26.874 INFO:teuthology.orchestra.run.smithi175.stdout:/ 2024-04-15T15:15:26.874 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/selinux 2024-04-15T15:15:26.874 INFO:teuthology.orchestra.run.smithi175.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:15:26.874 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/mqueue 2024-04-15T15:15:26.874 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/hugepages 2024-04-15T15:15:26.874 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/debug 2024-04-15T15:15:26.874 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/tracing 2024-04-15T15:15:26.874 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/fuse/connections 2024-04-15T15:15:26.874 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/config 2024-04-15T15:15:26.874 INFO:teuthology.orchestra.run.smithi175.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:15:26.874 INFO:teuthology.orchestra.run.smithi175.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:15:26.875 INFO:teuthology.orchestra.run.smithi175.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:15:26.875 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:15:26.875 INFO:teuthology.orchestra.run.smithi175.stdout:/run/user/1000 2024-04-15T15:15:26.875 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph 2024-04-15T15:15:26.875 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:15:26.875 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:15:26.875 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:15:26.875 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:15:26.875 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns 2024-04-15T15:15:26.875 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:15:26.875 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:15:26.875 INFO:teuthology.orchestra.run.smithi175.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-15T15:15:26.875 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:15:26.875 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:15:26.876 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:26.876 DEBUG:teuthology.orchestra.run.smithi175:> ls /sys/fs/fuse/connections 2024-04-15T15:15:26.929 INFO:teuthology.orchestra.run.smithi175.stdout:43 2024-04-15T15:15:26.929 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2024-04-15T15:15:26.929 DEBUG:teuthology.orchestra.run.smithi175:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-04-15T15:15:26.972 DEBUG:teuthology.orchestra.run.smithi175:> sudo modprobe fuse 2024-04-15T15:15:27.001 DEBUG:teuthology.orchestra.run.smithi175:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:15:27.058 INFO:teuthology.orchestra.run.smithi175.stdout:/proc 2024-04-15T15:15:27.058 INFO:teuthology.orchestra.run.smithi175.stdout:/sys 2024-04-15T15:15:27.058 INFO:teuthology.orchestra.run.smithi175.stdout:/dev 2024-04-15T15:15:27.058 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/security 2024-04-15T15:15:27.059 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/shm 2024-04-15T15:15:27.059 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/pts 2024-04-15T15:15:27.059 INFO:teuthology.orchestra.run.smithi175.stdout:/run 2024-04-15T15:15:27.059 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/cgroup 2024-04-15T15:15:27.059 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/pstore 2024-04-15T15:15:27.059 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/bpf 2024-04-15T15:15:27.059 INFO:teuthology.orchestra.run.smithi175.stdout:/ 2024-04-15T15:15:27.059 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/selinux 2024-04-15T15:15:27.059 INFO:teuthology.orchestra.run.smithi175.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:15:27.059 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/mqueue 2024-04-15T15:15:27.059 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/hugepages 2024-04-15T15:15:27.059 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/debug 2024-04-15T15:15:27.059 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/tracing 2024-04-15T15:15:27.059 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/fuse/connections 2024-04-15T15:15:27.060 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/config 2024-04-15T15:15:27.060 INFO:teuthology.orchestra.run.smithi175.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:15:27.060 INFO:teuthology.orchestra.run.smithi175.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:15:27.060 INFO:teuthology.orchestra.run.smithi175.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:15:27.060 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:15:27.060 INFO:teuthology.orchestra.run.smithi175.stdout:/run/user/1000 2024-04-15T15:15:27.060 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph 2024-04-15T15:15:27.060 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:15:27.060 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:15:27.060 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:15:27.060 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:15:27.060 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns 2024-04-15T15:15:27.060 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:15:27.060 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:15:27.060 INFO:teuthology.orchestra.run.smithi175.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-15T15:15:27.061 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:15:27.061 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:15:27.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:27.061 DEBUG:teuthology.orchestra.run.smithi175:> ls /sys/fs/fuse/connections 2024-04-15T15:15:27.062 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi175.stderr:2024-04-15T15:15:27.059+0000 7f8e0b84c480 -1 init, newargv = 0x5610334b7e50 newargc=15 2024-04-15T15:15:27.062 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi175.stderr:ceph-fuse[44300]: starting ceph client 2024-04-15T15:15:27.065 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi175.stderr:ceph-fuse[44300]: starting fuse 2024-04-15T15:15:27.077 INFO:teuthology.orchestra.run.smithi175.stdout:43 2024-04-15T15:15:27.077 INFO:teuthology.orchestra.run.smithi175.stdout:44 2024-04-15T15:15:27.077 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2024-04-15T15:15:27.077 DEBUG:teuthology.orchestra.run.smithi175:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:15:27.077 DEBUG:teuthology.orchestra.run.smithi175:> import glob 2024-04-15T15:15:27.077 DEBUG:teuthology.orchestra.run.smithi175:> import re 2024-04-15T15:15:27.077 DEBUG:teuthology.orchestra.run.smithi175:> import os 2024-04-15T15:15:27.078 DEBUG:teuthology.orchestra.run.smithi175:> import subprocess 2024-04-15T15:15:27.078 DEBUG:teuthology.orchestra.run.smithi175:> 2024-04-15T15:15:27.078 DEBUG:teuthology.orchestra.run.smithi175:> def _find_admin_socket(client_name): 2024-04-15T15:15:27.078 DEBUG:teuthology.orchestra.run.smithi175:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-04-15T15:15:27.078 DEBUG:teuthology.orchestra.run.smithi175:> files = glob.glob(asok_path) 2024-04-15T15:15:27.078 DEBUG:teuthology.orchestra.run.smithi175:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-04-15T15:15:27.078 DEBUG:teuthology.orchestra.run.smithi175:> 2024-04-15T15:15:27.078 DEBUG:teuthology.orchestra.run.smithi175:> # Given a non-glob path, it better be there 2024-04-15T15:15:27.078 DEBUG:teuthology.orchestra.run.smithi175:> if "*" not in asok_path: 2024-04-15T15:15:27.078 DEBUG:teuthology.orchestra.run.smithi175:> assert(len(files) == 1) 2024-04-15T15:15:27.078 DEBUG:teuthology.orchestra.run.smithi175:> return files[0] 2024-04-15T15:15:27.078 DEBUG:teuthology.orchestra.run.smithi175:> 2024-04-15T15:15:27.078 DEBUG:teuthology.orchestra.run.smithi175:> for f in files: 2024-04-15T15:15:27.078 DEBUG:teuthology.orchestra.run.smithi175:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:15:27.078 DEBUG:teuthology.orchestra.run.smithi175:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:15:27.078 DEBUG:teuthology.orchestra.run.smithi175:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:15:27.079 DEBUG:teuthology.orchestra.run.smithi175:> contents = proc_f.read() 2024-04-15T15:15:27.079 DEBUG:teuthology.orchestra.run.smithi175:> if mountpoint in contents: 2024-04-15T15:15:27.079 DEBUG:teuthology.orchestra.run.smithi175:> return f 2024-04-15T15:15:27.079 DEBUG:teuthology.orchestra.run.smithi175:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:15:27.079 DEBUG:teuthology.orchestra.run.smithi175:> 2024-04-15T15:15:27.079 DEBUG:teuthology.orchestra.run.smithi175:> print(_find_admin_socket("client.3")) 2024-04-15T15:15:27.079 DEBUG:teuthology.orchestra.run.smithi175:> ' 2024-04-15T15:15:27.199 INFO:teuthology.orchestra.run.smithi175.stdout:/var/run/ceph/ceph-client.3.44300.asok 2024-04-15T15:15:27.202 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:15:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:27.210 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.44300.asok 2024-04-15T15:15:27.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:27.210 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.44300.asok status 2024-04-15T15:15:27.349 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-04-15T15:15:27.349 INFO:teuthology.orchestra.run.smithi175.stdout: "metadata": { 2024-04-15T15:15:27.349 INFO:teuthology.orchestra.run.smithi175.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:15:27.349 INFO:teuthology.orchestra.run.smithi175.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:15:27.349 INFO:teuthology.orchestra.run.smithi175.stdout: "entity_id": "3", 2024-04-15T15:15:27.349 INFO:teuthology.orchestra.run.smithi175.stdout: "hostname": "smithi175", 2024-04-15T15:15:27.349 INFO:teuthology.orchestra.run.smithi175.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-04-15T15:15:27.349 INFO:teuthology.orchestra.run.smithi175.stdout: "pid": "44300", 2024-04-15T15:15:27.349 INFO:teuthology.orchestra.run.smithi175.stdout: "root": "/" 2024-04-15T15:15:27.350 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-15T15:15:27.350 INFO:teuthology.orchestra.run.smithi175.stdout: "dentry_count": 0, 2024-04-15T15:15:27.350 INFO:teuthology.orchestra.run.smithi175.stdout: "dentry_pinned_count": 0, 2024-04-15T15:15:27.350 INFO:teuthology.orchestra.run.smithi175.stdout: "id": 4571, 2024-04-15T15:15:27.350 INFO:teuthology.orchestra.run.smithi175.stdout: "inst": { 2024-04-15T15:15:27.350 INFO:teuthology.orchestra.run.smithi175.stdout: "name": { 2024-04-15T15:15:27.350 INFO:teuthology.orchestra.run.smithi175.stdout: "type": "client", 2024-04-15T15:15:27.350 INFO:teuthology.orchestra.run.smithi175.stdout: "num": 4571 2024-04-15T15:15:27.350 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-15T15:15:27.350 INFO:teuthology.orchestra.run.smithi175.stdout: "addr": { 2024-04-15T15:15:27.350 INFO:teuthology.orchestra.run.smithi175.stdout: "type": "v1", 2024-04-15T15:15:27.350 INFO:teuthology.orchestra.run.smithi175.stdout: "addr": "192.168.0.2:0", 2024-04-15T15:15:27.350 INFO:teuthology.orchestra.run.smithi175.stdout: "nonce": 4022620282 2024-04-15T15:15:27.351 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-15T15:15:27.351 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-15T15:15:27.351 INFO:teuthology.orchestra.run.smithi175.stdout: "addr": { 2024-04-15T15:15:27.351 INFO:teuthology.orchestra.run.smithi175.stdout: "type": "v1", 2024-04-15T15:15:27.351 INFO:teuthology.orchestra.run.smithi175.stdout: "addr": "192.168.0.2:0", 2024-04-15T15:15:27.351 INFO:teuthology.orchestra.run.smithi175.stdout: "nonce": 4022620282 2024-04-15T15:15:27.351 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-15T15:15:27.351 INFO:teuthology.orchestra.run.smithi175.stdout: "inst_str": "client.4571 192.168.0.2:0/4022620282", 2024-04-15T15:15:27.351 INFO:teuthology.orchestra.run.smithi175.stdout: "addr_str": "192.168.0.2:0/4022620282", 2024-04-15T15:15:27.351 INFO:teuthology.orchestra.run.smithi175.stdout: "inode_count": 1, 2024-04-15T15:15:27.351 INFO:teuthology.orchestra.run.smithi175.stdout: "mds_epoch": 9, 2024-04-15T15:15:27.351 INFO:teuthology.orchestra.run.smithi175.stdout: "osd_epoch": 24, 2024-04-15T15:15:27.351 INFO:teuthology.orchestra.run.smithi175.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:15:27.352 INFO:teuthology.orchestra.run.smithi175.stdout: "blocklisted": false, 2024-04-15T15:15:27.352 INFO:teuthology.orchestra.run.smithi175.stdout: "fs_name": "cephfs" 2024-04-15T15:15:27.352 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-04-15T15:15:27.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:27.358 DEBUG:teuthology.orchestra.run.smithi072:> stat --file-system '--printf=%T 2024-04-15T15:15:27.358 DEBUG:teuthology.orchestra.run.smithi072:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-15T15:15:27.378 INFO:teuthology.orchestra.run.smithi072.stdout:fuseblk 2024-04-15T15:15:27.378 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-15T15:15:27.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:27.378 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-15T15:15:27.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:27.452 DEBUG:teuthology.orchestra.run.smithi072:> stat --file-system '--printf=%T 2024-04-15T15:15:27.452 DEBUG:teuthology.orchestra.run.smithi072:> ' -- /home/ubuntu/cephtest/mnt.2 2024-04-15T15:15:27.510 INFO:teuthology.orchestra.run.smithi072.stdout:fuseblk 2024-04-15T15:15:27.510 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-04-15T15:15:27.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:27.510 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-04-15T15:15:27.582 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:27.583 DEBUG:teuthology.orchestra.run.smithi175:> stat --file-system '--printf=%T 2024-04-15T15:15:27.583 DEBUG:teuthology.orchestra.run.smithi175:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-15T15:15:27.599 INFO:teuthology.orchestra.run.smithi175.stdout:fuseblk 2024-04-15T15:15:27.599 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-15T15:15:27.600 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:27.600 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-15T15:15:27.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:27.668 DEBUG:teuthology.orchestra.run.smithi175:> stat --file-system '--printf=%T 2024-04-15T15:15:27.668 DEBUG:teuthology.orchestra.run.smithi175:> ' -- /home/ubuntu/cephtest/mnt.3 2024-04-15T15:15:27.724 INFO:teuthology.orchestra.run.smithi175.stdout:fuseblk 2024-04-15T15:15:27.725 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-04-15T15:15:27.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:27.725 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-04-15T15:15:27.793 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-04-15T15:15:27.810 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:27.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.919+0000 7f39b28c8640 1 Processor -- start 2024-04-15T15:15:27.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.919+0000 7f39b28c8640 1 -- start start 2024-04-15T15:15:27.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.920+0000 7f39b28c8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f39ac15c500 0x7f39ac15f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:27.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.920+0000 7f39b28c8640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39ac1624e0 con 0x7f39ac0cc0b0 2024-04-15T15:15:27.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.920+0000 7f39b28c8640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39ac1627c0 con 0x7f39ac15c120 2024-04-15T15:15:27.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.920+0000 7f39b28c8640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f39ac162aa0 con 0x7f39ac15c500 2024-04-15T15:15:27.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.920+0000 7f39ab7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f39ac15c500 0x7f39ac15f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:27.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.920+0000 7f39b0e65640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f39ac0cc0b0 0x7f39ac0c68d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:48054/0 (socket says 172.21.15.72:48054) 2024-04-15T15:15:27.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.920+0000 7f39b0e65640 1 -- 172.21.15.72:0/410171480 learned_addr learned my addr 172.21.15.72:0/410171480 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:27.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.921+0000 7f39ab7fe640 1 -- 172.21.15.72:0/410171480 >> v1:172.21.15.175:6789/0 conn(0x7f39ac15c120 legacy=0x7f39ac0cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:27.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.921+0000 7f39ab7fe640 1 -- 172.21.15.72:0/410171480 >> v1:172.21.15.72:6789/0 conn(0x7f39ac0cc0b0 legacy=0x7f39ac0c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:27.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.921+0000 7f39aaffd640 1 -- 172.21.15.72:0/410171480 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4086849317 0 0) 0x7f39ac1624e0 con 0x7f39ac0cc0b0 2024-04-15T15:15:27.924 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.921+0000 7f39ab7fe640 1 -- 172.21.15.72:0/410171480 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39ac162d80 con 0x7f39ac15c500 2024-04-15T15:15:27.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.921+0000 7f39ab7fe640 1 --2- 172.21.15.72:0/410171480 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39ac15c500 0x7f39ac15f7f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f39a00097a0 tx=0x7f39a002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c57c9f745ba1ce97 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:27.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.922+0000 7f39aaffd640 1 -- 172.21.15.72:0/410171480 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a0034070 con 0x7f39ac15c500 2024-04-15T15:15:27.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.922+0000 7f39aaffd640 1 -- 172.21.15.72:0/410171480 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39a0030610 con 0x7f39ac15c500 2024-04-15T15:15:27.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.922+0000 7f39aaffd640 1 -- 172.21.15.72:0/410171480 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a00309e0 con 0x7f39ac15c500 2024-04-15T15:15:27.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.922+0000 7f39b28c8640 1 -- 172.21.15.72:0/410171480 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39ac15c500 msgr2=0x7f39ac15f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:27.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.922+0000 7f39b28c8640 1 --2- 172.21.15.72:0/410171480 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39ac15c500 0x7f39ac15f7f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f39a00097a0 tx=0x7f39a002b960 comp rx=0 tx=0).stop 2024-04-15T15:15:27.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.922+0000 7f39b28c8640 1 -- 172.21.15.72:0/410171480 shutdown_connections 2024-04-15T15:15:27.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.922+0000 7f39b28c8640 1 --2- 172.21.15.72:0/410171480 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39ac15c500 0x7f39ac15f7f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:27.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.922+0000 7f39b28c8640 1 -- 172.21.15.72:0/410171480 >> 172.21.15.72:0/410171480 conn(0x7f39ac0cd4a0 msgr2=0x7f39ac0cd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:27.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.922+0000 7f39b28c8640 1 -- 172.21.15.72:0/410171480 shutdown_connections 2024-04-15T15:15:27.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.922+0000 7f39b28c8640 1 -- 172.21.15.72:0/410171480 wait complete. 2024-04-15T15:15:27.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.923+0000 7f39b28c8640 1 Processor -- start 2024-04-15T15:15:27.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.924+0000 7f39b28c8640 1 -- start start 2024-04-15T15:15:27.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.924+0000 7f39b28c8640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f39ac0cc0b0 0x7f39ac14a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:27.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.924+0000 7f39b28c8640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f39ac15c120 0x7f39ac0b3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:27.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.924+0000 7f39b28c8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39ac15c500 0x7f39ac160680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:27.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.924+0000 7f39b28c8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f39ac06d520 con 0x7f39ac15c500 2024-04-15T15:15:27.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.924+0000 7f39b28c8640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f39ac06d3a0 con 0x7f39ac0cc0b0 2024-04-15T15:15:27.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.924+0000 7f39b28c8640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f39ac06d6a0 con 0x7f39ac15c120 2024-04-15T15:15:27.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.925+0000 7f39b0e65640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39ac15c500 0x7f39ac160680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:27.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.925+0000 7f39b0e65640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39ac15c500 0x7f39ac160680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42722/0 (socket says 172.21.15.72:42722) 2024-04-15T15:15:27.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.925+0000 7f39b0e65640 1 -- 172.21.15.72:0/2220063577 learned_addr learned my addr 172.21.15.72:0/2220063577 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:27.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.925+0000 7f39ab7fe640 1 --2- 172.21.15.72:0/2220063577 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f39ac15c120 0x7f39ac0b3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:27.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.925+0000 7f39abfff640 1 --2- 172.21.15.72:0/2220063577 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f39ac0cc0b0 0x7f39ac14a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:27.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.925+0000 7f39b0e65640 1 -- 172.21.15.72:0/2220063577 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f39ac15c120 msgr2=0x7f39ac0b3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:27.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.925+0000 7f39b0e65640 1 --2- 172.21.15.72:0/2220063577 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f39ac15c120 0x7f39ac0b3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:27.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.925+0000 7f39b0e65640 1 -- 172.21.15.72:0/2220063577 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f39ac0cc0b0 msgr2=0x7f39ac14a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:27.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.925+0000 7f39b0e65640 1 --2- 172.21.15.72:0/2220063577 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f39ac0cc0b0 0x7f39ac14a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:27.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.925+0000 7f39b0e65640 1 -- 172.21.15.72:0/2220063577 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39ac0b5750 con 0x7f39ac15c500 2024-04-15T15:15:27.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.925+0000 7f39abfff640 1 --2- 172.21.15.72:0/2220063577 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f39ac0cc0b0 0x7f39ac14a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:27.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.925+0000 7f39b0e65640 1 --2- 172.21.15.72:0/2220063577 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39ac15c500 0x7f39ac160680 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f39a400c7a0 tx=0x7f39a400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:27.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.925+0000 7f39a8ff9640 1 -- 172.21.15.72:0/2220063577 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a4018070 con 0x7f39ac15c500 2024-04-15T15:15:27.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.925+0000 7f39a8ff9640 1 -- 172.21.15.72:0/2220063577 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39a400a640 con 0x7f39ac15c500 2024-04-15T15:15:27.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.925+0000 7f39b28c8640 1 -- 172.21.15.72:0/2220063577 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39ac0b5a30 con 0x7f39ac15c500 2024-04-15T15:15:27.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.925+0000 7f39a8ff9640 1 -- 172.21.15.72:0/2220063577 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a400aa70 con 0x7f39ac15c500 2024-04-15T15:15:27.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.925+0000 7f39b28c8640 1 -- 172.21.15.72:0/2220063577 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39ac0b5ff0 con 0x7f39ac15c500 2024-04-15T15:15:27.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.926+0000 7f39b28c8640 1 -- 172.21.15.72:0/2220063577 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39ac0689e0 con 0x7f39ac15c500 2024-04-15T15:15:27.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.927+0000 7f39a8ff9640 1 -- 172.21.15.72:0/2220063577 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f39a401b070 con 0x7f39ac15c500 2024-04-15T15:15:27.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.927+0000 7f39a8ff9640 1 --2- 172.21.15.72:0/2220063577 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f399806c8f0 0x7f399806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:27.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.927+0000 7f39a8ff9640 1 -- 172.21.15.72:0/2220063577 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f39a408b570 con 0x7f39ac15c500 2024-04-15T15:15:27.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.927+0000 7f39abfff640 1 --2- 172.21.15.72:0/2220063577 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f399806c8f0 0x7f399806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:27.931 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.928+0000 7f39abfff640 1 --2- 172.21.15.72:0/2220063577 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f399806c8f0 0x7f399806edb0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f399c000f60 tx=0x7f399c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:27.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:27.932+0000 7f39a8ff9640 1 -- 172.21.15.72:0/2220063577 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f39a40595d0 con 0x7f39ac15c500 2024-04-15T15:15:28.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.109+0000 7f39b28c8640 1 -- 172.21.15.72:0/2220063577 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f39ac06cef0 con 0x7f39ac15c500 2024-04-15T15:15:28.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.110+0000 7f39a8ff9640 1 -- 172.21.15.72:0/2220063577 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7f39a4058ed0 con 0x7f39ac15c500 2024-04-15T15:15:28.113 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:28.113 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":24,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:14:56.451113+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:14:49.438366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:14:50.368379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:14:54.081157+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:28.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.112+0000 7f39b28c8640 1 -- 172.21.15.72:0/2220063577 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f399806c8f0 msgr2=0x7f399806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.112+0000 7f39b28c8640 1 --2- 172.21.15.72:0/2220063577 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f399806c8f0 0x7f399806edb0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f399c000f60 tx=0x7f399c009040 comp rx=0 tx=0).stop 2024-04-15T15:15:28.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.112+0000 7f39b28c8640 1 -- 172.21.15.72:0/2220063577 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39ac15c500 msgr2=0x7f39ac160680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.112+0000 7f39b28c8640 1 --2- 172.21.15.72:0/2220063577 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39ac15c500 0x7f39ac160680 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f39a400c7a0 tx=0x7f39a400cc60 comp rx=0 tx=0).stop 2024-04-15T15:15:28.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.112+0000 7f39b28c8640 1 -- 172.21.15.72:0/2220063577 shutdown_connections 2024-04-15T15:15:28.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.112+0000 7f39b28c8640 1 --2- 172.21.15.72:0/2220063577 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f399806c8f0 0x7f399806edb0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.112+0000 7f39b28c8640 1 --2- 172.21.15.72:0/2220063577 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39ac15c500 0x7f39ac160680 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.112+0000 7f39b28c8640 1 --2- 172.21.15.72:0/2220063577 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f39ac15c120 0x7f39ac0b3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.112+0000 7f39b28c8640 1 --2- 172.21.15.72:0/2220063577 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f39ac0cc0b0 0x7f39ac14a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.112+0000 7f39b28c8640 1 -- 172.21.15.72:0/2220063577 >> 172.21.15.72:0/2220063577 conn(0x7f39ac0cd4a0 msgr2=0x7f39ac0cd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:28.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.113+0000 7f39b28c8640 1 -- 172.21.15.72:0/2220063577 shutdown_connections 2024-04-15T15:15:28.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.113+0000 7f39b28c8640 1 -- 172.21.15.72:0/2220063577 wait complete. 2024-04-15T15:15:28.126 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:39.359102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:14:49.438366+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:14:50.368379+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:14:54.081157+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:15:28.127 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:28.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.230+0000 7f623aa4a640 1 Processor -- start 2024-04-15T15:15:28.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.230+0000 7f623aa4a640 1 -- start start 2024-04-15T15:15:28.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.231+0000 7f623aa4a640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f623415d830 0x7f62340b57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.231+0000 7f623aa4a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f62340b5d30 0x7f62340cf160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.231+0000 7f623aa4a640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62340b61e0 con 0x7f62340cf6a0 2024-04-15T15:15:28.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.231+0000 7f623aa4a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f62340b64c0 con 0x7f62340b5d30 2024-04-15T15:15:28.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.231+0000 7f623aa4a640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f62340b67a0 con 0x7f623415d830 2024-04-15T15:15:28.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.231+0000 7f62337fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f62340b5d30 0x7f62340cf160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.231+0000 7f62337fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f62340b5d30 0x7f62340cf160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42734/0 (socket says 172.21.15.72:42734) 2024-04-15T15:15:28.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.231+0000 7f62337fe640 1 -- 172.21.15.72:0/3441187797 learned_addr learned my addr 172.21.15.72:0/3441187797 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:28.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.232+0000 7f6233fff640 1 --2- 172.21.15.72:0/3441187797 >> v2:172.21.15.175:3300/0 conn(0x7f623415d830 0x7f62340b57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.232+0000 7f62337fe640 1 -- 172.21.15.72:0/3441187797 >> v2:172.21.15.175:3300/0 conn(0x7f623415d830 msgr2=0x7f62340b57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:28.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.232+0000 7f62337fe640 1 --2- 172.21.15.72:0/3441187797 >> v2:172.21.15.175:3300/0 conn(0x7f623415d830 0x7f62340b57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.232+0000 7f62337fe640 1 -- 172.21.15.72:0/3441187797 >> v1:172.21.15.175:6789/0 conn(0x7f62340cf6a0 legacy=0x7f62340d12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.232+0000 7f62337fe640 1 -- 172.21.15.72:0/3441187797 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62340b6a80 con 0x7f62340b5d30 2024-04-15T15:15:28.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.233+0000 7f62337fe640 1 --2- 172.21.15.72:0/3441187797 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62340b5d30 0x7f62340cf160 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f623406c950 tx=0x7f622802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62a625d78965e43a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.233+0000 7f6232ffd640 1 -- 172.21.15.72:0/3441187797 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6228035020 con 0x7f62340b5d30 2024-04-15T15:15:28.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.233+0000 7f6232ffd640 1 -- 172.21.15.72:0/3441187797 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6228031870 con 0x7f62340b5d30 2024-04-15T15:15:28.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.233+0000 7f6232ffd640 1 -- 172.21.15.72:0/3441187797 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6228031c40 con 0x7f62340b5d30 2024-04-15T15:15:28.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.233+0000 7f623aa4a640 1 -- 172.21.15.72:0/3441187797 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62340b5d30 msgr2=0x7f62340cf160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.233+0000 7f623aa4a640 1 --2- 172.21.15.72:0/3441187797 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62340b5d30 0x7f62340cf160 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f623406c950 tx=0x7f622802b910 comp rx=0 tx=0).stop 2024-04-15T15:15:28.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.233+0000 7f623aa4a640 1 -- 172.21.15.72:0/3441187797 shutdown_connections 2024-04-15T15:15:28.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.233+0000 7f623aa4a640 1 --2- 172.21.15.72:0/3441187797 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f62340b5d30 0x7f62340cf160 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.233+0000 7f623aa4a640 1 --2- 172.21.15.72:0/3441187797 >> v2:172.21.15.175:3300/0 conn(0x7f623415d830 0x7f62340b57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.233+0000 7f623aa4a640 1 -- 172.21.15.72:0/3441187797 >> 172.21.15.72:0/3441187797 conn(0x7f62340d24d0 msgr2=0x7f62340d28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:28.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.233+0000 7f623aa4a640 1 -- 172.21.15.72:0/3441187797 shutdown_connections 2024-04-15T15:15:28.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.233+0000 7f623aa4a640 1 -- 172.21.15.72:0/3441187797 wait complete. 2024-04-15T15:15:28.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.234+0000 7f623aa4a640 1 Processor -- start 2024-04-15T15:15:28.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.235+0000 7f623aa4a640 1 -- start start 2024-04-15T15:15:28.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.235+0000 7f623aa4a640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f62340b5d30 0x7f6234146eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.235+0000 7f623aa4a640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f62340cf6a0 0x7f623415bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.235+0000 7f623aa4a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f623415d830 0x7f623415c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.235+0000 7f623aa4a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f623415ced0 con 0x7f623415d830 2024-04-15T15:15:28.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.235+0000 7f623aa4a640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f623415cd50 con 0x7f62340b5d30 2024-04-15T15:15:28.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.235+0000 7f623aa4a640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f623415f2b0 con 0x7f62340cf6a0 2024-04-15T15:15:28.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.235+0000 7f6238fe7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f623415d830 0x7f623415c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.236+0000 7f6238fe7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f623415d830 0x7f623415c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42740/0 (socket says 172.21.15.72:42740) 2024-04-15T15:15:28.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.236+0000 7f6238fe7640 1 -- 172.21.15.72:0/2235442475 learned_addr learned my addr 172.21.15.72:0/2235442475 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:28.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.236+0000 7f62337fe640 1 --2- 172.21.15.72:0/2235442475 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f62340cf6a0 0x7f623415bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.236+0000 7f6233fff640 1 --2- 172.21.15.72:0/2235442475 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f62340b5d30 0x7f6234146eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.236+0000 7f6238fe7640 1 -- 172.21.15.72:0/2235442475 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f62340cf6a0 msgr2=0x7f623415bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.236+0000 7f6238fe7640 1 --2- 172.21.15.72:0/2235442475 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f62340cf6a0 0x7f623415bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.236+0000 7f6238fe7640 1 -- 172.21.15.72:0/2235442475 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f62340b5d30 msgr2=0x7f6234146eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.236+0000 7f6238fe7640 1 --2- 172.21.15.72:0/2235442475 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f62340b5d30 0x7f6234146eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.236+0000 7f6238fe7640 1 -- 172.21.15.72:0/2235442475 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6234158720 con 0x7f623415d830 2024-04-15T15:15:28.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.236+0000 7f6233fff640 1 --2- 172.21.15.72:0/2235442475 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f62340b5d30 0x7f6234146eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:28.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.236+0000 7f6238fe7640 1 --2- 172.21.15.72:0/2235442475 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f623415d830 0x7f623415c4a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f622c00c7a0 tx=0x7f622c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.236+0000 7f6230ff9640 1 -- 172.21.15.72:0/2235442475 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f622c003b70 con 0x7f623415d830 2024-04-15T15:15:28.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.236+0000 7f6230ff9640 1 -- 172.21.15.72:0/2235442475 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f622c003d30 con 0x7f623415d830 2024-04-15T15:15:28.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.236+0000 7f6230ff9640 1 -- 172.21.15.72:0/2235442475 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f622c00aa90 con 0x7f623415d830 2024-04-15T15:15:28.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.236+0000 7f623aa4a640 1 -- 172.21.15.72:0/2235442475 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6234159990 con 0x7f623415d830 2024-04-15T15:15:28.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.236+0000 7f623aa4a640 1 -- 172.21.15.72:0/2235442475 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6234159f50 con 0x7f623415d830 2024-04-15T15:15:28.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.237+0000 7f623aa4a640 1 -- 172.21.15.72:0/2235442475 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6234075c50 con 0x7f623415d830 2024-04-15T15:15:28.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.238+0000 7f6230ff9640 1 -- 172.21.15.72:0/2235442475 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f622c00ac30 con 0x7f623415d830 2024-04-15T15:15:28.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.238+0000 7f6230ff9640 1 --2- 172.21.15.72:0/2235442475 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f622006c8f0 0x7f622006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.238+0000 7f6230ff9640 1 -- 172.21.15.72:0/2235442475 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f622c08bd50 con 0x7f623415d830 2024-04-15T15:15:28.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.238+0000 7f6233fff640 1 --2- 172.21.15.72:0/2235442475 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f622006c8f0 0x7f622006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.239+0000 7f6233fff640 1 --2- 172.21.15.72:0/2235442475 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f622006c8f0 0x7f622006edb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f62340ce630 tx=0x7f6224008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.242+0000 7f6230ff9640 1 -- 172.21.15.72:0/2235442475 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f622c08f030 con 0x7f623415d830 2024-04-15T15:15:28.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.417+0000 7f623aa4a640 1 -- 172.21.15.72:0/2235442475 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f623406f720 con 0x7f623415d830 2024-04-15T15:15:28.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.418+0000 7f6230ff9640 1 -- 172.21.15.72:0/2235442475 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7f622c059e40 con 0x7f623415d830 2024-04-15T15:15:28.420 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-15T15:15:28.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.419+0000 7f623aa4a640 1 -- 172.21.15.72:0/2235442475 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f622006c8f0 msgr2=0x7f622006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.419+0000 7f623aa4a640 1 --2- 172.21.15.72:0/2235442475 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f622006c8f0 0x7f622006edb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f62340ce630 tx=0x7f6224008040 comp rx=0 tx=0).stop 2024-04-15T15:15:28.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.419+0000 7f623aa4a640 1 -- 172.21.15.72:0/2235442475 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f623415d830 msgr2=0x7f623415c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.420+0000 7f623aa4a640 1 --2- 172.21.15.72:0/2235442475 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f623415d830 0x7f623415c4a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f622c00c7a0 tx=0x7f622c00cc60 comp rx=0 tx=0).stop 2024-04-15T15:15:28.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.420+0000 7f623aa4a640 1 -- 172.21.15.72:0/2235442475 shutdown_connections 2024-04-15T15:15:28.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.420+0000 7f623aa4a640 1 --2- 172.21.15.72:0/2235442475 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f622006c8f0 0x7f622006edb0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.420+0000 7f623aa4a640 1 --2- 172.21.15.72:0/2235442475 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f623415d830 0x7f623415c4a0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.420+0000 7f623aa4a640 1 --2- 172.21.15.72:0/2235442475 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f62340cf6a0 0x7f623415bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.420+0000 7f623aa4a640 1 --2- 172.21.15.72:0/2235442475 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f62340b5d30 0x7f6234146eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.420+0000 7f623aa4a640 1 -- 172.21.15.72:0/2235442475 >> 172.21.15.72:0/2235442475 conn(0x7f62340d24d0 msgr2=0x7f62341561b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:28.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.420+0000 7f623aa4a640 1 -- 172.21.15.72:0/2235442475 shutdown_connections 2024-04-15T15:15:28.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.420+0000 7f623aa4a640 1 -- 172.21.15.72:0/2235442475 wait complete. 2024-04-15T15:15:28.433 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:15:28.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.536+0000 7f74484a0640 1 Processor -- start 2024-04-15T15:15:28.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.536+0000 7f74484a0640 1 -- start start 2024-04-15T15:15:28.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.537+0000 7f74484a0640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f74400cf2c0 0x7f74400d36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.537+0000 7f74484a0640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f74400b75a0 0x7f74400b7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.537+0000 7f74484a0640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f744006c250 con 0x7f74400cf6a0 2024-04-15T15:15:28.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.537+0000 7f74484a0640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f744006c4f0 con 0x7f74400cf2c0 2024-04-15T15:15:28.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.537+0000 7f74484a0640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f744006c7d0 con 0x7f74400b75a0 2024-04-15T15:15:28.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.537+0000 7f7446a3d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f74400cf6a0 0x7f74400cfaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:48064/0 (socket says 172.21.15.72:48064) 2024-04-15T15:15:28.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.538+0000 7f7446a3d640 1 -- 172.21.15.72:0/1416812483 learned_addr learned my addr 172.21.15.72:0/1416812483 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:28.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.538+0000 7f7445a3b640 1 --2- 172.21.15.72:0/1416812483 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f74400b75a0 0x7f74400b7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.538+0000 7f744623c640 1 --2- 172.21.15.72:0/1416812483 >> v2:172.21.15.175:3300/0 conn(0x7f74400cf2c0 0x7f74400d36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.538+0000 7f744523a640 1 -- 172.21.15.72:0/1416812483 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 365166563 0 0) 0x7f744006c250 con 0x7f74400cf6a0 2024-04-15T15:15:28.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.538+0000 7f744523a640 1 -- 172.21.15.72:0/1416812483 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7428003520 con 0x7f74400cf6a0 2024-04-15T15:15:28.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.538+0000 7f7445a3b640 1 -- 172.21.15.72:0/1416812483 >> v2:172.21.15.175:3300/0 conn(0x7f74400cf2c0 msgr2=0x7f74400d36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:28.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.538+0000 7f7445a3b640 1 --2- 172.21.15.72:0/1416812483 >> v2:172.21.15.175:3300/0 conn(0x7f74400cf2c0 0x7f74400d36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.538+0000 7f7445a3b640 1 -- 172.21.15.72:0/1416812483 >> v1:172.21.15.72:6789/0 conn(0x7f74400cf6a0 legacy=0x7f74400cfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.538+0000 7f744523a640 1 -- 172.21.15.72:0/1416812483 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 266426570 0 0) 0x7f7428003520 con 0x7f74400cf6a0 2024-04-15T15:15:28.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.538+0000 7f7445a3b640 1 -- 172.21.15.72:0/1416812483 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f744006cab0 con 0x7f74400b75a0 2024-04-15T15:15:28.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.538+0000 7f744623c640 1 --2- 172.21.15.72:0/1416812483 >> v2:172.21.15.175:3300/0 conn(0x7f74400cf2c0 0x7f74400d36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:28.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.539+0000 7f7445a3b640 1 --2- 172.21.15.72:0/1416812483 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f74400b75a0 0x7f74400b7a00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f74340097a0 tx=0x7f743402b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec4e1a584613b24 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.539+0000 7f744523a640 1 -- 172.21.15.72:0/1416812483 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7434037070 con 0x7f74400b75a0 2024-04-15T15:15:28.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.539+0000 7f744523a640 1 -- 172.21.15.72:0/1416812483 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7434030510 con 0x7f74400b75a0 2024-04-15T15:15:28.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.539+0000 7f744523a640 1 -- 172.21.15.72:0/1416812483 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7438001670 con 0x7f74400b75a0 2024-04-15T15:15:28.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.539+0000 7f74484a0640 1 -- 172.21.15.72:0/1416812483 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f74400b75a0 msgr2=0x7f74400b7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.539+0000 7f74484a0640 1 --2- 172.21.15.72:0/1416812483 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f74400b75a0 0x7f74400b7a00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f74340097a0 tx=0x7f743402b790 comp rx=0 tx=0).stop 2024-04-15T15:15:28.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.540+0000 7f74484a0640 1 -- 172.21.15.72:0/1416812483 shutdown_connections 2024-04-15T15:15:28.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.540+0000 7f74484a0640 1 --2- 172.21.15.72:0/1416812483 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f74400b75a0 0x7f74400b7a00 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.540+0000 7f74484a0640 1 --2- 172.21.15.72:0/1416812483 >> v2:172.21.15.175:3300/0 conn(0x7f74400cf2c0 0x7f74400d36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.540+0000 7f74484a0640 1 -- 172.21.15.72:0/1416812483 >> 172.21.15.72:0/1416812483 conn(0x7f74400d4750 msgr2=0x7f74400d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:28.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.540+0000 7f74484a0640 1 -- 172.21.15.72:0/1416812483 shutdown_connections 2024-04-15T15:15:28.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.540+0000 7f74484a0640 1 -- 172.21.15.72:0/1416812483 wait complete. 2024-04-15T15:15:28.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.541+0000 7f74484a0640 1 Processor -- start 2024-04-15T15:15:28.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.541+0000 7f74484a0640 1 -- start start 2024-04-15T15:15:28.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.541+0000 7f74484a0640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f74400b75a0 0x7f74400cd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.541+0000 7f74484a0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74400cf2c0 0x7f74400cc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.541+0000 7f74484a0640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74400cf6a0 0x7f744014fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.541+0000 7f74484a0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f74400b7f40 con 0x7f74400cf2c0 2024-04-15T15:15:28.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.541+0000 7f74484a0640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f74400d31b0 con 0x7f74400cf6a0 2024-04-15T15:15:28.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.541+0000 7f74484a0640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f74400d3c10 con 0x7f74400b75a0 2024-04-15T15:15:28.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.542+0000 7f7445a3b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74400cf2c0 0x7f74400cc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.542+0000 7f7445a3b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74400cf2c0 0x7f74400cc4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42754/0 (socket says 172.21.15.72:42754) 2024-04-15T15:15:28.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.542+0000 7f7445a3b640 1 -- 172.21.15.72:0/372648774 learned_addr learned my addr 172.21.15.72:0/372648774 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:28.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.542+0000 7f7446a3d640 1 --2- 172.21.15.72:0/372648774 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74400cf6a0 0x7f744014fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.542+0000 7f744623c640 1 --2- 172.21.15.72:0/372648774 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f74400b75a0 0x7f74400cd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.542+0000 7f7445a3b640 1 -- 172.21.15.72:0/372648774 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f74400b75a0 msgr2=0x7f74400cd2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.542+0000 7f7445a3b640 1 --2- 172.21.15.72:0/372648774 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f74400b75a0 0x7f74400cd2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.542+0000 7f7445a3b640 1 -- 172.21.15.72:0/372648774 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74400cf6a0 msgr2=0x7f744014fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.542+0000 7f7445a3b640 1 --2- 172.21.15.72:0/372648774 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74400cf6a0 0x7f744014fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.542+0000 7f7445a3b640 1 -- 172.21.15.72:0/372648774 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74400cd980 con 0x7f74400cf2c0 2024-04-15T15:15:28.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.542+0000 7f7446a3d640 1 --2- 172.21.15.72:0/372648774 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74400cf6a0 0x7f744014fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:28.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.542+0000 7f744623c640 1 --2- 172.21.15.72:0/372648774 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f74400b75a0 0x7f74400cd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:28.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.542+0000 7f7445a3b640 1 --2- 172.21.15.72:0/372648774 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74400cf2c0 0x7f74400cc4f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7434000c00 tx=0x7f74340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.542+0000 7f743effd640 1 -- 172.21.15.72:0/372648774 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7434041020 con 0x7f74400cf2c0 2024-04-15T15:15:28.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.542+0000 7f743effd640 1 -- 172.21.15.72:0/372648774 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f743403c870 con 0x7f74400cf2c0 2024-04-15T15:15:28.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.542+0000 7f74484a0640 1 -- 172.21.15.72:0/372648774 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74400cdc60 con 0x7f74400cf2c0 2024-04-15T15:15:28.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.542+0000 7f743effd640 1 -- 172.21.15.72:0/372648774 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7434037040 con 0x7f74400cf2c0 2024-04-15T15:15:28.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.542+0000 7f74484a0640 1 -- 172.21.15.72:0/372648774 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74400ce220 con 0x7f74400cf2c0 2024-04-15T15:15:28.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.543+0000 7f74484a0640 1 -- 172.21.15.72:0/372648774 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7440075c50 con 0x7f74400cf2c0 2024-04-15T15:15:28.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.544+0000 7f743effd640 1 -- 172.21.15.72:0/372648774 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7434044020 con 0x7f74400cf2c0 2024-04-15T15:15:28.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.544+0000 7f743effd640 1 --2- 172.21.15.72:0/372648774 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f742c06c8f0 0x7f742c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.544+0000 7f744623c640 1 --2- 172.21.15.72:0/372648774 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f742c06c8f0 0x7f742c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.544+0000 7f743effd640 1 -- 172.21.15.72:0/372648774 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f74340ad930 con 0x7f74400cf2c0 2024-04-15T15:15:28.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.545+0000 7f744623c640 1 --2- 172.21.15.72:0/372648774 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f742c06c8f0 0x7f742c06edb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7430007520 tx=0x7f743000b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.549+0000 7f743effd640 1 -- 172.21.15.72:0/372648774 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f743404b030 con 0x7f74400cf2c0 2024-04-15T15:15:28.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.722+0000 7f74484a0640 1 -- 172.21.15.72:0/372648774 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f74400d44e0 con 0x7f74400cf2c0 2024-04-15T15:15:28.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.722+0000 7f743effd640 1 -- 172.21.15.72:0/372648774 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7f743407b990 con 0x7f74400cf2c0 2024-04-15T15:15:28.725 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:28.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.724+0000 7f74484a0640 1 -- 172.21.15.72:0/372648774 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f742c06c8f0 msgr2=0x7f742c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.724+0000 7f74484a0640 1 --2- 172.21.15.72:0/372648774 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f742c06c8f0 0x7f742c06edb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7430007520 tx=0x7f743000b040 comp rx=0 tx=0).stop 2024-04-15T15:15:28.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.724+0000 7f74484a0640 1 -- 172.21.15.72:0/372648774 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74400cf2c0 msgr2=0x7f74400cc4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.724+0000 7f74484a0640 1 --2- 172.21.15.72:0/372648774 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74400cf2c0 0x7f74400cc4f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7434000c00 tx=0x7f74340046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.724+0000 7f74484a0640 1 -- 172.21.15.72:0/372648774 shutdown_connections 2024-04-15T15:15:28.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.724+0000 7f74484a0640 1 --2- 172.21.15.72:0/372648774 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f742c06c8f0 0x7f742c06edb0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.724+0000 7f74484a0640 1 --2- 172.21.15.72:0/372648774 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74400cf6a0 0x7f744014fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.724+0000 7f74484a0640 1 --2- 172.21.15.72:0/372648774 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74400cf2c0 0x7f74400cc4f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.724+0000 7f74484a0640 1 --2- 172.21.15.72:0/372648774 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f74400b75a0 0x7f74400cd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.724+0000 7f74484a0640 1 -- 172.21.15.72:0/372648774 >> 172.21.15.72:0/372648774 conn(0x7f74400d4750 msgr2=0x7f744006bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:28.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.724+0000 7f74484a0640 1 -- 172.21.15.72:0/372648774 shutdown_connections 2024-04-15T15:15:28.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.724+0000 7f74484a0640 1 -- 172.21.15.72:0/372648774 wait complete. 2024-04-15T15:15:28.738 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:15:28.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.841+0000 7f5a0b477640 1 Processor -- start 2024-04-15T15:15:28.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.842+0000 7f5a0b477640 1 -- start start 2024-04-15T15:15:28.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.842+0000 7f5a0b477640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5a040ce2f0 0x7f5a0415c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.842+0000 7f5a0b477640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f5a040d08f0 0x7f5a040cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.842+0000 7f5a0b477640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a0406bbb0 con 0x7f5a040ce6d0 2024-04-15T15:15:28.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.842+0000 7f5a0b477640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f5a0406b960 con 0x7f5a040d08f0 2024-04-15T15:15:28.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.842+0000 7f5a0b477640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f5a0406be90 con 0x7f5a040ce2f0 2024-04-15T15:15:28.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.843+0000 7f5a09213640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5a040ce2f0 0x7f5a0415c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.843+0000 7f5a08a12640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f5a040d08f0 0x7f5a040cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.843+0000 7f5a08a12640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f5a040d08f0 0x7f5a040cc0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.72:57496/0 (socket says 172.21.15.72:57496) 2024-04-15T15:15:28.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.843+0000 7f5a09a14640 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f5a040ce6d0 0x7f5a04155600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.72:59624/0 (socket says 172.21.15.72:59624) 2024-04-15T15:15:28.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.843+0000 7f5a09213640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5a040ce2f0 0x7f5a0415c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3301/0 says I am v2:172.21.15.72:39202/0 (socket says 172.21.15.72:39202) 2024-04-15T15:15:28.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.843+0000 7f5a08a12640 1 -- 172.21.15.72:0/993933427 learned_addr learned my addr 172.21.15.72:0/993933427 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:28.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.843+0000 7f5a09213640 1 -- 172.21.15.72:0/993933427 >> v2:172.21.15.175:3300/0 conn(0x7f5a040d08f0 msgr2=0x7f5a040cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:28.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.843+0000 7f5a09213640 1 --2- 172.21.15.72:0/993933427 >> v2:172.21.15.175:3300/0 conn(0x7f5a040d08f0 0x7f5a040cc0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.843+0000 7f5a09213640 1 -- 172.21.15.72:0/993933427 >> v1:172.21.15.175:6789/0 conn(0x7f5a040ce6d0 legacy=0x7f5a04155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.843+0000 7f5a09213640 1 -- 172.21.15.72:0/993933427 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a0406c170 con 0x7f5a040ce2f0 2024-04-15T15:15:28.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.843+0000 7f5a08a12640 1 --2- 172.21.15.72:0/993933427 >> v2:172.21.15.175:3300/0 conn(0x7f5a040d08f0 0x7f5a040cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:28.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.844+0000 7f5a09213640 1 --2- 172.21.15.72:0/993933427 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5a040ce2f0 0x7f5a0415c530 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5a00009d90 tx=0x7f5a0002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c3299da6b5da403 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.844+0000 7f59fbfff640 1 -- 172.21.15.72:0/993933427 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a00039070 con 0x7f5a040ce2f0 2024-04-15T15:15:28.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.844+0000 7f59fbfff640 1 -- 172.21.15.72:0/993933427 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a0002cc50 con 0x7f5a040ce2f0 2024-04-15T15:15:28.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.844+0000 7f59fbfff640 1 -- 172.21.15.72:0/993933427 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a00034440 con 0x7f5a040ce2f0 2024-04-15T15:15:28.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.845+0000 7f5a0b477640 1 -- 172.21.15.72:0/993933427 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5a040ce2f0 msgr2=0x7f5a0415c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.845+0000 7f5a0b477640 1 --2- 172.21.15.72:0/993933427 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5a040ce2f0 0x7f5a0415c530 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5a00009d90 tx=0x7f5a0002bea0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.845+0000 7f5a0b477640 1 -- 172.21.15.72:0/993933427 shutdown_connections 2024-04-15T15:15:28.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.845+0000 7f5a0b477640 1 --2- 172.21.15.72:0/993933427 >> v2:172.21.15.175:3300/0 conn(0x7f5a040d08f0 0x7f5a040cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.845+0000 7f5a0b477640 1 --2- 172.21.15.72:0/993933427 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5a040ce2f0 0x7f5a0415c530 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.845+0000 7f5a0b477640 1 -- 172.21.15.72:0/993933427 >> 172.21.15.72:0/993933427 conn(0x7f5a040b31d0 msgr2=0x7f5a040b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:28.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.845+0000 7f5a0b477640 1 -- 172.21.15.72:0/993933427 shutdown_connections 2024-04-15T15:15:28.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.845+0000 7f5a0b477640 1 -- 172.21.15.72:0/993933427 wait complete. 2024-04-15T15:15:28.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.846+0000 7f5a0b477640 1 Processor -- start 2024-04-15T15:15:28.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.846+0000 7f5a0b477640 1 -- start start 2024-04-15T15:15:28.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.847+0000 7f5a0b477640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5a040ce2f0 0x7f5a0414ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.847+0000 7f5a0b477640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a040ce6d0 0x7f5a04157ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.847+0000 7f5a0b477640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5a040d08f0 0x7f5a0415d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.847+0000 7f5a0b477640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f5a04155d10 con 0x7f5a040ce2f0 2024-04-15T15:15:28.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.847+0000 7f5a0b477640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0415ca70 con 0x7f5a040ce6d0 2024-04-15T15:15:28.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.847+0000 7f5a0b477640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f5a04155e90 con 0x7f5a040d08f0 2024-04-15T15:15:28.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.847+0000 7f5a09213640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5a040ce2f0 0x7f5a0414ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.847+0000 7f5a09213640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5a040ce2f0 0x7f5a0414ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42770/0 (socket says 172.21.15.72:42770) 2024-04-15T15:15:28.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.847+0000 7f5a09213640 1 -- 172.21.15.72:0/3421249018 learned_addr learned my addr 172.21.15.72:0/3421249018 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:28.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.847+0000 7f5a08a12640 1 --2- 172.21.15.72:0/3421249018 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a040ce6d0 0x7f5a04157ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.847+0000 7f5a09a14640 1 --2- 172.21.15.72:0/3421249018 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5a040d08f0 0x7f5a0415d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.847+0000 7f5a09213640 1 -- 172.21.15.72:0/3421249018 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5a040d08f0 msgr2=0x7f5a0415d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.847+0000 7f5a09213640 1 --2- 172.21.15.72:0/3421249018 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5a040d08f0 0x7f5a0415d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.847+0000 7f5a09213640 1 -- 172.21.15.72:0/3421249018 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a040ce6d0 msgr2=0x7f5a04157ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.847+0000 7f5a09213640 1 --2- 172.21.15.72:0/3421249018 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a040ce6d0 0x7f5a04157ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.847+0000 7f5a09213640 1 -- 172.21.15.72:0/3421249018 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a04156e00 con 0x7f5a040ce2f0 2024-04-15T15:15:28.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.847+0000 7f5a09213640 1 --2- 172.21.15.72:0/3421249018 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5a040ce2f0 0x7f5a0414ee60 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f5a00009d90 tx=0x7f5a00004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.847+0000 7f59f9ffb640 1 -- 172.21.15.72:0/3421249018 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a00039050 con 0x7f5a040ce2f0 2024-04-15T15:15:28.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.847+0000 7f59f9ffb640 1 -- 172.21.15.72:0/3421249018 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a00045070 con 0x7f5a040ce2f0 2024-04-15T15:15:28.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.847+0000 7f59f9ffb640 1 -- 172.21.15.72:0/3421249018 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a0003e610 con 0x7f5a040ce2f0 2024-04-15T15:15:28.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.847+0000 7f5a0b477640 1 -- 172.21.15.72:0/3421249018 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a041570e0 con 0x7f5a040ce2f0 2024-04-15T15:15:28.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.848+0000 7f5a0b477640 1 -- 172.21.15.72:0/3421249018 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a041576a0 con 0x7f5a040ce2f0 2024-04-15T15:15:28.853 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.848+0000 7f5a0b477640 1 -- 172.21.15.72:0/3421249018 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a04075c50 con 0x7f5a040ce2f0 2024-04-15T15:15:28.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.849+0000 7f59f9ffb640 1 -- 172.21.15.72:0/3421249018 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5a0003e7b0 con 0x7f5a040ce2f0 2024-04-15T15:15:28.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.849+0000 7f59f9ffb640 1 --2- 172.21.15.72:0/3421249018 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f59e806c8f0 0x7f59e806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.850+0000 7f5a08a12640 1 --2- 172.21.15.72:0/3421249018 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f59e806c8f0 0x7f59e806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.850+0000 7f59f9ffb640 1 -- 172.21.15.72:0/3421249018 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f5a000ae9b0 con 0x7f5a040ce2f0 2024-04-15T15:15:28.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.850+0000 7f5a08a12640 1 --2- 172.21.15.72:0/3421249018 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f59e806c8f0 0x7f59e806edb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5a041f49e0 tx=0x7f59f4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:28.854+0000 7f59f9ffb640 1 -- 172.21.15.72:0/3421249018 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5a0007cb30 con 0x7f5a040ce2f0 2024-04-15T15:15:29.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.030+0000 7f5a0b477640 1 -- 172.21.15.72:0/3421249018 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5a040cc6a0 con 0x7f5a040ce2f0 2024-04-15T15:15:29.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.030+0000 7f59f9ffb640 1 -- 172.21.15.72:0/3421249018 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7f5a0007c430 con 0x7f5a040ce2f0 2024-04-15T15:15:29.033 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-04-15T15:15:29.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.032+0000 7f5a0b477640 1 -- 172.21.15.72:0/3421249018 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f59e806c8f0 msgr2=0x7f59e806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.032+0000 7f5a0b477640 1 --2- 172.21.15.72:0/3421249018 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f59e806c8f0 0x7f59e806edb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5a041f49e0 tx=0x7f59f4008040 comp rx=0 tx=0).stop 2024-04-15T15:15:29.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.032+0000 7f5a0b477640 1 -- 172.21.15.72:0/3421249018 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5a040ce2f0 msgr2=0x7f5a0414ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.032+0000 7f5a0b477640 1 --2- 172.21.15.72:0/3421249018 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5a040ce2f0 0x7f5a0414ee60 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f5a00009d90 tx=0x7f5a00004a40 comp rx=0 tx=0).stop 2024-04-15T15:15:29.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.032+0000 7f5a0b477640 1 -- 172.21.15.72:0/3421249018 shutdown_connections 2024-04-15T15:15:29.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.032+0000 7f5a0b477640 1 --2- 172.21.15.72:0/3421249018 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f59e806c8f0 0x7f59e806edb0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.032+0000 7f5a0b477640 1 --2- 172.21.15.72:0/3421249018 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5a040d08f0 0x7f5a0415d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.032+0000 7f5a0b477640 1 --2- 172.21.15.72:0/3421249018 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5a040ce6d0 0x7f5a04157ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.032+0000 7f5a0b477640 1 --2- 172.21.15.72:0/3421249018 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5a040ce2f0 0x7f5a0414ee60 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.032+0000 7f5a0b477640 1 -- 172.21.15.72:0/3421249018 >> 172.21.15.72:0/3421249018 conn(0x7f5a040b31d0 msgr2=0x7f5a0415e540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:29.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.032+0000 7f5a0b477640 1 -- 172.21.15.72:0/3421249018 shutdown_connections 2024-04-15T15:15:29.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.032+0000 7f5a0b477640 1 -- 172.21.15.72:0/3421249018 wait complete. 2024-04-15T15:15:29.046 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:15:29.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.150+0000 7fe2f62b3640 1 Processor -- start 2024-04-15T15:15:29.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.150+0000 7fe2f62b3640 1 -- start start 2024-04-15T15:15:29.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.151+0000 7fe2f62b3640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe2f0161ee0 0x7fe2f00c95f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.151+0000 7fe2f62b3640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe2f00b67c0 0x7fe2f00d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.151+0000 7fe2f62b3640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2f00ccfe0 con 0x7fe2f00b6c20 2024-04-15T15:15:29.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.151+0000 7fe2f62b3640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fe2f01622c0 con 0x7fe2f0161ee0 2024-04-15T15:15:29.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.151+0000 7fe2f62b3640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fe2f00cd2c0 con 0x7fe2f00b67c0 2024-04-15T15:15:29.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.151+0000 7fe2effff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe2f0161ee0 0x7fe2f00c95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.151+0000 7fe2f4850640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fe2f00b6c20 0x7fe2f00ce080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:48076/0 (socket says 172.21.15.72:48076) 2024-04-15T15:15:29.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.151+0000 7fe2f4850640 1 -- 172.21.15.72:0/3417909295 learned_addr learned my addr 172.21.15.72:0/3417909295 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:29.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.151+0000 7fe2ef7fe640 1 --2- 172.21.15.72:0/3417909295 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe2f00b67c0 0x7fe2f00d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.151+0000 7fe2effff640 1 -- 172.21.15.72:0/3417909295 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe2f00b67c0 msgr2=0x7fe2f00d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:29.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.151+0000 7fe2effff640 1 --2- 172.21.15.72:0/3417909295 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe2f00b67c0 0x7fe2f00d1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.151+0000 7fe2effff640 1 -- 172.21.15.72:0/3417909295 >> v1:172.21.15.72:6789/0 conn(0x7fe2f00b6c20 legacy=0x7fe2f00ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.151+0000 7fe2effff640 1 -- 172.21.15.72:0/3417909295 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2f00cd5a0 con 0x7fe2f0161ee0 2024-04-15T15:15:29.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.151+0000 7fe2eeffd640 1 -- 172.21.15.72:0/3417909295 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1844034741 0 0) 0x7fe2f00ccfe0 con 0x7fe2f00b6c20 2024-04-15T15:15:29.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.152+0000 7fe2effff640 1 --2- 172.21.15.72:0/3417909295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2f0161ee0 0x7fe2f00c95f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe2e00097d0 tx=0x7fe2e002b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e41f3c63ed0d4a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:29.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.152+0000 7fe2eeffd640 1 -- 172.21.15.72:0/3417909295 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e0035020 con 0x7fe2f0161ee0 2024-04-15T15:15:29.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.152+0000 7fe2eeffd640 1 -- 172.21.15.72:0/3417909295 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2e0031800 con 0x7fe2f0161ee0 2024-04-15T15:15:29.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.152+0000 7fe2eeffd640 1 -- 172.21.15.72:0/3417909295 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e0031bd0 con 0x7fe2f0161ee0 2024-04-15T15:15:29.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.152+0000 7fe2f62b3640 1 -- 172.21.15.72:0/3417909295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2f0161ee0 msgr2=0x7fe2f00c95f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.153+0000 7fe2f62b3640 1 --2- 172.21.15.72:0/3417909295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2f0161ee0 0x7fe2f00c95f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe2e00097d0 tx=0x7fe2e002b8a0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.153+0000 7fe2f62b3640 1 -- 172.21.15.72:0/3417909295 shutdown_connections 2024-04-15T15:15:29.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.153+0000 7fe2f62b3640 1 --2- 172.21.15.72:0/3417909295 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe2f00b67c0 0x7fe2f00d1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.153+0000 7fe2f62b3640 1 --2- 172.21.15.72:0/3417909295 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2f0161ee0 0x7fe2f00c95f0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.153+0000 7fe2f62b3640 1 -- 172.21.15.72:0/3417909295 >> 172.21.15.72:0/3417909295 conn(0x7fe2f00b4010 msgr2=0x7fe2f00b4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:29.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.153+0000 7fe2f62b3640 1 -- 172.21.15.72:0/3417909295 shutdown_connections 2024-04-15T15:15:29.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.153+0000 7fe2f62b3640 1 -- 172.21.15.72:0/3417909295 wait complete. 2024-04-15T15:15:29.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.154+0000 7fe2f62b3640 1 Processor -- start 2024-04-15T15:15:29.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.154+0000 7fe2f62b3640 1 -- start start 2024-04-15T15:15:29.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.155+0000 7fe2f62b3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2f00b67c0 0x7fe2f0149c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.155+0000 7fe2f62b3640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe2f00b6c20 0x7fe2f015cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.155+0000 7fe2f62b3640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe2f0161ee0 0x7fe2f014a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.155+0000 7fe2f62b3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f00ccb70 con 0x7fe2f00b67c0 2024-04-15T15:15:29.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.155+0000 7fe2f62b3640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fe2f00c9b30 con 0x7fe2f00b6c20 2024-04-15T15:15:29.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.155+0000 7fe2f62b3640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fe2f00cccf0 con 0x7fe2f0161ee0 2024-04-15T15:15:29.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.155+0000 7fe2effff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2f00b67c0 0x7fe2f0149c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.155+0000 7fe2effff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2f00b67c0 0x7fe2f0149c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42794/0 (socket says 172.21.15.72:42794) 2024-04-15T15:15:29.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.155+0000 7fe2effff640 1 -- 172.21.15.72:0/3801508467 learned_addr learned my addr 172.21.15.72:0/3801508467 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:29.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.155+0000 7fe2effff640 1 -- 172.21.15.72:0/3801508467 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe2f0161ee0 msgr2=0x7fe2f014a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.155+0000 7fe2effff640 1 --2- 172.21.15.72:0/3801508467 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe2f0161ee0 0x7fe2f014a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.155+0000 7fe2effff640 1 -- 172.21.15.72:0/3801508467 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe2f00b6c20 msgr2=0x7fe2f015cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.155+0000 7fe2effff640 1 --2- 172.21.15.72:0/3801508467 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe2f00b6c20 0x7fe2f015cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.155+0000 7fe2effff640 1 -- 172.21.15.72:0/3801508467 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2f00c79c0 con 0x7fe2f00b67c0 2024-04-15T15:15:29.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.155+0000 7fe2effff640 1 --2- 172.21.15.72:0/3801508467 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2f00b67c0 0x7fe2f0149c40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe2e00097d0 tx=0x7fe2e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:29.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.155+0000 7fe2ecff9640 1 -- 172.21.15.72:0/3801508467 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e0041070 con 0x7fe2f00b67c0 2024-04-15T15:15:29.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.155+0000 7fe2ecff9640 1 -- 172.21.15.72:0/3801508467 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2e003ac80 con 0x7fe2f00b67c0 2024-04-15T15:15:29.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.155+0000 7fe2f62b3640 1 -- 172.21.15.72:0/3801508467 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2f00c7ca0 con 0x7fe2f00b67c0 2024-04-15T15:15:29.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.155+0000 7fe2ecff9640 1 -- 172.21.15.72:0/3801508467 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e0035040 con 0x7fe2f00b67c0 2024-04-15T15:15:29.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.155+0000 7fe2f62b3640 1 -- 172.21.15.72:0/3801508467 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2f0157090 con 0x7fe2f00b67c0 2024-04-15T15:15:29.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.156+0000 7fe2f62b3640 1 -- 172.21.15.72:0/3801508467 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2f0075c50 con 0x7fe2f00b67c0 2024-04-15T15:15:29.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.157+0000 7fe2ecff9640 1 -- 172.21.15.72:0/3801508467 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe2e0038020 con 0x7fe2f00b67c0 2024-04-15T15:15:29.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.157+0000 7fe2ecff9640 1 --2- 172.21.15.72:0/3801508467 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe2dc06c8f0 0x7fe2dc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.157+0000 7fe2ecff9640 1 -- 172.21.15.72:0/3801508467 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fe2e00adab0 con 0x7fe2f00b67c0 2024-04-15T15:15:29.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.157+0000 7fe2ef7fe640 1 --2- 172.21.15.72:0/3801508467 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe2dc06c8f0 0x7fe2dc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.158+0000 7fe2ef7fe640 1 --2- 172.21.15.72:0/3801508467 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe2dc06c8f0 0x7fe2dc06edb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0160090 tx=0x7fe2e4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:29.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.162+0000 7fe2ecff9640 1 -- 172.21.15.72:0/3801508467 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe2e007bc30 con 0x7fe2f00b67c0 2024-04-15T15:15:29.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.335+0000 7fe2f62b3640 1 -- 172.21.15.72:0/3801508467 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fe2f00d1e50 con 0x7fe2f00b67c0 2024-04-15T15:15:29.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.335+0000 7fe2ecff9640 1 -- 172.21.15.72:0/3801508467 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7fe2e004de80 con 0x7fe2f00b67c0 2024-04-15T15:15:29.338 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:29.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.337+0000 7fe2f62b3640 1 -- 172.21.15.72:0/3801508467 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe2dc06c8f0 msgr2=0x7fe2dc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.337+0000 7fe2f62b3640 1 --2- 172.21.15.72:0/3801508467 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe2dc06c8f0 0x7fe2dc06edb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0160090 tx=0x7fe2e4008040 comp rx=0 tx=0).stop 2024-04-15T15:15:29.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.337+0000 7fe2f62b3640 1 -- 172.21.15.72:0/3801508467 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2f00b67c0 msgr2=0x7fe2f0149c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.337+0000 7fe2f62b3640 1 --2- 172.21.15.72:0/3801508467 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2f00b67c0 0x7fe2f0149c40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe2e00097d0 tx=0x7fe2e00046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.337+0000 7fe2f62b3640 1 -- 172.21.15.72:0/3801508467 shutdown_connections 2024-04-15T15:15:29.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.337+0000 7fe2f62b3640 1 --2- 172.21.15.72:0/3801508467 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe2dc06c8f0 0x7fe2dc06edb0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.337+0000 7fe2f62b3640 1 --2- 172.21.15.72:0/3801508467 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe2f0161ee0 0x7fe2f014a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.337+0000 7fe2f62b3640 1 --2- 172.21.15.72:0/3801508467 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe2f00b6c20 0x7fe2f015cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.337+0000 7fe2f62b3640 1 --2- 172.21.15.72:0/3801508467 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2f00b67c0 0x7fe2f0149c40 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.337+0000 7fe2f62b3640 1 -- 172.21.15.72:0/3801508467 >> 172.21.15.72:0/3801508467 conn(0x7fe2f00b4010 msgr2=0x7fe2f00cb020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:29.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.337+0000 7fe2f62b3640 1 -- 172.21.15.72:0/3801508467 shutdown_connections 2024-04-15T15:15:29.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.337+0000 7fe2f62b3640 1 -- 172.21.15.72:0/3801508467 wait complete. 2024-04-15T15:15:29.351 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:29.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.454+0000 7fae8ffff640 1 Processor -- start 2024-04-15T15:15:29.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.454+0000 7fae8ffff640 1 -- start start 2024-04-15T15:15:29.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.455+0000 7fae8ffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fae9015e6e0 0x7fae900d1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.455+0000 7fae8ffff640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae900cf9c0 con 0x7fae900d49a0 2024-04-15T15:15:29.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.455+0000 7fae8ffff640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae90162690 con 0x7fae900b3990 2024-04-15T15:15:29.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.455+0000 7fae8ffff640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fae90163210 con 0x7fae9015e6e0 2024-04-15T15:15:29.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.455+0000 7fae8effd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fae9015e6e0 0x7fae900d1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.455+0000 7fae8f7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fae900d49a0 0x7fae900cfea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:48078/0 (socket says 172.21.15.72:48078) 2024-04-15T15:15:29.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.455+0000 7fae8f7fe640 1 -- 172.21.15.72:0/4261326364 learned_addr learned my addr 172.21.15.72:0/4261326364 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:29.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.455+0000 7fae8effd640 1 -- 172.21.15.72:0/4261326364 >> v1:172.21.15.175:6789/0 conn(0x7fae900b3990 legacy=0x7fae900d4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:29.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.455+0000 7fae8effd640 1 -- 172.21.15.72:0/4261326364 >> v1:172.21.15.72:6789/0 conn(0x7fae900d49a0 legacy=0x7fae900cfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.455+0000 7fae8effd640 1 -- 172.21.15.72:0/4261326364 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae90162970 con 0x7fae9015e6e0 2024-04-15T15:15:29.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.455+0000 7fae8e7fc640 1 -- 172.21.15.72:0/4261326364 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1695460120 0 0) 0x7fae900cf9c0 con 0x7fae900d49a0 2024-04-15T15:15:29.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.456+0000 7fae8effd640 1 --2- 172.21.15.72:0/4261326364 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae9015e6e0 0x7fae900d1fa0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fae800097d0 tx=0x7fae8002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63bd47ef3087c6a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:29.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.456+0000 7fae8e7fc640 1 -- 172.21.15.72:0/4261326364 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae80035020 con 0x7fae9015e6e0 2024-04-15T15:15:29.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.456+0000 7fae8e7fc640 1 -- 172.21.15.72:0/4261326364 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae80031830 con 0x7fae9015e6e0 2024-04-15T15:15:29.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.456+0000 7fae8e7fc640 1 -- 172.21.15.72:0/4261326364 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae80031c00 con 0x7fae9015e6e0 2024-04-15T15:15:29.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.456+0000 7fae8ffff640 1 -- 172.21.15.72:0/4261326364 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae9015e6e0 msgr2=0x7fae900d1fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.456+0000 7fae8ffff640 1 --2- 172.21.15.72:0/4261326364 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae9015e6e0 0x7fae900d1fa0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fae800097d0 tx=0x7fae8002b8d0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.456+0000 7fae8ffff640 1 -- 172.21.15.72:0/4261326364 shutdown_connections 2024-04-15T15:15:29.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.456+0000 7fae8ffff640 1 --2- 172.21.15.72:0/4261326364 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae9015e6e0 0x7fae900d1fa0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.456+0000 7fae8ffff640 1 -- 172.21.15.72:0/4261326364 >> 172.21.15.72:0/4261326364 conn(0x7fae900b9c30 msgr2=0x7fae9009d720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:29.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.456+0000 7fae8ffff640 1 -- 172.21.15.72:0/4261326364 shutdown_connections 2024-04-15T15:15:29.461 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.456+0000 7fae8ffff640 1 -- 172.21.15.72:0/4261326364 wait complete. 2024-04-15T15:15:29.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.458+0000 7fae8ffff640 1 Processor -- start 2024-04-15T15:15:29.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.458+0000 7fae8ffff640 1 -- start start 2024-04-15T15:15:29.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.458+0000 7fae8ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae900b3990 0x7fae9015c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.458+0000 7fae8ffff640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fae900d49a0 0x7fae900b65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.458+0000 7fae8ffff640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fae9015e6e0 0x7fae9006d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.458+0000 7fae8ffff640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fae9015eac0 con 0x7fae900b3990 2024-04-15T15:15:29.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.458+0000 7fae8ffff640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fae90162f30 con 0x7fae9015e6e0 2024-04-15T15:15:29.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.458+0000 7fae8ffff640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fae901622b0 con 0x7fae900d49a0 2024-04-15T15:15:29.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.459+0000 7fae8effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae900b3990 0x7fae9015c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.459+0000 7fae8effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae900b3990 0x7fae9015c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42806/0 (socket says 172.21.15.72:42806) 2024-04-15T15:15:29.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.459+0000 7fae8effd640 1 -- 172.21.15.72:0/4159140890 learned_addr learned my addr 172.21.15.72:0/4159140890 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:29.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.459+0000 7fae8f7fe640 1 --2- 172.21.15.72:0/4159140890 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fae9015e6e0 0x7fae9006d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.459+0000 7fae867fc640 1 --2- 172.21.15.72:0/4159140890 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fae900d49a0 0x7fae900b65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.459+0000 7fae8effd640 1 -- 172.21.15.72:0/4159140890 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fae900d49a0 msgr2=0x7fae900b65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.459+0000 7fae8effd640 1 --2- 172.21.15.72:0/4159140890 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fae900d49a0 0x7fae900b65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.459+0000 7fae8effd640 1 -- 172.21.15.72:0/4159140890 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fae9015e6e0 msgr2=0x7fae9006d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.459+0000 7fae8effd640 1 --2- 172.21.15.72:0/4159140890 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fae9015e6e0 0x7fae9006d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.459+0000 7fae8effd640 1 -- 172.21.15.72:0/4159140890 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae9006c5c0 con 0x7fae900b3990 2024-04-15T15:15:29.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.459+0000 7fae8effd640 1 --2- 172.21.15.72:0/4159140890 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae900b3990 0x7fae9015c750 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fae8002b910 tx=0x7fae800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:29.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.459+0000 7fae87fff640 1 -- 172.21.15.72:0/4159140890 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae8003a9f0 con 0x7fae900b3990 2024-04-15T15:15:29.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.459+0000 7fae87fff640 1 -- 172.21.15.72:0/4159140890 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae8003ab90 con 0x7fae900b3990 2024-04-15T15:15:29.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.459+0000 7fae8ffff640 1 -- 172.21.15.72:0/4159140890 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae9006c8a0 con 0x7fae900b3990 2024-04-15T15:15:29.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.459+0000 7fae87fff640 1 -- 172.21.15.72:0/4159140890 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae800443f0 con 0x7fae900b3990 2024-04-15T15:15:29.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.459+0000 7fae8ffff640 1 -- 172.21.15.72:0/4159140890 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae900b3f60 con 0x7fae900b3990 2024-04-15T15:15:29.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.460+0000 7fae8ffff640 1 -- 172.21.15.72:0/4159140890 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae90075b40 con 0x7fae900b3990 2024-04-15T15:15:29.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.461+0000 7fae87fff640 1 -- 172.21.15.72:0/4159140890 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fae8004b060 con 0x7fae900b3990 2024-04-15T15:15:29.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.461+0000 7fae87fff640 1 --2- 172.21.15.72:0/4159140890 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fae7006c920 0x7fae7006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.461+0000 7fae87fff640 1 -- 172.21.15.72:0/4159140890 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fae800ada90 con 0x7fae900b3990 2024-04-15T15:15:29.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.461+0000 7fae867fc640 1 --2- 172.21.15.72:0/4159140890 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fae7006c920 0x7fae7006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.462+0000 7fae867fc640 1 --2- 172.21.15.72:0/4159140890 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fae7006c920 0x7fae7006ede0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fae9015dc30 tx=0x7fae78009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:29.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.465+0000 7fae87fff640 1 -- 172.21.15.72:0/4159140890 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fae8003f070 con 0x7fae900b3990 2024-04-15T15:15:29.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.640+0000 7fae8ffff640 1 -- 172.21.15.72:0/4159140890 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fae900ceb40 con 0x7fae900b3990 2024-04-15T15:15:29.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.641+0000 7fae87fff640 1 -- 172.21.15.72:0/4159140890 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7fae8002f2e0 con 0x7fae900b3990 2024-04-15T15:15:29.646 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:29.646 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":24,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:14:56.451113+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:14:49.438366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:14:50.368379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:14:54.081157+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:29.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.643+0000 7fae8ffff640 1 -- 172.21.15.72:0/4159140890 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fae7006c920 msgr2=0x7fae7006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.643+0000 7fae8ffff640 1 --2- 172.21.15.72:0/4159140890 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fae7006c920 0x7fae7006ede0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fae9015dc30 tx=0x7fae78009040 comp rx=0 tx=0).stop 2024-04-15T15:15:29.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.643+0000 7fae8ffff640 1 -- 172.21.15.72:0/4159140890 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae900b3990 msgr2=0x7fae9015c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.643+0000 7fae8ffff640 1 --2- 172.21.15.72:0/4159140890 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae900b3990 0x7fae9015c750 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fae8002b910 tx=0x7fae800046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.644+0000 7fae8ffff640 1 -- 172.21.15.72:0/4159140890 shutdown_connections 2024-04-15T15:15:29.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.644+0000 7fae8ffff640 1 --2- 172.21.15.72:0/4159140890 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fae7006c920 0x7fae7006ede0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.644+0000 7fae8ffff640 1 --2- 172.21.15.72:0/4159140890 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fae9015e6e0 0x7fae9006d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.644+0000 7fae8ffff640 1 --2- 172.21.15.72:0/4159140890 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fae900d49a0 0x7fae900b65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.644+0000 7fae8ffff640 1 --2- 172.21.15.72:0/4159140890 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae900b3990 0x7fae9015c750 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.644+0000 7fae8ffff640 1 -- 172.21.15.72:0/4159140890 >> 172.21.15.72:0/4159140890 conn(0x7fae900b9c30 msgr2=0x7fae900b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:29.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.644+0000 7fae8ffff640 1 -- 172.21.15.72:0/4159140890 shutdown_connections 2024-04-15T15:15:29.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.644+0000 7fae8ffff640 1 -- 172.21.15.72:0/4159140890 wait complete. 2024-04-15T15:15:29.658 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:39.359102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:14:49.438366+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:14:50.368379+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:14:54.081157+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:15:29.659 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:29.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.762+0000 7f1131b59640 1 Processor -- start 2024-04-15T15:15:29.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.763+0000 7f1131b59640 1 -- start start 2024-04-15T15:15:29.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.763+0000 7f1131b59640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f112c0ce2f0 0x7f112c15c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.763+0000 7f1131b59640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f112c0d08f0 0x7f112c0cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.763+0000 7f1131b59640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f112c06bbb0 con 0x7f112c0ce6d0 2024-04-15T15:15:29.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.763+0000 7f1131b59640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f112c06b960 con 0x7f112c0d08f0 2024-04-15T15:15:29.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.763+0000 7f1131b59640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f112c06be90 con 0x7f112c0ce2f0 2024-04-15T15:15:29.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.764+0000 7f112affd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f112c0d08f0 0x7f112c0cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.764+0000 7f112bfff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f112c0ce6d0 0x7f112c155600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:48086/0 (socket says 172.21.15.72:48086) 2024-04-15T15:15:29.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.764+0000 7f112bfff640 1 -- 172.21.15.72:0/2095800240 learned_addr learned my addr 172.21.15.72:0/2095800240 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:29.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.764+0000 7f112b7fe640 1 --2- 172.21.15.72:0/2095800240 >> v2:172.21.15.175:3300/0 conn(0x7f112c0ce2f0 0x7f112c15c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.764+0000 7f112affd640 1 -- 172.21.15.72:0/2095800240 >> v2:172.21.15.175:3300/0 conn(0x7f112c0ce2f0 msgr2=0x7f112c15c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:29.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.764+0000 7f112affd640 1 --2- 172.21.15.72:0/2095800240 >> v2:172.21.15.175:3300/0 conn(0x7f112c0ce2f0 0x7f112c15c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.764+0000 7f112affd640 1 -- 172.21.15.72:0/2095800240 >> v1:172.21.15.72:6789/0 conn(0x7f112c0ce6d0 legacy=0x7f112c155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.764+0000 7f112affd640 1 -- 172.21.15.72:0/2095800240 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f112c06c170 con 0x7f112c0d08f0 2024-04-15T15:15:29.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.764+0000 7f112a7fc640 1 -- 172.21.15.72:0/2095800240 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1860490788 0 0) 0x7f112c06bbb0 con 0x7f112c0ce6d0 2024-04-15T15:15:29.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.764+0000 7f112b7fe640 1 --2- 172.21.15.72:0/2095800240 >> v2:172.21.15.175:3300/0 conn(0x7f112c0ce2f0 0x7f112c15c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:29.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.765+0000 7f112affd640 1 --2- 172.21.15.72:0/2095800240 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f112c0d08f0 0x7f112c0cc0b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f112c0cdf10 tx=0x7f111402e0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1930f7915702750 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:29.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.765+0000 7f112a7fc640 1 -- 172.21.15.72:0/2095800240 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f111403b070 con 0x7f112c0d08f0 2024-04-15T15:15:29.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.765+0000 7f112a7fc640 1 -- 172.21.15.72:0/2095800240 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f111402ee50 con 0x7f112c0d08f0 2024-04-15T15:15:29.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.765+0000 7f112a7fc640 1 -- 172.21.15.72:0/2095800240 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1114036660 con 0x7f112c0d08f0 2024-04-15T15:15:29.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.765+0000 7f1131b59640 1 -- 172.21.15.72:0/2095800240 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f112c0d08f0 msgr2=0x7f112c0cc0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.765+0000 7f1131b59640 1 --2- 172.21.15.72:0/2095800240 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f112c0d08f0 0x7f112c0cc0b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f112c0cdf10 tx=0x7f111402e0a0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.765+0000 7f1131b59640 1 -- 172.21.15.72:0/2095800240 shutdown_connections 2024-04-15T15:15:29.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.765+0000 7f1131b59640 1 --2- 172.21.15.72:0/2095800240 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f112c0d08f0 0x7f112c0cc0b0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.765+0000 7f1131b59640 1 --2- 172.21.15.72:0/2095800240 >> v2:172.21.15.175:3300/0 conn(0x7f112c0ce2f0 0x7f112c15c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.765+0000 7f1131b59640 1 -- 172.21.15.72:0/2095800240 >> 172.21.15.72:0/2095800240 conn(0x7f112c0b31d0 msgr2=0x7f112c0b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:29.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.765+0000 7f1131b59640 1 -- 172.21.15.72:0/2095800240 shutdown_connections 2024-04-15T15:15:29.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.765+0000 7f1131b59640 1 -- 172.21.15.72:0/2095800240 wait complete. 2024-04-15T15:15:29.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.766+0000 7f1131b59640 1 Processor -- start 2024-04-15T15:15:29.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.767+0000 7f1131b59640 1 -- start start 2024-04-15T15:15:29.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.767+0000 7f1131b59640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f112c0ce2f0 0x7f112c146660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.767+0000 7f1131b59640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f112c0ce6d0 0x7f112c0b0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.767+0000 7f1131b59640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f112c0d08f0 0x7f112c0ae370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.767+0000 7f1131b59640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f112c155d10 con 0x7f112c0ce2f0 2024-04-15T15:15:29.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.767+0000 7f1131b59640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f112c15ca70 con 0x7f112c0ce6d0 2024-04-15T15:15:29.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.767+0000 7f1131b59640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f112c155e90 con 0x7f112c0d08f0 2024-04-15T15:15:29.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.767+0000 7f112b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f112c0ce2f0 0x7f112c146660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.768+0000 7f112b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f112c0ce2f0 0x7f112c146660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42818/0 (socket says 172.21.15.72:42818) 2024-04-15T15:15:29.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.768+0000 7f112b7fe640 1 -- 172.21.15.72:0/719003292 learned_addr learned my addr 172.21.15.72:0/719003292 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:29.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.768+0000 7f112bfff640 1 --2- 172.21.15.72:0/719003292 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f112c0d08f0 0x7f112c0ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.768+0000 7f112b7fe640 1 -- 172.21.15.72:0/719003292 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f112c0d08f0 msgr2=0x7f112c0ae370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.768+0000 7f112b7fe640 1 --2- 172.21.15.72:0/719003292 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f112c0d08f0 0x7f112c0ae370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.768+0000 7f112affd640 1 --2- 172.21.15.72:0/719003292 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f112c0ce6d0 0x7f112c0b0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.768+0000 7f112b7fe640 1 -- 172.21.15.72:0/719003292 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f112c0ce6d0 msgr2=0x7f112c0b0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.768+0000 7f112b7fe640 1 --2- 172.21.15.72:0/719003292 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f112c0ce6d0 0x7f112c0b0ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.768+0000 7f112b7fe640 1 -- 172.21.15.72:0/719003292 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f112c0b1010 con 0x7f112c0ce2f0 2024-04-15T15:15:29.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.768+0000 7f112b7fe640 1 --2- 172.21.15.72:0/719003292 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f112c0ce2f0 0x7f112c146660 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f111c00ce00 tx=0x7f111c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:29.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.768+0000 7f1123fff640 1 -- 172.21.15.72:0/719003292 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f111c00f890 con 0x7f112c0ce2f0 2024-04-15T15:15:29.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.768+0000 7f1123fff640 1 -- 172.21.15.72:0/719003292 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f111c00fa30 con 0x7f112c0ce2f0 2024-04-15T15:15:29.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.768+0000 7f1123fff640 1 -- 172.21.15.72:0/719003292 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f111c00fe40 con 0x7f112c0ce2f0 2024-04-15T15:15:29.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.768+0000 7f1131b59640 1 -- 172.21.15.72:0/719003292 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f112c0afd10 con 0x7f112c0ce2f0 2024-04-15T15:15:29.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.768+0000 7f1131b59640 1 -- 172.21.15.72:0/719003292 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f112c0b02d0 con 0x7f112c0ce2f0 2024-04-15T15:15:29.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.769+0000 7f1131b59640 1 -- 172.21.15.72:0/719003292 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f112c075c50 con 0x7f112c0ce2f0 2024-04-15T15:15:29.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.770+0000 7f1123fff640 1 -- 172.21.15.72:0/719003292 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f111c018400 con 0x7f112c0ce2f0 2024-04-15T15:15:29.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.770+0000 7f1123fff640 1 --2- 172.21.15.72:0/719003292 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f110c06c8f0 0x7f110c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.770+0000 7f1123fff640 1 -- 172.21.15.72:0/719003292 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f111c08cc60 con 0x7f112c0ce2f0 2024-04-15T15:15:29.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.770+0000 7f112affd640 1 --2- 172.21.15.72:0/719003292 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f110c06c8f0 0x7f110c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.771+0000 7f112affd640 1 --2- 172.21.15.72:0/719003292 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f110c06c8f0 0x7f110c06edb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1114038000 tx=0x7f1114032000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:29.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.775+0000 7f1123fff640 1 -- 172.21.15.72:0/719003292 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f111c014030 con 0x7f112c0ce2f0 2024-04-15T15:15:29.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.948+0000 7f1131b59640 1 -- 172.21.15.72:0/719003292 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f112c15ff90 con 0x7f112c0ce2f0 2024-04-15T15:15:29.951 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.948+0000 7f1123fff640 1 -- 172.21.15.72:0/719003292 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7f111c00ae20 con 0x7f112c0ce2f0 2024-04-15T15:15:29.951 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-15T15:15:29.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.950+0000 7f1131b59640 1 -- 172.21.15.72:0/719003292 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f110c06c8f0 msgr2=0x7f110c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.950+0000 7f1131b59640 1 --2- 172.21.15.72:0/719003292 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f110c06c8f0 0x7f110c06edb0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1114038000 tx=0x7f1114032000 comp rx=0 tx=0).stop 2024-04-15T15:15:29.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.950+0000 7f1131b59640 1 -- 172.21.15.72:0/719003292 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f112c0ce2f0 msgr2=0x7f112c146660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.950+0000 7f1131b59640 1 --2- 172.21.15.72:0/719003292 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f112c0ce2f0 0x7f112c146660 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f111c00ce00 tx=0x7f111c00a650 comp rx=0 tx=0).stop 2024-04-15T15:15:29.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.950+0000 7f1131b59640 1 -- 172.21.15.72:0/719003292 shutdown_connections 2024-04-15T15:15:29.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.950+0000 7f1131b59640 1 --2- 172.21.15.72:0/719003292 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f110c06c8f0 0x7f110c06edb0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.950+0000 7f1131b59640 1 --2- 172.21.15.72:0/719003292 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f112c0d08f0 0x7f112c0ae370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.950+0000 7f1131b59640 1 --2- 172.21.15.72:0/719003292 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f112c0ce6d0 0x7f112c0b0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.950+0000 7f1131b59640 1 --2- 172.21.15.72:0/719003292 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f112c0ce2f0 0x7f112c146660 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.950+0000 7f1131b59640 1 -- 172.21.15.72:0/719003292 >> 172.21.15.72:0/719003292 conn(0x7f112c0b31d0 msgr2=0x7f112c15e600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:29.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.950+0000 7f1131b59640 1 -- 172.21.15.72:0/719003292 shutdown_connections 2024-04-15T15:15:29.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:29.951+0000 7f1131b59640 1 -- 172.21.15.72:0/719003292 wait complete. 2024-04-15T15:15:29.964 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:15:30.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.068+0000 7f4c2427b640 1 Processor -- start 2024-04-15T15:15:30.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.068+0000 7f4c2427b640 1 -- start start 2024-04-15T15:15:30.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.069+0000 7f4c2427b640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f4c1c15e610 0x7f4c1c0d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.069+0000 7f4c2427b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4c1c06c400 0x7f4c1c06c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.069+0000 7f4c2427b640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c1c0d2320 con 0x7f4c1c15e9f0 2024-04-15T15:15:30.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.069+0000 7f4c2427b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f4c1c0d31d0 con 0x7f4c1c06c400 2024-04-15T15:15:30.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.069+0000 7f4c2427b640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f4c1c0d25c0 con 0x7f4c1c15e610 2024-04-15T15:15:30.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.070+0000 7f4c22818640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4c1c06c400 0x7f4c1c06c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.070+0000 7f4c22818640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4c1c06c400 0x7f4c1c06c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42830/0 (socket says 172.21.15.72:42830) 2024-04-15T15:15:30.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.070+0000 7f4c22818640 1 -- 172.21.15.72:0/885638771 learned_addr learned my addr 172.21.15.72:0/885638771 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:30.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.070+0000 7f4c22017640 1 --2- 172.21.15.72:0/885638771 >> v2:172.21.15.175:3300/0 conn(0x7f4c1c15e610 0x7f4c1c0d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.070+0000 7f4c22818640 1 -- 172.21.15.72:0/885638771 >> v2:172.21.15.175:3300/0 conn(0x7f4c1c15e610 msgr2=0x7f4c1c0d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:30.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.070+0000 7f4c22818640 1 --2- 172.21.15.72:0/885638771 >> v2:172.21.15.175:3300/0 conn(0x7f4c1c15e610 0x7f4c1c0d0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.070+0000 7f4c22818640 1 -- 172.21.15.72:0/885638771 >> v1:172.21.15.175:6789/0 conn(0x7f4c1c15e9f0 legacy=0x7f4c1c0d3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.070+0000 7f4c22818640 1 -- 172.21.15.72:0/885638771 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c1c0d28a0 con 0x7f4c1c06c400 2024-04-15T15:15:30.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.071+0000 7f4c22818640 1 --2- 172.21.15.72:0/885638771 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c1c06c400 0x7f4c1c06c890 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c0b72a0 tx=0x7f4c1402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b647245a982d99d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:30.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.071+0000 7f4c21015640 1 -- 172.21.15.72:0/885638771 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1402f850 con 0x7f4c1c06c400 2024-04-15T15:15:30.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.071+0000 7f4c21015640 1 -- 172.21.15.72:0/885638771 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c1402f9f0 con 0x7f4c1c06c400 2024-04-15T15:15:30.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.071+0000 7f4c21015640 1 -- 172.21.15.72:0/885638771 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1402fdc0 con 0x7f4c1c06c400 2024-04-15T15:15:30.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.071+0000 7f4c2427b640 1 -- 172.21.15.72:0/885638771 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c1c06c400 msgr2=0x7f4c1c06c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.071+0000 7f4c2427b640 1 --2- 172.21.15.72:0/885638771 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c1c06c400 0x7f4c1c06c890 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c0b72a0 tx=0x7f4c1402ba30 comp rx=0 tx=0).stop 2024-04-15T15:15:30.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.071+0000 7f4c2427b640 1 -- 172.21.15.72:0/885638771 shutdown_connections 2024-04-15T15:15:30.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.071+0000 7f4c2427b640 1 --2- 172.21.15.72:0/885638771 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c1c06c400 0x7f4c1c06c890 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.071+0000 7f4c2427b640 1 --2- 172.21.15.72:0/885638771 >> v2:172.21.15.175:3300/0 conn(0x7f4c1c15e610 0x7f4c1c0d0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.071+0000 7f4c2427b640 1 -- 172.21.15.72:0/885638771 >> 172.21.15.72:0/885638771 conn(0x7f4c1c0d4750 msgr2=0x7f4c1c0d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:30.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.072+0000 7f4c2427b640 1 -- 172.21.15.72:0/885638771 shutdown_connections 2024-04-15T15:15:30.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.072+0000 7f4c2427b640 1 -- 172.21.15.72:0/885638771 wait complete. 2024-04-15T15:15:30.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.073+0000 7f4c2427b640 1 Processor -- start 2024-04-15T15:15:30.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.073+0000 7f4c2427b640 1 -- start start 2024-04-15T15:15:30.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.073+0000 7f4c2427b640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4c1c06c400 0x7f4c1c0cc600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.073+0000 7f4c2427b640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4c1c15e610 0x7f4c1c0cb180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.073+0000 7f4c2427b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c1c15e9f0 0x7f4c1c0b4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.073+0000 7f4c2427b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f4c1c06d130 con 0x7f4c1c15e9f0 2024-04-15T15:15:30.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.073+0000 7f4c2427b640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f4c1c0d2e60 con 0x7f4c1c06c400 2024-04-15T15:15:30.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.073+0000 7f4c2427b640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f4c1c0d0bd0 con 0x7f4c1c15e610 2024-04-15T15:15:30.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.074+0000 7f4c22818640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c1c15e9f0 0x7f4c1c0b4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.074+0000 7f4c22818640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c1c15e9f0 0x7f4c1c0b4a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42834/0 (socket says 172.21.15.72:42834) 2024-04-15T15:15:30.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.074+0000 7f4c22818640 1 -- 172.21.15.72:0/2859146297 learned_addr learned my addr 172.21.15.72:0/2859146297 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:30.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.074+0000 7f4c21816640 1 --2- 172.21.15.72:0/2859146297 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4c1c15e610 0x7f4c1c0cb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.074+0000 7f4c22017640 1 --2- 172.21.15.72:0/2859146297 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4c1c06c400 0x7f4c1c0cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.074+0000 7f4c22818640 1 -- 172.21.15.72:0/2859146297 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4c1c15e610 msgr2=0x7f4c1c0cb180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.074+0000 7f4c22818640 1 --2- 172.21.15.72:0/2859146297 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4c1c15e610 0x7f4c1c0cb180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.074+0000 7f4c22818640 1 -- 172.21.15.72:0/2859146297 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4c1c06c400 msgr2=0x7f4c1c0cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.074+0000 7f4c22818640 1 --2- 172.21.15.72:0/2859146297 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4c1c06c400 0x7f4c1c0cc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.074+0000 7f4c22818640 1 -- 172.21.15.72:0/2859146297 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c1c157900 con 0x7f4c1c15e9f0 2024-04-15T15:15:30.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.074+0000 7f4c22017640 1 --2- 172.21.15.72:0/2859146297 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4c1c06c400 0x7f4c1c0cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:30.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.074+0000 7f4c22818640 1 --2- 172.21.15.72:0/2859146297 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c1c15e9f0 0x7f4c1c0b4a30 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4c1402ba70 tx=0x7f4c14004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:30.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.074+0000 7f4c1affd640 1 -- 172.21.15.72:0/2859146297 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c14037c40 con 0x7f4c1c15e9f0 2024-04-15T15:15:30.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.074+0000 7f4c1affd640 1 -- 172.21.15.72:0/2859146297 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c14046070 con 0x7f4c1c15e9f0 2024-04-15T15:15:30.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.074+0000 7f4c1affd640 1 -- 172.21.15.72:0/2859146297 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c14041400 con 0x7f4c1c15e9f0 2024-04-15T15:15:30.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.074+0000 7f4c2427b640 1 -- 172.21.15.72:0/2859146297 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c1c0b4f70 con 0x7f4c1c15e9f0 2024-04-15T15:15:30.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.074+0000 7f4c2427b640 1 -- 172.21.15.72:0/2859146297 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c1c157ba0 con 0x7f4c1c15e9f0 2024-04-15T15:15:30.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.075+0000 7f4c2427b640 1 -- 172.21.15.72:0/2859146297 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c1c075c50 con 0x7f4c1c15e9f0 2024-04-15T15:15:30.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.076+0000 7f4c1affd640 1 -- 172.21.15.72:0/2859146297 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4c14040070 con 0x7f4c1c15e9f0 2024-04-15T15:15:30.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.076+0000 7f4c1affd640 1 --2- 172.21.15.72:0/2859146297 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4c0806c8f0 0x7f4c0806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.076+0000 7f4c22017640 1 --2- 172.21.15.72:0/2859146297 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4c0806c8f0 0x7f4c0806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.076+0000 7f4c1affd640 1 -- 172.21.15.72:0/2859146297 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f4c140ae5d0 con 0x7f4c1c15e9f0 2024-04-15T15:15:30.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.077+0000 7f4c22017640 1 --2- 172.21.15.72:0/2859146297 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4c0806c8f0 0x7f4c0806edb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c007520 tx=0x7f4c0c00b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:30.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.081+0000 7f4c1affd640 1 -- 172.21.15.72:0/2859146297 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4c14033030 con 0x7f4c1c15e9f0 2024-04-15T15:15:30.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.256+0000 7f4c2427b640 1 -- 172.21.15.72:0/2859146297 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4c1c158320 con 0x7f4c1c15e9f0 2024-04-15T15:15:30.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.257+0000 7f4c1affd640 1 -- 172.21.15.72:0/2859146297 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7f4c1407c930 con 0x7f4c1c15e9f0 2024-04-15T15:15:30.259 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:30.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.259+0000 7f4c2427b640 1 -- 172.21.15.72:0/2859146297 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4c0806c8f0 msgr2=0x7f4c0806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.259+0000 7f4c2427b640 1 --2- 172.21.15.72:0/2859146297 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4c0806c8f0 0x7f4c0806edb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c007520 tx=0x7f4c0c00b040 comp rx=0 tx=0).stop 2024-04-15T15:15:30.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.259+0000 7f4c2427b640 1 -- 172.21.15.72:0/2859146297 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c1c15e9f0 msgr2=0x7f4c1c0b4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.259+0000 7f4c2427b640 1 --2- 172.21.15.72:0/2859146297 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c1c15e9f0 0x7f4c1c0b4a30 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4c1402ba70 tx=0x7f4c14004730 comp rx=0 tx=0).stop 2024-04-15T15:15:30.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.259+0000 7f4c2427b640 1 -- 172.21.15.72:0/2859146297 shutdown_connections 2024-04-15T15:15:30.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.259+0000 7f4c2427b640 1 --2- 172.21.15.72:0/2859146297 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4c0806c8f0 0x7f4c0806edb0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.259+0000 7f4c2427b640 1 --2- 172.21.15.72:0/2859146297 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c1c15e9f0 0x7f4c1c0b4a30 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.259+0000 7f4c2427b640 1 --2- 172.21.15.72:0/2859146297 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4c1c15e610 0x7f4c1c0cb180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.259+0000 7f4c2427b640 1 --2- 172.21.15.72:0/2859146297 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4c1c06c400 0x7f4c1c0cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.259+0000 7f4c2427b640 1 -- 172.21.15.72:0/2859146297 >> 172.21.15.72:0/2859146297 conn(0x7f4c1c0d4750 msgr2=0x7f4c1c0d4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:30.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.259+0000 7f4c2427b640 1 -- 172.21.15.72:0/2859146297 shutdown_connections 2024-04-15T15:15:30.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.259+0000 7f4c2427b640 1 -- 172.21.15.72:0/2859146297 wait complete. 2024-04-15T15:15:30.272 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:15:30.379 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.376+0000 7feb22463640 1 Processor -- start 2024-04-15T15:15:30.379 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.376+0000 7feb22463640 1 -- start start 2024-04-15T15:15:30.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.377+0000 7feb22463640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7feb1c15c650 0x7feb1c0b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.377+0000 7feb22463640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7feb1c0d0a60 0x7feb1c0d0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.377+0000 7feb22463640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb1c0d02d0 con 0x7feb1c0b36f0 2024-04-15T15:15:30.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.377+0000 7feb22463640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7feb1c15ca30 con 0x7feb1c0d0a60 2024-04-15T15:15:30.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.377+0000 7feb22463640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7feb1c0b26b0 con 0x7feb1c15c650 2024-04-15T15:15:30.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.377+0000 7feb1b7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7feb1c0b36f0 0x7feb1c0b5290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:48098/0 (socket says 172.21.15.72:48098) 2024-04-15T15:15:30.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.377+0000 7feb1b7fe640 1 -- 172.21.15.72:0/1882698986 learned_addr learned my addr 172.21.15.72:0/1882698986 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:30.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.377+0000 7feb20a00640 1 --2- 172.21.15.72:0/1882698986 >> v2:172.21.15.175:3300/0 conn(0x7feb1c0d0a60 0x7feb1c0d0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.377+0000 7feb1bfff640 1 --2- 172.21.15.72:0/1882698986 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7feb1c15c650 0x7feb1c0b1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.377+0000 7feb1affd640 1 -- 172.21.15.72:0/1882698986 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1377678181 0 0) 0x7feb1c0d02d0 con 0x7feb1c0b36f0 2024-04-15T15:15:30.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.378+0000 7feb1affd640 1 -- 172.21.15.72:0/1882698986 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb08003520 con 0x7feb1c0b36f0 2024-04-15T15:15:30.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.378+0000 7feb1affd640 1 -- 172.21.15.72:0/1882698986 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1148485112 0 0) 0x7feb08003520 con 0x7feb1c0b36f0 2024-04-15T15:15:30.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.378+0000 7feb1affd640 1 -- 172.21.15.72:0/1882698986 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7feb1c15c650 msgr2=0x7feb1c0b1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:30.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.378+0000 7feb1affd640 1 --2- 172.21.15.72:0/1882698986 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7feb1c15c650 0x7feb1c0b1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.378+0000 7feb1affd640 1 -- 172.21.15.72:0/1882698986 >> v2:172.21.15.175:3300/0 conn(0x7feb1c0d0a60 msgr2=0x7feb1c0d0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:30.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.378+0000 7feb1affd640 1 --2- 172.21.15.72:0/1882698986 >> v2:172.21.15.175:3300/0 conn(0x7feb1c0d0a60 0x7feb1c0d0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.378+0000 7feb1affd640 1 -- 172.21.15.72:0/1882698986 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb1c0b2990 con 0x7feb1c0b36f0 2024-04-15T15:15:30.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.378+0000 7feb1affd640 1 -- 172.21.15.72:0/1882698986 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7feb04001670 con 0x7feb1c0b36f0 2024-04-15T15:15:30.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.378+0000 7feb1affd640 1 -- 172.21.15.72:0/1882698986 >> v1:172.21.15.72:6789/0 conn(0x7feb1c0b36f0 legacy=0x7feb1c0b5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.378+0000 7feb1affd640 1 --2- 172.21.15.72:0/1882698986 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb080039b0 0x7feb08003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.378+0000 7feb1affd640 1 --2- 172.21.15.72:0/1882698986 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7feb1c15c650 0x7feb08004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.378+0000 7feb1affd640 1 --2- 172.21.15.72:0/1882698986 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feb08004d40 0x7feb080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.378+0000 7feb1affd640 1 -- 172.21.15.72:0/1882698986 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7feb1c15ca30 con 0x7feb080039b0 2024-04-15T15:15:30.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.378+0000 7feb1affd640 1 -- 172.21.15.72:0/1882698986 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7feb1c0b26b0 con 0x7feb08004d40 2024-04-15T15:15:30.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.378+0000 7feb1affd640 1 -- 172.21.15.72:0/1882698986 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7feb0800b9c0 con 0x7feb1c15c650 2024-04-15T15:15:30.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.378+0000 7feb1bfff640 1 --2- 172.21.15.72:0/1882698986 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb080039b0 0x7feb08003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.378+0000 7feb20a00640 1 --2- 172.21.15.72:0/1882698986 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feb08004d40 0x7feb080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.378+0000 7feb1b7fe640 1 --2- 172.21.15.72:0/1882698986 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7feb1c15c650 0x7feb08004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.378+0000 7feb1bfff640 1 -- 172.21.15.72:0/1882698986 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7feb1c15c650 msgr2=0x7feb08004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:30.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.378+0000 7feb1bfff640 1 --2- 172.21.15.72:0/1882698986 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7feb1c15c650 0x7feb08004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.378+0000 7feb1bfff640 1 -- 172.21.15.72:0/1882698986 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feb08004d40 msgr2=0x7feb080051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:30.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.378+0000 7feb1bfff640 1 --2- 172.21.15.72:0/1882698986 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feb08004d40 0x7feb080051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.378+0000 7feb1bfff640 1 -- 172.21.15.72:0/1882698986 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb08003520 con 0x7feb080039b0 2024-04-15T15:15:30.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.378+0000 7feb1b7fe640 1 --2- 172.21.15.72:0/1882698986 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7feb1c15c650 0x7feb08004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:30.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.378+0000 7feb20a00640 1 --2- 172.21.15.72:0/1882698986 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feb08004d40 0x7feb080051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:30.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.379+0000 7feb1bfff640 1 --2- 172.21.15.72:0/1882698986 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb080039b0 0x7feb08003dd0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7feb0c009fc0 tx=0x7feb08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=999a61ed66bf4041 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:30.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.379+0000 7feb1affd640 1 -- 172.21.15.72:0/1882698986 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c039070 con 0x7feb080039b0 2024-04-15T15:15:30.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.379+0000 7feb1affd640 1 -- 172.21.15.72:0/1882698986 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb0c02cc20 con 0x7feb080039b0 2024-04-15T15:15:30.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.379+0000 7feb1affd640 1 -- 172.21.15.72:0/1882698986 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0c009c10 con 0x7feb080039b0 2024-04-15T15:15:30.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.380+0000 7feb22463640 1 -- 172.21.15.72:0/1882698986 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb080039b0 msgr2=0x7feb08003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.380+0000 7feb22463640 1 --2- 172.21.15.72:0/1882698986 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb080039b0 0x7feb08003dd0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7feb0c009fc0 tx=0x7feb08005830 comp rx=0 tx=0).stop 2024-04-15T15:15:30.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.380+0000 7feb20a00640 1 -- 172.21.15.72:0/1882698986 reap_dead start 2024-04-15T15:15:30.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.380+0000 7feb22463640 1 -- 172.21.15.72:0/1882698986 shutdown_connections 2024-04-15T15:15:30.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.380+0000 7feb22463640 1 -- 172.21.15.72:0/1882698986 >> 172.21.15.72:0/1882698986 conn(0x7feb1c0ceec0 msgr2=0x7feb1c0cf2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:30.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.380+0000 7feb22463640 1 -- 172.21.15.72:0/1882698986 shutdown_connections 2024-04-15T15:15:30.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.380+0000 7feb22463640 1 -- 172.21.15.72:0/1882698986 wait complete. 2024-04-15T15:15:30.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.381+0000 7feb22463640 1 Processor -- start 2024-04-15T15:15:30.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.381+0000 7feb22463640 1 -- start start 2024-04-15T15:15:30.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.382+0000 7feb22463640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feb080039b0 0x7feb1c0c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.382+0000 7feb22463640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7feb1c15f760 0x7feb1c15fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.382+0000 7feb22463640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb1c1f2880 0x7feb1c1f2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.382+0000 7feb22463640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7feb1c0b59a0 con 0x7feb1c1f2880 2024-04-15T15:15:30.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.382+0000 7feb22463640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7feb1c0b2c70 con 0x7feb080039b0 2024-04-15T15:15:30.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.382+0000 7feb22463640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7feb1c0b1d40 con 0x7feb1c15f760 2024-04-15T15:15:30.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.382+0000 7feb20a00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb1c1f2880 0x7feb1c1f2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.382+0000 7feb20a00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb1c1f2880 0x7feb1c1f2d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42852/0 (socket says 172.21.15.72:42852) 2024-04-15T15:15:30.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.382+0000 7feb20a00640 1 -- 172.21.15.72:0/3120956955 learned_addr learned my addr 172.21.15.72:0/3120956955 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:30.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.382+0000 7feb1b7fe640 1 --2- 172.21.15.72:0/3120956955 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7feb1c15f760 0x7feb1c15fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.382+0000 7feb1bfff640 1 --2- 172.21.15.72:0/3120956955 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feb080039b0 0x7feb1c0c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.382+0000 7feb20a00640 1 -- 172.21.15.72:0/3120956955 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7feb1c15f760 msgr2=0x7feb1c15fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.382+0000 7feb20a00640 1 --2- 172.21.15.72:0/3120956955 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7feb1c15f760 0x7feb1c15fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.382+0000 7feb20a00640 1 -- 172.21.15.72:0/3120956955 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feb080039b0 msgr2=0x7feb1c0c5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.382+0000 7feb20a00640 1 --2- 172.21.15.72:0/3120956955 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feb080039b0 0x7feb1c0c5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.382+0000 7feb20a00640 1 -- 172.21.15.72:0/3120956955 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb1c1f3450 con 0x7feb1c1f2880 2024-04-15T15:15:30.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.382+0000 7feb20a00640 1 --2- 172.21.15.72:0/3120956955 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb1c1f2880 0x7feb1c1f2d00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7feb1c0d3540 tx=0x7feb14011860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:30.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.383+0000 7feb18ff9640 1 -- 172.21.15.72:0/3120956955 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb14019020 con 0x7feb1c1f2880 2024-04-15T15:15:30.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.383+0000 7feb18ff9640 1 -- 172.21.15.72:0/3120956955 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb14004460 con 0x7feb1c1f2880 2024-04-15T15:15:30.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.383+0000 7feb22463640 1 -- 172.21.15.72:0/3120956955 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb1c1f3730 con 0x7feb1c1f2880 2024-04-15T15:15:30.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.383+0000 7feb18ff9640 1 -- 172.21.15.72:0/3120956955 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb14003850 con 0x7feb1c1f2880 2024-04-15T15:15:30.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.383+0000 7feb22463640 1 -- 172.21.15.72:0/3120956955 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb1c1f3cf0 con 0x7feb1c1f2880 2024-04-15T15:15:30.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.384+0000 7feb22463640 1 -- 172.21.15.72:0/3120956955 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb1c075c50 con 0x7feb1c1f2880 2024-04-15T15:15:30.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.384+0000 7feb18ff9640 1 -- 172.21.15.72:0/3120956955 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7feb1401c020 con 0x7feb1c1f2880 2024-04-15T15:15:30.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.385+0000 7feb18ff9640 1 --2- 172.21.15.72:0/3120956955 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7feafc06c920 0x7feafc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.385+0000 7feb18ff9640 1 -- 172.21.15.72:0/3120956955 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7feb1408bd50 con 0x7feb1c1f2880 2024-04-15T15:15:30.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.385+0000 7feb1bfff640 1 --2- 172.21.15.72:0/3120956955 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7feafc06c920 0x7feafc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.385+0000 7feb1bfff640 1 --2- 172.21.15.72:0/3120956955 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7feafc06c920 0x7feafc06ede0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7feb0c036000 tx=0x7feb0c02e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:30.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.389+0000 7feb18ff9640 1 -- 172.21.15.72:0/3120956955 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7feb1408f030 con 0x7feb1c1f2880 2024-04-15T15:15:30.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.564+0000 7feb22463640 1 -- 172.21.15.72:0/3120956955 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7feb1c0d0010 con 0x7feb1c1f2880 2024-04-15T15:15:30.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.564+0000 7feb18ff9640 1 -- 172.21.15.72:0/3120956955 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7feb14023030 con 0x7feb1c1f2880 2024-04-15T15:15:30.567 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-04-15T15:15:30.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.566+0000 7feb22463640 1 -- 172.21.15.72:0/3120956955 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7feafc06c920 msgr2=0x7feafc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.566+0000 7feb22463640 1 --2- 172.21.15.72:0/3120956955 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7feafc06c920 0x7feafc06ede0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7feb0c036000 tx=0x7feb0c02e000 comp rx=0 tx=0).stop 2024-04-15T15:15:30.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.566+0000 7feb22463640 1 -- 172.21.15.72:0/3120956955 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb1c1f2880 msgr2=0x7feb1c1f2d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.566+0000 7feb22463640 1 --2- 172.21.15.72:0/3120956955 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb1c1f2880 0x7feb1c1f2d00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7feb1c0d3540 tx=0x7feb14011860 comp rx=0 tx=0).stop 2024-04-15T15:15:30.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.566+0000 7feb22463640 1 -- 172.21.15.72:0/3120956955 shutdown_connections 2024-04-15T15:15:30.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.566+0000 7feb22463640 1 --2- 172.21.15.72:0/3120956955 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7feafc06c920 0x7feafc06ede0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.566+0000 7feb22463640 1 --2- 172.21.15.72:0/3120956955 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7feb1c1f2880 0x7feb1c1f2d00 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.566+0000 7feb22463640 1 --2- 172.21.15.72:0/3120956955 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7feb1c15f760 0x7feb1c15fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.566+0000 7feb22463640 1 --2- 172.21.15.72:0/3120956955 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7feb080039b0 0x7feb1c0c5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.566+0000 7feb22463640 1 -- 172.21.15.72:0/3120956955 >> 172.21.15.72:0/3120956955 conn(0x7feb1c0ceec0 msgr2=0x7feb1c06ffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:30.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.566+0000 7feb22463640 1 -- 172.21.15.72:0/3120956955 shutdown_connections 2024-04-15T15:15:30.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.567+0000 7feb22463640 1 -- 172.21.15.72:0/3120956955 wait complete. 2024-04-15T15:15:30.580 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:15:30.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.684+0000 7f18fdf04640 1 Processor -- start 2024-04-15T15:15:30.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.684+0000 7f18fdf04640 1 -- start start 2024-04-15T15:15:30.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.684+0000 7f18fdf04640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f18f80ce2f0 0x7f18f815c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.685+0000 7f18fdf04640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18f806bbb0 con 0x7f18f80d08f0 2024-04-15T15:15:30.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.685+0000 7f18fdf04640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18f806be90 con 0x7f18f80cc7c0 2024-04-15T15:15:30.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.685+0000 7f18fdf04640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f18f806b960 con 0x7f18f80ce2f0 2024-04-15T15:15:30.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.685+0000 7f18f77fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f18f80ce2f0 0x7f18f815c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.685+0000 7f18f6ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f18f80d08f0 0x7f18f80cc0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:48100/0 (socket says 172.21.15.72:48100) 2024-04-15T15:15:30.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.685+0000 7f18f6ffd640 1 -- 172.21.15.72:0/3596043825 learned_addr learned my addr 172.21.15.72:0/3596043825 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:30.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.685+0000 7f18f77fe640 1 -- 172.21.15.72:0/3596043825 >> v1:172.21.15.175:6789/0 conn(0x7f18f80cc7c0 legacy=0x7f18f8155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:30.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.685+0000 7f18f77fe640 1 -- 172.21.15.72:0/3596043825 >> v1:172.21.15.72:6789/0 conn(0x7f18f80d08f0 legacy=0x7f18f80cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.685+0000 7f18f67fc640 1 -- 172.21.15.72:0/3596043825 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 200810032 0 0) 0x7f18f806bbb0 con 0x7f18f80d08f0 2024-04-15T15:15:30.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.685+0000 7f18f77fe640 1 -- 172.21.15.72:0/3596043825 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18f806c170 con 0x7f18f80ce2f0 2024-04-15T15:15:30.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.686+0000 7f18f77fe640 1 --2- 172.21.15.72:0/3596043825 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18f80ce2f0 0x7f18f815c530 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f18f806bae0 tx=0x7f18e802e0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8876a3f228329179 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:30.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.686+0000 7f18f67fc640 1 -- 172.21.15.72:0/3596043825 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e802ea50 con 0x7f18f80ce2f0 2024-04-15T15:15:30.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.686+0000 7f18f67fc640 1 -- 172.21.15.72:0/3596043825 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18e802ebf0 con 0x7f18f80ce2f0 2024-04-15T15:15:30.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.686+0000 7f18f67fc640 1 -- 172.21.15.72:0/3596043825 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e8009540 con 0x7f18f80ce2f0 2024-04-15T15:15:30.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.686+0000 7f18fdf04640 1 -- 172.21.15.72:0/3596043825 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18f80ce2f0 msgr2=0x7f18f815c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.686+0000 7f18fdf04640 1 --2- 172.21.15.72:0/3596043825 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18f80ce2f0 0x7f18f815c530 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f18f806bae0 tx=0x7f18e802e0b0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.686+0000 7f18fdf04640 1 -- 172.21.15.72:0/3596043825 shutdown_connections 2024-04-15T15:15:30.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.686+0000 7f18fdf04640 1 --2- 172.21.15.72:0/3596043825 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18f80ce2f0 0x7f18f815c530 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.686+0000 7f18fdf04640 1 -- 172.21.15.72:0/3596043825 >> 172.21.15.72:0/3596043825 conn(0x7f18f80b31d0 msgr2=0x7f18f80b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:30.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.686+0000 7f18fdf04640 1 -- 172.21.15.72:0/3596043825 shutdown_connections 2024-04-15T15:15:30.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.686+0000 7f18fdf04640 1 -- 172.21.15.72:0/3596043825 wait complete. 2024-04-15T15:15:30.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.688+0000 7f18fdf04640 1 Processor -- start 2024-04-15T15:15:30.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.688+0000 7f18fdf04640 1 -- start start 2024-04-15T15:15:30.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.688+0000 7f18fdf04640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f18f80cc7c0 0x7f18f80b1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.688+0000 7f18fdf04640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18f80ce2f0 0x7f18f814d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.688+0000 7f18fdf04640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18f80d08f0 0x7f18f809d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.689+0000 7f18fdf04640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f18f806edd0 con 0x7f18f80d08f0 2024-04-15T15:15:30.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.689+0000 7f18fdf04640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f18f815ca70 con 0x7f18f80ce2f0 2024-04-15T15:15:30.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.689+0000 7f18fdf04640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f18f806ef50 con 0x7f18f80cc7c0 2024-04-15T15:15:30.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.689+0000 7f18f7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18f80d08f0 0x7f18f809d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.689+0000 7f18f7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18f80d08f0 0x7f18f809d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42876/0 (socket says 172.21.15.72:42876) 2024-04-15T15:15:30.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.689+0000 7f18f7fff640 1 -- 172.21.15.72:0/1465668014 learned_addr learned my addr 172.21.15.72:0/1465668014 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:30.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.689+0000 7f18f7fff640 1 -- 172.21.15.72:0/1465668014 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f18f80cc7c0 msgr2=0x7f18f80b1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.689+0000 7f18f7fff640 1 --2- 172.21.15.72:0/1465668014 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f18f80cc7c0 0x7f18f80b1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.694 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.689+0000 7f18f7fff640 1 -- 172.21.15.72:0/1465668014 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18f80ce2f0 msgr2=0x7f18f814d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.689+0000 7f18f6ffd640 1 --2- 172.21.15.72:0/1465668014 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18f80ce2f0 0x7f18f814d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.689+0000 7f18f7fff640 1 --2- 172.21.15.72:0/1465668014 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18f80ce2f0 0x7f18f814d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.689+0000 7f18f7fff640 1 -- 172.21.15.72:0/1465668014 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18f8159140 con 0x7f18f80d08f0 2024-04-15T15:15:30.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.689+0000 7f18f7fff640 1 --2- 172.21.15.72:0/1465668014 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18f80d08f0 0x7f18f809d540 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f18f000c360 tx=0x7f18f000c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:30.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.689+0000 7f18effff640 1 -- 172.21.15.72:0/1465668014 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f0017070 con 0x7f18f80d08f0 2024-04-15T15:15:30.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.689+0000 7f18effff640 1 -- 172.21.15.72:0/1465668014 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18f00035f0 con 0x7f18f80d08f0 2024-04-15T15:15:30.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.689+0000 7f18effff640 1 -- 172.21.15.72:0/1465668014 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f0003a20 con 0x7f18f80d08f0 2024-04-15T15:15:30.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.689+0000 7f18fdf04640 1 -- 172.21.15.72:0/1465668014 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18f80b2390 con 0x7f18f80d08f0 2024-04-15T15:15:30.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.689+0000 7f18fdf04640 1 -- 172.21.15.72:0/1465668014 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18f809a410 con 0x7f18f80d08f0 2024-04-15T15:15:30.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.690+0000 7f18fdf04640 1 -- 172.21.15.72:0/1465668014 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18f8075c50 con 0x7f18f80d08f0 2024-04-15T15:15:30.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.691+0000 7f18effff640 1 -- 172.21.15.72:0/1465668014 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f18f001a070 con 0x7f18f80d08f0 2024-04-15T15:15:30.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.691+0000 7f18effff640 1 --2- 172.21.15.72:0/1465668014 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f18d806c8f0 0x7f18d806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.691+0000 7f18effff640 1 -- 172.21.15.72:0/1465668014 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f18f008b1f0 con 0x7f18f80d08f0 2024-04-15T15:15:30.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.691+0000 7f18f77fe640 1 --2- 172.21.15.72:0/1465668014 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f18d806c8f0 0x7f18d806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.692+0000 7f18f77fe640 1 --2- 172.21.15.72:0/1465668014 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f18d806c8f0 0x7f18d806edb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f18e8037000 tx=0x7f18e8030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:30.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.695+0000 7f18effff640 1 -- 172.21.15.72:0/1465668014 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f18f8159140 con 0x7f18f80d08f0 2024-04-15T15:15:30.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.868+0000 7f18fdf04640 1 -- 172.21.15.72:0/1465668014 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f18f815a8a0 con 0x7f18f80d08f0 2024-04-15T15:15:30.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.868+0000 7f18effff640 1 -- 172.21.15.72:0/1465668014 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7f18f005f030 con 0x7f18f80d08f0 2024-04-15T15:15:30.871 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:30.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.870+0000 7f18fdf04640 1 -- 172.21.15.72:0/1465668014 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f18d806c8f0 msgr2=0x7f18d806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.870+0000 7f18fdf04640 1 --2- 172.21.15.72:0/1465668014 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f18d806c8f0 0x7f18d806edb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f18e8037000 tx=0x7f18e8030000 comp rx=0 tx=0).stop 2024-04-15T15:15:30.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.870+0000 7f18fdf04640 1 -- 172.21.15.72:0/1465668014 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18f80d08f0 msgr2=0x7f18f809d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.870+0000 7f18fdf04640 1 --2- 172.21.15.72:0/1465668014 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18f80d08f0 0x7f18f809d540 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f18f000c360 tx=0x7f18f000c820 comp rx=0 tx=0).stop 2024-04-15T15:15:30.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.870+0000 7f18fdf04640 1 -- 172.21.15.72:0/1465668014 shutdown_connections 2024-04-15T15:15:30.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.870+0000 7f18fdf04640 1 --2- 172.21.15.72:0/1465668014 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f18d806c8f0 0x7f18d806edb0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.870+0000 7f18fdf04640 1 --2- 172.21.15.72:0/1465668014 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18f80d08f0 0x7f18f809d540 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.870+0000 7f18fdf04640 1 --2- 172.21.15.72:0/1465668014 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f18f80ce2f0 0x7f18f814d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.870+0000 7f18fdf04640 1 --2- 172.21.15.72:0/1465668014 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f18f80cc7c0 0x7f18f80b1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.870+0000 7f18fdf04640 1 -- 172.21.15.72:0/1465668014 >> 172.21.15.72:0/1465668014 conn(0x7f18f80b31d0 msgr2=0x7f18f815f0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:30.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.870+0000 7f18fdf04640 1 -- 172.21.15.72:0/1465668014 shutdown_connections 2024-04-15T15:15:30.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.870+0000 7f18fdf04640 1 -- 172.21.15.72:0/1465668014 wait complete. 2024-04-15T15:15:30.884 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:30.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.985+0000 7f10bf07b640 1 Processor -- start 2024-04-15T15:15:30.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.985+0000 7f10bf07b640 1 -- start start 2024-04-15T15:15:30.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.986+0000 7f10bf07b640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f10b8156d30 0x7f10b80d11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.986+0000 7f10bf07b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10b80d0100 con 0x7f10b80cd160 2024-04-15T15:15:30.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.986+0000 7f10bf07b640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10b80d03e0 con 0x7f10b80cd5d0 2024-04-15T15:15:30.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.986+0000 7f10bf07b640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f10b80d06c0 con 0x7f10b8156d30 2024-04-15T15:15:30.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.986+0000 7f10b7fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f10b80cd160 0x7f10b80cf360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:48114/0 (socket says 172.21.15.72:48114) 2024-04-15T15:15:30.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.986+0000 7f10b7fff640 1 -- 172.21.15.72:0/2772379791 learned_addr learned my addr 172.21.15.72:0/2772379791 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:30.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.986+0000 7f10b77fe640 1 -- 172.21.15.72:0/2772379791 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1120817030 0 0) 0x7f10b80d0100 con 0x7f10b80cd160 2024-04-15T15:15:30.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.986+0000 7f10b77fe640 1 -- 172.21.15.72:0/2772379791 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10a4003520 con 0x7f10b80cd160 2024-04-15T15:15:30.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.986+0000 7f10bce17640 1 --2- 172.21.15.72:0/2772379791 >> v2:172.21.15.175:3300/0 conn(0x7f10b8156d30 0x7f10b80d11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.987+0000 7f10b77fe640 1 -- 172.21.15.72:0/2772379791 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2581351029 0 0) 0x7f10a4003520 con 0x7f10b80cd160 2024-04-15T15:15:30.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.987+0000 7f10b77fe640 1 -- 172.21.15.72:0/2772379791 >> v2:172.21.15.175:3300/0 conn(0x7f10b8156d30 msgr2=0x7f10b80d11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:30.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.987+0000 7f10b77fe640 1 --2- 172.21.15.72:0/2772379791 >> v2:172.21.15.175:3300/0 conn(0x7f10b8156d30 0x7f10b80d11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.987+0000 7f10b77fe640 1 -- 172.21.15.72:0/2772379791 >> v1:172.21.15.175:6789/0 conn(0x7f10b80cd5d0 legacy=0x7f10b815dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.987+0000 7f10b77fe640 1 -- 172.21.15.72:0/2772379791 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10b80d09a0 con 0x7f10b80cd160 2024-04-15T15:15:30.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.987+0000 7f10b77fe640 1 -- 172.21.15.72:0/2772379791 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f10a00019f0 con 0x7f10b80cd160 2024-04-15T15:15:30.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.987+0000 7f10b77fe640 1 -- 172.21.15.72:0/2772379791 >> v1:172.21.15.72:6789/0 conn(0x7f10b80cd160 legacy=0x7f10b80cf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.987+0000 7f10b77fe640 1 --2- 172.21.15.72:0/2772379791 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f10a40039b0 0x7f10a4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.987+0000 7f10b77fe640 1 --2- 172.21.15.72:0/2772379791 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f10a4004800 0x7f10a4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.987+0000 7f10b77fe640 1 --2- 172.21.15.72:0/2772379791 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10a40051c0 0x7f10a400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.987+0000 7f10b77fe640 1 -- 172.21.15.72:0/2772379791 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f10b80d06c0 con 0x7f10a40051c0 2024-04-15T15:15:30.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.987+0000 7f10b77fe640 1 -- 172.21.15.72:0/2772379791 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f10a400c060 con 0x7f10a4004800 2024-04-15T15:15:30.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.987+0000 7f10b77fe640 1 -- 172.21.15.72:0/2772379791 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f10a400c260 con 0x7f10a40039b0 2024-04-15T15:15:30.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.987+0000 7f10bd618640 1 --2- 172.21.15.72:0/2772379791 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10a40051c0 0x7f10a400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.987+0000 7f10b7fff640 1 --2- 172.21.15.72:0/2772379791 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f10a4004800 0x7f10a4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.987+0000 7f10bce17640 1 --2- 172.21.15.72:0/2772379791 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f10a40039b0 0x7f10a4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.987+0000 7f10bd618640 1 -- 172.21.15.72:0/2772379791 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f10a40039b0 msgr2=0x7f10a4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:30.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.987+0000 7f10bd618640 1 --2- 172.21.15.72:0/2772379791 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f10a40039b0 0x7f10a4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.987+0000 7f10bd618640 1 -- 172.21.15.72:0/2772379791 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f10a4004800 msgr2=0x7f10a4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:30.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.987+0000 7f10bd618640 1 --2- 172.21.15.72:0/2772379791 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f10a4004800 0x7f10a4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.987+0000 7f10bd618640 1 -- 172.21.15.72:0/2772379791 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10a4003520 con 0x7f10a40051c0 2024-04-15T15:15:30.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.988+0000 7f10bd618640 1 -- 172.21.15.72:0/2772379791 reap_dead start 2024-04-15T15:15:30.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.988+0000 7f10bd618640 1 --2- 172.21.15.72:0/2772379791 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10a40051c0 0x7f10a400b9c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f10b0001750 tx=0x7f10a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c867f0059de19025 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:30.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.988+0000 7f10b77fe640 1 -- 172.21.15.72:0/2772379791 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b0001370 con 0x7f10a40051c0 2024-04-15T15:15:30.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.988+0000 7f10b77fe640 1 -- 172.21.15.72:0/2772379791 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10b0005310 con 0x7f10a40051c0 2024-04-15T15:15:30.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.988+0000 7f10b77fe640 1 -- 172.21.15.72:0/2772379791 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b0003830 con 0x7f10a40051c0 2024-04-15T15:15:30.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.989+0000 7f10bf07b640 1 -- 172.21.15.72:0/2772379791 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10a40051c0 msgr2=0x7f10a400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.989+0000 7f10bf07b640 1 --2- 172.21.15.72:0/2772379791 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10a40051c0 0x7f10a400b9c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f10b0001750 tx=0x7f10a4005830 comp rx=0 tx=0).stop 2024-04-15T15:15:30.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.989+0000 7f10bf07b640 1 -- 172.21.15.72:0/2772379791 shutdown_connections 2024-04-15T15:15:30.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.989+0000 7f10bf07b640 1 --2- 172.21.15.72:0/2772379791 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10a40051c0 0x7f10a400b9c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.989+0000 7f10bf07b640 1 -- 172.21.15.72:0/2772379791 >> 172.21.15.72:0/2772379791 conn(0x7f10b80b43c0 msgr2=0x7f10b80b47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:30.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.989+0000 7f10bf07b640 1 -- 172.21.15.72:0/2772379791 shutdown_connections 2024-04-15T15:15:30.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.989+0000 7f10bf07b640 1 -- 172.21.15.72:0/2772379791 wait complete. 2024-04-15T15:15:30.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.990+0000 7f10bf07b640 1 Processor -- start 2024-04-15T15:15:30.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.990+0000 7f10bf07b640 1 -- start start 2024-04-15T15:15:30.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.991+0000 7f10bf07b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10a40051c0 0x7f10b80b2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.991+0000 7f10bf07b640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f10b81499c0 0x7f10b80b6c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.991+0000 7f10bf07b640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f10b8149e40 0x7f10b80b7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.991+0000 7f10bf07b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f10b80cce90 con 0x7f10a40051c0 2024-04-15T15:15:30.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.991+0000 7f10bf07b640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f10b80ccd10 con 0x7f10b81499c0 2024-04-15T15:15:30.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.991+0000 7f10bf07b640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f10b80d11a0 con 0x7f10b8149e40 2024-04-15T15:15:30.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.991+0000 7f10bce17640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10a40051c0 0x7f10b80b2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.991+0000 7f10bce17640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10a40051c0 0x7f10b80b2ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42894/0 (socket says 172.21.15.72:42894) 2024-04-15T15:15:30.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.991+0000 7f10bce17640 1 -- 172.21.15.72:0/1914425950 learned_addr learned my addr 172.21.15.72:0/1914425950 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:30.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.991+0000 7f10bd618640 1 --2- 172.21.15.72:0/1914425950 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f10b8149e40 0x7f10b80b7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.991+0000 7f10b7fff640 1 --2- 172.21.15.72:0/1914425950 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f10b81499c0 0x7f10b80b6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.991+0000 7f10bce17640 1 -- 172.21.15.72:0/1914425950 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f10b8149e40 msgr2=0x7f10b80b7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.991+0000 7f10bce17640 1 --2- 172.21.15.72:0/1914425950 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f10b8149e40 0x7f10b80b7190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.991+0000 7f10bce17640 1 -- 172.21.15.72:0/1914425950 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f10b81499c0 msgr2=0x7f10b80b6c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.991+0000 7f10bce17640 1 --2- 172.21.15.72:0/1914425950 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f10b81499c0 0x7f10b80b6c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.991+0000 7f10bce17640 1 -- 172.21.15.72:0/1914425950 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10b80b5450 con 0x7f10a40051c0 2024-04-15T15:15:30.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.991+0000 7f10b7fff640 1 --2- 172.21.15.72:0/1914425950 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f10b81499c0 0x7f10b80b6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:30.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.991+0000 7f10bce17640 1 --2- 172.21.15.72:0/1914425950 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10a40051c0 0x7f10b80b2ee0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f10a8011910 tx=0x7f10a8011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:30.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.991+0000 7f10b57fa640 1 -- 172.21.15.72:0/1914425950 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a800faf0 con 0x7f10a40051c0 2024-04-15T15:15:30.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.991+0000 7f10b57fa640 1 -- 172.21.15.72:0/1914425950 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10a800fc90 con 0x7f10a40051c0 2024-04-15T15:15:30.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.992+0000 7f10b57fa640 1 -- 172.21.15.72:0/1914425950 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a8014440 con 0x7f10a40051c0 2024-04-15T15:15:30.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.992+0000 7f10bf07b640 1 -- 172.21.15.72:0/1914425950 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10b806e440 con 0x7f10a40051c0 2024-04-15T15:15:30.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.992+0000 7f10bf07b640 1 -- 172.21.15.72:0/1914425950 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10b806ea00 con 0x7f10a40051c0 2024-04-15T15:15:30.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.993+0000 7f10bf07b640 1 -- 172.21.15.72:0/1914425950 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10b80689e0 con 0x7f10a40051c0 2024-04-15T15:15:30.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.993+0000 7f10b57fa640 1 -- 172.21.15.72:0/1914425950 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f10a8012040 con 0x7f10a40051c0 2024-04-15T15:15:30.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.993+0000 7f10b57fa640 1 --2- 172.21.15.72:0/1914425950 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f109806c8f0 0x7f109806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.994+0000 7f10b7fff640 1 --2- 172.21.15.72:0/1914425950 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f109806c8f0 0x7f109806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.994+0000 7f10b57fa640 1 -- 172.21.15.72:0/1914425950 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f10a808c570 con 0x7f10a40051c0 2024-04-15T15:15:30.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.994+0000 7f10b7fff640 1 --2- 172.21.15.72:0/1914425950 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f109806c8f0 0x7f109806edb0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f10a0004c20 tx=0x7f10a0006040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:31.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:30.998+0000 7f10b57fa640 1 -- 172.21.15.72:0/1914425950 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f10a8019030 con 0x7f10a40051c0 2024-04-15T15:15:31.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.177+0000 7f10bf07b640 1 -- 172.21.15.72:0/1914425950 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f10b8074e30 con 0x7f10a40051c0 2024-04-15T15:15:31.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.179+0000 7f10b57fa640 1 -- 172.21.15.72:0/1914425950 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7f10a805a660 con 0x7f10a40051c0 2024-04-15T15:15:31.181 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:31.182 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":24,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:14:56.451113+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:14:49.438366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:14:50.368379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:14:54.081157+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:31.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.181+0000 7f10bf07b640 1 -- 172.21.15.72:0/1914425950 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f109806c8f0 msgr2=0x7f109806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.181+0000 7f10bf07b640 1 --2- 172.21.15.72:0/1914425950 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f109806c8f0 0x7f109806edb0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f10a0004c20 tx=0x7f10a0006040 comp rx=0 tx=0).stop 2024-04-15T15:15:31.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.181+0000 7f10bf07b640 1 -- 172.21.15.72:0/1914425950 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10a40051c0 msgr2=0x7f10b80b2ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.181+0000 7f10bf07b640 1 --2- 172.21.15.72:0/1914425950 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10a40051c0 0x7f10b80b2ee0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f10a8011910 tx=0x7f10a8011dd0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.181+0000 7f10bf07b640 1 -- 172.21.15.72:0/1914425950 shutdown_connections 2024-04-15T15:15:31.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.181+0000 7f10bf07b640 1 --2- 172.21.15.72:0/1914425950 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f109806c8f0 0x7f109806edb0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.181+0000 7f10bf07b640 1 --2- 172.21.15.72:0/1914425950 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f10b8149e40 0x7f10b80b7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.181+0000 7f10bf07b640 1 --2- 172.21.15.72:0/1914425950 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f10b81499c0 0x7f10b80b6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.181+0000 7f10bf07b640 1 --2- 172.21.15.72:0/1914425950 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10a40051c0 0x7f10b80b2ee0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.181+0000 7f10bf07b640 1 -- 172.21.15.72:0/1914425950 >> 172.21.15.72:0/1914425950 conn(0x7f10b80b43c0 msgr2=0x7f10b8155570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:31.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.181+0000 7f10bf07b640 1 -- 172.21.15.72:0/1914425950 shutdown_connections 2024-04-15T15:15:31.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.181+0000 7f10bf07b640 1 -- 172.21.15.72:0/1914425950 wait complete. 2024-04-15T15:15:31.195 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:39.359102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:14:49.438366+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:14:50.368379+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:14:54.081157+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:15:31.195 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:31.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.298+0000 7f7598277640 1 Processor -- start 2024-04-15T15:15:31.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.298+0000 7f7598277640 1 -- start start 2024-04-15T15:15:31.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.299+0000 7f7598277640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f75900ce2f0 0x7f759015c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.299+0000 7f7598277640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f75900cc7c0 0x7f7590155600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.299+0000 7f7598277640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f759006bbb0 con 0x7f75900d08f0 2024-04-15T15:15:31.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.299+0000 7f7598277640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f759006b960 con 0x7f75900ce2f0 2024-04-15T15:15:31.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.299+0000 7f7598277640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f759006be90 con 0x7f75900cc7c0 2024-04-15T15:15:31.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.299+0000 7f7596013640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f75900ce2f0 0x7f759015c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.299+0000 7f7596013640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f75900ce2f0 0x7f759015c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42898/0 (socket says 172.21.15.72:42898) 2024-04-15T15:15:31.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.299+0000 7f7596013640 1 -- 172.21.15.72:0/2606825647 learned_addr learned my addr 172.21.15.72:0/2606825647 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:31.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.299+0000 7f7596814640 1 --2- 172.21.15.72:0/2606825647 >> v2:172.21.15.175:3300/0 conn(0x7f75900cc7c0 0x7f7590155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.299+0000 7f7596013640 1 -- 172.21.15.72:0/2606825647 >> v2:172.21.15.175:3300/0 conn(0x7f75900cc7c0 msgr2=0x7f7590155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:31.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.299+0000 7f7596013640 1 --2- 172.21.15.72:0/2606825647 >> v2:172.21.15.175:3300/0 conn(0x7f75900cc7c0 0x7f7590155600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.299+0000 7f7596013640 1 -- 172.21.15.72:0/2606825647 >> v1:172.21.15.175:6789/0 conn(0x7f75900d08f0 legacy=0x7f75900cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:31.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.299+0000 7f7596013640 1 -- 172.21.15.72:0/2606825647 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f759006c170 con 0x7f75900ce2f0 2024-04-15T15:15:31.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.300+0000 7f7596013640 1 --2- 172.21.15.72:0/2606825647 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75900ce2f0 0x7f759015c530 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f759015f050 tx=0x7f758c02e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ca1f2eaa620e524 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:31.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.300+0000 7f7595011640 1 -- 172.21.15.72:0/2606825647 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f758c03b070 con 0x7f75900ce2f0 2024-04-15T15:15:31.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.300+0000 7f7595011640 1 -- 172.21.15.72:0/2606825647 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f758c036440 con 0x7f75900ce2f0 2024-04-15T15:15:31.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.300+0000 7f7595011640 1 -- 172.21.15.72:0/2606825647 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f758c036810 con 0x7f75900ce2f0 2024-04-15T15:15:31.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.300+0000 7f7598277640 1 -- 172.21.15.72:0/2606825647 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75900ce2f0 msgr2=0x7f759015c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.301+0000 7f7598277640 1 --2- 172.21.15.72:0/2606825647 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75900ce2f0 0x7f759015c530 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f759015f050 tx=0x7f758c02e0e0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.301+0000 7f7598277640 1 -- 172.21.15.72:0/2606825647 shutdown_connections 2024-04-15T15:15:31.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.301+0000 7f7598277640 1 --2- 172.21.15.72:0/2606825647 >> v2:172.21.15.175:3300/0 conn(0x7f75900cc7c0 0x7f7590155600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.301+0000 7f7598277640 1 --2- 172.21.15.72:0/2606825647 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75900ce2f0 0x7f759015c530 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.301+0000 7f7598277640 1 -- 172.21.15.72:0/2606825647 >> 172.21.15.72:0/2606825647 conn(0x7f75900b31d0 msgr2=0x7f75900b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:31.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.301+0000 7f7598277640 1 -- 172.21.15.72:0/2606825647 shutdown_connections 2024-04-15T15:15:31.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.301+0000 7f7598277640 1 -- 172.21.15.72:0/2606825647 wait complete. 2024-04-15T15:15:31.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.302+0000 7f7598277640 1 Processor -- start 2024-04-15T15:15:31.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.302+0000 7f7598277640 1 -- start start 2024-04-15T15:15:31.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.302+0000 7f7598277640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75900cc7c0 0x7f75900b1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.302+0000 7f7598277640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75900ce2f0 0x7f759014d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.302+0000 7f7598277640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f75900d08f0 0x7f75900ae370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.303+0000 7f7598277640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f75900ce6d0 con 0x7f75900cc7c0 2024-04-15T15:15:31.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.303+0000 7f7598277640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f759015ca70 con 0x7f75900ce2f0 2024-04-15T15:15:31.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.303+0000 7f7598277640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f75900ce850 con 0x7f75900d08f0 2024-04-15T15:15:31.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.303+0000 7f7596013640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75900cc7c0 0x7f75900b1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.303+0000 7f7596013640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75900cc7c0 0x7f75900b1540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42910/0 (socket says 172.21.15.72:42910) 2024-04-15T15:15:31.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.303+0000 7f7596013640 1 -- 172.21.15.72:0/2998961586 learned_addr learned my addr 172.21.15.72:0/2998961586 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:31.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.303+0000 7f7596013640 1 -- 172.21.15.72:0/2998961586 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f75900d08f0 msgr2=0x7f75900ae370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.303+0000 7f7596013640 1 --2- 172.21.15.72:0/2998961586 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f75900d08f0 0x7f75900ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.303+0000 7f7596013640 1 -- 172.21.15.72:0/2998961586 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75900ce2f0 msgr2=0x7f759014d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.303+0000 7f7595812640 1 --2- 172.21.15.72:0/2998961586 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75900ce2f0 0x7f759014d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.303+0000 7f7596013640 1 --2- 172.21.15.72:0/2998961586 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75900ce2f0 0x7f759014d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.303+0000 7f7596013640 1 -- 172.21.15.72:0/2998961586 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f759015afa0 con 0x7f75900cc7c0 2024-04-15T15:15:31.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.303+0000 7f7595812640 1 --2- 172.21.15.72:0/2998961586 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75900ce2f0 0x7f759014d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:31.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.303+0000 7f7596013640 1 --2- 172.21.15.72:0/2998961586 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75900cc7c0 0x7f75900b1540 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f758c02e120 tx=0x7f758c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:31.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.303+0000 7f7586ffd640 1 -- 172.21.15.72:0/2998961586 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f758c045020 con 0x7f75900cc7c0 2024-04-15T15:15:31.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.303+0000 7f7586ffd640 1 -- 172.21.15.72:0/2998961586 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f758c0346c0 con 0x7f75900cc7c0 2024-04-15T15:15:31.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.303+0000 7f7598277640 1 -- 172.21.15.72:0/2998961586 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f759009a260 con 0x7f75900cc7c0 2024-04-15T15:15:31.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.303+0000 7f7586ffd640 1 -- 172.21.15.72:0/2998961586 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f758c03b040 con 0x7f75900cc7c0 2024-04-15T15:15:31.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.303+0000 7f7598277640 1 -- 172.21.15.72:0/2998961586 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75900b1f20 con 0x7f75900cc7c0 2024-04-15T15:15:31.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.304+0000 7f7598277640 1 -- 172.21.15.72:0/2998961586 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7590075c50 con 0x7f75900cc7c0 2024-04-15T15:15:31.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.305+0000 7f7586ffd640 1 -- 172.21.15.72:0/2998961586 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f758c048020 con 0x7f75900cc7c0 2024-04-15T15:15:31.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.305+0000 7f7586ffd640 1 --2- 172.21.15.72:0/2998961586 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f757406c8f0 0x7f757406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.305+0000 7f7586ffd640 1 -- 172.21.15.72:0/2998961586 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f758c0b0a40 con 0x7f75900cc7c0 2024-04-15T15:15:31.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.305+0000 7f7595812640 1 --2- 172.21.15.72:0/2998961586 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f757406c8f0 0x7f757406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.305+0000 7f7595812640 1 --2- 172.21.15.72:0/2998961586 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f757406c8f0 0x7f757406edb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f759006ef30 tx=0x7f7580008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:31.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.310+0000 7f7586ffd640 1 -- 172.21.15.72:0/2998961586 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f758c04f030 con 0x7f75900cc7c0 2024-04-15T15:15:31.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.495+0000 7f7598277640 1 -- 172.21.15.72:0/2998961586 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f759006fa20 con 0x7f75900cc7c0 2024-04-15T15:15:31.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.495+0000 7f7586ffd640 1 -- 172.21.15.72:0/2998961586 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7f758c0b0e20 con 0x7f75900cc7c0 2024-04-15T15:15:31.498 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-15T15:15:31.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.497+0000 7f7598277640 1 -- 172.21.15.72:0/2998961586 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f757406c8f0 msgr2=0x7f757406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.497+0000 7f7598277640 1 --2- 172.21.15.72:0/2998961586 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f757406c8f0 0x7f757406edb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f759006ef30 tx=0x7f7580008040 comp rx=0 tx=0).stop 2024-04-15T15:15:31.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.497+0000 7f7598277640 1 -- 172.21.15.72:0/2998961586 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75900cc7c0 msgr2=0x7f75900b1540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.497+0000 7f7598277640 1 --2- 172.21.15.72:0/2998961586 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75900cc7c0 0x7f75900b1540 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f758c02e120 tx=0x7f758c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.497+0000 7f7598277640 1 -- 172.21.15.72:0/2998961586 shutdown_connections 2024-04-15T15:15:31.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.497+0000 7f7598277640 1 --2- 172.21.15.72:0/2998961586 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f757406c8f0 0x7f757406edb0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.497+0000 7f7598277640 1 --2- 172.21.15.72:0/2998961586 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f75900d08f0 0x7f75900ae370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.497+0000 7f7598277640 1 --2- 172.21.15.72:0/2998961586 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75900ce2f0 0x7f759014d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.497+0000 7f7598277640 1 --2- 172.21.15.72:0/2998961586 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75900cc7c0 0x7f75900b1540 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.497+0000 7f7598277640 1 -- 172.21.15.72:0/2998961586 >> 172.21.15.72:0/2998961586 conn(0x7f75900b31d0 msgr2=0x7f759015e530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:31.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.497+0000 7f7598277640 1 -- 172.21.15.72:0/2998961586 shutdown_connections 2024-04-15T15:15:31.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.497+0000 7f7598277640 1 -- 172.21.15.72:0/2998961586 wait complete. 2024-04-15T15:15:31.511 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:15:31.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.617+0000 7f60d5e2a640 1 Processor -- start 2024-04-15T15:15:31.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.617+0000 7f60d5e2a640 1 -- start start 2024-04-15T15:15:31.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.618+0000 7f60d5e2a640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f60d00cc7c0 0x7f60d0155600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.618+0000 7f60d5e2a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60d006bbb0 con 0x7f60d00ce2f0 2024-04-15T15:15:31.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.618+0000 7f60d5e2a640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60d006be90 con 0x7f60d00d08f0 2024-04-15T15:15:31.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.618+0000 7f60d5e2a640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f60d006b960 con 0x7f60d00cc7c0 2024-04-15T15:15:31.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.618+0000 7f60cf7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f60d00ce2f0 0x7f60d015c530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:48122/0 (socket says 172.21.15.72:48122) 2024-04-15T15:15:31.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.618+0000 7f60cf7fe640 1 -- 172.21.15.72:0/704101328 learned_addr learned my addr 172.21.15.72:0/704101328 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:31.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.618+0000 7f60cffff640 1 --2- 172.21.15.72:0/704101328 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f60d00cc7c0 0x7f60d0155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.619+0000 7f60ce7fc640 1 -- 172.21.15.72:0/704101328 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 284996808 0 0) 0x7f60d006bbb0 con 0x7f60d00ce2f0 2024-04-15T15:15:31.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.619+0000 7f60ce7fc640 1 -- 172.21.15.72:0/704101328 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f60b8003520 con 0x7f60d00ce2f0 2024-04-15T15:15:31.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.619+0000 7f60ce7fc640 1 -- 172.21.15.72:0/704101328 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2551560275 0 0) 0x7f60b8003520 con 0x7f60d00ce2f0 2024-04-15T15:15:31.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.619+0000 7f60ce7fc640 1 -- 172.21.15.72:0/704101328 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f60d00cc7c0 msgr2=0x7f60d0155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:31.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.619+0000 7f60ce7fc640 1 --2- 172.21.15.72:0/704101328 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f60d00cc7c0 0x7f60d0155600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.619+0000 7f60ce7fc640 1 -- 172.21.15.72:0/704101328 >> v1:172.21.15.175:6789/0 conn(0x7f60d00d08f0 legacy=0x7f60d00cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.619+0000 7f60ce7fc640 1 -- 172.21.15.72:0/704101328 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60d006c170 con 0x7f60d00ce2f0 2024-04-15T15:15:31.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.619+0000 7f60ce7fc640 1 -- 172.21.15.72:0/704101328 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f60c0001670 con 0x7f60d00ce2f0 2024-04-15T15:15:31.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.619+0000 7f60ce7fc640 1 -- 172.21.15.72:0/704101328 >> v1:172.21.15.72:6789/0 conn(0x7f60d00ce2f0 legacy=0x7f60d015c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.619+0000 7f60ce7fc640 1 --2- 172.21.15.72:0/704101328 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60b80039e0 0x7f60b8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.619+0000 7f60ce7fc640 1 --2- 172.21.15.72:0/704101328 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60b8004390 0x7f60b8004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.619+0000 7f60ce7fc640 1 --2- 172.21.15.72:0/704101328 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f60d00cc7c0 0x7f60b80050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.619+0000 7f60ce7fc640 1 -- 172.21.15.72:0/704101328 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f60d006b960 con 0x7f60b8004390 2024-04-15T15:15:31.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.619+0000 7f60ce7fc640 1 -- 172.21.15.72:0/704101328 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f60b800b9c0 con 0x7f60b80039e0 2024-04-15T15:15:31.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.619+0000 7f60ce7fc640 1 -- 172.21.15.72:0/704101328 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f60b800ddd0 con 0x7f60d00cc7c0 2024-04-15T15:15:31.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.619+0000 7f60ceffd640 1 --2- 172.21.15.72:0/704101328 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60b8004390 0x7f60b8004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.619+0000 7f60cf7fe640 1 --2- 172.21.15.72:0/704101328 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60b80039e0 0x7f60b8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.619+0000 7f60cffff640 1 --2- 172.21.15.72:0/704101328 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f60d00cc7c0 0x7f60b80050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.619+0000 7f60ceffd640 1 -- 172.21.15.72:0/704101328 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f60d00cc7c0 msgr2=0x7f60b80050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:31.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.619+0000 7f60ceffd640 1 --2- 172.21.15.72:0/704101328 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f60d00cc7c0 0x7f60b80050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.619+0000 7f60ceffd640 1 -- 172.21.15.72:0/704101328 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60b80039e0 msgr2=0x7f60b8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:31.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.619+0000 7f60ceffd640 1 --2- 172.21.15.72:0/704101328 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60b80039e0 0x7f60b8003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.619+0000 7f60ceffd640 1 -- 172.21.15.72:0/704101328 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60b8003520 con 0x7f60b8004390 2024-04-15T15:15:31.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.619+0000 7f60cffff640 1 --2- 172.21.15.72:0/704101328 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f60d00cc7c0 0x7f60b80050c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:31.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.619+0000 7f60cf7fe640 1 --2- 172.21.15.72:0/704101328 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60b80039e0 0x7f60b8003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:31.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.620+0000 7f60ceffd640 1 --2- 172.21.15.72:0/704101328 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60b8004390 0x7f60b8004860 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f60c4001860 tx=0x7f60b8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=248b5dfde1628c8f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:31.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.620+0000 7f60ce7fc640 1 -- 172.21.15.72:0/704101328 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c4038070 con 0x7f60b8004390 2024-04-15T15:15:31.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.620+0000 7f60ce7fc640 1 -- 172.21.15.72:0/704101328 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60c4033410 con 0x7f60b8004390 2024-04-15T15:15:31.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.620+0000 7f60ce7fc640 1 -- 172.21.15.72:0/704101328 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c4001430 con 0x7f60b8004390 2024-04-15T15:15:31.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.621+0000 7f60d5e2a640 1 -- 172.21.15.72:0/704101328 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60b8004390 msgr2=0x7f60b8004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.621+0000 7f60d5e2a640 1 --2- 172.21.15.72:0/704101328 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60b8004390 0x7f60b8004860 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f60c4001860 tx=0x7f60b8005830 comp rx=0 tx=0).stop 2024-04-15T15:15:31.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.621+0000 7f60cffff640 1 -- 172.21.15.72:0/704101328 reap_dead start 2024-04-15T15:15:31.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.621+0000 7f60d5e2a640 1 -- 172.21.15.72:0/704101328 shutdown_connections 2024-04-15T15:15:31.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.621+0000 7f60d5e2a640 1 -- 172.21.15.72:0/704101328 >> 172.21.15.72:0/704101328 conn(0x7f60d00b31d0 msgr2=0x7f60d00b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:31.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.621+0000 7f60d5e2a640 1 -- 172.21.15.72:0/704101328 shutdown_connections 2024-04-15T15:15:31.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.621+0000 7f60d5e2a640 1 -- 172.21.15.72:0/704101328 wait complete. 2024-04-15T15:15:31.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.622+0000 7f60d5e2a640 1 Processor -- start 2024-04-15T15:15:31.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.622+0000 7f60d5e2a640 1 -- start start 2024-04-15T15:15:31.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.623+0000 7f60d5e2a640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60b8004390 0x7f60d00b2730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.623+0000 7f60d5e2a640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f60d00afd40 0x7f60d00b01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.623+0000 7f60d5e2a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60d0071c20 0x7f60d00720a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.623+0000 7f60d5e2a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f60d006ef50 con 0x7f60d0071c20 2024-04-15T15:15:31.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.623+0000 7f60d5e2a640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f60d006edd0 con 0x7f60b8004390 2024-04-15T15:15:31.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.623+0000 7f60d5e2a640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f60d006f0d0 con 0x7f60d00afd40 2024-04-15T15:15:31.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.623+0000 7f60cffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60d0071c20 0x7f60d00720a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.623+0000 7f60cffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60d0071c20 0x7f60d00720a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42930/0 (socket says 172.21.15.72:42930) 2024-04-15T15:15:31.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.623+0000 7f60cffff640 1 -- 172.21.15.72:0/4248842746 learned_addr learned my addr 172.21.15.72:0/4248842746 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:31.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.623+0000 7f60ceffd640 1 --2- 172.21.15.72:0/4248842746 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f60d00afd40 0x7f60d00b01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.623+0000 7f60cf7fe640 1 --2- 172.21.15.72:0/4248842746 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60b8004390 0x7f60d00b2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.623+0000 7f60cffff640 1 -- 172.21.15.72:0/4248842746 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f60d00afd40 msgr2=0x7f60d00b01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.623+0000 7f60cffff640 1 --2- 172.21.15.72:0/4248842746 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f60d00afd40 0x7f60d00b01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.623+0000 7f60cffff640 1 -- 172.21.15.72:0/4248842746 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60b8004390 msgr2=0x7f60d00b2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.623+0000 7f60cffff640 1 --2- 172.21.15.72:0/4248842746 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60b8004390 0x7f60d00b2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.623+0000 7f60cffff640 1 -- 172.21.15.72:0/4248842746 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60d0072990 con 0x7f60d0071c20 2024-04-15T15:15:31.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.623+0000 7f60ceffd640 1 --2- 172.21.15.72:0/4248842746 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f60d00afd40 0x7f60d00b01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:31.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.623+0000 7f60cf7fe640 1 --2- 172.21.15.72:0/4248842746 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60b8004390 0x7f60d00b2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:31.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.623+0000 7f60cffff640 1 --2- 172.21.15.72:0/4248842746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60d0071c20 0x7f60d00720a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f60d00ccbd0 tx=0x7f60c8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:31.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.623+0000 7f60b7fff640 1 -- 172.21.15.72:0/4248842746 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c8010e20 con 0x7f60d0071c20 2024-04-15T15:15:31.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.623+0000 7f60b7fff640 1 -- 172.21.15.72:0/4248842746 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60c8003990 con 0x7f60d0071c20 2024-04-15T15:15:31.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.623+0000 7f60d5e2a640 1 -- 172.21.15.72:0/4248842746 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60d0072c70 con 0x7f60d0071c20 2024-04-15T15:15:31.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.623+0000 7f60b7fff640 1 -- 172.21.15.72:0/4248842746 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c8003dc0 con 0x7f60d0071c20 2024-04-15T15:15:31.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.624+0000 7f60d5e2a640 1 -- 172.21.15.72:0/4248842746 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60d0073230 con 0x7f60d0071c20 2024-04-15T15:15:31.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.625+0000 7f60d5e2a640 1 -- 172.21.15.72:0/4248842746 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60d0075c50 con 0x7f60d0071c20 2024-04-15T15:15:31.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.625+0000 7f60b7fff640 1 -- 172.21.15.72:0/4248842746 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f60c801e020 con 0x7f60d0071c20 2024-04-15T15:15:31.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.625+0000 7f60b7fff640 1 --2- 172.21.15.72:0/4248842746 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f60bc06c8f0 0x7f60bc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.625+0000 7f60b7fff640 1 -- 172.21.15.72:0/4248842746 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6735+0+0 (secure 0 0 0) 0x7f60c808c660 con 0x7f60d0071c20 2024-04-15T15:15:31.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.625+0000 7f60cf7fe640 1 --2- 172.21.15.72:0/4248842746 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f60bc06c8f0 0x7f60bc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.626+0000 7f60cf7fe640 1 --2- 172.21.15.72:0/4248842746 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f60bc06c8f0 0x7f60bc06edb0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f60c0000c00 tx=0x7f60c0001180 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:31.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.630+0000 7f60b7fff640 1 -- 172.21.15.72:0/4248842746 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f60c805a040 con 0x7f60d0071c20 2024-04-15T15:15:31.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.806+0000 7f60d5e2a640 1 -- 172.21.15.72:0/4248842746 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f60d0159500 con 0x7f60d0071c20 2024-04-15T15:15:31.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.806+0000 7f60b7fff640 1 -- 172.21.15.72:0/4248842746 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7f60c805e340 con 0x7f60d0071c20 2024-04-15T15:15:31.809 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:31.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.808+0000 7f60d5e2a640 1 -- 172.21.15.72:0/4248842746 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f60bc06c8f0 msgr2=0x7f60bc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.808+0000 7f60d5e2a640 1 --2- 172.21.15.72:0/4248842746 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f60bc06c8f0 0x7f60bc06edb0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f60c0000c00 tx=0x7f60c0001180 comp rx=0 tx=0).stop 2024-04-15T15:15:31.811 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.808+0000 7f60d5e2a640 1 -- 172.21.15.72:0/4248842746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60d0071c20 msgr2=0x7f60d00720a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.808+0000 7f60d5e2a640 1 --2- 172.21.15.72:0/4248842746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60d0071c20 0x7f60d00720a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f60d00ccbd0 tx=0x7f60c8009f90 comp rx=0 tx=0).stop 2024-04-15T15:15:31.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.808+0000 7f60d5e2a640 1 -- 172.21.15.72:0/4248842746 shutdown_connections 2024-04-15T15:15:31.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.808+0000 7f60d5e2a640 1 --2- 172.21.15.72:0/4248842746 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f60bc06c8f0 0x7f60bc06edb0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.808+0000 7f60d5e2a640 1 --2- 172.21.15.72:0/4248842746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60d0071c20 0x7f60d00720a0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.808+0000 7f60d5e2a640 1 --2- 172.21.15.72:0/4248842746 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f60d00afd40 0x7f60d00b01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.808+0000 7f60d5e2a640 1 --2- 172.21.15.72:0/4248842746 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60b8004390 0x7f60d00b2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.808+0000 7f60d5e2a640 1 -- 172.21.15.72:0/4248842746 >> 172.21.15.72:0/4248842746 conn(0x7f60d00b31d0 msgr2=0x7f60d00b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:31.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.809+0000 7f60d5e2a640 1 -- 172.21.15.72:0/4248842746 shutdown_connections 2024-04-15T15:15:31.812 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.809+0000 7f60d5e2a640 1 -- 172.21.15.72:0/4248842746 wait complete. 2024-04-15T15:15:31.822 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:15:31.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.925+0000 7f1c098f2640 1 Processor -- start 2024-04-15T15:15:31.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.925+0000 7f1c098f2640 1 -- start start 2024-04-15T15:15:31.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.926+0000 7f1c098f2640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1c040cf160 0x7f1c040cf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.926+0000 7f1c098f2640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1c0415e2a0 0x7f1c0415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.926+0000 7f1c098f2640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c040b61e0 con 0x7f1c0415d830 2024-04-15T15:15:31.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.926+0000 7f1c098f2640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f1c040b64c0 con 0x7f1c040cf160 2024-04-15T15:15:31.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.926+0000 7f1c098f2640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f1c040b67a0 con 0x7f1c0415e2a0 2024-04-15T15:15:31.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.926+0000 7f1c02ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f1c0415d830 0x7f1c040b57f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49972/0 (socket says 172.21.15.72:49972) 2024-04-15T15:15:31.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.926+0000 7f1c02ffd640 1 -- 172.21.15.72:0/1998960458 learned_addr learned my addr 172.21.15.72:0/1998960458 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:31.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.927+0000 7f1c037fe640 1 --2- 172.21.15.72:0/1998960458 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1c0415e2a0 0x7f1c0415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.927+0000 7f1c01ffb640 1 -- 172.21.15.72:0/1998960458 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2890712010 0 0) 0x7f1c040b61e0 con 0x7f1c0415d830 2024-04-15T15:15:31.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.927+0000 7f1c027fc640 1 --2- 172.21.15.72:0/1998960458 >> v2:172.21.15.175:3300/0 conn(0x7f1c040cf160 0x7f1c040cf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.927+0000 7f1c01ffb640 1 -- 172.21.15.72:0/1998960458 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1bec003520 con 0x7f1c0415d830 2024-04-15T15:15:31.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.927+0000 7f1c01ffb640 1 -- 172.21.15.72:0/1998960458 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2414162087 0 0) 0x7f1bec003520 con 0x7f1c0415d830 2024-04-15T15:15:31.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.927+0000 7f1c01ffb640 1 -- 172.21.15.72:0/1998960458 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1c0415e2a0 msgr2=0x7f1c0415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:31.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.927+0000 7f1c01ffb640 1 --2- 172.21.15.72:0/1998960458 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1c0415e2a0 0x7f1c0415e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.927+0000 7f1c01ffb640 1 -- 172.21.15.72:0/1998960458 >> v2:172.21.15.175:3300/0 conn(0x7f1c040cf160 msgr2=0x7f1c040cf5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:31.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.927+0000 7f1c01ffb640 1 --2- 172.21.15.72:0/1998960458 >> v2:172.21.15.175:3300/0 conn(0x7f1c040cf160 0x7f1c040cf5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.927+0000 7f1c01ffb640 1 -- 172.21.15.72:0/1998960458 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c040b6a80 con 0x7f1c0415d830 2024-04-15T15:15:31.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.927+0000 7f1c01ffb640 1 -- 172.21.15.72:0/1998960458 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f1bf4001670 con 0x7f1c0415d830 2024-04-15T15:15:31.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.927+0000 7f1c01ffb640 1 -- 172.21.15.72:0/1998960458 >> v1:172.21.15.72:6789/0 conn(0x7f1c0415d830 legacy=0x7f1c040b57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.927+0000 7f1c01ffb640 1 --2- 172.21.15.72:0/1998960458 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1c0415e2a0 0x7f1bec0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.927+0000 7f1c01ffb640 1 --2- 172.21.15.72:0/1998960458 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1bec003f30 0x7f1bec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.927+0000 7f1c01ffb640 1 --2- 172.21.15.72:0/1998960458 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1bec004d40 0x7f1bec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.927+0000 7f1c01ffb640 1 -- 172.21.15.72:0/1998960458 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f1c040b64c0 con 0x7f1bec004d40 2024-04-15T15:15:31.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.927+0000 7f1c01ffb640 1 -- 172.21.15.72:0/1998960458 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f1c040b67a0 con 0x7f1bec003f30 2024-04-15T15:15:31.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.927+0000 7f1c01ffb640 1 -- 172.21.15.72:0/1998960458 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f1bec00ba10 con 0x7f1c0415e2a0 2024-04-15T15:15:31.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.927+0000 7f1c037fe640 1 --2- 172.21.15.72:0/1998960458 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1bec004d40 0x7f1bec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.927+0000 7f1c027fc640 1 --2- 172.21.15.72:0/1998960458 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1bec003f30 0x7f1bec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.927+0000 7f1c02ffd640 1 --2- 172.21.15.72:0/1998960458 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1c0415e2a0 0x7f1bec0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.927+0000 7f1c037fe640 1 -- 172.21.15.72:0/1998960458 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1c0415e2a0 msgr2=0x7f1bec0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:31.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.927+0000 7f1c037fe640 1 --2- 172.21.15.72:0/1998960458 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1c0415e2a0 0x7f1bec0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.927+0000 7f1c037fe640 1 -- 172.21.15.72:0/1998960458 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1bec003f30 msgr2=0x7f1bec004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:31.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.927+0000 7f1c037fe640 1 --2- 172.21.15.72:0/1998960458 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1bec003f30 0x7f1bec004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.927+0000 7f1c037fe640 1 -- 172.21.15.72:0/1998960458 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bec003520 con 0x7f1bec004d40 2024-04-15T15:15:31.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.927+0000 7f1c02ffd640 1 --2- 172.21.15.72:0/1998960458 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1c0415e2a0 0x7f1bec0039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:31.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.928+0000 7f1c037fe640 1 --2- 172.21.15.72:0/1998960458 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1bec004d40 0x7f1bec0051c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc00e170 tx=0x7f1bec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4c5b4c70cab70da server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:31.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.928+0000 7f1c01ffb640 1 -- 172.21.15.72:0/1998960458 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc039070 con 0x7f1bec004d40 2024-04-15T15:15:31.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.928+0000 7f1c01ffb640 1 -- 172.21.15.72:0/1998960458 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bfc02cbf0 con 0x7f1bec004d40 2024-04-15T15:15:31.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.928+0000 7f1c01ffb640 1 -- 172.21.15.72:0/1998960458 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc009c10 con 0x7f1bec004d40 2024-04-15T15:15:31.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.929+0000 7f1c098f2640 1 -- 172.21.15.72:0/1998960458 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1bec004d40 msgr2=0x7f1bec0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.929+0000 7f1c098f2640 1 --2- 172.21.15.72:0/1998960458 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1bec004d40 0x7f1bec0051c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc00e170 tx=0x7f1bec005830 comp rx=0 tx=0).stop 2024-04-15T15:15:31.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.929+0000 7f1c037fe640 1 -- 172.21.15.72:0/1998960458 reap_dead start 2024-04-15T15:15:31.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.929+0000 7f1c098f2640 1 -- 172.21.15.72:0/1998960458 shutdown_connections 2024-04-15T15:15:31.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.929+0000 7f1c098f2640 1 -- 172.21.15.72:0/1998960458 >> 172.21.15.72:0/1998960458 conn(0x7f1c040d24d0 msgr2=0x7f1c040d28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:31.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.929+0000 7f1c098f2640 1 -- 172.21.15.72:0/1998960458 shutdown_connections 2024-04-15T15:15:31.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.929+0000 7f1c098f2640 1 -- 172.21.15.72:0/1998960458 wait complete. 2024-04-15T15:15:31.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.930+0000 7f1c098f2640 1 Processor -- start 2024-04-15T15:15:31.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.930+0000 7f1c098f2640 1 -- start start 2024-04-15T15:15:31.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.931+0000 7f1c098f2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c04151000 0x7f1c04156380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.931+0000 7f1c098f2640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1c041513e0 0x7f1c0414f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.931+0000 7f1c098f2640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1c0414b960 0x7f1c0414bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.931+0000 7f1c098f2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f1c0415cd50 con 0x7f1c04151000 2024-04-15T15:15:31.939 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.931+0000 7f1c098f2640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f1c040b5f00 con 0x7f1c0414b960 2024-04-15T15:15:31.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.931+0000 7f1c098f2640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f1c0415ced0 con 0x7f1c041513e0 2024-04-15T15:15:31.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.931+0000 7f1c02ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c04151000 0x7f1c04156380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.931+0000 7f1c02ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c04151000 0x7f1c04156380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54696/0 (socket says 172.21.15.72:54696) 2024-04-15T15:15:31.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.931+0000 7f1c02ffd640 1 -- 172.21.15.72:0/3386908445 learned_addr learned my addr 172.21.15.72:0/3386908445 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:31.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.931+0000 7f1c027fc640 1 --2- 172.21.15.72:0/3386908445 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1c041513e0 0x7f1c0414f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.931+0000 7f1c037fe640 1 --2- 172.21.15.72:0/3386908445 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1c0414b960 0x7f1c0414bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.931+0000 7f1c02ffd640 1 -- 172.21.15.72:0/3386908445 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1c041513e0 msgr2=0x7f1c0414f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.931+0000 7f1c02ffd640 1 --2- 172.21.15.72:0/3386908445 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1c041513e0 0x7f1c0414f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.931+0000 7f1c02ffd640 1 -- 172.21.15.72:0/3386908445 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1c0414b960 msgr2=0x7f1c0414bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.931+0000 7f1c02ffd640 1 --2- 172.21.15.72:0/3386908445 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1c0414b960 0x7f1c0414bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.931+0000 7f1c02ffd640 1 -- 172.21.15.72:0/3386908445 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c0414e8f0 con 0x7f1c04151000 2024-04-15T15:15:31.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.931+0000 7f1c02ffd640 1 --2- 172.21.15.72:0/3386908445 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c04151000 0x7f1c04156380 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1bf400eaf0 tx=0x7f1bf4005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:31.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.932+0000 7f1bebfff640 1 -- 172.21.15.72:0/3386908445 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf400c970 con 0x7f1c04151000 2024-04-15T15:15:31.940 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.932+0000 7f1bebfff640 1 -- 172.21.15.72:0/3386908445 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bf4018070 con 0x7f1c04151000 2024-04-15T15:15:31.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.932+0000 7f1bebfff640 1 -- 172.21.15.72:0/3386908445 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bf400cd10 con 0x7f1c04151000 2024-04-15T15:15:31.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.932+0000 7f1c098f2640 1 -- 172.21.15.72:0/3386908445 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c041568c0 con 0x7f1c04151000 2024-04-15T15:15:31.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.932+0000 7f1c098f2640 1 -- 172.21.15.72:0/3386908445 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c0414eb90 con 0x7f1c04151000 2024-04-15T15:15:31.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.933+0000 7f1c098f2640 1 -- 172.21.15.72:0/3386908445 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c04075c50 con 0x7f1c04151000 2024-04-15T15:15:31.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.933+0000 7f1bebfff640 1 -- 172.21.15.72:0/3386908445 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1bf401d460 con 0x7f1c04151000 2024-04-15T15:15:31.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.933+0000 7f1bebfff640 1 --2- 172.21.15.72:0/3386908445 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1bf006c8f0 0x7f1bf006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.934+0000 7f1bebfff640 1 -- 172.21.15.72:0/3386908445 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6735+0+0 (secure 0 0 0) 0x7f1bf408bfb0 con 0x7f1c04151000 2024-04-15T15:15:31.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.934+0000 7f1c027fc640 1 --2- 172.21.15.72:0/3386908445 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1bf006c8f0 0x7f1bf006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.934+0000 7f1c027fc640 1 --2- 172.21.15.72:0/3386908445 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1bf006c8f0 0x7f1bf006edb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1c04157fd0 tx=0x7f1bf8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:31.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:31.938+0000 7f1bebfff640 1 -- 172.21.15.72:0/3386908445 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1bf405a070 con 0x7f1c04151000 2024-04-15T15:15:32.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.117+0000 7f1c098f2640 1 -- 172.21.15.72:0/3386908445 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1c040cff00 con 0x7f1c04151000 2024-04-15T15:15:32.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.117+0000 7f1bebfff640 1 -- 172.21.15.72:0/3386908445 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7f1bf4059970 con 0x7f1c04151000 2024-04-15T15:15:32.120 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-04-15T15:15:32.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.119+0000 7f1c098f2640 1 -- 172.21.15.72:0/3386908445 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1bf006c8f0 msgr2=0x7f1bf006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.119+0000 7f1c098f2640 1 --2- 172.21.15.72:0/3386908445 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1bf006c8f0 0x7f1bf006edb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1c04157fd0 tx=0x7f1bf8008040 comp rx=0 tx=0).stop 2024-04-15T15:15:32.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.119+0000 7f1c098f2640 1 -- 172.21.15.72:0/3386908445 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c04151000 msgr2=0x7f1c04156380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.119+0000 7f1c098f2640 1 --2- 172.21.15.72:0/3386908445 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c04151000 0x7f1c04156380 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1bf400eaf0 tx=0x7f1bf4005e50 comp rx=0 tx=0).stop 2024-04-15T15:15:32.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.119+0000 7f1c098f2640 1 -- 172.21.15.72:0/3386908445 shutdown_connections 2024-04-15T15:15:32.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.119+0000 7f1c098f2640 1 --2- 172.21.15.72:0/3386908445 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1bf006c8f0 0x7f1bf006edb0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.119+0000 7f1c098f2640 1 --2- 172.21.15.72:0/3386908445 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1c0414b960 0x7f1c0414bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.119+0000 7f1c098f2640 1 --2- 172.21.15.72:0/3386908445 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1c041513e0 0x7f1c0414f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.119+0000 7f1c098f2640 1 --2- 172.21.15.72:0/3386908445 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c04151000 0x7f1c04156380 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.119+0000 7f1c098f2640 1 -- 172.21.15.72:0/3386908445 >> 172.21.15.72:0/3386908445 conn(0x7f1c040d24d0 msgr2=0x7f1c0415b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:32.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.119+0000 7f1c098f2640 1 -- 172.21.15.72:0/3386908445 shutdown_connections 2024-04-15T15:15:32.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.119+0000 7f1c098f2640 1 -- 172.21.15.72:0/3386908445 wait complete. 2024-04-15T15:15:32.133 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:15:32.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.238+0000 7ffae7c67640 1 Processor -- start 2024-04-15T15:15:32.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.238+0000 7ffae7c67640 1 -- start start 2024-04-15T15:15:32.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.238+0000 7ffae7c67640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffae00ce2f0 0x7ffae015c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.238+0000 7ffae7c67640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ffae00cc7c0 0x7ffae0155600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.238+0000 7ffae7c67640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffae006bbb0 con 0x7ffae00d08f0 2024-04-15T15:15:32.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.238+0000 7ffae7c67640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7ffae006b960 con 0x7ffae00cc7c0 2024-04-15T15:15:32.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.238+0000 7ffae7c67640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7ffae006be90 con 0x7ffae00ce2f0 2024-04-15T15:15:32.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.238+0000 7ffae5202640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ffae00d08f0 0x7ffae00cc0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49978/0 (socket says 172.21.15.72:49978) 2024-04-15T15:15:32.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.238+0000 7ffae5202640 1 -- 172.21.15.72:0/3715371240 learned_addr learned my addr 172.21.15.72:0/3715371240 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:32.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.238+0000 7ffae5a03640 1 --2- 172.21.15.72:0/3715371240 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffae00ce2f0 0x7ffae015c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.238+0000 7ffae6204640 1 --2- 172.21.15.72:0/3715371240 >> v2:172.21.15.175:3300/0 conn(0x7ffae00cc7c0 0x7ffae0155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.239+0000 7ffae6204640 1 -- 172.21.15.72:0/3715371240 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffae00ce2f0 msgr2=0x7ffae015c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:32.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.238+0000 7ffae4a01640 1 -- 172.21.15.72:0/3715371240 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3219385434 0 0) 0x7ffae006bbb0 con 0x7ffae00d08f0 2024-04-15T15:15:32.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.239+0000 7ffae6204640 1 --2- 172.21.15.72:0/3715371240 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffae00ce2f0 0x7ffae015c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.239+0000 7ffae6204640 1 -- 172.21.15.72:0/3715371240 >> v1:172.21.15.72:6789/0 conn(0x7ffae00d08f0 legacy=0x7ffae00cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.239+0000 7ffae6204640 1 -- 172.21.15.72:0/3715371240 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffae006c170 con 0x7ffae00cc7c0 2024-04-15T15:15:32.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.239+0000 7ffae5a03640 1 --2- 172.21.15.72:0/3715371240 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffae00ce2f0 0x7ffae015c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:32.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.239+0000 7ffae6204640 1 --2- 172.21.15.72:0/3715371240 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffae00cc7c0 0x7ffae0155600 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ffad80097a0 tx=0x7ffad802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b061ef4b19e1842 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:32.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.240+0000 7ffae4a01640 1 -- 172.21.15.72:0/3715371240 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad8035020 con 0x7ffae00cc7c0 2024-04-15T15:15:32.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.240+0000 7ffae4a01640 1 -- 172.21.15.72:0/3715371240 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffad8031830 con 0x7ffae00cc7c0 2024-04-15T15:15:32.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.240+0000 7ffae4a01640 1 -- 172.21.15.72:0/3715371240 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad8031c00 con 0x7ffae00cc7c0 2024-04-15T15:15:32.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.240+0000 7ffae7c67640 1 -- 172.21.15.72:0/3715371240 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffae00cc7c0 msgr2=0x7ffae0155600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.240+0000 7ffae7c67640 1 --2- 172.21.15.72:0/3715371240 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffae00cc7c0 0x7ffae0155600 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ffad80097a0 tx=0x7ffad802b8d0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.240+0000 7ffae7c67640 1 -- 172.21.15.72:0/3715371240 shutdown_connections 2024-04-15T15:15:32.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.240+0000 7ffae7c67640 1 --2- 172.21.15.72:0/3715371240 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffae00cc7c0 0x7ffae0155600 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.240+0000 7ffae7c67640 1 --2- 172.21.15.72:0/3715371240 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffae00ce2f0 0x7ffae015c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.240+0000 7ffae7c67640 1 -- 172.21.15.72:0/3715371240 >> 172.21.15.72:0/3715371240 conn(0x7ffae00b31d0 msgr2=0x7ffae00b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:32.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.241+0000 7ffae7c67640 1 -- 172.21.15.72:0/3715371240 shutdown_connections 2024-04-15T15:15:32.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.241+0000 7ffae7c67640 1 -- 172.21.15.72:0/3715371240 wait complete. 2024-04-15T15:15:32.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.242+0000 7ffae7c67640 1 Processor -- start 2024-04-15T15:15:32.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.242+0000 7ffae7c67640 1 -- start start 2024-04-15T15:15:32.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.243+0000 7ffae7c67640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffae00cc7c0 0x7ffae00b1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.243+0000 7ffae7c67640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffae00ce2f0 0x7ffae0072e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.243+0000 7ffae7c67640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffae00d08f0 0x7ffae00afb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.243+0000 7ffae7c67640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ffae00ce6d0 con 0x7ffae00ce2f0 2024-04-15T15:15:32.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.243+0000 7ffae7c67640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7ffae015ca70 con 0x7ffae00cc7c0 2024-04-15T15:15:32.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.243+0000 7ffae7c67640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7ffae00ce850 con 0x7ffae00d08f0 2024-04-15T15:15:32.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.243+0000 7ffae5202640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffae00ce2f0 0x7ffae0072e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.243+0000 7ffae5202640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffae00ce2f0 0x7ffae0072e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54706/0 (socket says 172.21.15.72:54706) 2024-04-15T15:15:32.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.243+0000 7ffae5202640 1 -- 172.21.15.72:0/1160243090 learned_addr learned my addr 172.21.15.72:0/1160243090 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:32.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.243+0000 7ffae6204640 1 --2- 172.21.15.72:0/1160243090 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffae00d08f0 0x7ffae00afb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.243+0000 7ffae5a03640 1 --2- 172.21.15.72:0/1160243090 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffae00cc7c0 0x7ffae00b1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.243+0000 7ffae5202640 1 -- 172.21.15.72:0/1160243090 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffae00d08f0 msgr2=0x7ffae00afb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.243+0000 7ffae5202640 1 --2- 172.21.15.72:0/1160243090 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffae00d08f0 0x7ffae00afb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.243+0000 7ffae5202640 1 -- 172.21.15.72:0/1160243090 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffae00cc7c0 msgr2=0x7ffae00b1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.243+0000 7ffae5202640 1 --2- 172.21.15.72:0/1160243090 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffae00cc7c0 0x7ffae00b1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.243+0000 7ffae5202640 1 -- 172.21.15.72:0/1160243090 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffae00b1a60 con 0x7ffae00ce2f0 2024-04-15T15:15:32.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.243+0000 7ffae5202640 1 --2- 172.21.15.72:0/1160243090 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffae00ce2f0 0x7ffae0072e70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ffad000c430 tx=0x7ffad000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:32.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.243+0000 7ffad67fc640 1 -- 172.21.15.72:0/1160243090 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad00074e0 con 0x7ffae00ce2f0 2024-04-15T15:15:32.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.243+0000 7ffad67fc640 1 -- 172.21.15.72:0/1160243090 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffad00047e0 con 0x7ffae00ce2f0 2024-04-15T15:15:32.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.243+0000 7ffae7c67640 1 -- 172.21.15.72:0/1160243090 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffae00b1ff0 con 0x7ffae00ce2f0 2024-04-15T15:15:32.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.243+0000 7ffad67fc640 1 -- 172.21.15.72:0/1160243090 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad0004bd0 con 0x7ffae00ce2f0 2024-04-15T15:15:32.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.243+0000 7ffae7c67640 1 -- 172.21.15.72:0/1160243090 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffae00b25b0 con 0x7ffae00ce2f0 2024-04-15T15:15:32.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.245+0000 7ffae7c67640 1 -- 172.21.15.72:0/1160243090 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffae00b17b0 con 0x7ffae00ce2f0 2024-04-15T15:15:32.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.245+0000 7ffad67fc640 1 -- 172.21.15.72:0/1160243090 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ffad0011070 con 0x7ffae00ce2f0 2024-04-15T15:15:32.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.245+0000 7ffad67fc640 1 --2- 172.21.15.72:0/1160243090 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ffac406c8f0 0x7ffac406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.245+0000 7ffae5a03640 1 --2- 172.21.15.72:0/1160243090 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ffac406c8f0 0x7ffac406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.245+0000 7ffad67fc640 1 -- 172.21.15.72:0/1160243090 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6735+0+0 (secure 0 0 0) 0x7ffad008bb80 con 0x7ffae00ce2f0 2024-04-15T15:15:32.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.246+0000 7ffae5a03640 1 --2- 172.21.15.72:0/1160243090 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ffac406c8f0 0x7ffac406edb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ffadc009bd0 tx=0x7ffadc00b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:32.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.250+0000 7ffad67fc640 1 -- 172.21.15.72:0/1160243090 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ffae00b17b0 con 0x7ffae00ce2f0 2024-04-15T15:15:32.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.433+0000 7ffae7c67640 1 -- 172.21.15.72:0/1160243090 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7ffae00b2780 con 0x7ffae00ce2f0 2024-04-15T15:15:32.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.434+0000 7ffad67fc640 1 -- 172.21.15.72:0/1160243090 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7ffad000e280 con 0x7ffae00ce2f0 2024-04-15T15:15:32.436 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:32.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.436+0000 7ffae7c67640 1 -- 172.21.15.72:0/1160243090 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ffac406c8f0 msgr2=0x7ffac406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.436+0000 7ffae7c67640 1 --2- 172.21.15.72:0/1160243090 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ffac406c8f0 0x7ffac406edb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ffadc009bd0 tx=0x7ffadc00b040 comp rx=0 tx=0).stop 2024-04-15T15:15:32.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.436+0000 7ffae7c67640 1 -- 172.21.15.72:0/1160243090 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffae00ce2f0 msgr2=0x7ffae0072e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.436+0000 7ffae7c67640 1 --2- 172.21.15.72:0/1160243090 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffae00ce2f0 0x7ffae0072e70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ffad000c430 tx=0x7ffad000c8f0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.436+0000 7ffae7c67640 1 -- 172.21.15.72:0/1160243090 shutdown_connections 2024-04-15T15:15:32.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.436+0000 7ffae7c67640 1 --2- 172.21.15.72:0/1160243090 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ffac406c8f0 0x7ffac406edb0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.436+0000 7ffae7c67640 1 --2- 172.21.15.72:0/1160243090 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ffae00d08f0 0x7ffae00afb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.436+0000 7ffae7c67640 1 --2- 172.21.15.72:0/1160243090 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffae00ce2f0 0x7ffae0072e70 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.436+0000 7ffae7c67640 1 --2- 172.21.15.72:0/1160243090 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ffae00cc7c0 0x7ffae00b1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.436+0000 7ffae7c67640 1 -- 172.21.15.72:0/1160243090 >> 172.21.15.72:0/1160243090 conn(0x7ffae00b31d0 msgr2=0x7ffae015aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:32.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.436+0000 7ffae7c67640 1 -- 172.21.15.72:0/1160243090 shutdown_connections 2024-04-15T15:15:32.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.436+0000 7ffae7c67640 1 -- 172.21.15.72:0/1160243090 wait complete. 2024-04-15T15:15:32.453 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:32.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.553+0000 7f199e80d640 1 Processor -- start 2024-04-15T15:15:32.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.553+0000 7f199e80d640 1 -- start start 2024-04-15T15:15:32.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.554+0000 7f199e80d640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f199815ad10 0x7f199815b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.554+0000 7f199e80d640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f19980cc5e0 0x7f199815b630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.554+0000 7f199e80d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f19980af740 0x7f199806c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.554+0000 7f199e80d640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f19980cf990 con 0x7f19980af740 2024-04-15T15:15:32.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.554+0000 7f199e80d640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f19980cd4b0 con 0x7f19980cc5e0 2024-04-15T15:15:32.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.554+0000 7f199e80d640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f19980cd790 con 0x7f199815ad10 2024-04-15T15:15:32.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.554+0000 7f199cdaa640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f19980af740 0x7f199806c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.554+0000 7f199cdaa640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f19980af740 0x7f199806c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54710/0 (socket says 172.21.15.72:54710) 2024-04-15T15:15:32.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.554+0000 7f199cdaa640 1 -- 172.21.15.72:0/496426014 learned_addr learned my addr 172.21.15.72:0/496426014 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:32.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.554+0000 7f199cdaa640 1 -- 172.21.15.72:0/496426014 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f199815ad10 msgr2=0x7f199815b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:32.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.554+0000 7f199cdaa640 1 --2- 172.21.15.72:0/496426014 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f199815ad10 0x7f199815b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.554+0000 7f199cdaa640 1 -- 172.21.15.72:0/496426014 >> v2:172.21.15.175:3300/0 conn(0x7f19980cc5e0 msgr2=0x7f199815b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:32.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.554+0000 7f199cdaa640 1 --2- 172.21.15.72:0/496426014 >> v2:172.21.15.175:3300/0 conn(0x7f19980cc5e0 0x7f199815b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.554+0000 7f199cdaa640 1 -- 172.21.15.72:0/496426014 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f199815bf10 con 0x7f19980af740 2024-04-15T15:15:32.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.555+0000 7f199cdaa640 1 --2- 172.21.15.72:0/496426014 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19980af740 0x7f199806c370 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f19980d12a0 tx=0x7f199002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c520958da7f5fe18 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:32.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.555+0000 7f1996ffd640 1 -- 172.21.15.72:0/496426014 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f199002ca70 con 0x7f19980af740 2024-04-15T15:15:32.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.555+0000 7f1996ffd640 1 -- 172.21.15.72:0/496426014 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f199002cc10 con 0x7f19980af740 2024-04-15T15:15:32.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.555+0000 7f1996ffd640 1 -- 172.21.15.72:0/496426014 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19900343f0 con 0x7f19980af740 2024-04-15T15:15:32.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.556+0000 7f199e80d640 1 -- 172.21.15.72:0/496426014 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19980af740 msgr2=0x7f199806c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.556+0000 7f199e80d640 1 --2- 172.21.15.72:0/496426014 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19980af740 0x7f199806c370 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f19980d12a0 tx=0x7f199002bed0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.556+0000 7f199e80d640 1 -- 172.21.15.72:0/496426014 shutdown_connections 2024-04-15T15:15:32.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.556+0000 7f199e80d640 1 --2- 172.21.15.72:0/496426014 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19980af740 0x7f199806c370 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.556+0000 7f199e80d640 1 --2- 172.21.15.72:0/496426014 >> v2:172.21.15.175:3300/0 conn(0x7f19980cc5e0 0x7f199815b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.556+0000 7f199e80d640 1 --2- 172.21.15.72:0/496426014 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f199815ad10 0x7f199815b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.556+0000 7f199e80d640 1 -- 172.21.15.72:0/496426014 >> 172.21.15.72:0/496426014 conn(0x7f19980b1dd0 msgr2=0x7f19980b21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:32.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.556+0000 7f199e80d640 1 -- 172.21.15.72:0/496426014 shutdown_connections 2024-04-15T15:15:32.560 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.556+0000 7f199e80d640 1 -- 172.21.15.72:0/496426014 wait complete. 2024-04-15T15:15:32.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.557+0000 7f199e80d640 1 Processor -- start 2024-04-15T15:15:32.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.557+0000 7f199e80d640 1 -- start start 2024-04-15T15:15:32.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.558+0000 7f199e80d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19980af740 0x7f199814e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.558+0000 7f199e80d640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f19980cc5e0 0x7f1998155500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.558+0000 7f199e80d640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f199815ad10 0x7f199814ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.558+0000 7f199e80d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f19980ce2d0 con 0x7f19980af740 2024-04-15T15:15:32.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.558+0000 7f199e80d640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f199815c1f0 con 0x7f199815ad10 2024-04-15T15:15:32.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.558+0000 7f199e80d640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f199815bb70 con 0x7f19980cc5e0 2024-04-15T15:15:32.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.558+0000 7f1997fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19980af740 0x7f199814e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.558+0000 7f1997fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19980af740 0x7f199814e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54712/0 (socket says 172.21.15.72:54712) 2024-04-15T15:15:32.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.558+0000 7f1997fff640 1 -- 172.21.15.72:0/3997994834 learned_addr learned my addr 172.21.15.72:0/3997994834 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:32.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.558+0000 7f19977fe640 1 --2- 172.21.15.72:0/3997994834 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f19980cc5e0 0x7f1998155500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.558+0000 7f199cdaa640 1 --2- 172.21.15.72:0/3997994834 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f199815ad10 0x7f199814ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.558+0000 7f1997fff640 1 -- 172.21.15.72:0/3997994834 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f19980cc5e0 msgr2=0x7f1998155500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.558+0000 7f1997fff640 1 --2- 172.21.15.72:0/3997994834 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f19980cc5e0 0x7f1998155500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.558+0000 7f1997fff640 1 -- 172.21.15.72:0/3997994834 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f199815ad10 msgr2=0x7f199814ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.558+0000 7f1997fff640 1 --2- 172.21.15.72:0/3997994834 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f199815ad10 0x7f199814ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.558+0000 7f1997fff640 1 -- 172.21.15.72:0/3997994834 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1998155ee0 con 0x7f19980af740 2024-04-15T15:15:32.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.558+0000 7f19977fe640 1 --2- 172.21.15.72:0/3997994834 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f19980cc5e0 0x7f1998155500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:32.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.558+0000 7f199cdaa640 1 --2- 172.21.15.72:0/3997994834 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f199815ad10 0x7f199814ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:32.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.558+0000 7f1997fff640 1 --2- 172.21.15.72:0/3997994834 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19980af740 0x7f199814e2c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f198800b440 tx=0x7f198800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:32.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.558+0000 7f1994ff9640 1 -- 172.21.15.72:0/3997994834 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1988015020 con 0x7f19980af740 2024-04-15T15:15:32.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.558+0000 7f1994ff9640 1 -- 172.21.15.72:0/3997994834 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1988011890 con 0x7f19980af740 2024-04-15T15:15:32.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.558+0000 7f1994ff9640 1 -- 172.21.15.72:0/3997994834 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1988011c80 con 0x7f19980af740 2024-04-15T15:15:32.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.558+0000 7f199e80d640 1 -- 172.21.15.72:0/3997994834 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19981561c0 con 0x7f19980af740 2024-04-15T15:15:32.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.559+0000 7f199e80d640 1 -- 172.21.15.72:0/3997994834 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1998156780 con 0x7f19980af740 2024-04-15T15:15:32.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.560+0000 7f199e80d640 1 -- 172.21.15.72:0/3997994834 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19980689e0 con 0x7f19980af740 2024-04-15T15:15:32.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.560+0000 7f1994ff9640 1 -- 172.21.15.72:0/3997994834 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1988018020 con 0x7f19980af740 2024-04-15T15:15:32.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.560+0000 7f1994ff9640 1 --2- 172.21.15.72:0/3997994834 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f198406c920 0x7f198406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.561+0000 7f1994ff9640 1 -- 172.21.15.72:0/3997994834 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f198808bfa0 con 0x7f19980af740 2024-04-15T15:15:32.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.561+0000 7f19977fe640 1 --2- 172.21.15.72:0/3997994834 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f198406c920 0x7f198406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.561+0000 7f19977fe640 1 --2- 172.21.15.72:0/3997994834 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f198406c920 0x7f198406ede0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f199815b540 tx=0x7f198c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:32.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.565+0000 7f1994ff9640 1 -- 172.21.15.72:0/3997994834 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f198801f020 con 0x7f19980af740 2024-04-15T15:15:32.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.739+0000 7f199e80d640 1 -- 172.21.15.72:0/3997994834 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1998153bb0 con 0x7f19980af740 2024-04-15T15:15:32.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.740+0000 7f1994ff9640 1 -- 172.21.15.72:0/3997994834 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18903 (secure 0 0 0) 0x7f198800d040 con 0x7f19980af740 2024-04-15T15:15:32.743 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:32.743 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":26,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:15:32.421728+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:14:49.438366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:14:50.368379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:14:54.081157+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":1}]},{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:32.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.742+0000 7f199e80d640 1 -- 172.21.15.72:0/3997994834 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f198406c920 msgr2=0x7f198406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.742+0000 7f199e80d640 1 --2- 172.21.15.72:0/3997994834 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f198406c920 0x7f198406ede0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f199815b540 tx=0x7f198c008040 comp rx=0 tx=0).stop 2024-04-15T15:15:32.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.742+0000 7f199e80d640 1 -- 172.21.15.72:0/3997994834 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19980af740 msgr2=0x7f199814e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.742+0000 7f199e80d640 1 --2- 172.21.15.72:0/3997994834 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19980af740 0x7f199814e2c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f198800b440 tx=0x7f198800b900 comp rx=0 tx=0).stop 2024-04-15T15:15:32.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.742+0000 7f199e80d640 1 -- 172.21.15.72:0/3997994834 shutdown_connections 2024-04-15T15:15:32.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.742+0000 7f199e80d640 1 --2- 172.21.15.72:0/3997994834 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f198406c920 0x7f198406ede0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.742+0000 7f199e80d640 1 --2- 172.21.15.72:0/3997994834 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f199815ad10 0x7f199814ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.742+0000 7f199e80d640 1 --2- 172.21.15.72:0/3997994834 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f19980cc5e0 0x7f1998155500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.742+0000 7f199e80d640 1 --2- 172.21.15.72:0/3997994834 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19980af740 0x7f199814e2c0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.742+0000 7f199e80d640 1 -- 172.21.15.72:0/3997994834 >> 172.21.15.72:0/3997994834 conn(0x7f19980b1dd0 msgr2=0x7f1998152ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:32.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.742+0000 7f199e80d640 1 -- 172.21.15.72:0/3997994834 shutdown_connections 2024-04-15T15:15:32.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.742+0000 7f199e80d640 1 -- 172.21.15.72:0/3997994834 wait complete. 2024-04-15T15:15:32.756 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:39.359102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:14:49.438366+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:14:50.368379+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:14:54.081157+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:15:32.757 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:32.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.860+0000 7fc3a688f640 1 Processor -- start 2024-04-15T15:15:32.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.860+0000 7fc3a688f640 1 -- start start 2024-04-15T15:15:32.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.862+0000 7fc3a688f640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc3a00ccb50 0x7fc3a006dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.862+0000 7fc3a688f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc3a015af20 0x7fc3a00ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.862+0000 7fc3a688f640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3a00cd470 con 0x7fc3a015e840 2024-04-15T15:15:32.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.862+0000 7fc3a688f640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fc3a006ba90 con 0x7fc3a015af20 2024-04-15T15:15:32.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.862+0000 7fc3a688f640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fc3a015c110 con 0x7fc3a00ccb50 2024-04-15T15:15:32.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.862+0000 7fc39f7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc3a015af20 0x7fc3a00ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.862+0000 7fc39f7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc3a015af20 0x7fc3a00ccf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54720/0 (socket says 172.21.15.72:54720) 2024-04-15T15:15:32.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.862+0000 7fc39f7fe640 1 -- 172.21.15.72:0/3010036358 learned_addr learned my addr 172.21.15.72:0/3010036358 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:32.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.862+0000 7fc39ffff640 1 --2- 172.21.15.72:0/3010036358 >> v2:172.21.15.175:3300/0 conn(0x7fc3a00ccb50 0x7fc3a006dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.862+0000 7fc39f7fe640 1 -- 172.21.15.72:0/3010036358 >> v2:172.21.15.175:3300/0 conn(0x7fc3a00ccb50 msgr2=0x7fc3a006dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:32.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.862+0000 7fc39f7fe640 1 --2- 172.21.15.72:0/3010036358 >> v2:172.21.15.175:3300/0 conn(0x7fc3a00ccb50 0x7fc3a006dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.862+0000 7fc39f7fe640 1 -- 172.21.15.72:0/3010036358 >> v1:172.21.15.175:6789/0 conn(0x7fc3a015e840 legacy=0x7fc3a015ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:32.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.862+0000 7fc39f7fe640 1 -- 172.21.15.72:0/3010036358 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3a00cfae0 con 0x7fc3a015af20 2024-04-15T15:15:32.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.862+0000 7fc39f7fe640 1 --2- 172.21.15.72:0/3010036358 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3a015af20 0x7fc3a00ccf30 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fc3a00cdf50 tx=0x7fc39402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f42e47d5c4271277 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:32.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.862+0000 7fc39effd640 1 -- 172.21.15.72:0/3010036358 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc39402e070 con 0x7fc3a015af20 2024-04-15T15:15:32.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.862+0000 7fc39effd640 1 -- 172.21.15.72:0/3010036358 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc394037070 con 0x7fc3a015af20 2024-04-15T15:15:32.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.862+0000 7fc39effd640 1 -- 172.21.15.72:0/3010036358 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc394031a00 con 0x7fc3a015af20 2024-04-15T15:15:32.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.863+0000 7fc3a688f640 1 -- 172.21.15.72:0/3010036358 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3a015af20 msgr2=0x7fc3a00ccf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.863+0000 7fc3a688f640 1 --2- 172.21.15.72:0/3010036358 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3a015af20 0x7fc3a00ccf30 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fc3a00cdf50 tx=0x7fc39402b880 comp rx=0 tx=0).stop 2024-04-15T15:15:32.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.863+0000 7fc3a688f640 1 -- 172.21.15.72:0/3010036358 shutdown_connections 2024-04-15T15:15:32.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.863+0000 7fc3a688f640 1 --2- 172.21.15.72:0/3010036358 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3a015af20 0x7fc3a00ccf30 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.863+0000 7fc3a688f640 1 --2- 172.21.15.72:0/3010036358 >> v2:172.21.15.175:3300/0 conn(0x7fc3a00ccb50 0x7fc3a006dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.863+0000 7fc3a688f640 1 -- 172.21.15.72:0/3010036358 >> 172.21.15.72:0/3010036358 conn(0x7fc3a00b31d0 msgr2=0x7fc3a00b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:32.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.863+0000 7fc3a688f640 1 -- 172.21.15.72:0/3010036358 shutdown_connections 2024-04-15T15:15:32.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.863+0000 7fc3a688f640 1 -- 172.21.15.72:0/3010036358 wait complete. 2024-04-15T15:15:32.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.864+0000 7fc3a688f640 1 Processor -- start 2024-04-15T15:15:32.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.864+0000 7fc3a688f640 1 -- start start 2024-04-15T15:15:32.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.865+0000 7fc3a688f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3a00ccb50 0x7fc3a0149d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.865+0000 7fc3a688f640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc3a015af20 0x7fc3a006bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.865+0000 7fc3a688f640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc3a015e840 0x7fc3a006c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.865+0000 7fc3a688f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a00cf8c0 con 0x7fc3a00ccb50 2024-04-15T15:15:32.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.865+0000 7fc3a688f640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a00cf740 con 0x7fc3a015e840 2024-04-15T15:15:32.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.865+0000 7fc3a688f640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fc3a00cc770 con 0x7fc3a015af20 2024-04-15T15:15:32.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.865+0000 7fc39ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3a00ccb50 0x7fc3a0149d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.865+0000 7fc39ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3a00ccb50 0x7fc3a0149d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54732/0 (socket says 172.21.15.72:54732) 2024-04-15T15:15:32.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.865+0000 7fc39ffff640 1 -- 172.21.15.72:0/33649411 learned_addr learned my addr 172.21.15.72:0/33649411 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:32.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.865+0000 7fc39f7fe640 1 --2- 172.21.15.72:0/33649411 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc3a015af20 0x7fc3a006bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.865+0000 7fc3a4e2c640 1 --2- 172.21.15.72:0/33649411 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc3a015e840 0x7fc3a006c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.865+0000 7fc39ffff640 1 -- 172.21.15.72:0/33649411 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc3a015af20 msgr2=0x7fc3a006bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.865+0000 7fc39ffff640 1 --2- 172.21.15.72:0/33649411 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc3a015af20 0x7fc3a006bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.865+0000 7fc39ffff640 1 -- 172.21.15.72:0/33649411 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc3a015e840 msgr2=0x7fc3a006c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.865+0000 7fc39ffff640 1 --2- 172.21.15.72:0/33649411 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc3a015e840 0x7fc3a006c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.865+0000 7fc39ffff640 1 -- 172.21.15.72:0/33649411 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3a015d6c0 con 0x7fc3a00ccb50 2024-04-15T15:15:32.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.865+0000 7fc39ffff640 1 --2- 172.21.15.72:0/33649411 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3a00ccb50 0x7fc3a0149d10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc39000fa70 tx=0x7fc39000ff30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:32.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.865+0000 7fc39cff9640 1 -- 172.21.15.72:0/33649411 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc390015070 con 0x7fc3a00ccb50 2024-04-15T15:15:32.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.866+0000 7fc39cff9640 1 -- 172.21.15.72:0/33649411 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc390007c90 con 0x7fc3a00ccb50 2024-04-15T15:15:32.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.866+0000 7fc3a688f640 1 -- 172.21.15.72:0/33649411 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3a015d9a0 con 0x7fc3a00ccb50 2024-04-15T15:15:32.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.866+0000 7fc39cff9640 1 -- 172.21.15.72:0/33649411 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc39001a680 con 0x7fc3a00ccb50 2024-04-15T15:15:32.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.866+0000 7fc3a688f640 1 -- 172.21.15.72:0/33649411 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3a015df60 con 0x7fc3a00ccb50 2024-04-15T15:15:32.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.867+0000 7fc3a688f640 1 -- 172.21.15.72:0/33649411 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3a00b2fd0 con 0x7fc3a00ccb50 2024-04-15T15:15:32.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.867+0000 7fc39cff9640 1 -- 172.21.15.72:0/33649411 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc39001a820 con 0x7fc3a00ccb50 2024-04-15T15:15:32.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.867+0000 7fc39cff9640 1 --2- 172.21.15.72:0/33649411 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc38c06c8f0 0x7fc38c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.867+0000 7fc39cff9640 1 -- 172.21.15.72:0/33649411 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fc39008e840 con 0x7fc3a00ccb50 2024-04-15T15:15:32.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.867+0000 7fc39f7fe640 1 --2- 172.21.15.72:0/33649411 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc38c06c8f0 0x7fc38c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.868+0000 7fc39f7fe640 1 --2- 172.21.15.72:0/33649411 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc38c06c8f0 0x7fc38c06edb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc394036000 tx=0x7fc39403d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:32.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:32.872+0000 7fc39cff9640 1 -- 172.21.15.72:0/33649411 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc3a00b2fd0 con 0x7fc3a00ccb50 2024-04-15T15:15:33.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.047+0000 7fc3a688f640 1 -- 172.21.15.72:0/33649411 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc3a0075c50 con 0x7fc3a00ccb50 2024-04-15T15:15:33.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.047+0000 7fc39cff9640 1 -- 172.21.15.72:0/33649411 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7fc39005c870 con 0x7fc3a00ccb50 2024-04-15T15:15:33.049 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-15T15:15:33.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.048+0000 7fc3a688f640 1 -- 172.21.15.72:0/33649411 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc38c06c8f0 msgr2=0x7fc38c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.048+0000 7fc3a688f640 1 --2- 172.21.15.72:0/33649411 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc38c06c8f0 0x7fc38c06edb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc394036000 tx=0x7fc39403d040 comp rx=0 tx=0).stop 2024-04-15T15:15:33.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.048+0000 7fc3a688f640 1 -- 172.21.15.72:0/33649411 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3a00ccb50 msgr2=0x7fc3a0149d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.048+0000 7fc3a688f640 1 --2- 172.21.15.72:0/33649411 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3a00ccb50 0x7fc3a0149d10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc39000fa70 tx=0x7fc39000ff30 comp rx=0 tx=0).stop 2024-04-15T15:15:33.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.048+0000 7fc3a688f640 1 -- 172.21.15.72:0/33649411 shutdown_connections 2024-04-15T15:15:33.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.048+0000 7fc3a688f640 1 --2- 172.21.15.72:0/33649411 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc38c06c8f0 0x7fc38c06edb0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.048+0000 7fc3a688f640 1 --2- 172.21.15.72:0/33649411 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc3a015e840 0x7fc3a006c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.048+0000 7fc3a688f640 1 --2- 172.21.15.72:0/33649411 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc3a015af20 0x7fc3a006bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.048+0000 7fc3a688f640 1 --2- 172.21.15.72:0/33649411 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3a00ccb50 0x7fc3a0149d10 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.048+0000 7fc3a688f640 1 -- 172.21.15.72:0/33649411 >> 172.21.15.72:0/33649411 conn(0x7fc3a00b31d0 msgr2=0x7fc3a006e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:33.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.048+0000 7fc3a688f640 1 -- 172.21.15.72:0/33649411 shutdown_connections 2024-04-15T15:15:33.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.048+0000 7fc3a688f640 1 -- 172.21.15.72:0/33649411 wait complete. 2024-04-15T15:15:33.062 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:15:33.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.166+0000 7f7992022640 1 Processor -- start 2024-04-15T15:15:33.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.167+0000 7f7992022640 1 -- start start 2024-04-15T15:15:33.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.167+0000 7f7992022640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f798c0d08f0 0x7f798c0cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.167+0000 7f7992022640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f798c0ce6d0 0x7f798c0cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.167+0000 7f7992022640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f798c06bbb0 con 0x7f798c0ce2f0 2024-04-15T15:15:33.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.167+0000 7f7992022640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f798c06b960 con 0x7f798c0d08f0 2024-04-15T15:15:33.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.167+0000 7f7992022640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f798c06be90 con 0x7f798c0ce6d0 2024-04-15T15:15:33.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.168+0000 7f798bfff640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f798c0ce6d0 0x7f798c0cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.168+0000 7f798affd640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f798c0d08f0 0x7f798c0cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.168+0000 7f798bfff640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f798c0ce6d0 0x7f798c0cc5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3301/0 says I am v2:172.21.15.72:48524/0 (socket says 172.21.15.72:48524) 2024-04-15T15:15:33.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.168+0000 7f798bfff640 1 -- 172.21.15.72:0/2606877584 learned_addr learned my addr 172.21.15.72:0/2606877584 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:33.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.168+0000 7f798b7fe640 1 --1- 172.21.15.72:0/2606877584 >> v1:172.21.15.175:6789/0 conn(0x7f798c0ce2f0 0x7f798c15c530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.72:38698/0 (socket says 172.21.15.72:38698) 2024-04-15T15:15:33.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.168+0000 7f798bfff640 1 -- 172.21.15.72:0/2606877584 >> v2:172.21.15.175:3300/0 conn(0x7f798c0d08f0 msgr2=0x7f798c0cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:33.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.168+0000 7f798bfff640 1 --2- 172.21.15.72:0/2606877584 >> v2:172.21.15.175:3300/0 conn(0x7f798c0d08f0 0x7f798c0cc0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.168+0000 7f798bfff640 1 -- 172.21.15.72:0/2606877584 >> v1:172.21.15.175:6789/0 conn(0x7f798c0ce2f0 legacy=0x7f798c15c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.168+0000 7f798bfff640 1 -- 172.21.15.72:0/2606877584 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f798c06c170 con 0x7f798c0ce6d0 2024-04-15T15:15:33.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.168+0000 7f798affd640 1 --2- 172.21.15.72:0/2606877584 >> v2:172.21.15.175:3300/0 conn(0x7f798c0d08f0 0x7f798c0cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:33.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.169+0000 7f798bfff640 1 --2- 172.21.15.72:0/2606877584 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f798c0ce6d0 0x7f798c0cc5f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f798400bb10 tx=0x7f798402e150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1dd80acfadfbf064 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:33.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.169+0000 7f798a7fc640 1 -- 172.21.15.72:0/2606877584 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f798403b070 con 0x7f798c0ce6d0 2024-04-15T15:15:33.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.169+0000 7f798a7fc640 1 -- 172.21.15.72:0/2606877584 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7984036440 con 0x7f798c0ce6d0 2024-04-15T15:15:33.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.169+0000 7f798a7fc640 1 -- 172.21.15.72:0/2606877584 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7984036810 con 0x7f798c0ce6d0 2024-04-15T15:15:33.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.170+0000 7f7992022640 1 -- 172.21.15.72:0/2606877584 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f798c0ce6d0 msgr2=0x7f798c0cc5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.170+0000 7f7992022640 1 --2- 172.21.15.72:0/2606877584 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f798c0ce6d0 0x7f798c0cc5f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f798400bb10 tx=0x7f798402e150 comp rx=0 tx=0).stop 2024-04-15T15:15:33.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.170+0000 7f7992022640 1 -- 172.21.15.72:0/2606877584 shutdown_connections 2024-04-15T15:15:33.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.170+0000 7f7992022640 1 --2- 172.21.15.72:0/2606877584 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f798c0ce6d0 0x7f798c0cc5f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.170+0000 7f7992022640 1 --2- 172.21.15.72:0/2606877584 >> v2:172.21.15.175:3300/0 conn(0x7f798c0d08f0 0x7f798c0cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.170+0000 7f7992022640 1 -- 172.21.15.72:0/2606877584 >> 172.21.15.72:0/2606877584 conn(0x7f798c0b31d0 msgr2=0x7f798c0b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:33.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.170+0000 7f7992022640 1 -- 172.21.15.72:0/2606877584 shutdown_connections 2024-04-15T15:15:33.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.170+0000 7f7992022640 1 -- 172.21.15.72:0/2606877584 wait complete. 2024-04-15T15:15:33.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.171+0000 7f7992022640 1 Processor -- start 2024-04-15T15:15:33.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.171+0000 7f7992022640 1 -- start start 2024-04-15T15:15:33.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.172+0000 7f7992022640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f798c0ce2f0 0x7f798c154aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.172+0000 7f7992022640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f798c0ce6d0 0x7f798c0afd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.175 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.172+0000 7f7992022640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f798c0d08f0 0x7f798c14d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.172+0000 7f7992022640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f798c155780 con 0x7f798c0ce2f0 2024-04-15T15:15:33.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.172+0000 7f7992022640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f798c155600 con 0x7f798c0d08f0 2024-04-15T15:15:33.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.172+0000 7f7992022640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f798c155900 con 0x7f798c0ce6d0 2024-04-15T15:15:33.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.172+0000 7f798b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f798c0ce2f0 0x7f798c154aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.172+0000 7f798b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f798c0ce2f0 0x7f798c154aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54748/0 (socket says 172.21.15.72:54748) 2024-04-15T15:15:33.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.172+0000 7f798b7fe640 1 -- 172.21.15.72:0/2631261408 learned_addr learned my addr 172.21.15.72:0/2631261408 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:33.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.172+0000 7f798affd640 1 --2- 172.21.15.72:0/2631261408 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f798c0ce6d0 0x7f798c0afd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.172+0000 7f798bfff640 1 --2- 172.21.15.72:0/2631261408 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f798c0d08f0 0x7f798c14d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.172+0000 7f798b7fe640 1 -- 172.21.15.72:0/2631261408 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f798c0ce6d0 msgr2=0x7f798c0afd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.172+0000 7f798b7fe640 1 --2- 172.21.15.72:0/2631261408 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f798c0ce6d0 0x7f798c0afd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.172+0000 7f798b7fe640 1 -- 172.21.15.72:0/2631261408 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f798c0d08f0 msgr2=0x7f798c14d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.172+0000 7f798b7fe640 1 --2- 172.21.15.72:0/2631261408 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f798c0d08f0 0x7f798c14d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.172+0000 7f798b7fe640 1 -- 172.21.15.72:0/2631261408 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f798c0b0280 con 0x7f798c0ce2f0 2024-04-15T15:15:33.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.172+0000 7f798b7fe640 1 --2- 172.21.15.72:0/2631261408 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f798c0ce2f0 0x7f798c154aa0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f797400c790 tx=0x7f797400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:33.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.172+0000 7f7983fff640 1 -- 172.21.15.72:0/2631261408 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7974017070 con 0x7f798c0ce2f0 2024-04-15T15:15:33.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.173+0000 7f7983fff640 1 -- 172.21.15.72:0/2631261408 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f797400a640 con 0x7f798c0ce2f0 2024-04-15T15:15:33.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.173+0000 7f7983fff640 1 -- 172.21.15.72:0/2631261408 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f797400a9f0 con 0x7f798c0ce2f0 2024-04-15T15:15:33.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.173+0000 7f7992022640 1 -- 172.21.15.72:0/2631261408 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f798c0b0500 con 0x7f798c0ce2f0 2024-04-15T15:15:33.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.173+0000 7f7992022640 1 -- 172.21.15.72:0/2631261408 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f798c0b0ac0 con 0x7f798c0ce2f0 2024-04-15T15:15:33.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.174+0000 7f7992022640 1 -- 172.21.15.72:0/2631261408 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f798c075c50 con 0x7f798c0ce2f0 2024-04-15T15:15:33.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.174+0000 7f7983fff640 1 -- 172.21.15.72:0/2631261408 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f797401a070 con 0x7f798c0ce2f0 2024-04-15T15:15:33.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.174+0000 7f7983fff640 1 --2- 172.21.15.72:0/2631261408 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f796c06c8f0 0x7f796c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.174+0000 7f7983fff640 1 -- 172.21.15.72:0/2631261408 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f797408b4a0 con 0x7f798c0ce2f0 2024-04-15T15:15:33.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.174+0000 7f798affd640 1 --2- 172.21.15.72:0/2631261408 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f796c06c8f0 0x7f796c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.175+0000 7f798affd640 1 --2- 172.21.15.72:0/2631261408 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f796c06c8f0 0x7f796c06edb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f797c009560 tx=0x7f797c009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:33.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.179+0000 7f7983fff640 1 -- 172.21.15.72:0/2631261408 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f798c0b0280 con 0x7f798c0ce2f0 2024-04-15T15:15:33.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.352+0000 7f7992022640 1 -- 172.21.15.72:0/2631261408 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f798c06cab0 con 0x7f798c0ce2f0 2024-04-15T15:15:33.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.352+0000 7f7983fff640 1 -- 172.21.15.72:0/2631261408 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f7974058e80 con 0x7f798c0ce2f0 2024-04-15T15:15:33.354 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:33.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.354+0000 7f7992022640 1 -- 172.21.15.72:0/2631261408 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f796c06c8f0 msgr2=0x7f796c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.354+0000 7f7992022640 1 --2- 172.21.15.72:0/2631261408 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f796c06c8f0 0x7f796c06edb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f797c009560 tx=0x7f797c009190 comp rx=0 tx=0).stop 2024-04-15T15:15:33.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.354+0000 7f7992022640 1 -- 172.21.15.72:0/2631261408 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f798c0ce2f0 msgr2=0x7f798c154aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.354+0000 7f7992022640 1 --2- 172.21.15.72:0/2631261408 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f798c0ce2f0 0x7f798c154aa0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f797400c790 tx=0x7f797400cc50 comp rx=0 tx=0).stop 2024-04-15T15:15:33.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.354+0000 7f7992022640 1 -- 172.21.15.72:0/2631261408 shutdown_connections 2024-04-15T15:15:33.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.354+0000 7f7992022640 1 --2- 172.21.15.72:0/2631261408 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f796c06c8f0 0x7f796c06edb0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.354+0000 7f7992022640 1 --2- 172.21.15.72:0/2631261408 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f798c0d08f0 0x7f798c14d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.354+0000 7f7992022640 1 --2- 172.21.15.72:0/2631261408 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f798c0ce6d0 0x7f798c0afd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.354+0000 7f7992022640 1 --2- 172.21.15.72:0/2631261408 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f798c0ce2f0 0x7f798c154aa0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.354+0000 7f7992022640 1 -- 172.21.15.72:0/2631261408 >> 172.21.15.72:0/2631261408 conn(0x7f798c0b31d0 msgr2=0x7f798c15d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:33.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.355+0000 7f7992022640 1 -- 172.21.15.72:0/2631261408 shutdown_connections 2024-04-15T15:15:33.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.355+0000 7f7992022640 1 -- 172.21.15.72:0/2631261408 wait complete. 2024-04-15T15:15:33.367 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:15:33.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.475+0000 7f707f088640 1 Processor -- start 2024-04-15T15:15:33.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.475+0000 7f707f088640 1 -- start start 2024-04-15T15:15:33.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.475+0000 7f707f088640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f70780cf160 0x7f70780cf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.475+0000 7f707f088640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f707815e2a0 0x7f707815e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.475+0000 7f707f088640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70780b61e0 con 0x7f707815d830 2024-04-15T15:15:33.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.475+0000 7f707f088640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f70780b64c0 con 0x7f707815e2a0 2024-04-15T15:15:33.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.475+0000 7f707f088640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f70780b67a0 con 0x7f70780cf160 2024-04-15T15:15:33.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.475+0000 7f707d625640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f707815e2a0 0x7f707815e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.476+0000 7f707ce24640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f707815d830 0x7f70780b57f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49988/0 (socket says 172.21.15.72:49988) 2024-04-15T15:15:33.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.476+0000 7f707ce24640 1 -- 172.21.15.72:0/3883194705 learned_addr learned my addr 172.21.15.72:0/3883194705 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:33.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.476+0000 7f7077fff640 1 --2- 172.21.15.72:0/3883194705 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f70780cf160 0x7f70780cf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.476+0000 7f707d625640 1 -- 172.21.15.72:0/3883194705 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f70780cf160 msgr2=0x7f70780cf5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:33.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.476+0000 7f707d625640 1 --2- 172.21.15.72:0/3883194705 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f70780cf160 0x7f70780cf5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.476+0000 7f707d625640 1 -- 172.21.15.72:0/3883194705 >> v1:172.21.15.72:6789/0 conn(0x7f707815d830 legacy=0x7f70780b57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.476+0000 7f707d625640 1 -- 172.21.15.72:0/3883194705 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70780b6a80 con 0x7f707815e2a0 2024-04-15T15:15:33.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.476+0000 7f70777fe640 1 -- 172.21.15.72:0/3883194705 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 90025331 0 0) 0x7f70780b61e0 con 0x7f707815d830 2024-04-15T15:15:33.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.477+0000 7f707d625640 1 --2- 172.21.15.72:0/3883194705 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f707815e2a0 0x7f707815e6c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7070009d90 tx=0x7f707002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8243242682ea77ca server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:33.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.477+0000 7f70777fe640 1 -- 172.21.15.72:0/3883194705 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7070039070 con 0x7f707815e2a0 2024-04-15T15:15:33.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.477+0000 7f70777fe640 1 -- 172.21.15.72:0/3883194705 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f707002cbc0 con 0x7f707815e2a0 2024-04-15T15:15:33.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.477+0000 7f70777fe640 1 -- 172.21.15.72:0/3883194705 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7070034440 con 0x7f707815e2a0 2024-04-15T15:15:33.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.477+0000 7f707f088640 1 -- 172.21.15.72:0/3883194705 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f707815e2a0 msgr2=0x7f707815e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.477+0000 7f707f088640 1 --2- 172.21.15.72:0/3883194705 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f707815e2a0 0x7f707815e6c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7070009d90 tx=0x7f707002be10 comp rx=0 tx=0).stop 2024-04-15T15:15:33.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.477+0000 7f707f088640 1 -- 172.21.15.72:0/3883194705 shutdown_connections 2024-04-15T15:15:33.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.477+0000 7f707f088640 1 --2- 172.21.15.72:0/3883194705 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f707815e2a0 0x7f707815e6c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.477+0000 7f707f088640 1 --2- 172.21.15.72:0/3883194705 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f70780cf160 0x7f70780cf5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.477+0000 7f707f088640 1 -- 172.21.15.72:0/3883194705 >> 172.21.15.72:0/3883194705 conn(0x7f70780d24d0 msgr2=0x7f70780d28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:33.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.477+0000 7f707f088640 1 -- 172.21.15.72:0/3883194705 shutdown_connections 2024-04-15T15:15:33.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.477+0000 7f707f088640 1 -- 172.21.15.72:0/3883194705 wait complete. 2024-04-15T15:15:33.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.478+0000 7f707f088640 1 Processor -- start 2024-04-15T15:15:33.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.478+0000 7f707f088640 1 -- start start 2024-04-15T15:15:33.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.479+0000 7f707f088640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f70780cf160 0x7f70780c72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.479+0000 7f707f088640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f707815d830 0x7f70780c80c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.479+0000 7f707f088640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f707815e2a0 0x7f70781e63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.479+0000 7f707f088640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f707815cd50 con 0x7f70780cf160 2024-04-15T15:15:33.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.479+0000 7f707f088640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f70780b5f00 con 0x7f707815e2a0 2024-04-15T15:15:33.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.479+0000 7f707f088640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f707815ced0 con 0x7f707815d830 2024-04-15T15:15:33.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.479+0000 7f707ce24640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f70780cf160 0x7f70780c72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.479+0000 7f707ce24640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f70780cf160 0x7f70780c72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54758/0 (socket says 172.21.15.72:54758) 2024-04-15T15:15:33.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.479+0000 7f707ce24640 1 -- 172.21.15.72:0/2422118576 learned_addr learned my addr 172.21.15.72:0/2422118576 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:33.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.480+0000 7f707ce24640 1 -- 172.21.15.72:0/2422118576 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f707815d830 msgr2=0x7f70780c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.480+0000 7f7077fff640 1 --2- 172.21.15.72:0/2422118576 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f707815d830 0x7f70780c80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.480+0000 7f707ce24640 1 --2- 172.21.15.72:0/2422118576 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f707815d830 0x7f70780c80c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.480+0000 7f707ce24640 1 -- 172.21.15.72:0/2422118576 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f707815e2a0 msgr2=0x7f70781e63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.480+0000 7f707ce24640 1 --2- 172.21.15.72:0/2422118576 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f707815e2a0 0x7f70781e63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.480+0000 7f707ce24640 1 -- 172.21.15.72:0/2422118576 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70781f7810 con 0x7f70780cf160 2024-04-15T15:15:33.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.480+0000 7f707ce24640 1 --2- 172.21.15.72:0/2422118576 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f70780cf160 0x7f70780c72f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f706800c4d0 tx=0x7f706800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:33.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.480+0000 7f70757fa640 1 -- 172.21.15.72:0/2422118576 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7068018070 con 0x7f70780cf160 2024-04-15T15:15:33.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.480+0000 7f70757fa640 1 -- 172.21.15.72:0/2422118576 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7068007500 con 0x7f70780cf160 2024-04-15T15:15:33.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.480+0000 7f707f088640 1 -- 172.21.15.72:0/2422118576 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70781f7af0 con 0x7f70780cf160 2024-04-15T15:15:33.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.480+0000 7f70757fa640 1 -- 172.21.15.72:0/2422118576 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7068004a20 con 0x7f70780cf160 2024-04-15T15:15:33.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.480+0000 7f707f088640 1 -- 172.21.15.72:0/2422118576 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70781f80b0 con 0x7f70780cf160 2024-04-15T15:15:33.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.481+0000 7f707f088640 1 -- 172.21.15.72:0/2422118576 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7078075c50 con 0x7f70780cf160 2024-04-15T15:15:33.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.481+0000 7f70757fa640 1 -- 172.21.15.72:0/2422118576 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f706801b070 con 0x7f70780cf160 2024-04-15T15:15:33.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.482+0000 7f70757fa640 1 --2- 172.21.15.72:0/2422118576 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f706406c8f0 0x7f706406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.482+0000 7f70757fa640 1 -- 172.21.15.72:0/2422118576 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f706808b290 con 0x7f70780cf160 2024-04-15T15:15:33.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.482+0000 7f7077fff640 1 --2- 172.21.15.72:0/2422118576 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f706406c8f0 0x7f706406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.482+0000 7f7077fff640 1 --2- 172.21.15.72:0/2422118576 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f706406c8f0 0x7f706406edb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7078070870 tx=0x7f706c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:33.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.486+0000 7f70757fa640 1 -- 172.21.15.72:0/2422118576 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f70680592c0 con 0x7f70780cf160 2024-04-15T15:15:33.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.664+0000 7f707f088640 1 -- 172.21.15.72:0/2422118576 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f707806d420 con 0x7f70780cf160 2024-04-15T15:15:33.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.664+0000 7f70757fa640 1 -- 172.21.15.72:0/2422118576 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f7068058bc0 con 0x7f70780cf160 2024-04-15T15:15:33.666 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-04-15T15:15:33.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.666+0000 7f707f088640 1 -- 172.21.15.72:0/2422118576 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f706406c8f0 msgr2=0x7f706406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.666+0000 7f707f088640 1 --2- 172.21.15.72:0/2422118576 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f706406c8f0 0x7f706406edb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7078070870 tx=0x7f706c008040 comp rx=0 tx=0).stop 2024-04-15T15:15:33.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.666+0000 7f707f088640 1 -- 172.21.15.72:0/2422118576 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f70780cf160 msgr2=0x7f70780c72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.666+0000 7f707f088640 1 --2- 172.21.15.72:0/2422118576 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f70780cf160 0x7f70780c72f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f706800c4d0 tx=0x7f706800c990 comp rx=0 tx=0).stop 2024-04-15T15:15:33.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.666+0000 7f707f088640 1 -- 172.21.15.72:0/2422118576 shutdown_connections 2024-04-15T15:15:33.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.666+0000 7f707f088640 1 --2- 172.21.15.72:0/2422118576 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f706406c8f0 0x7f706406edb0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.666+0000 7f707f088640 1 --2- 172.21.15.72:0/2422118576 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f707815e2a0 0x7f70781e63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.666+0000 7f707f088640 1 --2- 172.21.15.72:0/2422118576 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f707815d830 0x7f70780c80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.666+0000 7f707f088640 1 --2- 172.21.15.72:0/2422118576 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f70780cf160 0x7f70780c72f0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.666+0000 7f707f088640 1 -- 172.21.15.72:0/2422118576 >> 172.21.15.72:0/2422118576 conn(0x7f70780d24d0 msgr2=0x7f7078154a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:33.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.666+0000 7f707f088640 1 -- 172.21.15.72:0/2422118576 shutdown_connections 2024-04-15T15:15:33.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.666+0000 7f707f088640 1 -- 172.21.15.72:0/2422118576 wait complete. 2024-04-15T15:15:33.679 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:15:33.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.782+0000 7f57ccfa8640 1 Processor -- start 2024-04-15T15:15:33.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.782+0000 7f57ccfa8640 1 -- start start 2024-04-15T15:15:33.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.783+0000 7f57ccfa8640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f57c806c730 0x7f57c806cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.783+0000 7f57ccfa8640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57c80cf8f0 con 0x7f57c81667c0 2024-04-15T15:15:33.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.783+0000 7f57ccfa8640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57c80b3ae0 con 0x7f57c806c350 2024-04-15T15:15:33.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.783+0000 7f57ccfa8640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f57c80d27d0 con 0x7f57c806c730 2024-04-15T15:15:33.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.783+0000 7f57c6d9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f57c81667c0 0x7f57c8168bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49990/0 (socket says 172.21.15.72:49990) 2024-04-15T15:15:33.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.783+0000 7f57c6d9d640 1 -- 172.21.15.72:0/555677299 learned_addr learned my addr 172.21.15.72:0/555677299 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:33.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.784+0000 7f57c5d9b640 1 --2- 172.21.15.72:0/555677299 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f57c806c730 0x7f57c806cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.784+0000 7f57c559a640 1 -- 172.21.15.72:0/555677299 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 796395231 0 0) 0x7f57c80cf8f0 con 0x7f57c81667c0 2024-04-15T15:15:33.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.784+0000 7f57c559a640 1 -- 172.21.15.72:0/555677299 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57b0003520 con 0x7f57c81667c0 2024-04-15T15:15:33.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.784+0000 7f57c559a640 1 -- 172.21.15.72:0/555677299 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1346039113 0 0) 0x7f57b8001110 con 0x7f57c806c350 2024-04-15T15:15:33.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.784+0000 7f57c559a640 1 -- 172.21.15.72:0/555677299 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f57c80cf8f0 con 0x7f57c806c350 2024-04-15T15:15:33.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.784+0000 7f57c5d9b640 1 -- 172.21.15.72:0/555677299 >> v1:172.21.15.175:6789/0 conn(0x7f57c806c350 legacy=0x7f57c80d0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.784+0000 7f57c5d9b640 1 -- 172.21.15.72:0/555677299 >> v1:172.21.15.72:6789/0 conn(0x7f57c81667c0 legacy=0x7f57c8168bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.784+0000 7f57c559a640 1 -- 172.21.15.72:0/555677299 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2779579367 0 0) 0x7f57b0003520 con 0x7f57c81667c0 2024-04-15T15:15:33.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.784+0000 7f57c5d9b640 1 -- 172.21.15.72:0/555677299 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57c80d0e20 con 0x7f57c806c730 2024-04-15T15:15:33.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.785+0000 7f57c5d9b640 1 --2- 172.21.15.72:0/555677299 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f57c806c730 0x7f57c806cb90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f57bc0097d0 tx=0x7f57bc02b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8ccdd414073442f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:33.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.785+0000 7f57c559a640 1 -- 172.21.15.72:0/555677299 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57bc037070 con 0x7f57c806c730 2024-04-15T15:15:33.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.785+0000 7f57c559a640 1 -- 172.21.15.72:0/555677299 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57bc030510 con 0x7f57c806c730 2024-04-15T15:15:33.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.785+0000 7f57c559a640 1 -- 172.21.15.72:0/555677299 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57c00019f0 con 0x7f57c806c730 2024-04-15T15:15:33.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.785+0000 7f57ccfa8640 1 -- 172.21.15.72:0/555677299 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f57c806c730 msgr2=0x7f57c806cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.785+0000 7f57ccfa8640 1 --2- 172.21.15.72:0/555677299 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f57c806c730 0x7f57c806cb90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f57bc0097d0 tx=0x7f57bc02b770 comp rx=0 tx=0).stop 2024-04-15T15:15:33.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.786+0000 7f57ccfa8640 1 -- 172.21.15.72:0/555677299 shutdown_connections 2024-04-15T15:15:33.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.786+0000 7f57ccfa8640 1 --2- 172.21.15.72:0/555677299 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f57c806c730 0x7f57c806cb90 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.786+0000 7f57ccfa8640 1 -- 172.21.15.72:0/555677299 >> 172.21.15.72:0/555677299 conn(0x7f57c80d48c0 msgr2=0x7f57c80d4cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:33.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.786+0000 7f57ccfa8640 1 -- 172.21.15.72:0/555677299 shutdown_connections 2024-04-15T15:15:33.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.786+0000 7f57ccfa8640 1 -- 172.21.15.72:0/555677299 wait complete. 2024-04-15T15:15:33.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.787+0000 7f57ccfa8640 1 Processor -- start 2024-04-15T15:15:33.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.787+0000 7f57ccfa8640 1 -- start start 2024-04-15T15:15:33.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.788+0000 7f57ccfa8640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f57c806c350 0x7f57c81557b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.788+0000 7f57ccfa8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57c806c730 0x7f57c815f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.788+0000 7f57ccfa8640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f57c81667c0 0x7f57c8154f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.788+0000 7f57ccfa8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f57c80b3240 con 0x7f57c806c730 2024-04-15T15:15:33.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.788+0000 7f57ccfa8640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f57c80d3c70 con 0x7f57c81667c0 2024-04-15T15:15:33.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.788+0000 7f57ccfa8640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f57c80b3520 con 0x7f57c806c350 2024-04-15T15:15:33.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.788+0000 7f57c5d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57c806c730 0x7f57c815f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.788+0000 7f57c5d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57c806c730 0x7f57c815f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54762/0 (socket says 172.21.15.72:54762) 2024-04-15T15:15:33.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.788+0000 7f57c5d9b640 1 -- 172.21.15.72:0/2036069416 learned_addr learned my addr 172.21.15.72:0/2036069416 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:33.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.788+0000 7f57c6d9d640 1 --2- 172.21.15.72:0/2036069416 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f57c81667c0 0x7f57c8154f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.788+0000 7f57c659c640 1 --2- 172.21.15.72:0/2036069416 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f57c806c350 0x7f57c81557b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.788+0000 7f57c5d9b640 1 -- 172.21.15.72:0/2036069416 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f57c806c350 msgr2=0x7f57c81557b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.788+0000 7f57c5d9b640 1 --2- 172.21.15.72:0/2036069416 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f57c806c350 0x7f57c81557b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.788+0000 7f57c5d9b640 1 -- 172.21.15.72:0/2036069416 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f57c81667c0 msgr2=0x7f57c8154f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.788+0000 7f57c5d9b640 1 --2- 172.21.15.72:0/2036069416 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f57c81667c0 0x7f57c8154f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.788+0000 7f57c5d9b640 1 -- 172.21.15.72:0/2036069416 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57c815fab0 con 0x7f57c806c730 2024-04-15T15:15:33.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.788+0000 7f57c6d9d640 1 --2- 172.21.15.72:0/2036069416 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f57c81667c0 0x7f57c8154f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:33.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.788+0000 7f57c5d9b640 1 --2- 172.21.15.72:0/2036069416 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57c806c730 0x7f57c815f280 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f57bc000c00 tx=0x7f57bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:33.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.788+0000 7f57b6ffd640 1 -- 172.21.15.72:0/2036069416 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57bc041020 con 0x7f57c806c730 2024-04-15T15:15:33.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.788+0000 7f57b6ffd640 1 -- 172.21.15.72:0/2036069416 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57bc03c870 con 0x7f57c806c730 2024-04-15T15:15:33.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.789+0000 7f57ccfa8640 1 -- 172.21.15.72:0/2036069416 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57c815fd90 con 0x7f57c806c730 2024-04-15T15:15:33.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.789+0000 7f57b6ffd640 1 -- 172.21.15.72:0/2036069416 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57bc037040 con 0x7f57c806c730 2024-04-15T15:15:33.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.789+0000 7f57ccfa8640 1 -- 172.21.15.72:0/2036069416 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57c8160350 con 0x7f57c806c730 2024-04-15T15:15:33.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.790+0000 7f57ccfa8640 1 -- 172.21.15.72:0/2036069416 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57c8075c50 con 0x7f57c806c730 2024-04-15T15:15:33.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.790+0000 7f57b6ffd640 1 -- 172.21.15.72:0/2036069416 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f57bc044020 con 0x7f57c806c730 2024-04-15T15:15:33.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.790+0000 7f57b6ffd640 1 --2- 172.21.15.72:0/2036069416 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f57a806c920 0x7f57a806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.791+0000 7f57b6ffd640 1 -- 172.21.15.72:0/2036069416 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f57bc0ad6b0 con 0x7f57c806c730 2024-04-15T15:15:33.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.791+0000 7f57c659c640 1 --2- 172.21.15.72:0/2036069416 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f57a806c920 0x7f57a806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.791+0000 7f57c659c640 1 --2- 172.21.15.72:0/2036069416 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f57a806c920 0x7f57a806ede0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f57b80013b0 tx=0x7f57b8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:33.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.795+0000 7f57b6ffd640 1 -- 172.21.15.72:0/2036069416 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f57bc04b070 con 0x7f57c806c730 2024-04-15T15:15:33.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.969+0000 7f57ccfa8640 1 -- 172.21.15.72:0/2036069416 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f57c80b4800 con 0x7f57c806c730 2024-04-15T15:15:33.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.970+0000 7f57b6ffd640 1 -- 172.21.15.72:0/2036069416 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f57bc07b660 con 0x7f57c806c730 2024-04-15T15:15:33.971 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:33.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.971+0000 7f57ccfa8640 1 -- 172.21.15.72:0/2036069416 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f57a806c920 msgr2=0x7f57a806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.971+0000 7f57ccfa8640 1 --2- 172.21.15.72:0/2036069416 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f57a806c920 0x7f57a806ede0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f57b80013b0 tx=0x7f57b8009040 comp rx=0 tx=0).stop 2024-04-15T15:15:33.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.971+0000 7f57ccfa8640 1 -- 172.21.15.72:0/2036069416 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57c806c730 msgr2=0x7f57c815f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.971+0000 7f57ccfa8640 1 --2- 172.21.15.72:0/2036069416 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57c806c730 0x7f57c815f280 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f57bc000c00 tx=0x7f57bc0046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.972+0000 7f57ccfa8640 1 -- 172.21.15.72:0/2036069416 shutdown_connections 2024-04-15T15:15:33.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.972+0000 7f57ccfa8640 1 --2- 172.21.15.72:0/2036069416 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f57a806c920 0x7f57a806ede0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.972+0000 7f57ccfa8640 1 --2- 172.21.15.72:0/2036069416 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f57c81667c0 0x7f57c8154f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.972+0000 7f57ccfa8640 1 --2- 172.21.15.72:0/2036069416 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f57c806c730 0x7f57c815f280 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.972+0000 7f57ccfa8640 1 --2- 172.21.15.72:0/2036069416 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f57c806c350 0x7f57c81557b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.972+0000 7f57ccfa8640 1 -- 172.21.15.72:0/2036069416 >> 172.21.15.72:0/2036069416 conn(0x7f57c80d48c0 msgr2=0x7f57c8159ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:33.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.972+0000 7f57ccfa8640 1 -- 172.21.15.72:0/2036069416 shutdown_connections 2024-04-15T15:15:33.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:33.972+0000 7f57ccfa8640 1 -- 172.21.15.72:0/2036069416 wait complete. 2024-04-15T15:15:33.996 INFO:tasks.cephfs_test_runner:Starting test: test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce) That a file with ceph.quiesce.block is not quiesced. 2024-04-15T15:15:33.996 INFO:tasks.cephfs_test_runner:test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce) 2024-04-15T15:15:33.997 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:34.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.099+0000 7f0619474640 1 Processor -- start 2024-04-15T15:15:34.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.099+0000 7f0619474640 1 -- start start 2024-04-15T15:15:34.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.099+0000 7f0619474640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f06140d12b0 0x7f06140d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.099+0000 7f0619474640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0614162220 con 0x7f06140d2ea0 2024-04-15T15:15:34.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.099+0000 7f0619474640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06140d1820 con 0x7f0614161e40 2024-04-15T15:15:34.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.099+0000 7f0619474640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f06140b3f00 con 0x7f06140d12b0 2024-04-15T15:15:34.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.100+0000 7f06137fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f06140d2ea0 0x7f06140d45b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50000/0 (socket says 172.21.15.72:50000) 2024-04-15T15:15:34.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.100+0000 7f06137fe640 1 -- 172.21.15.72:0/2210270953 learned_addr learned my addr 172.21.15.72:0/2210270953 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:34.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.100+0000 7f06127fc640 1 --2- 172.21.15.72:0/2210270953 >> v2:172.21.15.72:3300/0 conn(0x7f06140d12b0 0x7f06140d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.100+0000 7f06127fc640 1 -- 172.21.15.72:0/2210270953 >> v1:172.21.15.175:6789/0 conn(0x7f0614161e40 legacy=0x7f06140b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.100+0000 7f06127fc640 1 -- 172.21.15.72:0/2210270953 >> v1:172.21.15.72:6789/0 conn(0x7f06140d2ea0 legacy=0x7f06140d45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.100+0000 7f0611ffb640 1 -- 172.21.15.72:0/2210270953 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 955480791 0 0) 0x7f0614162220 con 0x7f06140d2ea0 2024-04-15T15:15:34.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.100+0000 7f06127fc640 1 -- 172.21.15.72:0/2210270953 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06140d1b00 con 0x7f06140d12b0 2024-04-15T15:15:34.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.101+0000 7f06127fc640 1 --2- 172.21.15.72:0/2210270953 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06140d12b0 0x7f06140d2960 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f05fc0097d0 tx=0x7f05fc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84803352e75460cd server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:34.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.101+0000 7f0611ffb640 1 -- 172.21.15.72:0/2210270953 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05fc035020 con 0x7f06140d12b0 2024-04-15T15:15:34.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.101+0000 7f0611ffb640 1 -- 172.21.15.72:0/2210270953 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05fc031830 con 0x7f06140d12b0 2024-04-15T15:15:34.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.101+0000 7f0611ffb640 1 -- 172.21.15.72:0/2210270953 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05fc031c00 con 0x7f06140d12b0 2024-04-15T15:15:34.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.101+0000 7f0619474640 1 -- 172.21.15.72:0/2210270953 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06140d12b0 msgr2=0x7f06140d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.101+0000 7f0619474640 1 --2- 172.21.15.72:0/2210270953 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06140d12b0 0x7f06140d2960 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f05fc0097d0 tx=0x7f05fc02b8d0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.101+0000 7f0619474640 1 -- 172.21.15.72:0/2210270953 shutdown_connections 2024-04-15T15:15:34.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.101+0000 7f0619474640 1 --2- 172.21.15.72:0/2210270953 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06140d12b0 0x7f06140d2960 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.101+0000 7f0619474640 1 -- 172.21.15.72:0/2210270953 >> 172.21.15.72:0/2210270953 conn(0x7f061409d970 msgr2=0x7f06140b95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:34.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.101+0000 7f0619474640 1 -- 172.21.15.72:0/2210270953 shutdown_connections 2024-04-15T15:15:34.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.101+0000 7f0619474640 1 -- 172.21.15.72:0/2210270953 wait complete. 2024-04-15T15:15:34.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.103+0000 7f0619474640 1 Processor -- start 2024-04-15T15:15:34.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.103+0000 7f0619474640 1 -- start start 2024-04-15T15:15:34.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.103+0000 7f0619474640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06140d12b0 0x7f06140b69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.103+0000 7f0619474640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0614161e40 0x7f061414c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.103+0000 7f0619474640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f06140cd850 0x7f06140cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.103+0000 7f0619474640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f06140d2440 con 0x7f06140d12b0 2024-04-15T15:15:34.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.103+0000 7f0619474640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f06140d21f0 con 0x7f0614161e40 2024-04-15T15:15:34.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.103+0000 7f0619474640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f06140d4cc0 con 0x7f06140cd850 2024-04-15T15:15:34.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.103+0000 7f0612ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06140d12b0 0x7f06140b69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.103+0000 7f0612ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06140d12b0 0x7f06140b69f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54784/0 (socket says 172.21.15.72:54784) 2024-04-15T15:15:34.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.104+0000 7f0612ffd640 1 -- 172.21.15.72:0/2150519402 learned_addr learned my addr 172.21.15.72:0/2150519402 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:34.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.104+0000 7f06137fe640 1 --2- 172.21.15.72:0/2150519402 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f06140cd850 0x7f06140cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.104+0000 7f0612ffd640 1 -- 172.21.15.72:0/2150519402 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f06140cd850 msgr2=0x7f06140cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.104+0000 7f06127fc640 1 --2- 172.21.15.72:0/2150519402 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0614161e40 0x7f061414c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.104+0000 7f0612ffd640 1 --2- 172.21.15.72:0/2150519402 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f06140cd850 0x7f06140cdd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.104+0000 7f0612ffd640 1 -- 172.21.15.72:0/2150519402 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0614161e40 msgr2=0x7f061414c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.104+0000 7f0612ffd640 1 --2- 172.21.15.72:0/2150519402 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0614161e40 0x7f061414c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.104+0000 7f0612ffd640 1 -- 172.21.15.72:0/2150519402 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06140b5970 con 0x7f06140d12b0 2024-04-15T15:15:34.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.104+0000 7f0612ffd640 1 --2- 172.21.15.72:0/2150519402 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06140d12b0 0x7f06140b69f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f060c00c760 tx=0x7f060c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:34.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.104+0000 7f060b7fe640 1 -- 172.21.15.72:0/2150519402 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060c003b70 con 0x7f06140d12b0 2024-04-15T15:15:34.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.104+0000 7f060b7fe640 1 -- 172.21.15.72:0/2150519402 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f060c003d30 con 0x7f06140d12b0 2024-04-15T15:15:34.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.104+0000 7f0619474640 1 -- 172.21.15.72:0/2150519402 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06140b5c50 con 0x7f06140d12b0 2024-04-15T15:15:34.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.104+0000 7f060b7fe640 1 -- 172.21.15.72:0/2150519402 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060c00aa90 con 0x7f06140d12b0 2024-04-15T15:15:34.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.104+0000 7f0619474640 1 -- 172.21.15.72:0/2150519402 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06140b6210 con 0x7f06140d12b0 2024-04-15T15:15:34.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.105+0000 7f0619474640 1 -- 172.21.15.72:0/2150519402 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06140d35f0 con 0x7f06140d12b0 2024-04-15T15:15:34.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.105+0000 7f060b7fe640 1 -- 172.21.15.72:0/2150519402 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f060c00ac30 con 0x7f06140d12b0 2024-04-15T15:15:34.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.106+0000 7f060b7fe640 1 --2- 172.21.15.72:0/2150519402 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f05f406c8f0 0x7f05f406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.106+0000 7f060b7fe640 1 -- 172.21.15.72:0/2150519402 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f060c08beb0 con 0x7f06140d12b0 2024-04-15T15:15:34.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.106+0000 7f06127fc640 1 --2- 172.21.15.72:0/2150519402 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f05f406c8f0 0x7f05f406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.106+0000 7f06127fc640 1 --2- 172.21.15.72:0/2150519402 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f05f406c8f0 0x7f05f406edb0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f05fc036000 tx=0x7f05fc008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:34.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.110+0000 7f060b7fe640 1 -- 172.21.15.72:0/2150519402 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f06140d35f0 con 0x7f06140d12b0 2024-04-15T15:15:34.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.286+0000 7f0619474640 1 -- 172.21.15.72:0/2150519402 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f06140689e0 con 0x7f06140d12b0 2024-04-15T15:15:34.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.287+0000 7f060b7fe640 1 -- 172.21.15.72:0/2150519402 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18903 (secure 0 0 0) 0x7f060c059ee0 con 0x7f06140d12b0 2024-04-15T15:15:34.289 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:34.289 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":26,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:15:32.421728+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:14:49.438366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:14:50.368379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:14:54.081157+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":1}]},{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:34.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.289+0000 7f0619474640 1 -- 172.21.15.72:0/2150519402 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f05f406c8f0 msgr2=0x7f05f406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.289+0000 7f0619474640 1 --2- 172.21.15.72:0/2150519402 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f05f406c8f0 0x7f05f406edb0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f05fc036000 tx=0x7f05fc008030 comp rx=0 tx=0).stop 2024-04-15T15:15:34.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.289+0000 7f0619474640 1 -- 172.21.15.72:0/2150519402 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06140d12b0 msgr2=0x7f06140b69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.289+0000 7f0619474640 1 --2- 172.21.15.72:0/2150519402 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06140d12b0 0x7f06140b69f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f060c00c760 tx=0x7f060c00cc20 comp rx=0 tx=0).stop 2024-04-15T15:15:34.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.289+0000 7f0619474640 1 -- 172.21.15.72:0/2150519402 shutdown_connections 2024-04-15T15:15:34.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.289+0000 7f0619474640 1 --2- 172.21.15.72:0/2150519402 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f05f406c8f0 0x7f05f406edb0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.289+0000 7f0619474640 1 --2- 172.21.15.72:0/2150519402 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f06140cd850 0x7f06140cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.289+0000 7f0619474640 1 --2- 172.21.15.72:0/2150519402 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0614161e40 0x7f061414c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.289+0000 7f0619474640 1 --2- 172.21.15.72:0/2150519402 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f06140d12b0 0x7f06140b69f0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.289+0000 7f0619474640 1 -- 172.21.15.72:0/2150519402 >> 172.21.15.72:0/2150519402 conn(0x7f061409d970 msgr2=0x7f06140b7e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:34.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.289+0000 7f0619474640 1 -- 172.21.15.72:0/2150519402 shutdown_connections 2024-04-15T15:15:34.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.289+0000 7f0619474640 1 -- 172.21.15.72:0/2150519402 wait complete. 2024-04-15T15:15:34.302 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:39.359102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:14:49.438366+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:14:50.368379+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:14:54.081157+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:15:34.303 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:34.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.407+0000 7fd538f3d640 1 Processor -- start 2024-04-15T15:15:34.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.407+0000 7fd538f3d640 1 -- start start 2024-04-15T15:15:34.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.408+0000 7fd538f3d640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd5340cf5f0 0x7fd53415e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.408+0000 7fd538f3d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5340cfa80 con 0x7fd5340cf160 2024-04-15T15:15:34.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.408+0000 7fd538f3d640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5340b61e0 con 0x7fd53415d830 2024-04-15T15:15:34.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.408+0000 7fd538f3d640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fd5340b64c0 con 0x7fd5340cf5f0 2024-04-15T15:15:34.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.409+0000 7fd531d9b640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fd5340cf160 0x7fd53415e2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50016/0 (socket says 172.21.15.72:50016) 2024-04-15T15:15:34.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.409+0000 7fd531d9b640 1 -- 172.21.15.72:0/403397788 learned_addr learned my addr 172.21.15.72:0/403397788 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:34.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.409+0000 7fd53159a640 1 -- 172.21.15.72:0/403397788 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 69661834 0 0) 0x7fd5340cfa80 con 0x7fd5340cf160 2024-04-15T15:15:34.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.409+0000 7fd53159a640 1 -- 172.21.15.72:0/403397788 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd51c003520 con 0x7fd5340cf160 2024-04-15T15:15:34.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.409+0000 7fd532d9d640 1 --2- 172.21.15.72:0/403397788 >> v2:172.21.15.175:3300/0 conn(0x7fd5340cf5f0 0x7fd53415e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.409+0000 7fd53159a640 1 -- 172.21.15.72:0/403397788 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2410035963 0 0) 0x7fd51c003520 con 0x7fd5340cf160 2024-04-15T15:15:34.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.409+0000 7fd53159a640 1 -- 172.21.15.72:0/403397788 >> v2:172.21.15.175:3300/0 conn(0x7fd5340cf5f0 msgr2=0x7fd53415e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:34.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.409+0000 7fd53159a640 1 --2- 172.21.15.72:0/403397788 >> v2:172.21.15.175:3300/0 conn(0x7fd5340cf5f0 0x7fd53415e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.409+0000 7fd53159a640 1 -- 172.21.15.72:0/403397788 >> v1:172.21.15.175:6789/0 conn(0x7fd53415d830 legacy=0x7fd5340b57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:34.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.409+0000 7fd53159a640 1 -- 172.21.15.72:0/403397788 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5340b67a0 con 0x7fd5340cf160 2024-04-15T15:15:34.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.409+0000 7fd53159a640 1 -- 172.21.15.72:0/403397788 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7fd528001670 con 0x7fd5340cf160 2024-04-15T15:15:34.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.409+0000 7fd53159a640 1 -- 172.21.15.72:0/403397788 >> v1:172.21.15.72:6789/0 conn(0x7fd5340cf160 legacy=0x7fd53415e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.409+0000 7fd53159a640 1 --2- 172.21.15.72:0/403397788 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd51c0039b0 0x7fd51c003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.409+0000 7fd53159a640 1 --2- 172.21.15.72:0/403397788 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd51c004800 0x7fd51c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.409+0000 7fd53159a640 1 --2- 172.21.15.72:0/403397788 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd51c0052e0 0x7fd51c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.409+0000 7fd53159a640 1 -- 172.21.15.72:0/403397788 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd5340b64c0 con 0x7fd51c0039b0 2024-04-15T15:15:34.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.409+0000 7fd53159a640 1 -- 172.21.15.72:0/403397788 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fd51c00c190 con 0x7fd51c0052e0 2024-04-15T15:15:34.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.409+0000 7fd53159a640 1 -- 172.21.15.72:0/403397788 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fd51c00c390 con 0x7fd51c004800 2024-04-15T15:15:34.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.409+0000 7fd53259c640 1 --2- 172.21.15.72:0/403397788 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd51c0039b0 0x7fd51c003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.409+0000 7fd531d9b640 1 --2- 172.21.15.72:0/403397788 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd51c004800 0x7fd51c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.409+0000 7fd532d9d640 1 --2- 172.21.15.72:0/403397788 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd51c0052e0 0x7fd51c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.409+0000 7fd53259c640 1 -- 172.21.15.72:0/403397788 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd51c004800 msgr2=0x7fd51c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:34.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.409+0000 7fd53259c640 1 --2- 172.21.15.72:0/403397788 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd51c004800 0x7fd51c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.410+0000 7fd53259c640 1 -- 172.21.15.72:0/403397788 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd51c0052e0 msgr2=0x7fd51c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:34.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.410+0000 7fd53259c640 1 --2- 172.21.15.72:0/403397788 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd51c0052e0 0x7fd51c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.410+0000 7fd53259c640 1 -- 172.21.15.72:0/403397788 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd51c003520 con 0x7fd51c0039b0 2024-04-15T15:15:34.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.410+0000 7fd532d9d640 1 -- 172.21.15.72:0/403397788 reap_dead start 2024-04-15T15:15:34.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.410+0000 7fd531d9b640 1 --2- 172.21.15.72:0/403397788 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd51c004800 0x7fd51c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:34.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.410+0000 7fd53259c640 1 --2- 172.21.15.72:0/403397788 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd51c0039b0 0x7fd51c003db0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd524003d80 tx=0x7fd51c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96bc976313fdd1c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:34.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.410+0000 7fd53159a640 1 -- 172.21.15.72:0/403397788 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd524037070 con 0x7fd51c0039b0 2024-04-15T15:15:34.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.410+0000 7fd53159a640 1 -- 172.21.15.72:0/403397788 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd524032600 con 0x7fd51c0039b0 2024-04-15T15:15:34.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.410+0000 7fd53159a640 1 -- 172.21.15.72:0/403397788 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd524003980 con 0x7fd51c0039b0 2024-04-15T15:15:34.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.411+0000 7fd538f3d640 1 -- 172.21.15.72:0/403397788 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd51c0039b0 msgr2=0x7fd51c003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.411+0000 7fd538f3d640 1 --2- 172.21.15.72:0/403397788 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd51c0039b0 0x7fd51c003db0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd524003d80 tx=0x7fd51c005830 comp rx=0 tx=0).stop 2024-04-15T15:15:34.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.411+0000 7fd538f3d640 1 -- 172.21.15.72:0/403397788 shutdown_connections 2024-04-15T15:15:34.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.411+0000 7fd538f3d640 1 --2- 172.21.15.72:0/403397788 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd51c0039b0 0x7fd51c003db0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.411+0000 7fd538f3d640 1 -- 172.21.15.72:0/403397788 >> 172.21.15.72:0/403397788 conn(0x7fd5340d24d0 msgr2=0x7fd5340d28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:34.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.411+0000 7fd538f3d640 1 -- 172.21.15.72:0/403397788 shutdown_connections 2024-04-15T15:15:34.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.411+0000 7fd538f3d640 1 -- 172.21.15.72:0/403397788 wait complete. 2024-04-15T15:15:34.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.412+0000 7fd538f3d640 1 Processor -- start 2024-04-15T15:15:34.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.412+0000 7fd538f3d640 1 -- start start 2024-04-15T15:15:34.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.413+0000 7fd538f3d640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd51c0039b0 0x7fd53413d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.413+0000 7fd538f3d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd5341560d0 0x7fd5341506f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.413+0000 7fd538f3d640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd534150c30 0x7fd53414db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.413+0000 7fd538f3d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd53415cd50 con 0x7fd5341560d0 2024-04-15T15:15:34.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.413+0000 7fd538f3d640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fd53415dd80 con 0x7fd51c0039b0 2024-04-15T15:15:34.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.413+0000 7fd538f3d640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fd53415ced0 con 0x7fd534150c30 2024-04-15T15:15:34.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.413+0000 7fd531d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd5341560d0 0x7fd5341506f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.413+0000 7fd531d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd5341560d0 0x7fd5341506f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54808/0 (socket says 172.21.15.72:54808) 2024-04-15T15:15:34.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.413+0000 7fd531d9b640 1 -- 172.21.15.72:0/3809755845 learned_addr learned my addr 172.21.15.72:0/3809755845 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:34.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.413+0000 7fd53259c640 1 --2- 172.21.15.72:0/3809755845 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd51c0039b0 0x7fd53413d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.413+0000 7fd531d9b640 1 -- 172.21.15.72:0/3809755845 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd534150c30 msgr2=0x7fd53414db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.413+0000 7fd532d9d640 1 --2- 172.21.15.72:0/3809755845 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd534150c30 0x7fd53414db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.413+0000 7fd531d9b640 1 --2- 172.21.15.72:0/3809755845 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd534150c30 0x7fd53414db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.413+0000 7fd531d9b640 1 -- 172.21.15.72:0/3809755845 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd51c0039b0 msgr2=0x7fd53413d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.413+0000 7fd531d9b640 1 --2- 172.21.15.72:0/3809755845 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd51c0039b0 0x7fd53413d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.413+0000 7fd531d9b640 1 -- 172.21.15.72:0/3809755845 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd53413dce0 con 0x7fd5341560d0 2024-04-15T15:15:34.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.413+0000 7fd53259c640 1 --2- 172.21.15.72:0/3809755845 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd51c0039b0 0x7fd53413d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:34.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.414+0000 7fd531d9b640 1 --2- 172.21.15.72:0/3809755845 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd5341560d0 0x7fd5341506f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd52800e7e0 tx=0x7fd52800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:34.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.414+0000 7fd522ffd640 1 -- 172.21.15.72:0/3809755845 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd52800c8d0 con 0x7fd5341560d0 2024-04-15T15:15:34.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.414+0000 7fd522ffd640 1 -- 172.21.15.72:0/3809755845 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd528018070 con 0x7fd5341560d0 2024-04-15T15:15:34.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.414+0000 7fd538f3d640 1 -- 172.21.15.72:0/3809755845 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd534155030 con 0x7fd5341560d0 2024-04-15T15:15:34.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.414+0000 7fd522ffd640 1 -- 172.21.15.72:0/3809755845 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd52800ccb0 con 0x7fd5341560d0 2024-04-15T15:15:34.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.414+0000 7fd538f3d640 1 -- 172.21.15.72:0/3809755845 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5341555f0 con 0x7fd5341560d0 2024-04-15T15:15:34.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.415+0000 7fd538f3d640 1 -- 172.21.15.72:0/3809755845 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd534075c50 con 0x7fd5341560d0 2024-04-15T15:15:34.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.415+0000 7fd522ffd640 1 -- 172.21.15.72:0/3809755845 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd52801d460 con 0x7fd5341560d0 2024-04-15T15:15:34.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.416+0000 7fd522ffd640 1 --2- 172.21.15.72:0/3809755845 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd51406c920 0x7fd51406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.416+0000 7fd522ffd640 1 -- 172.21.15.72:0/3809755845 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fd52808bef0 con 0x7fd5341560d0 2024-04-15T15:15:34.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.416+0000 7fd53259c640 1 --2- 172.21.15.72:0/3809755845 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd51406c920 0x7fd51406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.416+0000 7fd53259c640 1 --2- 172.21.15.72:0/3809755845 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd51406c920 0x7fd51406ede0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd524034000 tx=0x7fd524009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:34.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.420+0000 7fd522ffd640 1 -- 172.21.15.72:0/3809755845 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd528059e90 con 0x7fd5341560d0 2024-04-15T15:15:34.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.594+0000 7fd538f3d640 1 -- 172.21.15.72:0/3809755845 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd5340b6e10 con 0x7fd5341560d0 2024-04-15T15:15:34.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.594+0000 7fd522ffd640 1 -- 172.21.15.72:0/3809755845 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7fd528059790 con 0x7fd5341560d0 2024-04-15T15:15:34.596 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-15T15:15:34.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.596+0000 7fd538f3d640 1 -- 172.21.15.72:0/3809755845 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd51406c920 msgr2=0x7fd51406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.596+0000 7fd538f3d640 1 --2- 172.21.15.72:0/3809755845 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd51406c920 0x7fd51406ede0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd524034000 tx=0x7fd524009030 comp rx=0 tx=0).stop 2024-04-15T15:15:34.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.596+0000 7fd538f3d640 1 -- 172.21.15.72:0/3809755845 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd5341560d0 msgr2=0x7fd5341506f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.596+0000 7fd538f3d640 1 --2- 172.21.15.72:0/3809755845 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd5341560d0 0x7fd5341506f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd52800e7e0 tx=0x7fd52800eca0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.596+0000 7fd538f3d640 1 -- 172.21.15.72:0/3809755845 shutdown_connections 2024-04-15T15:15:34.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.596+0000 7fd538f3d640 1 --2- 172.21.15.72:0/3809755845 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd51406c920 0x7fd51406ede0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.596+0000 7fd538f3d640 1 --2- 172.21.15.72:0/3809755845 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd534150c30 0x7fd53414db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.596+0000 7fd538f3d640 1 --2- 172.21.15.72:0/3809755845 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd5341560d0 0x7fd5341506f0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.596+0000 7fd538f3d640 1 --2- 172.21.15.72:0/3809755845 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd51c0039b0 0x7fd53413d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.596+0000 7fd538f3d640 1 -- 172.21.15.72:0/3809755845 >> 172.21.15.72:0/3809755845 conn(0x7fd5340d24d0 msgr2=0x7fd5341530b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:34.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.596+0000 7fd538f3d640 1 -- 172.21.15.72:0/3809755845 shutdown_connections 2024-04-15T15:15:34.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.596+0000 7fd538f3d640 1 -- 172.21.15.72:0/3809755845 wait complete. 2024-04-15T15:15:34.609 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:15:34.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.713+0000 7fc25831c640 1 Processor -- start 2024-04-15T15:15:34.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.713+0000 7fc25831c640 1 -- start start 2024-04-15T15:15:34.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.714+0000 7fc25831c640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc25015c630 0x7fc25006cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.714+0000 7fc25831c640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc2500ccb90 0x7fc2500ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.714+0000 7fc25831c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc25006bd50 con 0x7fc25006e620 2024-04-15T15:15:34.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.714+0000 7fc25831c640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fc25006d2f0 con 0x7fc25015c630 2024-04-15T15:15:34.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.714+0000 7fc25831c640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fc2500cd530 con 0x7fc2500ccb90 2024-04-15T15:15:34.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.714+0000 7fc2568b9640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fc25006e620 0x7fc25006eab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50032/0 (socket says 172.21.15.72:50032) 2024-04-15T15:15:34.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.714+0000 7fc2568b9640 1 -- 172.21.15.72:0/887136243 learned_addr learned my addr 172.21.15.72:0/887136243 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:34.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.714+0000 7fc2558b7640 1 --2- 172.21.15.72:0/887136243 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc2500ccb90 0x7fc2500ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.714+0000 7fc2560b8640 1 --2- 172.21.15.72:0/887136243 >> v2:172.21.15.175:3300/0 conn(0x7fc25015c630 0x7fc25006cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.714+0000 7fc2550b6640 1 -- 172.21.15.72:0/887136243 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3501357501 0 0) 0x7fc25006bd50 con 0x7fc25006e620 2024-04-15T15:15:34.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.714+0000 7fc2550b6640 1 -- 172.21.15.72:0/887136243 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc23c003520 con 0x7fc25006e620 2024-04-15T15:15:34.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.715+0000 7fc2550b6640 1 -- 172.21.15.72:0/887136243 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 755213496 0 0) 0x7fc23c003520 con 0x7fc25006e620 2024-04-15T15:15:34.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.715+0000 7fc2550b6640 1 -- 172.21.15.72:0/887136243 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc2500ccb90 msgr2=0x7fc2500ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:34.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.715+0000 7fc2550b6640 1 --2- 172.21.15.72:0/887136243 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc2500ccb90 0x7fc2500ccff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.715+0000 7fc2550b6640 1 -- 172.21.15.72:0/887136243 >> v2:172.21.15.175:3300/0 conn(0x7fc25015c630 msgr2=0x7fc25006cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:34.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.715+0000 7fc2550b6640 1 --2- 172.21.15.72:0/887136243 >> v2:172.21.15.175:3300/0 conn(0x7fc25015c630 0x7fc25006cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.715+0000 7fc2550b6640 1 -- 172.21.15.72:0/887136243 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc25006bff0 con 0x7fc25006e620 2024-04-15T15:15:34.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.715+0000 7fc2550b6640 1 -- 172.21.15.72:0/887136243 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7fc2480019f0 con 0x7fc25006e620 2024-04-15T15:15:34.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.715+0000 7fc2550b6640 1 -- 172.21.15.72:0/887136243 >> v1:172.21.15.72:6789/0 conn(0x7fc25006e620 legacy=0x7fc25006eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.715+0000 7fc2550b6640 1 --2- 172.21.15.72:0/887136243 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc23c0039b0 0x7fc23c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.715+0000 7fc2550b6640 1 --2- 172.21.15.72:0/887136243 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc23c004800 0x7fc23c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.715+0000 7fc2550b6640 1 --2- 172.21.15.72:0/887136243 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc2500ccb90 0x7fc23c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.715+0000 7fc2550b6640 1 -- 172.21.15.72:0/887136243 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fc25006d2f0 con 0x7fc23c004800 2024-04-15T15:15:34.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.715+0000 7fc2550b6640 1 -- 172.21.15.72:0/887136243 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fc23c00b9c0 con 0x7fc23c0039b0 2024-04-15T15:15:34.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.715+0000 7fc2550b6640 1 -- 172.21.15.72:0/887136243 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fc23c00bb40 con 0x7fc2500ccb90 2024-04-15T15:15:34.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.715+0000 7fc2558b7640 1 --2- 172.21.15.72:0/887136243 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc23c004800 0x7fc23c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.715+0000 7fc2560b8640 1 --2- 172.21.15.72:0/887136243 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc23c0039b0 0x7fc23c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.715+0000 7fc2568b9640 1 --2- 172.21.15.72:0/887136243 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc2500ccb90 0x7fc23c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.715+0000 7fc2558b7640 1 -- 172.21.15.72:0/887136243 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc2500ccb90 msgr2=0x7fc23c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:34.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.715+0000 7fc2558b7640 1 --2- 172.21.15.72:0/887136243 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc2500ccb90 0x7fc23c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.715+0000 7fc2558b7640 1 -- 172.21.15.72:0/887136243 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc23c0039b0 msgr2=0x7fc23c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:34.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.715+0000 7fc2558b7640 1 --2- 172.21.15.72:0/887136243 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc23c0039b0 0x7fc23c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.715+0000 7fc2558b7640 1 -- 172.21.15.72:0/887136243 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc23c003520 con 0x7fc23c004800 2024-04-15T15:15:34.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.715+0000 7fc2568b9640 1 --2- 172.21.15.72:0/887136243 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc2500ccb90 0x7fc23c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:34.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.715+0000 7fc2560b8640 1 --2- 172.21.15.72:0/887136243 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc23c0039b0 0x7fc23c003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:34.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.716+0000 7fc2558b7640 1 --2- 172.21.15.72:0/887136243 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc23c004800 0x7fc23c004c80 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc240009ac0 tx=0x7fc23c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c5f9b43e018757a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:34.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.716+0000 7fc2550b6640 1 -- 172.21.15.72:0/887136243 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc240039070 con 0x7fc23c004800 2024-04-15T15:15:34.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.716+0000 7fc2550b6640 1 -- 172.21.15.72:0/887136243 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc24002c6b0 con 0x7fc23c004800 2024-04-15T15:15:34.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.716+0000 7fc2550b6640 1 -- 172.21.15.72:0/887136243 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc240009690 con 0x7fc23c004800 2024-04-15T15:15:34.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.717+0000 7fc25831c640 1 -- 172.21.15.72:0/887136243 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc23c004800 msgr2=0x7fc23c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.717+0000 7fc25831c640 1 --2- 172.21.15.72:0/887136243 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc23c004800 0x7fc23c004c80 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc240009ac0 tx=0x7fc23c005830 comp rx=0 tx=0).stop 2024-04-15T15:15:34.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.717+0000 7fc2568b9640 1 -- 172.21.15.72:0/887136243 reap_dead start 2024-04-15T15:15:34.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.717+0000 7fc25831c640 1 -- 172.21.15.72:0/887136243 shutdown_connections 2024-04-15T15:15:34.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.717+0000 7fc25831c640 1 -- 172.21.15.72:0/887136243 >> 172.21.15.72:0/887136243 conn(0x7fc2500b31d0 msgr2=0x7fc2500b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:34.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.717+0000 7fc25831c640 1 -- 172.21.15.72:0/887136243 shutdown_connections 2024-04-15T15:15:34.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.717+0000 7fc25831c640 1 -- 172.21.15.72:0/887136243 wait complete. 2024-04-15T15:15:34.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.718+0000 7fc25831c640 1 Processor -- start 2024-04-15T15:15:34.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.718+0000 7fc25831c640 1 -- start start 2024-04-15T15:15:34.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.719+0000 7fc25831c640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc23c004800 0x7fc250154710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.719+0000 7fc25831c640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc2501e6ad0 0x7fc2501e6f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.719+0000 7fc25831c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc2500720c0 0x7fc250072540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.719+0000 7fc25831c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fc25006c5b0 con 0x7fc2500720c0 2024-04-15T15:15:34.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.719+0000 7fc25831c640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fc25006c920 con 0x7fc23c004800 2024-04-15T15:15:34.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.719+0000 7fc25831c640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fc25015ca10 con 0x7fc2501e6ad0 2024-04-15T15:15:34.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.719+0000 7fc2568b9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc2500720c0 0x7fc250072540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.719+0000 7fc2568b9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc2500720c0 0x7fc250072540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54834/0 (socket says 172.21.15.72:54834) 2024-04-15T15:15:34.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.719+0000 7fc2568b9640 1 -- 172.21.15.72:0/136684059 learned_addr learned my addr 172.21.15.72:0/136684059 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:34.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.719+0000 7fc2560b8640 1 --2- 172.21.15.72:0/136684059 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc23c004800 0x7fc250154710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.719+0000 7fc2558b7640 1 --2- 172.21.15.72:0/136684059 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc2501e6ad0 0x7fc2501e6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.719+0000 7fc2568b9640 1 -- 172.21.15.72:0/136684059 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc2501e6ad0 msgr2=0x7fc2501e6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.719+0000 7fc2568b9640 1 --2- 172.21.15.72:0/136684059 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc2501e6ad0 0x7fc2501e6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.719+0000 7fc2568b9640 1 -- 172.21.15.72:0/136684059 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc23c004800 msgr2=0x7fc250154710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.719+0000 7fc2568b9640 1 --2- 172.21.15.72:0/136684059 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc23c004800 0x7fc250154710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.719+0000 7fc2568b9640 1 -- 172.21.15.72:0/136684059 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc250072a80 con 0x7fc2500720c0 2024-04-15T15:15:34.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.719+0000 7fc2560b8640 1 --2- 172.21.15.72:0/136684059 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc23c004800 0x7fc250154710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:34.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.719+0000 7fc2558b7640 1 --2- 172.21.15.72:0/136684059 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc2501e6ad0 0x7fc2501e6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:34.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.719+0000 7fc2568b9640 1 --2- 172.21.15.72:0/136684059 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc2500720c0 0x7fc250072540 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc25006bc00 tx=0x7fc24800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:34.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.719+0000 7fc246ffd640 1 -- 172.21.15.72:0/136684059 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc248007a80 con 0x7fc2500720c0 2024-04-15T15:15:34.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.719+0000 7fc246ffd640 1 -- 172.21.15.72:0/136684059 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc24800eda0 con 0x7fc2500720c0 2024-04-15T15:15:34.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.719+0000 7fc25831c640 1 -- 172.21.15.72:0/136684059 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc250072d10 con 0x7fc2500720c0 2024-04-15T15:15:34.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.719+0000 7fc246ffd640 1 -- 172.21.15.72:0/136684059 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc248014670 con 0x7fc2500720c0 2024-04-15T15:15:34.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.719+0000 7fc25831c640 1 -- 172.21.15.72:0/136684059 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2500732d0 con 0x7fc2500720c0 2024-04-15T15:15:34.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.720+0000 7fc25831c640 1 -- 172.21.15.72:0/136684059 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc250075c50 con 0x7fc2500720c0 2024-04-15T15:15:34.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.721+0000 7fc246ffd640 1 -- 172.21.15.72:0/136684059 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc248014810 con 0x7fc2500720c0 2024-04-15T15:15:34.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.721+0000 7fc246ffd640 1 --2- 172.21.15.72:0/136684059 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc23406c8f0 0x7fc23406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.721+0000 7fc246ffd640 1 -- 172.21.15.72:0/136684059 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fc24808be00 con 0x7fc2500720c0 2024-04-15T15:15:34.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.721+0000 7fc2560b8640 1 --2- 172.21.15.72:0/136684059 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc23406c8f0 0x7fc23406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.722+0000 7fc2560b8640 1 --2- 172.21.15.72:0/136684059 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc23406c8f0 0x7fc23406edb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc24c009bd0 tx=0x7fc24c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:34.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.726+0000 7fc246ffd640 1 -- 172.21.15.72:0/136684059 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc248059ec0 con 0x7fc2500720c0 2024-04-15T15:15:34.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.900+0000 7fc25831c640 1 -- 172.21.15.72:0/136684059 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc2500aec70 con 0x7fc2500720c0 2024-04-15T15:15:34.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.901+0000 7fc246ffd640 1 -- 172.21.15.72:0/136684059 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7fc2480597c0 con 0x7fc2500720c0 2024-04-15T15:15:34.902 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:34.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.902+0000 7fc25831c640 1 -- 172.21.15.72:0/136684059 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc23406c8f0 msgr2=0x7fc23406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.902+0000 7fc25831c640 1 --2- 172.21.15.72:0/136684059 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc23406c8f0 0x7fc23406edb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc24c009bd0 tx=0x7fc24c008040 comp rx=0 tx=0).stop 2024-04-15T15:15:34.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.902+0000 7fc25831c640 1 -- 172.21.15.72:0/136684059 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc2500720c0 msgr2=0x7fc250072540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.902+0000 7fc25831c640 1 --2- 172.21.15.72:0/136684059 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc2500720c0 0x7fc250072540 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc25006bc00 tx=0x7fc24800e530 comp rx=0 tx=0).stop 2024-04-15T15:15:34.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.903+0000 7fc25831c640 1 -- 172.21.15.72:0/136684059 shutdown_connections 2024-04-15T15:15:34.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.903+0000 7fc25831c640 1 --2- 172.21.15.72:0/136684059 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc23406c8f0 0x7fc23406edb0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.903+0000 7fc25831c640 1 --2- 172.21.15.72:0/136684059 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc2500720c0 0x7fc250072540 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.903+0000 7fc25831c640 1 --2- 172.21.15.72:0/136684059 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc2501e6ad0 0x7fc2501e6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.903+0000 7fc25831c640 1 --2- 172.21.15.72:0/136684059 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc23c004800 0x7fc250154710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.903+0000 7fc25831c640 1 -- 172.21.15.72:0/136684059 >> 172.21.15.72:0/136684059 conn(0x7fc2500b31d0 msgr2=0x7fc2500cf820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:34.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.903+0000 7fc25831c640 1 -- 172.21.15.72:0/136684059 shutdown_connections 2024-04-15T15:15:34.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:34.903+0000 7fc25831c640 1 -- 172.21.15.72:0/136684059 wait complete. 2024-04-15T15:15:34.915 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:15:35.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.020+0000 7f2e5ee31640 1 Processor -- start 2024-04-15T15:15:35.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.020+0000 7f2e5ee31640 1 -- start start 2024-04-15T15:15:35.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.021+0000 7f2e5ee31640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2e5815c250 0x7f2e5806d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.021+0000 7f2e5ee31640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2e580d0010 0x7f2e580d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.021+0000 7f2e5ee31640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e5806df30 con 0x7f2e5815f9f0 2024-04-15T15:15:35.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.021+0000 7f2e5ee31640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f2e580cd1a0 con 0x7f2e5815c250 2024-04-15T15:15:35.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.021+0000 7f2e5ee31640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f2e580d09b0 con 0x7f2e580d0010 2024-04-15T15:15:35.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.021+0000 7f2e5cbcd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2e5815c250 0x7f2e5806d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.021+0000 7f2e57fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f2e5815f9f0 0x7f2e580cca90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50038/0 (socket says 172.21.15.72:50038) 2024-04-15T15:15:35.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.021+0000 7f2e57fff640 1 -- 172.21.15.72:0/796216048 learned_addr learned my addr 172.21.15.72:0/796216048 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:35.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.022+0000 7f2e5d3ce640 1 --2- 172.21.15.72:0/796216048 >> v2:172.21.15.175:3300/0 conn(0x7f2e580d0010 0x7f2e580d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.022+0000 7f2e5cbcd640 1 -- 172.21.15.72:0/796216048 >> v2:172.21.15.175:3300/0 conn(0x7f2e580d0010 msgr2=0x7f2e580d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:35.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.022+0000 7f2e5cbcd640 1 --2- 172.21.15.72:0/796216048 >> v2:172.21.15.175:3300/0 conn(0x7f2e580d0010 0x7f2e580d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.022+0000 7f2e5cbcd640 1 -- 172.21.15.72:0/796216048 >> v1:172.21.15.72:6789/0 conn(0x7f2e5815f9f0 legacy=0x7f2e580cca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.022+0000 7f2e5cbcd640 1 -- 172.21.15.72:0/796216048 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e5806e210 con 0x7f2e5815c250 2024-04-15T15:15:35.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.022+0000 7f2e577fe640 1 -- 172.21.15.72:0/796216048 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 244811901 0 0) 0x7f2e5806df30 con 0x7f2e5815f9f0 2024-04-15T15:15:35.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.022+0000 7f2e5cbcd640 1 --2- 172.21.15.72:0/796216048 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e5815c250 0x7f2e5806d6e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2e5815fe80 tx=0x7f2e4802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2076d5dc31ca0a31 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:35.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.022+0000 7f2e577fe640 1 -- 172.21.15.72:0/796216048 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e48039070 con 0x7f2e5815c250 2024-04-15T15:15:35.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.023+0000 7f2e577fe640 1 -- 172.21.15.72:0/796216048 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e4802cbf0 con 0x7f2e5815c250 2024-04-15T15:15:35.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.023+0000 7f2e577fe640 1 -- 172.21.15.72:0/796216048 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e48034440 con 0x7f2e5815c250 2024-04-15T15:15:35.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.023+0000 7f2e5ee31640 1 -- 172.21.15.72:0/796216048 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e5815c250 msgr2=0x7f2e5806d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.023+0000 7f2e5ee31640 1 --2- 172.21.15.72:0/796216048 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e5815c250 0x7f2e5806d6e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2e5815fe80 tx=0x7f2e4802be40 comp rx=0 tx=0).stop 2024-04-15T15:15:35.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.023+0000 7f2e5ee31640 1 -- 172.21.15.72:0/796216048 shutdown_connections 2024-04-15T15:15:35.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.023+0000 7f2e5ee31640 1 --2- 172.21.15.72:0/796216048 >> v2:172.21.15.175:3300/0 conn(0x7f2e580d0010 0x7f2e580d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.023+0000 7f2e5ee31640 1 --2- 172.21.15.72:0/796216048 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e5815c250 0x7f2e5806d6e0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.023+0000 7f2e5ee31640 1 -- 172.21.15.72:0/796216048 >> 172.21.15.72:0/796216048 conn(0x7f2e580ceec0 msgr2=0x7f2e580cf2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:35.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.023+0000 7f2e5ee31640 1 -- 172.21.15.72:0/796216048 shutdown_connections 2024-04-15T15:15:35.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.023+0000 7f2e5ee31640 1 -- 172.21.15.72:0/796216048 wait complete. 2024-04-15T15:15:35.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.024+0000 7f2e5ee31640 1 Processor -- start 2024-04-15T15:15:35.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.024+0000 7f2e5ee31640 1 -- start start 2024-04-15T15:15:35.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.025+0000 7f2e5ee31640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2e580d0010 0x7f2e580b6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.025+0000 7f2e5ee31640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e5815c250 0x7f2e58157000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.025+0000 7f2e5ee31640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2e5815f9f0 0x7f2e5815edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.025+0000 7f2e5ee31640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2e580b38f0 con 0x7f2e5815c250 2024-04-15T15:15:35.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.025+0000 7f2e5ee31640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f2e580b3610 con 0x7f2e5815f9f0 2024-04-15T15:15:35.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.025+0000 7f2e5ee31640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f2e5806dc20 con 0x7f2e580d0010 2024-04-15T15:15:35.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.025+0000 7f2e57fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e5815c250 0x7f2e58157000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.025+0000 7f2e57fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e5815c250 0x7f2e58157000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54858/0 (socket says 172.21.15.72:54858) 2024-04-15T15:15:35.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.025+0000 7f2e57fff640 1 -- 172.21.15.72:0/1996822260 learned_addr learned my addr 172.21.15.72:0/1996822260 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:35.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.025+0000 7f2e5cbcd640 1 --2- 172.21.15.72:0/1996822260 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2e580d0010 0x7f2e580b6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.025+0000 7f2e5d3ce640 1 --2- 172.21.15.72:0/1996822260 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2e5815f9f0 0x7f2e5815edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.025+0000 7f2e57fff640 1 -- 172.21.15.72:0/1996822260 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2e580d0010 msgr2=0x7f2e580b6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.025+0000 7f2e57fff640 1 --2- 172.21.15.72:0/1996822260 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2e580d0010 0x7f2e580b6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.026+0000 7f2e57fff640 1 -- 172.21.15.72:0/1996822260 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2e5815f9f0 msgr2=0x7f2e5815edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.026+0000 7f2e57fff640 1 --2- 172.21.15.72:0/1996822260 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2e5815f9f0 0x7f2e5815edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.026+0000 7f2e57fff640 1 -- 172.21.15.72:0/1996822260 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e580b52b0 con 0x7f2e5815c250 2024-04-15T15:15:35.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.026+0000 7f2e5d3ce640 1 --2- 172.21.15.72:0/1996822260 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2e5815f9f0 0x7f2e5815edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:35.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.026+0000 7f2e57fff640 1 --2- 172.21.15.72:0/1996822260 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e5815c250 0x7f2e58157000 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f2e4000c430 tx=0x7f2e4000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:35.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.026+0000 7f2e557fa640 1 -- 172.21.15.72:0/1996822260 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e400074e0 con 0x7f2e5815c250 2024-04-15T15:15:35.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.026+0000 7f2e557fa640 1 -- 172.21.15.72:0/1996822260 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e400047e0 con 0x7f2e5815c250 2024-04-15T15:15:35.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.026+0000 7f2e557fa640 1 -- 172.21.15.72:0/1996822260 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e40004bd0 con 0x7f2e5815c250 2024-04-15T15:15:35.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.026+0000 7f2e5ee31640 1 -- 172.21.15.72:0/1996822260 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e580b5590 con 0x7f2e5815c250 2024-04-15T15:15:35.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.026+0000 7f2e5ee31640 1 -- 172.21.15.72:0/1996822260 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e580b5b50 con 0x7f2e5815c250 2024-04-15T15:15:35.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.027+0000 7f2e5ee31640 1 -- 172.21.15.72:0/1996822260 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e581630c0 con 0x7f2e5815c250 2024-04-15T15:15:35.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.027+0000 7f2e557fa640 1 -- 172.21.15.72:0/1996822260 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2e40011070 con 0x7f2e5815c250 2024-04-15T15:15:35.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.028+0000 7f2e557fa640 1 --2- 172.21.15.72:0/1996822260 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2e3806c8f0 0x7f2e3806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.028+0000 7f2e557fa640 1 -- 172.21.15.72:0/1996822260 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f2e4008bb80 con 0x7f2e5815c250 2024-04-15T15:15:35.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.028+0000 7f2e5cbcd640 1 --2- 172.21.15.72:0/1996822260 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2e3806c8f0 0x7f2e3806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.028+0000 7f2e5cbcd640 1 --2- 172.21.15.72:0/1996822260 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2e3806c8f0 0x7f2e3806edb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2e48036000 tx=0x7f2e48030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:35.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.032+0000 7f2e557fa640 1 -- 172.21.15.72:0/1996822260 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2e581630c0 con 0x7f2e5815c250 2024-04-15T15:15:35.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.208+0000 7f2e5ee31640 1 -- 172.21.15.72:0/1996822260 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2e58075c50 con 0x7f2e5815c250 2024-04-15T15:15:35.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.209+0000 7f2e557fa640 1 -- 172.21.15.72:0/1996822260 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f2e4000e280 con 0x7f2e5815c250 2024-04-15T15:15:35.210 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-04-15T15:15:35.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.211+0000 7f2e5ee31640 1 -- 172.21.15.72:0/1996822260 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2e3806c8f0 msgr2=0x7f2e3806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.211+0000 7f2e5ee31640 1 --2- 172.21.15.72:0/1996822260 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2e3806c8f0 0x7f2e3806edb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2e48036000 tx=0x7f2e48030000 comp rx=0 tx=0).stop 2024-04-15T15:15:35.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.211+0000 7f2e5ee31640 1 -- 172.21.15.72:0/1996822260 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e5815c250 msgr2=0x7f2e58157000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.211+0000 7f2e5ee31640 1 --2- 172.21.15.72:0/1996822260 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e5815c250 0x7f2e58157000 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f2e4000c430 tx=0x7f2e4000c8f0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.211+0000 7f2e5ee31640 1 -- 172.21.15.72:0/1996822260 shutdown_connections 2024-04-15T15:15:35.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.211+0000 7f2e5ee31640 1 --2- 172.21.15.72:0/1996822260 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2e3806c8f0 0x7f2e3806edb0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.211+0000 7f2e5ee31640 1 --2- 172.21.15.72:0/1996822260 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2e5815f9f0 0x7f2e5815edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.211+0000 7f2e5ee31640 1 --2- 172.21.15.72:0/1996822260 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e5815c250 0x7f2e58157000 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.211+0000 7f2e5ee31640 1 --2- 172.21.15.72:0/1996822260 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2e580d0010 0x7f2e580b6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.211+0000 7f2e5ee31640 1 -- 172.21.15.72:0/1996822260 >> 172.21.15.72:0/1996822260 conn(0x7f2e580ceec0 msgr2=0x7f2e58161550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:35.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.211+0000 7f2e5ee31640 1 -- 172.21.15.72:0/1996822260 shutdown_connections 2024-04-15T15:15:35.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.211+0000 7f2e5ee31640 1 -- 172.21.15.72:0/1996822260 wait complete. 2024-04-15T15:15:35.224 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:15:35.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.328+0000 7f4143831640 1 Processor -- start 2024-04-15T15:15:35.329 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.328+0000 7f4143831640 1 -- start start 2024-04-15T15:15:35.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.329+0000 7f4143831640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f413c15e770 0x7f413c0cf040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.329+0000 7f4143831640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f413c0b4dc0 0x7f413c0cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.329+0000 7f4143831640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f413c0cfc60 0x7f413c162440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.329+0000 7f4143831640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f413c0cf580 con 0x7f413c15e770 2024-04-15T15:15:35.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.329+0000 7f4143831640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f413c0ae160 con 0x7f413c0cfc60 2024-04-15T15:15:35.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.329+0000 7f4143831640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f413c0b5310 con 0x7f413c0b4dc0 2024-04-15T15:15:35.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.329+0000 7f41415cd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f413c15e770 0x7f413c0cf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.329+0000 7f41415cd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f413c15e770 0x7f413c0cf040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54870/0 (socket says 172.21.15.72:54870) 2024-04-15T15:15:35.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.329+0000 7f41415cd640 1 -- 172.21.15.72:0/2789253344 learned_addr learned my addr 172.21.15.72:0/2789253344 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:35.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.329+0000 7f4140dcc640 1 --2- 172.21.15.72:0/2789253344 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f413c0b4dc0 0x7f413c0cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.329+0000 7f4141dce640 1 --2- 172.21.15.72:0/2789253344 >> v2:172.21.15.175:3300/0 conn(0x7f413c0cfc60 0x7f413c162440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.329+0000 7f41415cd640 1 -- 172.21.15.72:0/2789253344 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f413c0b4dc0 msgr2=0x7f413c0cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:35.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.329+0000 7f41415cd640 1 --2- 172.21.15.72:0/2789253344 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f413c0b4dc0 0x7f413c0cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.329+0000 7f41415cd640 1 -- 172.21.15.72:0/2789253344 >> v2:172.21.15.175:3300/0 conn(0x7f413c0cfc60 msgr2=0x7f413c162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:35.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.329+0000 7f41415cd640 1 --2- 172.21.15.72:0/2789253344 >> v2:172.21.15.175:3300/0 conn(0x7f413c0cfc60 0x7f413c162440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.329+0000 7f41415cd640 1 -- 172.21.15.72:0/2789253344 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f413c0b55f0 con 0x7f413c15e770 2024-04-15T15:15:35.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.330+0000 7f41415cd640 1 --2- 172.21.15.72:0/2789253344 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f413c15e770 0x7f413c0cf040 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f413c0b5220 tx=0x7f413802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7a976b02ebc1ea3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:35.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.330+0000 7f4133fff640 1 -- 172.21.15.72:0/2789253344 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413802ca70 con 0x7f413c15e770 2024-04-15T15:15:35.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.330+0000 7f4133fff640 1 -- 172.21.15.72:0/2789253344 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f413802cc10 con 0x7f413c15e770 2024-04-15T15:15:35.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.330+0000 7f4133fff640 1 -- 172.21.15.72:0/2789253344 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41380343f0 con 0x7f413c15e770 2024-04-15T15:15:35.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.331+0000 7f4143831640 1 -- 172.21.15.72:0/2789253344 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f413c15e770 msgr2=0x7f413c0cf040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.331+0000 7f4143831640 1 --2- 172.21.15.72:0/2789253344 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f413c15e770 0x7f413c0cf040 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f413c0b5220 tx=0x7f413802bed0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.331+0000 7f4143831640 1 -- 172.21.15.72:0/2789253344 shutdown_connections 2024-04-15T15:15:35.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.331+0000 7f4143831640 1 --2- 172.21.15.72:0/2789253344 >> v2:172.21.15.175:3300/0 conn(0x7f413c0cfc60 0x7f413c162440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.331+0000 7f4143831640 1 --2- 172.21.15.72:0/2789253344 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f413c0b4dc0 0x7f413c0cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.331+0000 7f4143831640 1 --2- 172.21.15.72:0/2789253344 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f413c15e770 0x7f413c0cf040 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.331+0000 7f4143831640 1 -- 172.21.15.72:0/2789253344 >> 172.21.15.72:0/2789253344 conn(0x7f413c0d13a0 msgr2=0x7f413c0d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:35.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.331+0000 7f4143831640 1 -- 172.21.15.72:0/2789253344 shutdown_connections 2024-04-15T15:15:35.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.331+0000 7f4143831640 1 -- 172.21.15.72:0/2789253344 wait complete. 2024-04-15T15:15:35.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.332+0000 7f4143831640 1 Processor -- start 2024-04-15T15:15:35.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.332+0000 7f4143831640 1 -- start start 2024-04-15T15:15:35.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.333+0000 7f4143831640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f413c0b4dc0 0x7f413c147400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.333+0000 7f4143831640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f413c0cfc60 0x7f413c149c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.333+0000 7f4143831640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f413c15e770 0x7f413c14a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.333+0000 7f4143831640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f413c15eb50 con 0x7f413c0b4dc0 2024-04-15T15:15:35.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.333+0000 7f4143831640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f413c0b5bb0 con 0x7f413c15e770 2024-04-15T15:15:35.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.333+0000 7f4143831640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f413c15ecd0 con 0x7f413c0cfc60 2024-04-15T15:15:35.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.333+0000 7f41415cd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f413c0b4dc0 0x7f413c147400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.333+0000 7f41415cd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f413c0b4dc0 0x7f413c147400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54874/0 (socket says 172.21.15.72:54874) 2024-04-15T15:15:35.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.333+0000 7f41415cd640 1 -- 172.21.15.72:0/307223127 learned_addr learned my addr 172.21.15.72:0/307223127 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:35.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.333+0000 7f4140dcc640 1 --2- 172.21.15.72:0/307223127 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f413c0cfc60 0x7f413c149c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.333+0000 7f4141dce640 1 --2- 172.21.15.72:0/307223127 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f413c15e770 0x7f413c14a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.333+0000 7f41415cd640 1 -- 172.21.15.72:0/307223127 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f413c0cfc60 msgr2=0x7f413c149c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.333+0000 7f41415cd640 1 --2- 172.21.15.72:0/307223127 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f413c0cfc60 0x7f413c149c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.333+0000 7f41415cd640 1 -- 172.21.15.72:0/307223127 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f413c15e770 msgr2=0x7f413c14a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.333+0000 7f41415cd640 1 --2- 172.21.15.72:0/307223127 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f413c15e770 0x7f413c14a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.333+0000 7f41415cd640 1 -- 172.21.15.72:0/307223127 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f413c159730 con 0x7f413c0b4dc0 2024-04-15T15:15:35.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.333+0000 7f4141dce640 1 --2- 172.21.15.72:0/307223127 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f413c15e770 0x7f413c14a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:35.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.333+0000 7f41415cd640 1 --2- 172.21.15.72:0/307223127 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f413c0b4dc0 0x7f413c147400 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f4138036040 tx=0x7f4138004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:35.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.333+0000 7f4131ffb640 1 -- 172.21.15.72:0/307223127 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4138043020 con 0x7f413c0b4dc0 2024-04-15T15:15:35.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.333+0000 7f4131ffb640 1 -- 172.21.15.72:0/307223127 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f413803e450 con 0x7f413c0b4dc0 2024-04-15T15:15:35.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.333+0000 7f4143831640 1 -- 172.21.15.72:0/307223127 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f413c159a10 con 0x7f413c0b4dc0 2024-04-15T15:15:35.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.333+0000 7f4131ffb640 1 -- 172.21.15.72:0/307223127 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4138009510 con 0x7f413c0b4dc0 2024-04-15T15:15:35.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.334+0000 7f4143831640 1 -- 172.21.15.72:0/307223127 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f413c159fd0 con 0x7f413c0b4dc0 2024-04-15T15:15:35.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.335+0000 7f4143831640 1 -- 172.21.15.72:0/307223127 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f413c075c50 con 0x7f413c0b4dc0 2024-04-15T15:15:35.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.335+0000 7f4131ffb640 1 -- 172.21.15.72:0/307223127 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4138046020 con 0x7f413c0b4dc0 2024-04-15T15:15:35.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.336+0000 7f4131ffb640 1 --2- 172.21.15.72:0/307223127 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f412006c920 0x7f412006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.336+0000 7f4131ffb640 1 -- 172.21.15.72:0/307223127 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f41380ae4c0 con 0x7f413c0b4dc0 2024-04-15T15:15:35.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.336+0000 7f4140dcc640 1 --2- 172.21.15.72:0/307223127 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f412006c920 0x7f412006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.336+0000 7f4140dcc640 1 --2- 172.21.15.72:0/307223127 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f412006c920 0x7f412006ede0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f413c0ce5e0 tx=0x7f412c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:35.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.340+0000 7f4131ffb640 1 -- 172.21.15.72:0/307223127 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f413804d070 con 0x7f413c0b4dc0 2024-04-15T15:15:35.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.515+0000 7f4143831640 1 -- 172.21.15.72:0/307223127 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f413c15d020 con 0x7f413c0b4dc0 2024-04-15T15:15:35.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.515+0000 7f4131ffb640 1 -- 172.21.15.72:0/307223127 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f413807c4f0 con 0x7f413c0b4dc0 2024-04-15T15:15:35.517 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:35.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.517+0000 7f4143831640 1 -- 172.21.15.72:0/307223127 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f412006c920 msgr2=0x7f412006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.517+0000 7f4143831640 1 --2- 172.21.15.72:0/307223127 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f412006c920 0x7f412006ede0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f413c0ce5e0 tx=0x7f412c008040 comp rx=0 tx=0).stop 2024-04-15T15:15:35.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.517+0000 7f4143831640 1 -- 172.21.15.72:0/307223127 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f413c0b4dc0 msgr2=0x7f413c147400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.517+0000 7f4143831640 1 --2- 172.21.15.72:0/307223127 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f413c0b4dc0 0x7f413c147400 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f4138036040 tx=0x7f4138004a40 comp rx=0 tx=0).stop 2024-04-15T15:15:35.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.517+0000 7f4143831640 1 -- 172.21.15.72:0/307223127 shutdown_connections 2024-04-15T15:15:35.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.517+0000 7f4143831640 1 --2- 172.21.15.72:0/307223127 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f412006c920 0x7f412006ede0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.517+0000 7f4143831640 1 --2- 172.21.15.72:0/307223127 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f413c15e770 0x7f413c14a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.517+0000 7f4143831640 1 --2- 172.21.15.72:0/307223127 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f413c0cfc60 0x7f413c149c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.517+0000 7f4143831640 1 --2- 172.21.15.72:0/307223127 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f413c0b4dc0 0x7f413c147400 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.517+0000 7f4143831640 1 -- 172.21.15.72:0/307223127 >> 172.21.15.72:0/307223127 conn(0x7f413c0d13a0 msgr2=0x7f413c0b4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:35.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.517+0000 7f4143831640 1 -- 172.21.15.72:0/307223127 shutdown_connections 2024-04-15T15:15:35.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.517+0000 7f4143831640 1 -- 172.21.15.72:0/307223127 wait complete. 2024-04-15T15:15:35.530 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_blocked' 2024-04-15T15:15:35.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.631+0000 7f3582474640 1 Processor -- start 2024-04-15T15:15:35.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.631+0000 7f3582474640 1 -- start start 2024-04-15T15:15:35.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.632+0000 7f3582474640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f357c0cd2a0 0x7f357c0ad0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.632+0000 7f3582474640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f357c06f8f0 con 0x7f357c15eba0 2024-04-15T15:15:35.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.632+0000 7f3582474640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f357c15ef80 con 0x7f357c0af400 2024-04-15T15:15:35.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.632+0000 7f3582474640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f357c0ce2b0 con 0x7f357c0cd2a0 2024-04-15T15:15:35.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.632+0000 7f357bfff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f357c15eba0 0x7f357c06f1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50048/0 (socket says 172.21.15.72:50048) 2024-04-15T15:15:35.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.632+0000 7f357bfff640 1 -- 172.21.15.72:0/1414939093 learned_addr learned my addr 172.21.15.72:0/1414939093 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:35.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.632+0000 7f3580a11640 1 --2- 172.21.15.72:0/1414939093 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f357c0cd2a0 0x7f357c0ad0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.632+0000 7f357affd640 1 -- 172.21.15.72:0/1414939093 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1229339343 0 0) 0x7f357c06f8f0 con 0x7f357c15eba0 2024-04-15T15:15:35.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.632+0000 7f357affd640 1 -- 172.21.15.72:0/1414939093 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3564003520 con 0x7f357c15eba0 2024-04-15T15:15:35.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.632+0000 7f357affd640 1 -- 172.21.15.72:0/1414939093 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3414201817 0 0) 0x7f3564003520 con 0x7f357c15eba0 2024-04-15T15:15:35.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.632+0000 7f357affd640 1 -- 172.21.15.72:0/1414939093 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f357c0cd2a0 msgr2=0x7f357c0ad0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:35.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.632+0000 7f357affd640 1 --2- 172.21.15.72:0/1414939093 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f357c0cd2a0 0x7f357c0ad0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.632+0000 7f357affd640 1 -- 172.21.15.72:0/1414939093 >> v1:172.21.15.175:6789/0 conn(0x7f357c0af400 legacy=0x7f357c0ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.633+0000 7f357affd640 1 -- 172.21.15.72:0/1414939093 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f357c0ce590 con 0x7f357c15eba0 2024-04-15T15:15:35.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.633+0000 7f357affd640 1 -- 172.21.15.72:0/1414939093 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f356c001670 con 0x7f357c15eba0 2024-04-15T15:15:35.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.633+0000 7f357affd640 1 -- 172.21.15.72:0/1414939093 >> v1:172.21.15.72:6789/0 conn(0x7f357c15eba0 legacy=0x7f357c06f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.633+0000 7f357affd640 1 --2- 172.21.15.72:0/1414939093 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f357c0cd2a0 0x7f35640039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.633+0000 7f357affd640 1 --2- 172.21.15.72:0/1414939093 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3564003f20 0x7f3564004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.633+0000 7f357affd640 1 --2- 172.21.15.72:0/1414939093 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3564004d40 0x7f35640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.633+0000 7f357affd640 1 -- 172.21.15.72:0/1414939093 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f357c0ce2b0 con 0x7f3564004d40 2024-04-15T15:15:35.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.633+0000 7f357affd640 1 -- 172.21.15.72:0/1414939093 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f356400bad0 con 0x7f3564003f20 2024-04-15T15:15:35.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.633+0000 7f357affd640 1 -- 172.21.15.72:0/1414939093 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f356400bcd0 con 0x7f357c0cd2a0 2024-04-15T15:15:35.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.633+0000 7f3580a11640 1 --2- 172.21.15.72:0/1414939093 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3564004d40 0x7f35640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.633+0000 7f357bfff640 1 --2- 172.21.15.72:0/1414939093 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f357c0cd2a0 0x7f35640039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.633+0000 7f357b7fe640 1 --2- 172.21.15.72:0/1414939093 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3564003f20 0x7f3564004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.633+0000 7f3580a11640 1 -- 172.21.15.72:0/1414939093 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f357c0cd2a0 msgr2=0x7f35640039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:35.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.633+0000 7f3580a11640 1 --2- 172.21.15.72:0/1414939093 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f357c0cd2a0 0x7f35640039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.633+0000 7f3580a11640 1 -- 172.21.15.72:0/1414939093 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3564003f20 msgr2=0x7f3564004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:35.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.633+0000 7f3580a11640 1 --2- 172.21.15.72:0/1414939093 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3564003f20 0x7f3564004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.633+0000 7f3580a11640 1 -- 172.21.15.72:0/1414939093 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3564003520 con 0x7f3564004d40 2024-04-15T15:15:35.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.634+0000 7f3580a11640 1 --2- 172.21.15.72:0/1414939093 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3564004d40 0x7f35640051c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f3574009fc0 tx=0x7f3564005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b88892f13aaed8c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:35.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.634+0000 7f357affd640 1 -- 172.21.15.72:0/1414939093 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3574039070 con 0x7f3564004d40 2024-04-15T15:15:35.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.634+0000 7f357affd640 1 -- 172.21.15.72:0/1414939093 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f357402cc20 con 0x7f3564004d40 2024-04-15T15:15:35.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.634+0000 7f357affd640 1 -- 172.21.15.72:0/1414939093 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3574009c10 con 0x7f3564004d40 2024-04-15T15:15:35.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.634+0000 7f3582474640 1 -- 172.21.15.72:0/1414939093 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3564004d40 msgr2=0x7f35640051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.634+0000 7f3582474640 1 --2- 172.21.15.72:0/1414939093 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3564004d40 0x7f35640051c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f3574009fc0 tx=0x7f3564005830 comp rx=0 tx=0).stop 2024-04-15T15:15:35.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.634+0000 7f3580a11640 1 -- 172.21.15.72:0/1414939093 reap_dead start 2024-04-15T15:15:35.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.634+0000 7f3582474640 1 -- 172.21.15.72:0/1414939093 shutdown_connections 2024-04-15T15:15:35.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.634+0000 7f3582474640 1 -- 172.21.15.72:0/1414939093 >> 172.21.15.72:0/1414939093 conn(0x7f357c0b1dd0 msgr2=0x7f357c0b21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:35.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.635+0000 7f3582474640 1 -- 172.21.15.72:0/1414939093 shutdown_connections 2024-04-15T15:15:35.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.635+0000 7f3582474640 1 -- 172.21.15.72:0/1414939093 wait complete. 2024-04-15T15:15:35.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.636+0000 7f3582474640 1 Processor -- start 2024-04-15T15:15:35.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.636+0000 7f3582474640 1 -- start start 2024-04-15T15:15:35.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.637+0000 7f3582474640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f357c14b700 0x7f357c14fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.637+0000 7f3582474640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f357c16a990 0x7f357c14e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.637+0000 7f3582474640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f357c0b1480 0x7f357c14e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.637+0000 7f3582474640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f357c0b2b80 con 0x7f357c0b1480 2024-04-15T15:15:35.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.637+0000 7f3582474640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f357c06ecc0 con 0x7f357c16a990 2024-04-15T15:15:35.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.637+0000 7f3582474640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f357c0ceb50 con 0x7f357c14b700 2024-04-15T15:15:35.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.637+0000 7f3580a11640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f357c0b1480 0x7f357c14e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.637+0000 7f3580a11640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f357c0b1480 0x7f357c14e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54898/0 (socket says 172.21.15.72:54898) 2024-04-15T15:15:35.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.637+0000 7f3580a11640 1 -- 172.21.15.72:0/1067316055 learned_addr learned my addr 172.21.15.72:0/1067316055 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:35.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.637+0000 7f357b7fe640 1 --2- 172.21.15.72:0/1067316055 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f357c16a990 0x7f357c14e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.637+0000 7f3580a11640 1 -- 172.21.15.72:0/1067316055 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f357c14b700 msgr2=0x7f357c14fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.637+0000 7f357bfff640 1 --2- 172.21.15.72:0/1067316055 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f357c14b700 0x7f357c14fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.637+0000 7f3580a11640 1 --2- 172.21.15.72:0/1067316055 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f357c14b700 0x7f357c14fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.637+0000 7f3580a11640 1 -- 172.21.15.72:0/1067316055 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f357c16a990 msgr2=0x7f357c14e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.637+0000 7f3580a11640 1 --2- 172.21.15.72:0/1067316055 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f357c16a990 0x7f357c14e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.637+0000 7f3580a11640 1 -- 172.21.15.72:0/1067316055 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f357c0b09e0 con 0x7f357c0b1480 2024-04-15T15:15:35.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.637+0000 7f3580a11640 1 --2- 172.21.15.72:0/1067316055 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f357c0b1480 0x7f357c14e910 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f357402c690 tx=0x7f3574009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:35.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.637+0000 7f3578ff9640 1 -- 172.21.15.72:0/1067316055 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3574039030 con 0x7f357c0b1480 2024-04-15T15:15:35.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.637+0000 7f3578ff9640 1 -- 172.21.15.72:0/1067316055 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3574043070 con 0x7f357c0b1480 2024-04-15T15:15:35.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.637+0000 7f3582474640 1 -- 172.21.15.72:0/1067316055 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f357c16aeb0 con 0x7f357c0b1480 2024-04-15T15:15:35.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.637+0000 7f3578ff9640 1 -- 172.21.15.72:0/1067316055 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3574009c10 con 0x7f357c0b1480 2024-04-15T15:15:35.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.637+0000 7f3582474640 1 -- 172.21.15.72:0/1067316055 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f357c0b0c80 con 0x7f357c0b1480 2024-04-15T15:15:35.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.638+0000 7f3582474640 1 -- 172.21.15.72:0/1067316055 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f357c0cd960 con 0x7f357c0b1480 2024-04-15T15:15:35.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.639+0000 7f3578ff9640 1 -- 172.21.15.72:0/1067316055 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3574005d10 con 0x7f357c0b1480 2024-04-15T15:15:35.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.639+0000 7f3578ff9640 1 --2- 172.21.15.72:0/1067316055 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f356806c920 0x7f356806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.639+0000 7f3578ff9640 1 -- 172.21.15.72:0/1067316055 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f35740ae100 con 0x7f357c0b1480 2024-04-15T15:15:35.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.639+0000 7f357bfff640 1 --2- 172.21.15.72:0/1067316055 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f356806c920 0x7f356806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.640+0000 7f357bfff640 1 --2- 172.21.15.72:0/1067316055 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f356806c920 0x7f356806ede0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f356c001200 tx=0x7f356c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:35.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.644+0000 7f3578ff9640 1 -- 172.21.15.72:0/1067316055 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f357c0cd960 con 0x7f357c0b1480 2024-04-15T15:15:35.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:35.886+0000 7f3582474640 1 -- 172.21.15.72:0/1067316055 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_blocked"]} v 0) -- 0x7f357c0689e0 con 0x7f357c0b1480 2024-04-15T15:15:36.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.444+0000 7f3578ff9640 1 -- 172.21.15.72:0/1067316055 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_blocked"]}]=0 v64) ==== 140+0+0 (secure 0 0 0) 0x7f357407c130 con 0x7f357c0b1480 2024-04-15T15:15:36.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.446+0000 7f3582474640 1 -- 172.21.15.72:0/1067316055 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f356806c920 msgr2=0x7f356806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.446+0000 7f3582474640 1 --2- 172.21.15.72:0/1067316055 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f356806c920 0x7f356806ede0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f356c001200 tx=0x7f356c009040 comp rx=0 tx=0).stop 2024-04-15T15:15:36.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.446+0000 7f3582474640 1 -- 172.21.15.72:0/1067316055 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f357c0b1480 msgr2=0x7f357c14e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.446+0000 7f3582474640 1 --2- 172.21.15.72:0/1067316055 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f357c0b1480 0x7f357c14e910 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f357402c690 tx=0x7f3574009750 comp rx=0 tx=0).stop 2024-04-15T15:15:36.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.446+0000 7f3582474640 1 -- 172.21.15.72:0/1067316055 shutdown_connections 2024-04-15T15:15:36.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.446+0000 7f3582474640 1 --2- 172.21.15.72:0/1067316055 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f356806c920 0x7f356806ede0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.446+0000 7f3582474640 1 --2- 172.21.15.72:0/1067316055 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f357c0b1480 0x7f357c14e910 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.446+0000 7f3582474640 1 --2- 172.21.15.72:0/1067316055 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f357c16a990 0x7f357c14e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.446+0000 7f3582474640 1 --2- 172.21.15.72:0/1067316055 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f357c14b700 0x7f357c14fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.446+0000 7f3582474640 1 -- 172.21.15.72:0/1067316055 >> 172.21.15.72:0/1067316055 conn(0x7f357c0b1dd0 msgr2=0x7f357c15bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:36.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.447+0000 7f3582474640 1 -- 172.21.15.72:0/1067316055 shutdown_connections 2024-04-15T15:15:36.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.447+0000 7f3582474640 1 -- 172.21.15.72:0/1067316055 wait complete. 2024-04-15T15:15:36.460 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-15T15:15:36.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.567+0000 7fe6bcec5640 1 Processor -- start 2024-04-15T15:15:36.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.567+0000 7fe6bcec5640 1 -- start start 2024-04-15T15:15:36.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.568+0000 7fe6bcec5640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe6b80ccb50 0x7fe6b806dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.568+0000 7fe6bcec5640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe6b806bf20 0x7fe6b806c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.568+0000 7fe6bcec5640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6b815c110 con 0x7fe6b815af20 2024-04-15T15:15:36.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.568+0000 7fe6bcec5640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fe6b806ba90 con 0x7fe6b806bf20 2024-04-15T15:15:36.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.568+0000 7fe6bcec5640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fe6b80cfae0 con 0x7fe6b80ccb50 2024-04-15T15:15:36.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.568+0000 7fe6b6d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe6b806bf20 0x7fe6b806c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.568+0000 7fe6b6d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe6b806bf20 0x7fe6b806c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54908/0 (socket says 172.21.15.72:54908) 2024-04-15T15:15:36.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.568+0000 7fe6b6d9d640 1 -- 172.21.15.72:0/35116164 learned_addr learned my addr 172.21.15.72:0/35116164 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:36.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.568+0000 7fe6b6d9d640 1 -- 172.21.15.72:0/35116164 >> v2:172.21.15.175:3300/0 conn(0x7fe6b80ccb50 msgr2=0x7fe6b806dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:36.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.568+0000 7fe6b659c640 1 --2- 172.21.15.72:0/35116164 >> v2:172.21.15.175:3300/0 conn(0x7fe6b80ccb50 0x7fe6b806dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.568+0000 7fe6b6d9d640 1 --2- 172.21.15.72:0/35116164 >> v2:172.21.15.175:3300/0 conn(0x7fe6b80ccb50 0x7fe6b806dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.568+0000 7fe6b6d9d640 1 -- 172.21.15.72:0/35116164 >> v1:172.21.15.175:6789/0 conn(0x7fe6b815af20 legacy=0x7fe6b80ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:36.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.568+0000 7fe6b6d9d640 1 -- 172.21.15.72:0/35116164 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6b80cfdc0 con 0x7fe6b806bf20 2024-04-15T15:15:36.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.568+0000 7fe6b659c640 1 --2- 172.21.15.72:0/35116164 >> v2:172.21.15.175:3300/0 conn(0x7fe6b80ccb50 0x7fe6b806dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:36.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.569+0000 7fe6b6d9d640 1 --2- 172.21.15.72:0/35116164 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6b806bf20 0x7fe6b806c390 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe6b80cdf50 tx=0x7fe6b002cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=918f43a2e30856f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:36.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.569+0000 7fe6b5d9b640 1 -- 172.21.15.72:0/35116164 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b002d900 con 0x7fe6b806bf20 2024-04-15T15:15:36.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.569+0000 7fe6b5d9b640 1 -- 172.21.15.72:0/35116164 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6b002daa0 con 0x7fe6b806bf20 2024-04-15T15:15:36.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.569+0000 7fe6b5d9b640 1 -- 172.21.15.72:0/35116164 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b0008500 con 0x7fe6b806bf20 2024-04-15T15:15:36.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.569+0000 7fe6bcec5640 1 -- 172.21.15.72:0/35116164 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6b806bf20 msgr2=0x7fe6b806c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.569+0000 7fe6bcec5640 1 --2- 172.21.15.72:0/35116164 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6b806bf20 0x7fe6b806c390 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe6b80cdf50 tx=0x7fe6b002cf60 comp rx=0 tx=0).stop 2024-04-15T15:15:36.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.569+0000 7fe6bcec5640 1 -- 172.21.15.72:0/35116164 shutdown_connections 2024-04-15T15:15:36.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.570+0000 7fe6bcec5640 1 --2- 172.21.15.72:0/35116164 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6b806bf20 0x7fe6b806c390 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.570+0000 7fe6bcec5640 1 --2- 172.21.15.72:0/35116164 >> v2:172.21.15.175:3300/0 conn(0x7fe6b80ccb50 0x7fe6b806dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.570+0000 7fe6bcec5640 1 -- 172.21.15.72:0/35116164 >> 172.21.15.72:0/35116164 conn(0x7fe6b80b31d0 msgr2=0x7fe6b80b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:36.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.570+0000 7fe6bcec5640 1 -- 172.21.15.72:0/35116164 shutdown_connections 2024-04-15T15:15:36.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.570+0000 7fe6bcec5640 1 -- 172.21.15.72:0/35116164 wait complete. 2024-04-15T15:15:36.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.571+0000 7fe6bcec5640 1 Processor -- start 2024-04-15T15:15:36.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.571+0000 7fe6bcec5640 1 -- start start 2024-04-15T15:15:36.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.572+0000 7fe6bcec5640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe6b806bf20 0x7fe6b8153860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.572+0000 7fe6bcec5640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe6b80ccb50 0x7fe6b809d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.572+0000 7fe6bcec5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6b815af20 0x7fe6b8157fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.572+0000 7fe6bcec5640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b80cc8f0 con 0x7fe6b815af20 2024-04-15T15:15:36.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.572+0000 7fe6bcec5640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fe6b80cc770 con 0x7fe6b806bf20 2024-04-15T15:15:36.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.572+0000 7fe6bcec5640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fe6b815bcf0 con 0x7fe6b80ccb50 2024-04-15T15:15:36.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.572+0000 7fe6b6d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6b815af20 0x7fe6b8157fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.572+0000 7fe6b6d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6b815af20 0x7fe6b8157fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54910/0 (socket says 172.21.15.72:54910) 2024-04-15T15:15:36.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.572+0000 7fe6b6d9d640 1 -- 172.21.15.72:0/3560245089 learned_addr learned my addr 172.21.15.72:0/3560245089 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:36.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.572+0000 7fe6b659c640 1 --2- 172.21.15.72:0/3560245089 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe6b806bf20 0x7fe6b8153860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.572+0000 7fe6add9b640 1 --2- 172.21.15.72:0/3560245089 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe6b80ccb50 0x7fe6b809d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.572+0000 7fe6b6d9d640 1 -- 172.21.15.72:0/3560245089 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe6b80ccb50 msgr2=0x7fe6b809d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.572+0000 7fe6b6d9d640 1 --2- 172.21.15.72:0/3560245089 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe6b80ccb50 0x7fe6b809d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.572+0000 7fe6b6d9d640 1 -- 172.21.15.72:0/3560245089 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe6b806bf20 msgr2=0x7fe6b8153860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.572+0000 7fe6b6d9d640 1 --2- 172.21.15.72:0/3560245089 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe6b806bf20 0x7fe6b8153860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.572+0000 7fe6b6d9d640 1 -- 172.21.15.72:0/3560245089 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6b81571e0 con 0x7fe6b815af20 2024-04-15T15:15:36.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.572+0000 7fe6b6d9d640 1 --2- 172.21.15.72:0/3560245089 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6b815af20 0x7fe6b8157fa0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe6b000c140 tx=0x7fe6b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:36.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.572+0000 7fe6af7fe640 1 -- 172.21.15.72:0/3560245089 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b0044020 con 0x7fe6b815af20 2024-04-15T15:15:36.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.572+0000 7fe6af7fe640 1 -- 172.21.15.72:0/3560245089 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6b0009d10 con 0x7fe6b815af20 2024-04-15T15:15:36.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.573+0000 7fe6bcec5640 1 -- 172.21.15.72:0/3560245089 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6b81574c0 con 0x7fe6b815af20 2024-04-15T15:15:36.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.573+0000 7fe6af7fe640 1 -- 172.21.15.72:0/3560245089 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b00406b0 con 0x7fe6b815af20 2024-04-15T15:15:36.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.573+0000 7fe6bcec5640 1 -- 172.21.15.72:0/3560245089 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6b81584e0 con 0x7fe6b815af20 2024-04-15T15:15:36.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.574+0000 7fe6bcec5640 1 -- 172.21.15.72:0/3560245089 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6b8075c50 con 0x7fe6b815af20 2024-04-15T15:15:36.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.574+0000 7fe6af7fe640 1 -- 172.21.15.72:0/3560245089 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe6b0047020 con 0x7fe6b815af20 2024-04-15T15:15:36.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.574+0000 7fe6af7fe640 1 --2- 172.21.15.72:0/3560245089 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe69806c920 0x7fe69806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.574+0000 7fe6af7fe640 1 -- 172.21.15.72:0/3560245089 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fe6b00aecc0 con 0x7fe6b815af20 2024-04-15T15:15:36.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.575+0000 7fe6b659c640 1 --2- 172.21.15.72:0/3560245089 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe69806c920 0x7fe69806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.575+0000 7fe6b659c640 1 --2- 172.21.15.72:0/3560245089 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe69806c920 0x7fe69806ede0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe6b815f210 tx=0x7fe6a8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:36.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.579+0000 7fe6af7fe640 1 -- 172.21.15.72:0/3560245089 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe6b004e030 con 0x7fe6b815af20 2024-04-15T15:15:36.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.754+0000 7fe6bcec5640 1 -- 172.21.15.72:0/3560245089 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fe6b8153dd0 con 0x7fe6b815af20 2024-04-15T15:15:36.761 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.760+0000 7fe6af7fe640 1 -- 172.21.15.72:0/3560245089 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7fe6b007ccf0 con 0x7fe6b815af20 2024-04-15T15:15:36.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.762+0000 7fe6bcec5640 1 -- 172.21.15.72:0/3560245089 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe69806c920 msgr2=0x7fe69806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.762+0000 7fe6bcec5640 1 --2- 172.21.15.72:0/3560245089 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe69806c920 0x7fe69806ede0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe6b815f210 tx=0x7fe6a8008040 comp rx=0 tx=0).stop 2024-04-15T15:15:36.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.762+0000 7fe6bcec5640 1 -- 172.21.15.72:0/3560245089 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6b815af20 msgr2=0x7fe6b8157fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.762+0000 7fe6bcec5640 1 --2- 172.21.15.72:0/3560245089 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6b815af20 0x7fe6b8157fa0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe6b000c140 tx=0x7fe6b0004a40 comp rx=0 tx=0).stop 2024-04-15T15:15:36.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.762+0000 7fe6bcec5640 1 -- 172.21.15.72:0/3560245089 shutdown_connections 2024-04-15T15:15:36.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.762+0000 7fe6bcec5640 1 --2- 172.21.15.72:0/3560245089 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe69806c920 0x7fe69806ede0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.762+0000 7fe6bcec5640 1 --2- 172.21.15.72:0/3560245089 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6b815af20 0x7fe6b8157fa0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.762+0000 7fe6bcec5640 1 --2- 172.21.15.72:0/3560245089 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe6b80ccb50 0x7fe6b809d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.762+0000 7fe6bcec5640 1 --2- 172.21.15.72:0/3560245089 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe6b806bf20 0x7fe6b8153860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.762+0000 7fe6bcec5640 1 -- 172.21.15.72:0/3560245089 >> 172.21.15.72:0/3560245089 conn(0x7fe6b80b31d0 msgr2=0x7fe6b8073100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:36.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.762+0000 7fe6bcec5640 1 -- 172.21.15.72:0/3560245089 shutdown_connections 2024-04-15T15:15:36.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:36.762+0000 7fe6bcec5640 1 -- 172.21.15.72:0/3560245089 wait complete. 2024-04-15T15:15:36.774 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-15T15:15:36.775 DEBUG:teuthology.orchestra.run.smithi072:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-15T15:15:36.814 DEBUG:teuthology.orchestra.run.smithi175:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-15T15:15:36.861 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-15T15:15:36.861 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:15:36.861 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:15:36.879 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:15:36.961 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:15:36.978 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:37.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.119+0000 7f4a330e4640 1 Processor -- start 2024-04-15T15:15:37.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.119+0000 7f4a330e4640 1 -- start start 2024-04-15T15:15:37.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.120+0000 7f4a330e4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4a2c1569c0 0x7f4a2c15d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.120+0000 7f4a330e4640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f4a2c06c370 0x7f4a2c06c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.120+0000 7f4a330e4640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a2c15da80 con 0x7f4a2c156da0 2024-04-15T15:15:37.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.120+0000 7f4a330e4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f4a2c0cd280 con 0x7f4a2c1569c0 2024-04-15T15:15:37.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.120+0000 7f4a330e4640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f4a2c0cbad0 con 0x7f4a2c06c370 2024-04-15T15:15:37.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.120+0000 7f4a30e80640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4a2c1569c0 0x7f4a2c15d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.120+0000 7f4a30e80640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4a2c1569c0 0x7f4a2c15d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54922/0 (socket says 172.21.15.72:54922) 2024-04-15T15:15:37.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.120+0000 7f4a30e80640 1 -- 172.21.15.72:0/3836441679 learned_addr learned my addr 172.21.15.72:0/3836441679 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:37.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.120+0000 7f4a30e80640 1 -- 172.21.15.72:0/3836441679 >> v2:172.21.15.175:3300/0 conn(0x7f4a2c06c370 msgr2=0x7f4a2c06c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:37.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.120+0000 7f4a30e80640 1 --2- 172.21.15.72:0/3836441679 >> v2:172.21.15.175:3300/0 conn(0x7f4a2c06c370 0x7f4a2c06c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.120+0000 7f4a30e80640 1 -- 172.21.15.72:0/3836441679 >> v1:172.21.15.175:6789/0 conn(0x7f4a2c156da0 legacy=0x7f4a2c0ccb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:37.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.120+0000 7f4a30e80640 1 -- 172.21.15.72:0/3836441679 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a2c0cbdb0 con 0x7f4a2c1569c0 2024-04-15T15:15:37.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.121+0000 7f4a30e80640 1 --2- 172.21.15.72:0/3836441679 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a2c1569c0 0x7f4a2c15d540 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c06d020 tx=0x7f4a1c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6fb4f00be15c64c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:37.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.121+0000 7f4a2b7fe640 1 -- 172.21.15.72:0/3836441679 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c035020 con 0x7f4a2c1569c0 2024-04-15T15:15:37.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.121+0000 7f4a2b7fe640 1 -- 172.21.15.72:0/3836441679 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a1c031870 con 0x7f4a2c1569c0 2024-04-15T15:15:37.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.121+0000 7f4a2b7fe640 1 -- 172.21.15.72:0/3836441679 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c031c40 con 0x7f4a2c1569c0 2024-04-15T15:15:37.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.122+0000 7f4a330e4640 1 -- 172.21.15.72:0/3836441679 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a2c1569c0 msgr2=0x7f4a2c15d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.122+0000 7f4a330e4640 1 --2- 172.21.15.72:0/3836441679 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a2c1569c0 0x7f4a2c15d540 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c06d020 tx=0x7f4a1c02b910 comp rx=0 tx=0).stop 2024-04-15T15:15:37.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.122+0000 7f4a330e4640 1 -- 172.21.15.72:0/3836441679 shutdown_connections 2024-04-15T15:15:37.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.122+0000 7f4a330e4640 1 --2- 172.21.15.72:0/3836441679 >> v2:172.21.15.175:3300/0 conn(0x7f4a2c06c370 0x7f4a2c06c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.122+0000 7f4a330e4640 1 --2- 172.21.15.72:0/3836441679 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a2c1569c0 0x7f4a2c15d540 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.122+0000 7f4a330e4640 1 -- 172.21.15.72:0/3836441679 >> 172.21.15.72:0/3836441679 conn(0x7f4a2c0cd4a0 msgr2=0x7f4a2c0cd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:37.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.122+0000 7f4a330e4640 1 -- 172.21.15.72:0/3836441679 shutdown_connections 2024-04-15T15:15:37.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.122+0000 7f4a330e4640 1 -- 172.21.15.72:0/3836441679 wait complete. 2024-04-15T15:15:37.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.123+0000 7f4a330e4640 1 Processor -- start 2024-04-15T15:15:37.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.123+0000 7f4a330e4640 1 -- start start 2024-04-15T15:15:37.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.124+0000 7f4a330e4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a2c06c370 0x7f4a2c150060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.124+0000 7f4a330e4640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4a2c1569c0 0x7f4a2c142160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.124+0000 7f4a330e4640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4a2c156da0 0x7f4a2c1426a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.124+0000 7f4a330e4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f4a2c0cc650 con 0x7f4a2c06c370 2024-04-15T15:15:37.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.124+0000 7f4a330e4640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f4a2c0cc370 con 0x7f4a2c156da0 2024-04-15T15:15:37.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.124+0000 7f4a330e4640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f4a2c0b01a0 con 0x7f4a2c1569c0 2024-04-15T15:15:37.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.124+0000 7f4a30e80640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a2c06c370 0x7f4a2c150060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.124+0000 7f4a30e80640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a2c06c370 0x7f4a2c150060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54924/0 (socket says 172.21.15.72:54924) 2024-04-15T15:15:37.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.124+0000 7f4a30e80640 1 -- 172.21.15.72:0/3050559055 learned_addr learned my addr 172.21.15.72:0/3050559055 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:37.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.124+0000 7f4a2bfff640 1 --2- 172.21.15.72:0/3050559055 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4a2c1569c0 0x7f4a2c142160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.124+0000 7f4a31681640 1 --2- 172.21.15.72:0/3050559055 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4a2c156da0 0x7f4a2c1426a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.124+0000 7f4a30e80640 1 -- 172.21.15.72:0/3050559055 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4a2c1569c0 msgr2=0x7f4a2c142160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.124+0000 7f4a30e80640 1 --2- 172.21.15.72:0/3050559055 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4a2c1569c0 0x7f4a2c142160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.124+0000 7f4a30e80640 1 -- 172.21.15.72:0/3050559055 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4a2c156da0 msgr2=0x7f4a2c1426a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.124+0000 7f4a30e80640 1 --2- 172.21.15.72:0/3050559055 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4a2c156da0 0x7f4a2c1426a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.124+0000 7f4a30e80640 1 -- 172.21.15.72:0/3050559055 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a2c14f000 con 0x7f4a2c06c370 2024-04-15T15:15:37.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.125+0000 7f4a30e80640 1 --2- 172.21.15.72:0/3050559055 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a2c06c370 0x7f4a2c150060 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c02b950 tx=0x7f4a1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:37.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.125+0000 7f4a297fa640 1 -- 172.21.15.72:0/3050559055 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c03ad60 con 0x7f4a2c06c370 2024-04-15T15:15:37.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.125+0000 7f4a297fa640 1 -- 172.21.15.72:0/3050559055 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a1c0443f0 con 0x7f4a2c06c370 2024-04-15T15:15:37.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.125+0000 7f4a297fa640 1 -- 172.21.15.72:0/3050559055 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a1c0447e0 con 0x7f4a2c06c370 2024-04-15T15:15:37.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.125+0000 7f4a330e4640 1 -- 172.21.15.72:0/3050559055 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a2c14f2a0 con 0x7f4a2c06c370 2024-04-15T15:15:37.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.125+0000 7f4a330e4640 1 -- 172.21.15.72:0/3050559055 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a2c14f860 con 0x7f4a2c06c370 2024-04-15T15:15:37.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.126+0000 7f4a330e4640 1 -- 172.21.15.72:0/3050559055 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a2c0689e0 con 0x7f4a2c06c370 2024-04-15T15:15:37.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.126+0000 7f4a297fa640 1 -- 172.21.15.72:0/3050559055 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4a1c038020 con 0x7f4a2c06c370 2024-04-15T15:15:37.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.127+0000 7f4a297fa640 1 --2- 172.21.15.72:0/3050559055 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4a1806c8f0 0x7f4a1806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.127+0000 7f4a2bfff640 1 --2- 172.21.15.72:0/3050559055 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4a1806c8f0 0x7f4a1806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.127+0000 7f4a297fa640 1 -- 172.21.15.72:0/3050559055 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f4a1c0ae160 con 0x7f4a2c06c370 2024-04-15T15:15:37.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.127+0000 7f4a2bfff640 1 --2- 172.21.15.72:0/3050559055 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4a1806c8f0 0x7f4a1806edb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c0af710 tx=0x7f4a20009060 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:37.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.131+0000 7f4a297fa640 1 -- 172.21.15.72:0/3050559055 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4a1c03f070 con 0x7f4a2c06c370 2024-04-15T15:15:37.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.307+0000 7f4a330e4640 1 -- 172.21.15.72:0/3050559055 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4a2c0cb530 con 0x7f4a2c06c370 2024-04-15T15:15:37.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.308+0000 7f4a297fa640 1 -- 172.21.15.72:0/3050559055 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18903 (secure 0 0 0) 0x7f4a2c15da80 con 0x7f4a2c06c370 2024-04-15T15:15:37.310 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:37.310 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":26,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:15:32.421728+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:14:49.438366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:14:50.368379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:14:54.081157+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":1}]},{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:37.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.310+0000 7f4a330e4640 1 -- 172.21.15.72:0/3050559055 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4a1806c8f0 msgr2=0x7f4a1806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.310+0000 7f4a330e4640 1 --2- 172.21.15.72:0/3050559055 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4a1806c8f0 0x7f4a1806edb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c0af710 tx=0x7f4a20009060 comp rx=0 tx=0).stop 2024-04-15T15:15:37.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.310+0000 7f4a330e4640 1 -- 172.21.15.72:0/3050559055 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a2c06c370 msgr2=0x7f4a2c150060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.310+0000 7f4a330e4640 1 --2- 172.21.15.72:0/3050559055 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a2c06c370 0x7f4a2c150060 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4a1c02b950 tx=0x7f4a1c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.310+0000 7f4a330e4640 1 -- 172.21.15.72:0/3050559055 shutdown_connections 2024-04-15T15:15:37.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.310+0000 7f4a330e4640 1 --2- 172.21.15.72:0/3050559055 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4a1806c8f0 0x7f4a1806edb0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.310+0000 7f4a330e4640 1 --2- 172.21.15.72:0/3050559055 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4a2c156da0 0x7f4a2c1426a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.310+0000 7f4a330e4640 1 --2- 172.21.15.72:0/3050559055 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4a2c1569c0 0x7f4a2c142160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.310+0000 7f4a330e4640 1 --2- 172.21.15.72:0/3050559055 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4a2c06c370 0x7f4a2c150060 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.310+0000 7f4a330e4640 1 -- 172.21.15.72:0/3050559055 >> 172.21.15.72:0/3050559055 conn(0x7f4a2c0cd4a0 msgr2=0x7f4a2c0b09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:37.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.310+0000 7f4a330e4640 1 -- 172.21.15.72:0/3050559055 shutdown_connections 2024-04-15T15:15:37.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.310+0000 7f4a330e4640 1 -- 172.21.15.72:0/3050559055 wait complete. 2024-04-15T15:15:37.323 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:39.359102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:14:49.438366+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:14:50.368379+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:14:54.081157+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:15:37.324 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:37.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.428+0000 7fe7e79ba640 1 Processor -- start 2024-04-15T15:15:37.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.428+0000 7fe7e79ba640 1 -- start start 2024-04-15T15:15:37.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.429+0000 7fe7e79ba640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe7e00ce2f0 0x7fe7e015c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.429+0000 7fe7e79ba640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe7e00d08f0 0x7fe7e00cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.429+0000 7fe7e79ba640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7e006bbb0 con 0x7fe7e00ce6d0 2024-04-15T15:15:37.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.429+0000 7fe7e79ba640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fe7e006b960 con 0x7fe7e00ce2f0 2024-04-15T15:15:37.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.429+0000 7fe7e79ba640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fe7e006be90 con 0x7fe7e00d08f0 2024-04-15T15:15:37.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.429+0000 7fe7e4f55640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe7e00d08f0 0x7fe7e00cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.429+0000 7fe7e5756640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe7e00ce2f0 0x7fe7e015c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.429+0000 7fe7e4f55640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe7e00d08f0 0x7fe7e00cc0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3301/0 says I am v2:172.21.15.72:48720/0 (socket says 172.21.15.72:48720) 2024-04-15T15:15:37.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.429+0000 7fe7e4f55640 1 -- 172.21.15.72:0/2333666906 learned_addr learned my addr 172.21.15.72:0/2333666906 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:37.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.429+0000 7fe7e4f55640 1 -- 172.21.15.72:0/2333666906 >> v2:172.21.15.175:3300/0 conn(0x7fe7e00ce2f0 msgr2=0x7fe7e015c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:37.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.429+0000 7fe7e4f55640 1 --2- 172.21.15.72:0/2333666906 >> v2:172.21.15.175:3300/0 conn(0x7fe7e00ce2f0 0x7fe7e015c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.429+0000 7fe7e4f55640 1 -- 172.21.15.72:0/2333666906 >> v1:172.21.15.175:6789/0 conn(0x7fe7e00ce6d0 legacy=0x7fe7e0155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.429+0000 7fe7e4f55640 1 -- 172.21.15.72:0/2333666906 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7e006c170 con 0x7fe7e00d08f0 2024-04-15T15:15:37.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.430+0000 7fe7e4f55640 1 --2- 172.21.15.72:0/2333666906 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe7e00d08f0 0x7fe7e00cc0b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0009870 tx=0x7fe7d002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a533b707379e33ab server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:37.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.430+0000 7fe7d7fff640 1 -- 172.21.15.72:0/2333666906 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d0034070 con 0x7fe7e00d08f0 2024-04-15T15:15:37.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.430+0000 7fe7d7fff640 1 -- 172.21.15.72:0/2333666906 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7d0030840 con 0x7fe7e00d08f0 2024-04-15T15:15:37.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.430+0000 7fe7d7fff640 1 -- 172.21.15.72:0/2333666906 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d0030c10 con 0x7fe7e00d08f0 2024-04-15T15:15:37.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.431+0000 7fe7e79ba640 1 -- 172.21.15.72:0/2333666906 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe7e00d08f0 msgr2=0x7fe7e00cc0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.431+0000 7fe7e79ba640 1 --2- 172.21.15.72:0/2333666906 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe7e00d08f0 0x7fe7e00cc0b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe7d0009870 tx=0x7fe7d002b980 comp rx=0 tx=0).stop 2024-04-15T15:15:37.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.431+0000 7fe7e79ba640 1 -- 172.21.15.72:0/2333666906 shutdown_connections 2024-04-15T15:15:37.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.431+0000 7fe7e79ba640 1 --2- 172.21.15.72:0/2333666906 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe7e00d08f0 0x7fe7e00cc0b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.431+0000 7fe7e79ba640 1 --2- 172.21.15.72:0/2333666906 >> v2:172.21.15.175:3300/0 conn(0x7fe7e00ce2f0 0x7fe7e015c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.431+0000 7fe7e79ba640 1 -- 172.21.15.72:0/2333666906 >> 172.21.15.72:0/2333666906 conn(0x7fe7e00b31d0 msgr2=0x7fe7e00b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:37.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.431+0000 7fe7e79ba640 1 -- 172.21.15.72:0/2333666906 shutdown_connections 2024-04-15T15:15:37.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.431+0000 7fe7e79ba640 1 -- 172.21.15.72:0/2333666906 wait complete. 2024-04-15T15:15:37.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.432+0000 7fe7e79ba640 1 Processor -- start 2024-04-15T15:15:37.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.432+0000 7fe7e79ba640 1 -- start start 2024-04-15T15:15:37.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.433+0000 7fe7e79ba640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe7e00ce2f0 0x7fe7e014a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.433+0000 7fe7e79ba640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7e00ce6d0 0x7fe7e00b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.433+0000 7fe7e79ba640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe7e00d08f0 0x7fe7e01583d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.433+0000 7fe7e79ba640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e0155d10 con 0x7fe7e00ce6d0 2024-04-15T15:15:37.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.433+0000 7fe7e79ba640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e015ca70 con 0x7fe7e00ce2f0 2024-04-15T15:15:37.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.433+0000 7fe7e79ba640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fe7e0155e90 con 0x7fe7e00d08f0 2024-04-15T15:15:37.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.433+0000 7fe7e4f55640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7e00ce6d0 0x7fe7e00b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.433+0000 7fe7e4f55640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7e00ce6d0 0x7fe7e00b03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54928/0 (socket says 172.21.15.72:54928) 2024-04-15T15:15:37.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.433+0000 7fe7e4f55640 1 -- 172.21.15.72:0/2393969821 learned_addr learned my addr 172.21.15.72:0/2393969821 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:37.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.433+0000 7fe7e5756640 1 --2- 172.21.15.72:0/2393969821 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe7e00ce2f0 0x7fe7e014a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.433+0000 7fe7e5f57640 1 --2- 172.21.15.72:0/2393969821 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe7e00d08f0 0x7fe7e01583d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.433+0000 7fe7e4f55640 1 -- 172.21.15.72:0/2393969821 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe7e00d08f0 msgr2=0x7fe7e01583d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.433+0000 7fe7e4f55640 1 --2- 172.21.15.72:0/2393969821 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe7e00d08f0 0x7fe7e01583d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.433+0000 7fe7e4f55640 1 -- 172.21.15.72:0/2393969821 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe7e00ce2f0 msgr2=0x7fe7e014a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.433+0000 7fe7e4f55640 1 --2- 172.21.15.72:0/2393969821 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe7e00ce2f0 0x7fe7e014a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.433+0000 7fe7e4f55640 1 -- 172.21.15.72:0/2393969821 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7e0158db0 con 0x7fe7e00ce6d0 2024-04-15T15:15:37.437 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.434+0000 7fe7e4f55640 1 --2- 172.21.15.72:0/2393969821 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7e00ce6d0 0x7fe7e00b03c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe7d002b9c0 tx=0x7fe7d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:37.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.434+0000 7fe7d5ffb640 1 -- 172.21.15.72:0/2393969821 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d0034040 con 0x7fe7e00ce6d0 2024-04-15T15:15:37.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.434+0000 7fe7d5ffb640 1 -- 172.21.15.72:0/2393969821 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7d0039d90 con 0x7fe7e00ce6d0 2024-04-15T15:15:37.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.434+0000 7fe7d5ffb640 1 -- 172.21.15.72:0/2393969821 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d0043630 con 0x7fe7e00ce6d0 2024-04-15T15:15:37.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.434+0000 7fe7e79ba640 1 -- 172.21.15.72:0/2393969821 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7e00af8e0 con 0x7fe7e00ce6d0 2024-04-15T15:15:37.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.434+0000 7fe7e79ba640 1 -- 172.21.15.72:0/2393969821 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7e014aca0 con 0x7fe7e00ce6d0 2024-04-15T15:15:37.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.435+0000 7fe7e79ba640 1 -- 172.21.15.72:0/2393969821 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7e006edd0 con 0x7fe7e00ce6d0 2024-04-15T15:15:37.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.435+0000 7fe7d5ffb640 1 -- 172.21.15.72:0/2393969821 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe7d0037070 con 0x7fe7e00ce6d0 2024-04-15T15:15:37.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.436+0000 7fe7d5ffb640 1 --2- 172.21.15.72:0/2393969821 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe7c406c8f0 0x7fe7c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.436+0000 7fe7d5ffb640 1 -- 172.21.15.72:0/2393969821 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fe7d00ae120 con 0x7fe7e00ce6d0 2024-04-15T15:15:37.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.436+0000 7fe7e5756640 1 --2- 172.21.15.72:0/2393969821 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe7c406c8f0 0x7fe7c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.436+0000 7fe7e5756640 1 --2- 172.21.15.72:0/2393969821 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe7c406c8f0 0x7fe7c406edb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc0113e0 tx=0x7fe7dc00b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:37.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.440+0000 7fe7d5ffb640 1 -- 172.21.15.72:0/2393969821 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe7e006edd0 con 0x7fe7e00ce6d0 2024-04-15T15:15:37.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.616+0000 7fe7e79ba640 1 -- 172.21.15.72:0/2393969821 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe7e00d03c0 con 0x7fe7e00ce6d0 2024-04-15T15:15:37.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.616+0000 7fe7d5ffb640 1 -- 172.21.15.72:0/2393969821 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7fe7e00d03c0 con 0x7fe7e00ce6d0 2024-04-15T15:15:37.618 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-15T15:15:37.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.618+0000 7fe7e79ba640 1 -- 172.21.15.72:0/2393969821 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe7c406c8f0 msgr2=0x7fe7c406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.618+0000 7fe7e79ba640 1 --2- 172.21.15.72:0/2393969821 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe7c406c8f0 0x7fe7c406edb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc0113e0 tx=0x7fe7dc00b040 comp rx=0 tx=0).stop 2024-04-15T15:15:37.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.618+0000 7fe7e79ba640 1 -- 172.21.15.72:0/2393969821 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7e00ce6d0 msgr2=0x7fe7e00b03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.618+0000 7fe7e79ba640 1 --2- 172.21.15.72:0/2393969821 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7e00ce6d0 0x7fe7e00b03c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe7d002b9c0 tx=0x7fe7d00046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.618+0000 7fe7e79ba640 1 -- 172.21.15.72:0/2393969821 shutdown_connections 2024-04-15T15:15:37.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.618+0000 7fe7e79ba640 1 --2- 172.21.15.72:0/2393969821 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe7c406c8f0 0x7fe7c406edb0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.618+0000 7fe7e79ba640 1 --2- 172.21.15.72:0/2393969821 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe7e00d08f0 0x7fe7e01583d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.618+0000 7fe7e79ba640 1 --2- 172.21.15.72:0/2393969821 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7e00ce6d0 0x7fe7e00b03c0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.618+0000 7fe7e79ba640 1 --2- 172.21.15.72:0/2393969821 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe7e00ce2f0 0x7fe7e014a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.618+0000 7fe7e79ba640 1 -- 172.21.15.72:0/2393969821 >> 172.21.15.72:0/2393969821 conn(0x7fe7e00b31d0 msgr2=0x7fe7e015e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:37.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.618+0000 7fe7e79ba640 1 -- 172.21.15.72:0/2393969821 shutdown_connections 2024-04-15T15:15:37.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.618+0000 7fe7e79ba640 1 -- 172.21.15.72:0/2393969821 wait complete. 2024-04-15T15:15:37.631 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:15:37.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.734+0000 7f78dfa14640 1 Processor -- start 2024-04-15T15:15:37.736 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.734+0000 7f78dfa14640 1 -- start start 2024-04-15T15:15:37.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.735+0000 7f78dfa14640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f78d80cc7c0 0x7f78d8155600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.735+0000 7f78dfa14640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78d806bbb0 con 0x7f78d80d08f0 2024-04-15T15:15:37.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.735+0000 7f78dfa14640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78d806be90 con 0x7f78d80ce2f0 2024-04-15T15:15:37.739 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.735+0000 7f78dfa14640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f78d806b960 con 0x7f78d80cc7c0 2024-04-15T15:15:37.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.735+0000 7f78dcfaf640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f78d80d08f0 0x7f78d80cc0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50062/0 (socket says 172.21.15.72:50062) 2024-04-15T15:15:37.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.735+0000 7f78dcfaf640 1 -- 172.21.15.72:0/633710707 learned_addr learned my addr 172.21.15.72:0/633710707 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:37.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.736+0000 7f78ddfb1640 1 --2- 172.21.15.72:0/633710707 >> v2:172.21.15.175:3300/0 conn(0x7f78d80cc7c0 0x7f78d8155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.736+0000 7f78cffff640 1 -- 172.21.15.72:0/633710707 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4003475997 0 0) 0x7f78d806bbb0 con 0x7f78d80d08f0 2024-04-15T15:15:37.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.736+0000 7f78cffff640 1 -- 172.21.15.72:0/633710707 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78c4003520 con 0x7f78d80d08f0 2024-04-15T15:15:37.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.736+0000 7f78cffff640 1 -- 172.21.15.72:0/633710707 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 282715919 0 0) 0x7f78c4003520 con 0x7f78d80d08f0 2024-04-15T15:15:37.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.736+0000 7f78cffff640 1 -- 172.21.15.72:0/633710707 >> v2:172.21.15.175:3300/0 conn(0x7f78d80cc7c0 msgr2=0x7f78d8155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:37.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.736+0000 7f78cffff640 1 --2- 172.21.15.72:0/633710707 >> v2:172.21.15.175:3300/0 conn(0x7f78d80cc7c0 0x7f78d8155600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.736+0000 7f78cffff640 1 -- 172.21.15.72:0/633710707 >> v1:172.21.15.175:6789/0 conn(0x7f78d80ce2f0 legacy=0x7f78d815c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.736+0000 7f78cffff640 1 -- 172.21.15.72:0/633710707 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78d806c170 con 0x7f78d80d08f0 2024-04-15T15:15:37.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.736+0000 7f78cffff640 1 -- 172.21.15.72:0/633710707 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f78c8001670 con 0x7f78d80d08f0 2024-04-15T15:15:37.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.736+0000 7f78cffff640 1 -- 172.21.15.72:0/633710707 >> v1:172.21.15.72:6789/0 conn(0x7f78d80d08f0 legacy=0x7f78d80cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.736+0000 7f78cffff640 1 --2- 172.21.15.72:0/633710707 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f78c40039b0 0x7f78c4003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.736+0000 7f78cffff640 1 --2- 172.21.15.72:0/633710707 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78c4004800 0x7f78c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.736+0000 7f78cffff640 1 --2- 172.21.15.72:0/633710707 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f78c40051c0 0x7f78c400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.736+0000 7f78cffff640 1 -- 172.21.15.72:0/633710707 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f78d806b960 con 0x7f78c4004800 2024-04-15T15:15:37.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.736+0000 7f78cffff640 1 -- 172.21.15.72:0/633710707 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f78c400c090 con 0x7f78c40039b0 2024-04-15T15:15:37.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.736+0000 7f78cffff640 1 -- 172.21.15.72:0/633710707 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f78c400c290 con 0x7f78c40051c0 2024-04-15T15:15:37.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.736+0000 7f78dcfaf640 1 --2- 172.21.15.72:0/633710707 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78c4004800 0x7f78c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.736+0000 7f78dd7b0640 1 --2- 172.21.15.72:0/633710707 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f78c40039b0 0x7f78c4003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.736+0000 7f78ddfb1640 1 --2- 172.21.15.72:0/633710707 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f78c40051c0 0x7f78c400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.736+0000 7f78dcfaf640 1 -- 172.21.15.72:0/633710707 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f78c40051c0 msgr2=0x7f78c400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:37.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.736+0000 7f78dcfaf640 1 --2- 172.21.15.72:0/633710707 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f78c40051c0 0x7f78c400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.736+0000 7f78dcfaf640 1 -- 172.21.15.72:0/633710707 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f78c40039b0 msgr2=0x7f78c4003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:37.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.736+0000 7f78dcfaf640 1 --2- 172.21.15.72:0/633710707 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f78c40039b0 0x7f78c4003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.736+0000 7f78dcfaf640 1 -- 172.21.15.72:0/633710707 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78c4003520 con 0x7f78c4004800 2024-04-15T15:15:37.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.736+0000 7f78ddfb1640 1 -- 172.21.15.72:0/633710707 reap_dead start 2024-04-15T15:15:37.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.737+0000 7f78dcfaf640 1 --2- 172.21.15.72:0/633710707 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78c4004800 0x7f78c4004c80 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f78c800c240 tx=0x7f78c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4474f5eaa3a7621a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:37.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.737+0000 7f78cffff640 1 -- 172.21.15.72:0/633710707 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c8007d10 con 0x7f78c4004800 2024-04-15T15:15:37.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.737+0000 7f78cffff640 1 -- 172.21.15.72:0/633710707 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78c8030070 con 0x7f78c4004800 2024-04-15T15:15:37.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.737+0000 7f78cffff640 1 -- 172.21.15.72:0/633710707 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c8005d50 con 0x7f78c4004800 2024-04-15T15:15:37.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.738+0000 7f78dfa14640 1 -- 172.21.15.72:0/633710707 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78c4004800 msgr2=0x7f78c4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.738+0000 7f78dfa14640 1 --2- 172.21.15.72:0/633710707 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78c4004800 0x7f78c4004c80 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f78c800c240 tx=0x7f78c4005830 comp rx=0 tx=0).stop 2024-04-15T15:15:37.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.738+0000 7f78dfa14640 1 -- 172.21.15.72:0/633710707 shutdown_connections 2024-04-15T15:15:37.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.738+0000 7f78dfa14640 1 --2- 172.21.15.72:0/633710707 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78c4004800 0x7f78c4004c80 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.738+0000 7f78dfa14640 1 -- 172.21.15.72:0/633710707 >> 172.21.15.72:0/633710707 conn(0x7f78d80b31d0 msgr2=0x7f78d80b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:37.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.738+0000 7f78dfa14640 1 -- 172.21.15.72:0/633710707 shutdown_connections 2024-04-15T15:15:37.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.738+0000 7f78dfa14640 1 -- 172.21.15.72:0/633710707 wait complete. 2024-04-15T15:15:37.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.739+0000 7f78dfa14640 1 Processor -- start 2024-04-15T15:15:37.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.739+0000 7f78dfa14640 1 -- start start 2024-04-15T15:15:37.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.740+0000 7f78dfa14640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f78c4004800 0x7f78d81457f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.740+0000 7f78dfa14640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78d8151240 0x7f78d80b2340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.740+0000 7f78dfa14640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f78d80b2880 0x7f78d80ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.740+0000 7f78dfa14640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f78d80ce850 con 0x7f78d8151240 2024-04-15T15:15:37.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.740+0000 7f78dfa14640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f78d80ce6d0 con 0x7f78c4004800 2024-04-15T15:15:37.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.740+0000 7f78dfa14640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f78d80ce9d0 con 0x7f78d80b2880 2024-04-15T15:15:37.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.740+0000 7f78dcfaf640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78d8151240 0x7f78d80b2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.740+0000 7f78dcfaf640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78d8151240 0x7f78d80b2340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54946/0 (socket says 172.21.15.72:54946) 2024-04-15T15:15:37.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.740+0000 7f78dcfaf640 1 -- 172.21.15.72:0/489972626 learned_addr learned my addr 172.21.15.72:0/489972626 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:37.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.740+0000 7f78ddfb1640 1 --2- 172.21.15.72:0/489972626 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f78d80b2880 0x7f78d80ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.740+0000 7f78dd7b0640 1 --2- 172.21.15.72:0/489972626 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f78c4004800 0x7f78d81457f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.740+0000 7f78dcfaf640 1 -- 172.21.15.72:0/489972626 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f78d80b2880 msgr2=0x7f78d80ccc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.740+0000 7f78dcfaf640 1 --2- 172.21.15.72:0/489972626 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f78d80b2880 0x7f78d80ccc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.740+0000 7f78dcfaf640 1 -- 172.21.15.72:0/489972626 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f78c4004800 msgr2=0x7f78d81457f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.740+0000 7f78dcfaf640 1 --2- 172.21.15.72:0/489972626 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f78c4004800 0x7f78d81457f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.740+0000 7f78dcfaf640 1 -- 172.21.15.72:0/489972626 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78d81501c0 con 0x7f78d8151240 2024-04-15T15:15:37.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.740+0000 7f78dcfaf640 1 --2- 172.21.15.72:0/489972626 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78d8151240 0x7f78d80b2340 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f78c802c4e0 tx=0x7f78d806c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:37.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.740+0000 7f78cdffb640 1 -- 172.21.15.72:0/489972626 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c8045070 con 0x7f78d8151240 2024-04-15T15:15:37.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.740+0000 7f78cdffb640 1 -- 172.21.15.72:0/489972626 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78c80049f0 con 0x7f78d8151240 2024-04-15T15:15:37.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.740+0000 7f78dfa14640 1 -- 172.21.15.72:0/489972626 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78d8150480 con 0x7f78d8151240 2024-04-15T15:15:37.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.740+0000 7f78cdffb640 1 -- 172.21.15.72:0/489972626 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c8007d10 con 0x7f78d8151240 2024-04-15T15:15:37.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.741+0000 7f78dfa14640 1 -- 172.21.15.72:0/489972626 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78d8150a40 con 0x7f78d8151240 2024-04-15T15:15:37.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.742+0000 7f78dfa14640 1 -- 172.21.15.72:0/489972626 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78d8075c50 con 0x7f78d8151240 2024-04-15T15:15:37.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.742+0000 7f78cdffb640 1 -- 172.21.15.72:0/489972626 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f78c8007070 con 0x7f78d8151240 2024-04-15T15:15:37.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.742+0000 7f78cdffb640 1 --2- 172.21.15.72:0/489972626 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f78bc06c8f0 0x7f78bc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.742+0000 7f78cdffb640 1 -- 172.21.15.72:0/489972626 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f78c80acbf0 con 0x7f78d8151240 2024-04-15T15:15:37.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.742+0000 7f78dd7b0640 1 --2- 172.21.15.72:0/489972626 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f78bc06c8f0 0x7f78bc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.747 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.743+0000 7f78dd7b0640 1 --2- 172.21.15.72:0/489972626 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f78bc06c8f0 0x7f78bc06edb0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f78d4000f80 tx=0x7f78d4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:37.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.747+0000 7f78cdffb640 1 -- 172.21.15.72:0/489972626 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f78c807ac20 con 0x7f78d8151240 2024-04-15T15:15:37.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.921+0000 7f78dfa14640 1 -- 172.21.15.72:0/489972626 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f78d806cab0 con 0x7f78d8151240 2024-04-15T15:15:37.923 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.921+0000 7f78cdffb640 1 -- 172.21.15.72:0/489972626 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f78c807a520 con 0x7f78d8151240 2024-04-15T15:15:37.923 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:37.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.923+0000 7f78dfa14640 1 -- 172.21.15.72:0/489972626 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f78bc06c8f0 msgr2=0x7f78bc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.923+0000 7f78dfa14640 1 --2- 172.21.15.72:0/489972626 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f78bc06c8f0 0x7f78bc06edb0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f78d4000f80 tx=0x7f78d4009040 comp rx=0 tx=0).stop 2024-04-15T15:15:37.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.923+0000 7f78dfa14640 1 -- 172.21.15.72:0/489972626 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78d8151240 msgr2=0x7f78d80b2340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.923+0000 7f78dfa14640 1 --2- 172.21.15.72:0/489972626 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78d8151240 0x7f78d80b2340 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f78c802c4e0 tx=0x7f78d806c730 comp rx=0 tx=0).stop 2024-04-15T15:15:37.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.923+0000 7f78dfa14640 1 -- 172.21.15.72:0/489972626 shutdown_connections 2024-04-15T15:15:37.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.923+0000 7f78dfa14640 1 --2- 172.21.15.72:0/489972626 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f78bc06c8f0 0x7f78bc06edb0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.923+0000 7f78dfa14640 1 --2- 172.21.15.72:0/489972626 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f78d80b2880 0x7f78d80ccc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.923+0000 7f78dfa14640 1 --2- 172.21.15.72:0/489972626 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78d8151240 0x7f78d80b2340 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.923+0000 7f78dfa14640 1 --2- 172.21.15.72:0/489972626 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f78c4004800 0x7f78d81457f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.923+0000 7f78dfa14640 1 -- 172.21.15.72:0/489972626 >> 172.21.15.72:0/489972626 conn(0x7f78d80b31d0 msgr2=0x7f78d8072920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:37.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.924+0000 7f78dfa14640 1 -- 172.21.15.72:0/489972626 shutdown_connections 2024-04-15T15:15:37.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:37.924+0000 7f78dfa14640 1 -- 172.21.15.72:0/489972626 wait complete. 2024-04-15T15:15:37.936 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:15:38.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.041+0000 7f5e74ed9640 1 Processor -- start 2024-04-15T15:15:38.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.041+0000 7f5e74ed9640 1 -- start start 2024-04-15T15:15:38.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.042+0000 7f5e74ed9640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5e7015af20 0x7f5e7006dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:38.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.042+0000 7f5e74ed9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5e7006bf20 0x7f5e7006c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:38.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.042+0000 7f5e74ed9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e7015c110 con 0x7f5e700ccb50 2024-04-15T15:15:38.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.042+0000 7f5e74ed9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f5e7006ba90 con 0x7f5e7006bf20 2024-04-15T15:15:38.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.042+0000 7f5e74ed9640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f5e700cfae0 con 0x7f5e7015af20 2024-04-15T15:15:38.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.042+0000 7f5e6ed9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5e7006bf20 0x7f5e7006c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:38.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.042+0000 7f5e6ed9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5e7006bf20 0x7f5e7006c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54948/0 (socket says 172.21.15.72:54948) 2024-04-15T15:15:38.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.042+0000 7f5e6ed9d640 1 -- 172.21.15.72:0/2351266190 learned_addr learned my addr 172.21.15.72:0/2351266190 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:38.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.042+0000 7f5e6e59c640 1 --1- 172.21.15.72:0/2351266190 >> v1:172.21.15.72:6789/0 conn(0x7f5e700ccb50 0x7f5e700ccf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50072/0 (socket says 172.21.15.72:50072) 2024-04-15T15:15:38.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.042+0000 7f5e6dd9b640 1 --2- 172.21.15.72:0/2351266190 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5e7015af20 0x7f5e7006dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:38.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.042+0000 7f5e6ed9d640 1 -- 172.21.15.72:0/2351266190 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5e7015af20 msgr2=0x7f5e7006dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:38.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.042+0000 7f5e6ed9d640 1 --2- 172.21.15.72:0/2351266190 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5e7015af20 0x7f5e7006dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.042+0000 7f5e6ed9d640 1 -- 172.21.15.72:0/2351266190 >> v1:172.21.15.72:6789/0 conn(0x7f5e700ccb50 legacy=0x7f5e700ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:38.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.043+0000 7f5e6ed9d640 1 -- 172.21.15.72:0/2351266190 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e700cfdc0 con 0x7f5e7006bf20 2024-04-15T15:15:38.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.043+0000 7f5e6dd9b640 1 --2- 172.21.15.72:0/2351266190 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5e7015af20 0x7f5e7006dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:38.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.043+0000 7f5e6ed9d640 1 --2- 172.21.15.72:0/2351266190 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e7006bf20 0x7f5e7006c3b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f5e6800d150 tx=0x7f5e6802e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddb4ef192caa9dc7 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:38.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.043+0000 7f5e6d59a640 1 -- 172.21.15.72:0/2351266190 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e6802ea80 con 0x7f5e7006bf20 2024-04-15T15:15:38.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.043+0000 7f5e6d59a640 1 -- 172.21.15.72:0/2351266190 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e6802ec20 con 0x7f5e7006bf20 2024-04-15T15:15:38.046 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.043+0000 7f5e6d59a640 1 -- 172.21.15.72:0/2351266190 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e68009600 con 0x7f5e7006bf20 2024-04-15T15:15:38.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.044+0000 7f5e74ed9640 1 -- 172.21.15.72:0/2351266190 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e7006bf20 msgr2=0x7f5e7006c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:38.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.044+0000 7f5e74ed9640 1 --2- 172.21.15.72:0/2351266190 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e7006bf20 0x7f5e7006c3b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f5e6800d150 tx=0x7f5e6802e0e0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.044+0000 7f5e74ed9640 1 -- 172.21.15.72:0/2351266190 shutdown_connections 2024-04-15T15:15:38.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.044+0000 7f5e74ed9640 1 --2- 172.21.15.72:0/2351266190 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e7006bf20 0x7f5e7006c3b0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.044+0000 7f5e74ed9640 1 --2- 172.21.15.72:0/2351266190 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5e7015af20 0x7f5e7006dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.044+0000 7f5e74ed9640 1 -- 172.21.15.72:0/2351266190 >> 172.21.15.72:0/2351266190 conn(0x7f5e700b31d0 msgr2=0x7f5e700b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:38.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.044+0000 7f5e74ed9640 1 -- 172.21.15.72:0/2351266190 shutdown_connections 2024-04-15T15:15:38.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.044+0000 7f5e74ed9640 1 -- 172.21.15.72:0/2351266190 wait complete. 2024-04-15T15:15:38.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.045+0000 7f5e74ed9640 1 Processor -- start 2024-04-15T15:15:38.049 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.045+0000 7f5e74ed9640 1 -- start start 2024-04-15T15:15:38.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.046+0000 7f5e74ed9640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5e7006bf20 0x7f5e7015ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:38.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.046+0000 7f5e74ed9640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5e700ccb50 0x7f5e701599f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:38.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.046+0000 7f5e74ed9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e7015af20 0x7f5e70158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:38.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.046+0000 7f5e74ed9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f5e700cc8f0 con 0x7f5e7015af20 2024-04-15T15:15:38.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.046+0000 7f5e74ed9640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f5e700cc770 con 0x7f5e700ccb50 2024-04-15T15:15:38.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.046+0000 7f5e74ed9640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f5e7015bcf0 con 0x7f5e7006bf20 2024-04-15T15:15:38.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.046+0000 7f5e6ed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e7015af20 0x7f5e70158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:38.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.046+0000 7f5e6ed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e7015af20 0x7f5e70158100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54950/0 (socket says 172.21.15.72:54950) 2024-04-15T15:15:38.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.046+0000 7f5e6ed9d640 1 -- 172.21.15.72:0/3709044384 learned_addr learned my addr 172.21.15.72:0/3709044384 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:38.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.046+0000 7f5e6e59c640 1 --2- 172.21.15.72:0/3709044384 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5e7006bf20 0x7f5e7015ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:38.051 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.046+0000 7f5e6dd9b640 1 --2- 172.21.15.72:0/3709044384 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5e700ccb50 0x7f5e701599f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:38.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.047+0000 7f5e6ed9d640 1 -- 172.21.15.72:0/3709044384 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5e7006bf20 msgr2=0x7f5e7015ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:38.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.047+0000 7f5e6ed9d640 1 --2- 172.21.15.72:0/3709044384 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5e7006bf20 0x7f5e7015ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.047+0000 7f5e6ed9d640 1 -- 172.21.15.72:0/3709044384 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5e700ccb50 msgr2=0x7f5e701599f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:38.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.047+0000 7f5e6ed9d640 1 --2- 172.21.15.72:0/3709044384 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5e700ccb50 0x7f5e701599f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.047+0000 7f5e6ed9d640 1 -- 172.21.15.72:0/3709044384 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e70158950 con 0x7f5e7015af20 2024-04-15T15:15:38.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.047+0000 7f5e6e59c640 1 --2- 172.21.15.72:0/3709044384 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5e7006bf20 0x7f5e7015ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:38.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.047+0000 7f5e6ed9d640 1 --2- 172.21.15.72:0/3709044384 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e7015af20 0x7f5e70158100 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f5e6800d150 tx=0x7f5e68004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:38.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.047+0000 7f5e5effd640 1 -- 172.21.15.72:0/3709044384 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e68045020 con 0x7f5e7015af20 2024-04-15T15:15:38.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.047+0000 7f5e5effd640 1 -- 172.21.15.72:0/3709044384 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e68040660 con 0x7f5e7015af20 2024-04-15T15:15:38.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.047+0000 7f5e5effd640 1 -- 172.21.15.72:0/3709044384 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e6802ea80 con 0x7f5e7015af20 2024-04-15T15:15:38.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.047+0000 7f5e74ed9640 1 -- 172.21.15.72:0/3709044384 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e70158c30 con 0x7f5e7015af20 2024-04-15T15:15:38.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.047+0000 7f5e74ed9640 1 -- 172.21.15.72:0/3709044384 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e701594d0 con 0x7f5e7015af20 2024-04-15T15:15:38.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.048+0000 7f5e74ed9640 1 -- 172.21.15.72:0/3709044384 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e70159020 con 0x7f5e7015af20 2024-04-15T15:15:38.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.048+0000 7f5e5effd640 1 -- 172.21.15.72:0/3709044384 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5e68048020 con 0x7f5e7015af20 2024-04-15T15:15:38.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.049+0000 7f5e5effd640 1 --2- 172.21.15.72:0/3709044384 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f5e5006c8f0 0x7f5e5006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:38.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.049+0000 7f5e5effd640 1 -- 172.21.15.72:0/3709044384 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f5e680b0a00 con 0x7f5e7015af20 2024-04-15T15:15:38.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.049+0000 7f5e6e59c640 1 --2- 172.21.15.72:0/3709044384 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f5e5006c8f0 0x7f5e5006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:38.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.049+0000 7f5e6e59c640 1 --2- 172.21.15.72:0/3709044384 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f5e5006c8f0 0x7f5e5006edb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5e60000f60 tx=0x7f5e60009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:38.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.053+0000 7f5e5effd640 1 -- 172.21.15.72:0/3709044384 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5e6804f030 con 0x7f5e7015af20 2024-04-15T15:15:38.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.228+0000 7f5e74ed9640 1 -- 172.21.15.72:0/3709044384 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5e70158e00 con 0x7f5e7015af20 2024-04-15T15:15:38.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.228+0000 7f5e5effd640 1 -- 172.21.15.72:0/3709044384 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f5e6807ea40 con 0x7f5e7015af20 2024-04-15T15:15:38.230 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-04-15T15:15:38.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.230+0000 7f5e74ed9640 1 -- 172.21.15.72:0/3709044384 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f5e5006c8f0 msgr2=0x7f5e5006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:38.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.230+0000 7f5e74ed9640 1 --2- 172.21.15.72:0/3709044384 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f5e5006c8f0 0x7f5e5006edb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5e60000f60 tx=0x7f5e60009040 comp rx=0 tx=0).stop 2024-04-15T15:15:38.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.230+0000 7f5e74ed9640 1 -- 172.21.15.72:0/3709044384 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e7015af20 msgr2=0x7f5e70158100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:38.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.230+0000 7f5e74ed9640 1 --2- 172.21.15.72:0/3709044384 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e7015af20 0x7f5e70158100 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f5e6800d150 tx=0x7f5e68004a40 comp rx=0 tx=0).stop 2024-04-15T15:15:38.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.230+0000 7f5e74ed9640 1 -- 172.21.15.72:0/3709044384 shutdown_connections 2024-04-15T15:15:38.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.230+0000 7f5e74ed9640 1 --2- 172.21.15.72:0/3709044384 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f5e5006c8f0 0x7f5e5006edb0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.230+0000 7f5e74ed9640 1 --2- 172.21.15.72:0/3709044384 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e7015af20 0x7f5e70158100 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.230+0000 7f5e74ed9640 1 --2- 172.21.15.72:0/3709044384 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5e700ccb50 0x7f5e701599f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.230+0000 7f5e74ed9640 1 --2- 172.21.15.72:0/3709044384 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5e7006bf20 0x7f5e7015ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.230+0000 7f5e74ed9640 1 -- 172.21.15.72:0/3709044384 >> 172.21.15.72:0/3709044384 conn(0x7f5e700b31d0 msgr2=0x7f5e70073270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:38.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.230+0000 7f5e74ed9640 1 -- 172.21.15.72:0/3709044384 shutdown_connections 2024-04-15T15:15:38.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.230+0000 7f5e74ed9640 1 -- 172.21.15.72:0/3709044384 wait complete. 2024-04-15T15:15:38.243 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:15:38.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.346+0000 7f689e8d3640 1 Processor -- start 2024-04-15T15:15:38.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.346+0000 7f689e8d3640 1 -- start start 2024-04-15T15:15:38.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.347+0000 7f689e8d3640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f689815e960 0x7f68980d1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:38.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.347+0000 7f689e8d3640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68980b65b0 con 0x7f68980d5010 2024-04-15T15:15:38.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.347+0000 7f689e8d3640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68980b6890 con 0x7f68980b11d0 2024-04-15T15:15:38.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.347+0000 7f689e8d3640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f68980b6b70 con 0x7f689815e960 2024-04-15T15:15:38.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.347+0000 7f689ce70640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f68980d5010 0x7f68980cffe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50074/0 (socket says 172.21.15.72:50074) 2024-04-15T15:15:38.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.347+0000 7f689ce70640 1 -- 172.21.15.72:0/72460779 learned_addr learned my addr 172.21.15.72:0/72460779 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:38.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.348+0000 7f6897fff640 1 --2- 172.21.15.72:0/72460779 >> v2:172.21.15.175:3300/0 conn(0x7f689815e960 0x7f68980d1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:38.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.348+0000 7f6896ffd640 1 -- 172.21.15.72:0/72460779 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1434440817 0 0) 0x7f68980b65b0 con 0x7f68980d5010 2024-04-15T15:15:38.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.348+0000 7f6896ffd640 1 -- 172.21.15.72:0/72460779 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6880003520 con 0x7f68980d5010 2024-04-15T15:15:38.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.348+0000 7f6896ffd640 1 -- 172.21.15.72:0/72460779 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 236935363 0 0) 0x7f6880003520 con 0x7f68980d5010 2024-04-15T15:15:38.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.348+0000 7f6896ffd640 1 -- 172.21.15.72:0/72460779 >> v2:172.21.15.175:3300/0 conn(0x7f689815e960 msgr2=0x7f68980d1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:38.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.348+0000 7f6896ffd640 1 --2- 172.21.15.72:0/72460779 >> v2:172.21.15.175:3300/0 conn(0x7f689815e960 0x7f68980d1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.348+0000 7f6896ffd640 1 -- 172.21.15.72:0/72460779 >> v1:172.21.15.175:6789/0 conn(0x7f68980b11d0 legacy=0x7f68980d4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:38.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.348+0000 7f6896ffd640 1 -- 172.21.15.72:0/72460779 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68980b6e50 con 0x7f68980d5010 2024-04-15T15:15:38.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.348+0000 7f6896ffd640 1 -- 172.21.15.72:0/72460779 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f68900019f0 con 0x7f68980d5010 2024-04-15T15:15:38.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.348+0000 7f6896ffd640 1 -- 172.21.15.72:0/72460779 >> v1:172.21.15.72:6789/0 conn(0x7f68980d5010 legacy=0x7f68980cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:38.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.348+0000 7f6896ffd640 1 --2- 172.21.15.72:0/72460779 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f68800039b0 0x7f6880003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:38.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.348+0000 7f6896ffd640 1 --2- 172.21.15.72:0/72460779 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6880004800 0x7f6880004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:38.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.348+0000 7f6896ffd640 1 --2- 172.21.15.72:0/72460779 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f68800051c0 0x7f688000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:38.353 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.348+0000 7f6896ffd640 1 -- 172.21.15.72:0/72460779 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f68980b6b70 con 0x7f6880004800 2024-04-15T15:15:38.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.348+0000 7f6896ffd640 1 -- 172.21.15.72:0/72460779 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f688000c060 con 0x7f68800039b0 2024-04-15T15:15:38.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.348+0000 7f6896ffd640 1 -- 172.21.15.72:0/72460779 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f688000c260 con 0x7f68800051c0 2024-04-15T15:15:38.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.348+0000 7f68977fe640 1 --2- 172.21.15.72:0/72460779 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6880004800 0x7f6880004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:38.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.348+0000 7f6897fff640 1 --2- 172.21.15.72:0/72460779 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f68800039b0 0x7f6880003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:38.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.348+0000 7f689ce70640 1 --2- 172.21.15.72:0/72460779 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f68800051c0 0x7f688000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:38.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.348+0000 7f68977fe640 1 -- 172.21.15.72:0/72460779 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f68800051c0 msgr2=0x7f688000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:38.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.348+0000 7f68977fe640 1 --2- 172.21.15.72:0/72460779 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f68800051c0 0x7f688000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.348+0000 7f68977fe640 1 -- 172.21.15.72:0/72460779 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f68800039b0 msgr2=0x7f6880003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:38.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.348+0000 7f68977fe640 1 --2- 172.21.15.72:0/72460779 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f68800039b0 0x7f6880003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.348+0000 7f68977fe640 1 -- 172.21.15.72:0/72460779 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6880003520 con 0x7f6880004800 2024-04-15T15:15:38.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.348+0000 7f689ce70640 1 -- 172.21.15.72:0/72460779 reap_dead start 2024-04-15T15:15:38.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.349+0000 7f68977fe640 1 --2- 172.21.15.72:0/72460779 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6880004800 0x7f6880004c80 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f688c001720 tx=0x7f6880005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aff7660376ba0933 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:38.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.349+0000 7f6896ffd640 1 -- 172.21.15.72:0/72460779 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688c001370 con 0x7f6880004800 2024-04-15T15:15:38.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.349+0000 7f6896ffd640 1 -- 172.21.15.72:0/72460779 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f688c0052d0 con 0x7f6880004800 2024-04-15T15:15:38.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.349+0000 7f6896ffd640 1 -- 172.21.15.72:0/72460779 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688c003830 con 0x7f6880004800 2024-04-15T15:15:38.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.350+0000 7f689e8d3640 1 -- 172.21.15.72:0/72460779 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6880004800 msgr2=0x7f6880004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:38.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.350+0000 7f689e8d3640 1 --2- 172.21.15.72:0/72460779 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6880004800 0x7f6880004c80 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f688c001720 tx=0x7f6880005830 comp rx=0 tx=0).stop 2024-04-15T15:15:38.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.350+0000 7f689e8d3640 1 -- 172.21.15.72:0/72460779 shutdown_connections 2024-04-15T15:15:38.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.350+0000 7f689e8d3640 1 --2- 172.21.15.72:0/72460779 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6880004800 0x7f6880004c80 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.350+0000 7f689e8d3640 1 -- 172.21.15.72:0/72460779 >> 172.21.15.72:0/72460779 conn(0x7f68980b9580 msgr2=0x7f68980b9990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:38.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.350+0000 7f689e8d3640 1 -- 172.21.15.72:0/72460779 shutdown_connections 2024-04-15T15:15:38.356 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.350+0000 7f689e8d3640 1 -- 172.21.15.72:0/72460779 wait complete. 2024-04-15T15:15:38.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.351+0000 7f689e8d3640 1 Processor -- start 2024-04-15T15:15:38.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.351+0000 7f689e8d3640 1 -- start start 2024-04-15T15:15:38.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.352+0000 7f689e8d3640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6880004800 0x7f689806b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:38.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.352+0000 7f689e8d3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6898146480 0x7f6898161700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:38.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.352+0000 7f689e8d3640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f68980cdd70 0x7f6898145700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:38.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.352+0000 7f689e8d3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f68980b61c0 con 0x7f6898146480 2024-04-15T15:15:38.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.352+0000 7f689e8d3640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f68980b6040 con 0x7f6880004800 2024-04-15T15:15:38.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.352+0000 7f689e8d3640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f68980b6340 con 0x7f68980cdd70 2024-04-15T15:15:38.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.352+0000 7f68977fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6898146480 0x7f6898161700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:38.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.352+0000 7f68977fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6898146480 0x7f6898161700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54980/0 (socket says 172.21.15.72:54980) 2024-04-15T15:15:38.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.352+0000 7f68977fe640 1 -- 172.21.15.72:0/3140373142 learned_addr learned my addr 172.21.15.72:0/3140373142 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:38.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.352+0000 7f6897fff640 1 --2- 172.21.15.72:0/3140373142 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6880004800 0x7f689806b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:38.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.352+0000 7f689ce70640 1 --2- 172.21.15.72:0/3140373142 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f68980cdd70 0x7f6898145700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:38.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.352+0000 7f68977fe640 1 -- 172.21.15.72:0/3140373142 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f68980cdd70 msgr2=0x7f6898145700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:38.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.352+0000 7f68977fe640 1 --2- 172.21.15.72:0/3140373142 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f68980cdd70 0x7f6898145700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.352+0000 7f68977fe640 1 -- 172.21.15.72:0/3140373142 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6880004800 msgr2=0x7f689806b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:38.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.352+0000 7f68977fe640 1 --2- 172.21.15.72:0/3140373142 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6880004800 0x7f689806b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.352+0000 7f68977fe640 1 -- 172.21.15.72:0/3140373142 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f689806bea0 con 0x7f6898146480 2024-04-15T15:15:38.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.353+0000 7f68977fe640 1 --2- 172.21.15.72:0/3140373142 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6898146480 0x7f6898161700 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f688c005040 tx=0x7f688c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:38.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.353+0000 7f6894ff9640 1 -- 172.21.15.72:0/3140373142 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688c041020 con 0x7f6898146480 2024-04-15T15:15:38.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.353+0000 7f6894ff9640 1 -- 172.21.15.72:0/3140373142 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f688c005480 con 0x7f6898146480 2024-04-15T15:15:38.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.353+0000 7f689e8d3640 1 -- 172.21.15.72:0/3140373142 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f689806e720 con 0x7f6898146480 2024-04-15T15:15:38.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.353+0000 7f6894ff9640 1 -- 172.21.15.72:0/3140373142 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f688c007a30 con 0x7f6898146480 2024-04-15T15:15:38.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.353+0000 7f689e8d3640 1 -- 172.21.15.72:0/3140373142 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68980cd570 con 0x7f6898146480 2024-04-15T15:15:38.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.354+0000 7f689e8d3640 1 -- 172.21.15.72:0/3140373142 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6898075c50 con 0x7f6898146480 2024-04-15T15:15:38.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.354+0000 7f6894ff9640 1 -- 172.21.15.72:0/3140373142 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f688c044020 con 0x7f6898146480 2024-04-15T15:15:38.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.355+0000 7f6894ff9640 1 --2- 172.21.15.72:0/3140373142 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f688406c920 0x7f688406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:38.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.355+0000 7f6897fff640 1 --2- 172.21.15.72:0/3140373142 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f688406c920 0x7f688406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:38.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.355+0000 7f6894ff9640 1 -- 172.21.15.72:0/3140373142 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f688c0ad8d0 con 0x7f6898146480 2024-04-15T15:15:38.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.355+0000 7f6897fff640 1 --2- 172.21.15.72:0/3140373142 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f688406c920 0x7f688406ede0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f68980b5080 tx=0x7f6888009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:38.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.359+0000 7f6894ff9640 1 -- 172.21.15.72:0/3140373142 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f688c04b030 con 0x7f6898146480 2024-04-15T15:15:38.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.534+0000 7f689e8d3640 1 -- 172.21.15.72:0/3140373142 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f68980d4c80 con 0x7f6898146480 2024-04-15T15:15:38.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.534+0000 7f6894ff9640 1 -- 172.21.15.72:0/3140373142 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f688c0abe20 con 0x7f6898146480 2024-04-15T15:15:38.536 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:38.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.536+0000 7f689e8d3640 1 -- 172.21.15.72:0/3140373142 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f688406c920 msgr2=0x7f688406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:38.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.536+0000 7f689e8d3640 1 --2- 172.21.15.72:0/3140373142 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f688406c920 0x7f688406ede0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f68980b5080 tx=0x7f6888009190 comp rx=0 tx=0).stop 2024-04-15T15:15:38.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.536+0000 7f689e8d3640 1 -- 172.21.15.72:0/3140373142 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6898146480 msgr2=0x7f6898161700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:38.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.536+0000 7f689e8d3640 1 --2- 172.21.15.72:0/3140373142 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6898146480 0x7f6898161700 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f688c005040 tx=0x7f688c001120 comp rx=0 tx=0).stop 2024-04-15T15:15:38.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.536+0000 7f689e8d3640 1 -- 172.21.15.72:0/3140373142 shutdown_connections 2024-04-15T15:15:38.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.536+0000 7f689e8d3640 1 --2- 172.21.15.72:0/3140373142 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f688406c920 0x7f688406ede0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.536+0000 7f689e8d3640 1 --2- 172.21.15.72:0/3140373142 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f68980cdd70 0x7f6898145700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.536+0000 7f689e8d3640 1 --2- 172.21.15.72:0/3140373142 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6898146480 0x7f6898161700 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.536+0000 7f689e8d3640 1 --2- 172.21.15.72:0/3140373142 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6880004800 0x7f689806b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.536+0000 7f689e8d3640 1 -- 172.21.15.72:0/3140373142 >> 172.21.15.72:0/3140373142 conn(0x7f68980b9580 msgr2=0x7f68980b4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:38.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.536+0000 7f689e8d3640 1 -- 172.21.15.72:0/3140373142 shutdown_connections 2024-04-15T15:15:38.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.536+0000 7f689e8d3640 1 -- 172.21.15.72:0/3140373142 wait complete. 2024-04-15T15:15:38.549 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:15:38.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.651+0000 7f7dacb00640 1 Processor -- start 2024-04-15T15:15:38.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.651+0000 7f7dacb00640 1 -- start start 2024-04-15T15:15:38.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.652+0000 7f7dacb00640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7da80cfe30 0x7f7da80cf040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:38.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.652+0000 7f7dacb00640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7da80ae160 con 0x7f7da815e770 2024-04-15T15:15:38.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.652+0000 7f7dacb00640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7da80b5310 con 0x7f7da80b4dc0 2024-04-15T15:15:38.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.652+0000 7f7dacb00640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f7da80cf580 con 0x7f7da80cfe30 2024-04-15T15:15:38.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.652+0000 7f7da659c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f7da815e770 0x7f7da80cf720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50090/0 (socket says 172.21.15.72:50090) 2024-04-15T15:15:38.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.652+0000 7f7da659c640 1 -- 172.21.15.72:0/1466294503 learned_addr learned my addr 172.21.15.72:0/1466294503 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:38.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.652+0000 7f7d9dd9b640 1 --2- 172.21.15.72:0/1466294503 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7da80cfe30 0x7f7da80cf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:38.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.652+0000 7f7da5d9b640 1 -- 172.21.15.72:0/1466294503 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2869321254 0 0) 0x7f7da80ae160 con 0x7f7da815e770 2024-04-15T15:15:38.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.652+0000 7f7da5d9b640 1 -- 172.21.15.72:0/1466294503 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d94003520 con 0x7f7da815e770 2024-04-15T15:15:38.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.652+0000 7f7da5d9b640 1 -- 172.21.15.72:0/1466294503 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2466421623 0 0) 0x7f7d94003520 con 0x7f7da815e770 2024-04-15T15:15:38.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.652+0000 7f7da5d9b640 1 -- 172.21.15.72:0/1466294503 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7da80cfe30 msgr2=0x7f7da80cf040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:38.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.652+0000 7f7da5d9b640 1 --2- 172.21.15.72:0/1466294503 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7da80cfe30 0x7f7da80cf040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.652+0000 7f7da5d9b640 1 -- 172.21.15.72:0/1466294503 >> v1:172.21.15.175:6789/0 conn(0x7f7da80b4dc0 legacy=0x7f7da80ae600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:38.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.652+0000 7f7da5d9b640 1 -- 172.21.15.72:0/1466294503 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7da80b55f0 con 0x7f7da815e770 2024-04-15T15:15:38.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.652+0000 7f7da5d9b640 1 -- 172.21.15.72:0/1466294503 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f7d98001670 con 0x7f7da815e770 2024-04-15T15:15:38.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.652+0000 7f7da5d9b640 1 -- 172.21.15.72:0/1466294503 >> v1:172.21.15.72:6789/0 conn(0x7f7da815e770 legacy=0x7f7da80cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:38.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.652+0000 7f7da5d9b640 1 --2- 172.21.15.72:0/1466294503 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d940039e0 0x7f7d94003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:38.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.652+0000 7f7da5d9b640 1 --2- 172.21.15.72:0/1466294503 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7d94004390 0x7f7d94004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:38.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.652+0000 7f7da5d9b640 1 --2- 172.21.15.72:0/1466294503 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7da80cfe30 0x7f7d940050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:38.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.652+0000 7f7da5d9b640 1 -- 172.21.15.72:0/1466294503 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f7da80cf580 con 0x7f7d940039e0 2024-04-15T15:15:38.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.652+0000 7f7da5d9b640 1 -- 172.21.15.72:0/1466294503 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f7d9400b9c0 con 0x7f7d94004390 2024-04-15T15:15:38.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.652+0000 7f7da5d9b640 1 -- 172.21.15.72:0/1466294503 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f7d9400bbc0 con 0x7f7da80cfe30 2024-04-15T15:15:38.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.653+0000 7f7da659c640 1 --2- 172.21.15.72:0/1466294503 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d940039e0 0x7f7d94003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:38.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.653+0000 7f7da6d9d640 1 --2- 172.21.15.72:0/1466294503 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7da80cfe30 0x7f7d940050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:38.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.653+0000 7f7d9dd9b640 1 --2- 172.21.15.72:0/1466294503 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7d94004390 0x7f7d94004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:38.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.653+0000 7f7da659c640 1 -- 172.21.15.72:0/1466294503 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7da80cfe30 msgr2=0x7f7d940050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:38.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.653+0000 7f7da659c640 1 --2- 172.21.15.72:0/1466294503 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7da80cfe30 0x7f7d940050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.653+0000 7f7da659c640 1 -- 172.21.15.72:0/1466294503 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7d94004390 msgr2=0x7f7d94004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:38.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.653+0000 7f7da659c640 1 --2- 172.21.15.72:0/1466294503 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7d94004390 0x7f7d94004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.653+0000 7f7da659c640 1 -- 172.21.15.72:0/1466294503 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d94003520 con 0x7f7d940039e0 2024-04-15T15:15:38.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.653+0000 7f7da6d9d640 1 --2- 172.21.15.72:0/1466294503 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7da80cfe30 0x7f7d940050c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:38.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.653+0000 7f7d9dd9b640 1 --2- 172.21.15.72:0/1466294503 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7d94004390 0x7f7d94004860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:38.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.654+0000 7f7da659c640 1 --2- 172.21.15.72:0/1466294503 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d940039e0 0x7f7d94003dc0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f7d9800c240 tx=0x7f7d94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a666917816192c90 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:38.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.654+0000 7f7da5d9b640 1 -- 172.21.15.72:0/1466294503 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d98007d30 con 0x7f7d940039e0 2024-04-15T15:15:38.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.654+0000 7f7da5d9b640 1 -- 172.21.15.72:0/1466294503 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d98030070 con 0x7f7d940039e0 2024-04-15T15:15:38.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.654+0000 7f7da5d9b640 1 -- 172.21.15.72:0/1466294503 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d9802cd60 con 0x7f7d940039e0 2024-04-15T15:15:38.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.654+0000 7f7dacb00640 1 -- 172.21.15.72:0/1466294503 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d940039e0 msgr2=0x7f7d94003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:38.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.654+0000 7f7dacb00640 1 --2- 172.21.15.72:0/1466294503 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d940039e0 0x7f7d94003dc0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f7d9800c240 tx=0x7f7d94005830 comp rx=0 tx=0).stop 2024-04-15T15:15:38.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.654+0000 7f7da6d9d640 1 -- 172.21.15.72:0/1466294503 reap_dead start 2024-04-15T15:15:38.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.654+0000 7f7dacb00640 1 -- 172.21.15.72:0/1466294503 shutdown_connections 2024-04-15T15:15:38.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.654+0000 7f7dacb00640 1 -- 172.21.15.72:0/1466294503 >> 172.21.15.72:0/1466294503 conn(0x7f7da80d13a0 msgr2=0x7f7da80d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:38.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.654+0000 7f7dacb00640 1 -- 172.21.15.72:0/1466294503 shutdown_connections 2024-04-15T15:15:38.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.654+0000 7f7dacb00640 1 -- 172.21.15.72:0/1466294503 wait complete. 2024-04-15T15:15:38.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.656+0000 7f7dacb00640 1 Processor -- start 2024-04-15T15:15:38.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.656+0000 7f7dacb00640 1 -- start start 2024-04-15T15:15:38.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.656+0000 7f7dacb00640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7d940039e0 0x7f7da814b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:38.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.656+0000 7f7dacb00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7da8148cf0 0x7f7da80cdcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:38.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.656+0000 7f7dacb00640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7da80ce200 0x7f7da814c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:38.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.656+0000 7f7dacb00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f7da815eb50 con 0x7f7da8148cf0 2024-04-15T15:15:38.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.656+0000 7f7dacb00640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f7da80b5bb0 con 0x7f7d940039e0 2024-04-15T15:15:38.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.656+0000 7f7dacb00640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f7da815ecd0 con 0x7f7da80ce200 2024-04-15T15:15:38.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.657+0000 7f7d9dd9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7da8148cf0 0x7f7da80cdcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:38.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.657+0000 7f7d9dd9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7da8148cf0 0x7f7da80cdcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55002/0 (socket says 172.21.15.72:55002) 2024-04-15T15:15:38.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.657+0000 7f7d9dd9b640 1 -- 172.21.15.72:0/2325475394 learned_addr learned my addr 172.21.15.72:0/2325475394 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:38.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.657+0000 7f7da6d9d640 1 --2- 172.21.15.72:0/2325475394 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7da80ce200 0x7f7da814c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:38.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.657+0000 7f7d9dd9b640 1 -- 172.21.15.72:0/2325475394 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7da80ce200 msgr2=0x7f7da814c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:38.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.657+0000 7f7da659c640 1 --2- 172.21.15.72:0/2325475394 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7d940039e0 0x7f7da814b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:38.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.657+0000 7f7d9dd9b640 1 --2- 172.21.15.72:0/2325475394 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7da80ce200 0x7f7da814c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.657+0000 7f7d9dd9b640 1 -- 172.21.15.72:0/2325475394 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7d940039e0 msgr2=0x7f7da814b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:38.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.657+0000 7f7d9dd9b640 1 --2- 172.21.15.72:0/2325475394 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7d940039e0 0x7f7da814b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.657+0000 7f7d9dd9b640 1 -- 172.21.15.72:0/2325475394 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7da814bca0 con 0x7f7da8148cf0 2024-04-15T15:15:38.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.657+0000 7f7da6d9d640 1 --2- 172.21.15.72:0/2325475394 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7da80ce200 0x7f7da814c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:38.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.657+0000 7f7da659c640 1 --2- 172.21.15.72:0/2325475394 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7d940039e0 0x7f7da814b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:38.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.657+0000 7f7d9dd9b640 1 --2- 172.21.15.72:0/2325475394 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7da8148cf0 0x7f7da80cdcc0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f7d900116d0 tx=0x7f7d90011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:38.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.657+0000 7f7d9f7fe640 1 -- 172.21.15.72:0/2325475394 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d9000f9c0 con 0x7f7da8148cf0 2024-04-15T15:15:38.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.657+0000 7f7d9f7fe640 1 -- 172.21.15.72:0/2325475394 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d9000fb60 con 0x7f7da8148cf0 2024-04-15T15:15:38.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.657+0000 7f7d9f7fe640 1 -- 172.21.15.72:0/2325475394 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d90015460 con 0x7f7da8148cf0 2024-04-15T15:15:38.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.657+0000 7f7dacb00640 1 -- 172.21.15.72:0/2325475394 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7da8149170 con 0x7f7da8148cf0 2024-04-15T15:15:38.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.657+0000 7f7dacb00640 1 -- 172.21.15.72:0/2325475394 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7da8149730 con 0x7f7da8148cf0 2024-04-15T15:15:38.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.658+0000 7f7dacb00640 1 -- 172.21.15.72:0/2325475394 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7da8075c50 con 0x7f7da8148cf0 2024-04-15T15:15:38.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.659+0000 7f7d9f7fe640 1 -- 172.21.15.72:0/2325475394 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7d90015600 con 0x7f7da8148cf0 2024-04-15T15:15:38.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.659+0000 7f7d9f7fe640 1 --2- 172.21.15.72:0/2325475394 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7d8806c920 0x7f7d8806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:38.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.659+0000 7f7d9f7fe640 1 -- 172.21.15.72:0/2325475394 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f7d9008c860 con 0x7f7da8148cf0 2024-04-15T15:15:38.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.659+0000 7f7da659c640 1 --2- 172.21.15.72:0/2325475394 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7d8806c920 0x7f7d8806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:38.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.660+0000 7f7da659c640 1 --2- 172.21.15.72:0/2325475394 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7d8806c920 0x7f7d8806ede0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7d9800c240 tx=0x7f7d98009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:38.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.663+0000 7f7d9f7fe640 1 -- 172.21.15.72:0/2325475394 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7d90019030 con 0x7f7da8148cf0 2024-04-15T15:15:38.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.851+0000 7f7dacb00640 1 -- 172.21.15.72:0/2325475394 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f7da806f8f0 con 0x7f7da8148cf0 2024-04-15T15:15:38.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.851+0000 7f7d9f7fe640 1 -- 172.21.15.72:0/2325475394 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7f7d9005a800 con 0x7f7da8148cf0 2024-04-15T15:15:38.856 INFO:teuthology.orchestra.run.smithi072.stderr:listed 0 entries 2024-04-15T15:15:38.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.853+0000 7f7dacb00640 1 -- 172.21.15.72:0/2325475394 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7d8806c920 msgr2=0x7f7d8806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:38.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.853+0000 7f7dacb00640 1 --2- 172.21.15.72:0/2325475394 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7d8806c920 0x7f7d8806ede0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7d9800c240 tx=0x7f7d98009030 comp rx=0 tx=0).stop 2024-04-15T15:15:38.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.853+0000 7f7dacb00640 1 -- 172.21.15.72:0/2325475394 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7da8148cf0 msgr2=0x7f7da80cdcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:38.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.853+0000 7f7dacb00640 1 --2- 172.21.15.72:0/2325475394 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7da8148cf0 0x7f7da80cdcc0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f7d900116d0 tx=0x7f7d90011b90 comp rx=0 tx=0).stop 2024-04-15T15:15:38.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.853+0000 7f7dacb00640 1 -- 172.21.15.72:0/2325475394 shutdown_connections 2024-04-15T15:15:38.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.853+0000 7f7dacb00640 1 --2- 172.21.15.72:0/2325475394 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7d8806c920 0x7f7d8806ede0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.853+0000 7f7dacb00640 1 --2- 172.21.15.72:0/2325475394 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7da80ce200 0x7f7da814c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.853+0000 7f7dacb00640 1 --2- 172.21.15.72:0/2325475394 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7da8148cf0 0x7f7da80cdcc0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.853+0000 7f7dacb00640 1 --2- 172.21.15.72:0/2325475394 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7d940039e0 0x7f7da814b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.853+0000 7f7dacb00640 1 -- 172.21.15.72:0/2325475394 >> 172.21.15.72:0/2325475394 conn(0x7f7da80d13a0 msgr2=0x7f7da806e200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:38.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.853+0000 7f7dacb00640 1 -- 172.21.15.72:0/2325475394 shutdown_connections 2024-04-15T15:15:38.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:38.853+0000 7f7dacb00640 1 -- 172.21.15.72:0/2325475394 wait complete. 2024-04-15T15:15:38.866 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:15:38.866 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:15:38.882 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:39.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.023+0000 7efc318c8640 1 Processor -- start 2024-04-15T15:15:39.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.023+0000 7efc318c8640 1 -- start start 2024-04-15T15:15:39.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.024+0000 7efc318c8640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efc2c0d45b0 0x7efc2c0d4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.024+0000 7efc318c8640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc2c162220 con 0x7efc2c161e40 2024-04-15T15:15:39.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.024+0000 7efc318c8640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc2c0d3070 con 0x7efc2c0d12b0 2024-04-15T15:15:39.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.024+0000 7efc318c8640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7efc2c0b3f00 con 0x7efc2c0d45b0 2024-04-15T15:15:39.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.024+0000 7efc2affd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7efc2c161e40 0x7efc2c0b37f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50098/0 (socket says 172.21.15.72:50098) 2024-04-15T15:15:39.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.024+0000 7efc2affd640 1 -- 172.21.15.72:0/3941472355 learned_addr learned my addr 172.21.15.72:0/3941472355 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:39.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.024+0000 7efc2b7fe640 1 --2- 172.21.15.72:0/3941472355 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efc2c0d45b0 0x7efc2c0d4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.024+0000 7efc29ffb640 1 -- 172.21.15.72:0/3941472355 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4163626287 0 0) 0x7efc2c162220 con 0x7efc2c161e40 2024-04-15T15:15:39.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.024+0000 7efc29ffb640 1 -- 172.21.15.72:0/3941472355 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc14003520 con 0x7efc2c161e40 2024-04-15T15:15:39.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.025+0000 7efc29ffb640 1 -- 172.21.15.72:0/3941472355 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 258131444 0 0) 0x7efc14003520 con 0x7efc2c161e40 2024-04-15T15:15:39.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.025+0000 7efc29ffb640 1 -- 172.21.15.72:0/3941472355 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efc2c0d45b0 msgr2=0x7efc2c0d4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:39.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.025+0000 7efc29ffb640 1 --2- 172.21.15.72:0/3941472355 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efc2c0d45b0 0x7efc2c0d4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.025+0000 7efc29ffb640 1 -- 172.21.15.72:0/3941472355 >> v1:172.21.15.175:6789/0 conn(0x7efc2c0d12b0 legacy=0x7efc2c0d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.025+0000 7efc29ffb640 1 -- 172.21.15.72:0/3941472355 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc2c0d1820 con 0x7efc2c161e40 2024-04-15T15:15:39.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.025+0000 7efc29ffb640 1 -- 172.21.15.72:0/3941472355 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7efc1c001670 con 0x7efc2c161e40 2024-04-15T15:15:39.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.025+0000 7efc29ffb640 1 -- 172.21.15.72:0/3941472355 >> v1:172.21.15.72:6789/0 conn(0x7efc2c161e40 legacy=0x7efc2c0b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.025+0000 7efc29ffb640 1 --2- 172.21.15.72:0/3941472355 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc140039e0 0x7efc14003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.025+0000 7efc29ffb640 1 --2- 172.21.15.72:0/3941472355 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efc2c0d45b0 0x7efc14004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.025+0000 7efc29ffb640 1 --2- 172.21.15.72:0/3941472355 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc14004d40 0x7efc140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.025+0000 7efc29ffb640 1 -- 172.21.15.72:0/3941472355 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7efc2c0b3f00 con 0x7efc14004d40 2024-04-15T15:15:39.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.025+0000 7efc29ffb640 1 -- 172.21.15.72:0/3941472355 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7efc1400baf0 con 0x7efc140039e0 2024-04-15T15:15:39.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.025+0000 7efc29ffb640 1 -- 172.21.15.72:0/3941472355 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7efc1400bcf0 con 0x7efc2c0d45b0 2024-04-15T15:15:39.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.025+0000 7efc2b7fe640 1 --2- 172.21.15.72:0/3941472355 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc14004d40 0x7efc140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.025+0000 7efc2affd640 1 --2- 172.21.15.72:0/3941472355 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc140039e0 0x7efc14003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.025+0000 7efc2a7fc640 1 --2- 172.21.15.72:0/3941472355 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efc2c0d45b0 0x7efc14004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.025+0000 7efc2b7fe640 1 -- 172.21.15.72:0/3941472355 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efc2c0d45b0 msgr2=0x7efc14004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:39.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.025+0000 7efc2b7fe640 1 --2- 172.21.15.72:0/3941472355 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efc2c0d45b0 0x7efc14004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.025+0000 7efc2b7fe640 1 -- 172.21.15.72:0/3941472355 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc140039e0 msgr2=0x7efc14003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:39.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.025+0000 7efc2b7fe640 1 --2- 172.21.15.72:0/3941472355 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc140039e0 0x7efc14003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.025+0000 7efc2b7fe640 1 -- 172.21.15.72:0/3941472355 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc14003520 con 0x7efc14004d40 2024-04-15T15:15:39.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.026+0000 7efc2b7fe640 1 --2- 172.21.15.72:0/3941472355 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc14004d40 0x7efc140051c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7efc24009fc0 tx=0x7efc14005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9590f328687f6ff server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:39.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.026+0000 7efc29ffb640 1 -- 172.21.15.72:0/3941472355 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc24039070 con 0x7efc14004d40 2024-04-15T15:15:39.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.026+0000 7efc29ffb640 1 -- 172.21.15.72:0/3941472355 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc2402cc20 con 0x7efc14004d40 2024-04-15T15:15:39.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.026+0000 7efc29ffb640 1 -- 172.21.15.72:0/3941472355 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc24009c10 con 0x7efc14004d40 2024-04-15T15:15:39.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.027+0000 7efc318c8640 1 -- 172.21.15.72:0/3941472355 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc14004d40 msgr2=0x7efc140051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.027+0000 7efc318c8640 1 --2- 172.21.15.72:0/3941472355 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc14004d40 0x7efc140051c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7efc24009fc0 tx=0x7efc14005830 comp rx=0 tx=0).stop 2024-04-15T15:15:39.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.027+0000 7efc2b7fe640 1 -- 172.21.15.72:0/3941472355 reap_dead start 2024-04-15T15:15:39.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.027+0000 7efc318c8640 1 -- 172.21.15.72:0/3941472355 shutdown_connections 2024-04-15T15:15:39.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.027+0000 7efc318c8640 1 -- 172.21.15.72:0/3941472355 >> 172.21.15.72:0/3941472355 conn(0x7efc2c09d970 msgr2=0x7efc2c0b95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:39.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.027+0000 7efc318c8640 1 -- 172.21.15.72:0/3941472355 shutdown_connections 2024-04-15T15:15:39.038 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.027+0000 7efc318c8640 1 -- 172.21.15.72:0/3941472355 wait complete. 2024-04-15T15:15:39.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.028+0000 7efc318c8640 1 Processor -- start 2024-04-15T15:15:39.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.028+0000 7efc318c8640 1 -- start start 2024-04-15T15:15:39.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.029+0000 7efc318c8640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc2c156f00 0x7efc2c0ce550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.040 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.029+0000 7efc318c8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc2c15d1c0 0x7efc2c15d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.029+0000 7efc318c8640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efc2c06b960 0x7efc2c06bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.029+0000 7efc318c8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7efc2c0d21f0 con 0x7efc2c15d1c0 2024-04-15T15:15:39.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.029+0000 7efc318c8640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7efc2c0d4f60 con 0x7efc2c156f00 2024-04-15T15:15:39.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.029+0000 7efc318c8640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7efc2c0d1de0 con 0x7efc2c06b960 2024-04-15T15:15:39.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.029+0000 7efc2a7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc2c15d1c0 0x7efc2c15d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.029+0000 7efc2a7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc2c15d1c0 0x7efc2c15d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55028/0 (socket says 172.21.15.72:55028) 2024-04-15T15:15:39.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.029+0000 7efc2a7fc640 1 -- 172.21.15.72:0/882471558 learned_addr learned my addr 172.21.15.72:0/882471558 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:39.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.029+0000 7efc2a7fc640 1 -- 172.21.15.72:0/882471558 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efc2c06b960 msgr2=0x7efc2c06bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.029+0000 7efc2a7fc640 1 --2- 172.21.15.72:0/882471558 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efc2c06b960 0x7efc2c06bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.029+0000 7efc2a7fc640 1 -- 172.21.15.72:0/882471558 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc2c156f00 msgr2=0x7efc2c0ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.029+0000 7efc2a7fc640 1 --2- 172.21.15.72:0/882471558 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc2c156f00 0x7efc2c0ce550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.029+0000 7efc2a7fc640 1 -- 172.21.15.72:0/882471558 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc2c1572e0 con 0x7efc2c15d1c0 2024-04-15T15:15:39.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.029+0000 7efc2a7fc640 1 --2- 172.21.15.72:0/882471558 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc2c15d1c0 0x7efc2c15d640 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7efc2000f820 tx=0x7efc2000fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:39.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.029+0000 7efc13fff640 1 -- 172.21.15.72:0/882471558 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc200050d0 con 0x7efc2c15d1c0 2024-04-15T15:15:39.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.029+0000 7efc13fff640 1 -- 172.21.15.72:0/882471558 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc20018070 con 0x7efc2c15d1c0 2024-04-15T15:15:39.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.029+0000 7efc13fff640 1 -- 172.21.15.72:0/882471558 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc20005490 con 0x7efc2c15d1c0 2024-04-15T15:15:39.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.029+0000 7efc318c8640 1 -- 172.21.15.72:0/882471558 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc2c157900 con 0x7efc2c15d1c0 2024-04-15T15:15:39.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.030+0000 7efc318c8640 1 -- 172.21.15.72:0/882471558 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc2c157ec0 con 0x7efc2c15d1c0 2024-04-15T15:15:39.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.031+0000 7efc318c8640 1 -- 172.21.15.72:0/882471558 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc2c0689e0 con 0x7efc2c15d1c0 2024-04-15T15:15:39.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.031+0000 7efc13fff640 1 -- 172.21.15.72:0/882471558 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efc2000d6b0 con 0x7efc2c15d1c0 2024-04-15T15:15:39.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.031+0000 7efc13fff640 1 --2- 172.21.15.72:0/882471558 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7efc1806c8f0 0x7efc1806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.031+0000 7efc2affd640 1 --2- 172.21.15.72:0/882471558 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7efc1806c8f0 0x7efc1806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.031+0000 7efc13fff640 1 -- 172.21.15.72:0/882471558 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7efc2008bd50 con 0x7efc2c15d1c0 2024-04-15T15:15:39.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.032+0000 7efc2affd640 1 --2- 172.21.15.72:0/882471558 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7efc1806c8f0 0x7efc1806edb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7efc1c0011e0 tx=0x7efc1c001be0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:39.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.036+0000 7efc13fff640 1 -- 172.21.15.72:0/882471558 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efc2008f030 con 0x7efc2c15d1c0 2024-04-15T15:15:39.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.213+0000 7efc318c8640 1 -- 172.21.15.72:0/882471558 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efc2c0d51b0 con 0x7efc2c15d1c0 2024-04-15T15:15:39.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.214+0000 7efc13fff640 1 -- 172.21.15.72:0/882471558 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18903 (secure 0 0 0) 0x7efc20059e10 con 0x7efc2c15d1c0 2024-04-15T15:15:39.216 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:39.216 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":26,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:15:32.421728+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:14:49.438366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:14:50.368379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:14:54.081157+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":1}]},{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:39.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.216+0000 7efc318c8640 1 -- 172.21.15.72:0/882471558 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7efc1806c8f0 msgr2=0x7efc1806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.216+0000 7efc318c8640 1 --2- 172.21.15.72:0/882471558 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7efc1806c8f0 0x7efc1806edb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7efc1c0011e0 tx=0x7efc1c001be0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.216+0000 7efc318c8640 1 -- 172.21.15.72:0/882471558 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc2c15d1c0 msgr2=0x7efc2c15d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.216+0000 7efc318c8640 1 --2- 172.21.15.72:0/882471558 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc2c15d1c0 0x7efc2c15d640 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7efc2000f820 tx=0x7efc2000fce0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.216+0000 7efc318c8640 1 -- 172.21.15.72:0/882471558 shutdown_connections 2024-04-15T15:15:39.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.216+0000 7efc318c8640 1 --2- 172.21.15.72:0/882471558 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7efc1806c8f0 0x7efc1806edb0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.216+0000 7efc318c8640 1 --2- 172.21.15.72:0/882471558 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efc2c06b960 0x7efc2c06bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.216+0000 7efc318c8640 1 --2- 172.21.15.72:0/882471558 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efc2c15d1c0 0x7efc2c15d640 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.216+0000 7efc318c8640 1 --2- 172.21.15.72:0/882471558 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efc2c156f00 0x7efc2c0ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.216+0000 7efc318c8640 1 -- 172.21.15.72:0/882471558 >> 172.21.15.72:0/882471558 conn(0x7efc2c09d970 msgr2=0x7efc2c160c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:39.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.217+0000 7efc318c8640 1 -- 172.21.15.72:0/882471558 shutdown_connections 2024-04-15T15:15:39.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.217+0000 7efc318c8640 1 -- 172.21.15.72:0/882471558 wait complete. 2024-04-15T15:15:39.229 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:39.359102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:14:49.438366+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:14:50.368379+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:14:54.081157+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:15:39.230 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:39.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.335+0000 7f976b131640 1 Processor -- start 2024-04-15T15:15:39.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.335+0000 7f976b131640 1 -- start start 2024-04-15T15:15:39.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.336+0000 7f976b131640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f97640afae0 0x7f976406c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.336+0000 7f976b131640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97640d07c0 con 0x7f97640b0050 2024-04-15T15:15:39.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.336+0000 7f976b131640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97640b3aa0 con 0x7f97640b5410 2024-04-15T15:15:39.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.336+0000 7f976b131640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f97640d0330 con 0x7f97640afae0 2024-04-15T15:15:39.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.336+0000 7f9763fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f97640b0050 0x7f97640cf960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50108/0 (socket says 172.21.15.72:50108) 2024-04-15T15:15:39.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.336+0000 7f9763fff640 1 -- 172.21.15.72:0/1815457327 learned_addr learned my addr 172.21.15.72:0/1815457327 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:39.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.336+0000 7f97637fe640 1 -- 172.21.15.72:0/1815457327 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 993215848 0 0) 0x7f97640d07c0 con 0x7f97640b0050 2024-04-15T15:15:39.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.336+0000 7f97637fe640 1 -- 172.21.15.72:0/1815457327 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9754003520 con 0x7f97640b0050 2024-04-15T15:15:39.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.336+0000 7f97696ce640 1 --2- 172.21.15.72:0/1815457327 >> v2:172.21.15.175:3300/0 conn(0x7f97640afae0 0x7f976406c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.336+0000 7f97637fe640 1 -- 172.21.15.72:0/1815457327 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1869629065 0 0) 0x7f9754003520 con 0x7f97640b0050 2024-04-15T15:15:39.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.336+0000 7f97637fe640 1 -- 172.21.15.72:0/1815457327 >> v2:172.21.15.175:3300/0 conn(0x7f97640afae0 msgr2=0x7f976406c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:39.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.336+0000 7f97637fe640 1 --2- 172.21.15.72:0/1815457327 >> v2:172.21.15.175:3300/0 conn(0x7f97640afae0 0x7f976406c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.336+0000 7f97637fe640 1 -- 172.21.15.72:0/1815457327 >> v1:172.21.15.175:6789/0 conn(0x7f97640b5410 legacy=0x7f97640b57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.336+0000 7f97637fe640 1 -- 172.21.15.72:0/1815457327 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97640b3d80 con 0x7f97640b0050 2024-04-15T15:15:39.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.336+0000 7f97637fe640 1 -- 172.21.15.72:0/1815457327 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f9750001670 con 0x7f97640b0050 2024-04-15T15:15:39.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.336+0000 7f97637fe640 1 -- 172.21.15.72:0/1815457327 >> v1:172.21.15.72:6789/0 conn(0x7f97640b0050 legacy=0x7f97640cf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.336+0000 7f97637fe640 1 --2- 172.21.15.72:0/1815457327 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97540039b0 0x7f9754003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.336+0000 7f97637fe640 1 --2- 172.21.15.72:0/1815457327 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9754004800 0x7f9754004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.336+0000 7f97637fe640 1 --2- 172.21.15.72:0/1815457327 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f97540052e0 0x7f975400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.337+0000 7f97637fe640 1 -- 172.21.15.72:0/1815457327 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f97640d0330 con 0x7f97540039b0 2024-04-15T15:15:39.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.337+0000 7f97637fe640 1 -- 172.21.15.72:0/1815457327 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f975400c1b0 con 0x7f97540052e0 2024-04-15T15:15:39.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.337+0000 7f97637fe640 1 -- 172.21.15.72:0/1815457327 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f975400c3b0 con 0x7f9754004800 2024-04-15T15:15:39.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.337+0000 7f9768ecd640 1 --2- 172.21.15.72:0/1815457327 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97540039b0 0x7f9754003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.337+0000 7f97696ce640 1 --2- 172.21.15.72:0/1815457327 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f97540052e0 0x7f975400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.337+0000 7f9763fff640 1 --2- 172.21.15.72:0/1815457327 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9754004800 0x7f9754004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.337+0000 7f9768ecd640 1 -- 172.21.15.72:0/1815457327 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9754004800 msgr2=0x7f9754004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:39.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.337+0000 7f9768ecd640 1 --2- 172.21.15.72:0/1815457327 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9754004800 0x7f9754004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.337+0000 7f9768ecd640 1 -- 172.21.15.72:0/1815457327 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f97540052e0 msgr2=0x7f975400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:39.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.337+0000 7f9768ecd640 1 --2- 172.21.15.72:0/1815457327 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f97540052e0 0x7f975400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.337+0000 7f9768ecd640 1 -- 172.21.15.72:0/1815457327 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9754003520 con 0x7f97540039b0 2024-04-15T15:15:39.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.337+0000 7f97696ce640 1 -- 172.21.15.72:0/1815457327 reap_dead start 2024-04-15T15:15:39.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.337+0000 7f9763fff640 1 --2- 172.21.15.72:0/1815457327 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9754004800 0x7f9754004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:39.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.338+0000 7f9768ecd640 1 --2- 172.21.15.72:0/1815457327 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97540039b0 0x7f9754003dd0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f9758001720 tx=0x7f9754005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9485318f96df25de server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:39.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.338+0000 7f97637fe640 1 -- 172.21.15.72:0/1815457327 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9758001370 con 0x7f97540039b0 2024-04-15T15:15:39.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.338+0000 7f97637fe640 1 -- 172.21.15.72:0/1815457327 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97580052d0 con 0x7f97540039b0 2024-04-15T15:15:39.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.338+0000 7f97637fe640 1 -- 172.21.15.72:0/1815457327 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9758003830 con 0x7f97540039b0 2024-04-15T15:15:39.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.338+0000 7f976b131640 1 -- 172.21.15.72:0/1815457327 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97540039b0 msgr2=0x7f9754003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.338+0000 7f976b131640 1 --2- 172.21.15.72:0/1815457327 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97540039b0 0x7f9754003dd0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f9758001720 tx=0x7f9754005830 comp rx=0 tx=0).stop 2024-04-15T15:15:39.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.338+0000 7f976b131640 1 -- 172.21.15.72:0/1815457327 shutdown_connections 2024-04-15T15:15:39.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.338+0000 7f976b131640 1 --2- 172.21.15.72:0/1815457327 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97540039b0 0x7f9754003dd0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.338+0000 7f976b131640 1 -- 172.21.15.72:0/1815457327 >> 172.21.15.72:0/1815457327 conn(0x7f97640d0a50 msgr2=0x7f97640d0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:39.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.338+0000 7f976b131640 1 -- 172.21.15.72:0/1815457327 shutdown_connections 2024-04-15T15:15:39.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.338+0000 7f976b131640 1 -- 172.21.15.72:0/1815457327 wait complete. 2024-04-15T15:15:39.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.340+0000 7f976b131640 1 Processor -- start 2024-04-15T15:15:39.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.340+0000 7f976b131640 1 -- start start 2024-04-15T15:15:39.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.340+0000 7f976b131640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f97540039b0 0x7f9764149570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.340+0000 7f976b131640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97641f81a0 0x7f97641f8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.340+0000 7f976b131640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f976415a990 0x7f976415ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.340+0000 7f976b131640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f976406db40 con 0x7f97641f81a0 2024-04-15T15:15:39.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.340+0000 7f976b131640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f97640d18f0 con 0x7f97540039b0 2024-04-15T15:15:39.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.340+0000 7f976b131640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f976406dcc0 con 0x7f976415a990 2024-04-15T15:15:39.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.341+0000 7f9763fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97641f81a0 0x7f97641f8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.341+0000 7f9763fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97641f81a0 0x7f97641f8620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55050/0 (socket says 172.21.15.72:55050) 2024-04-15T15:15:39.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.341+0000 7f9763fff640 1 -- 172.21.15.72:0/2598308211 learned_addr learned my addr 172.21.15.72:0/2598308211 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:39.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.341+0000 7f97696ce640 1 --2- 172.21.15.72:0/2598308211 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f976415a990 0x7f976415ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.341+0000 7f9768ecd640 1 --2- 172.21.15.72:0/2598308211 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f97540039b0 0x7f9764149570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.341+0000 7f9763fff640 1 -- 172.21.15.72:0/2598308211 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f976415a990 msgr2=0x7f976415ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.341+0000 7f9763fff640 1 --2- 172.21.15.72:0/2598308211 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f976415a990 0x7f976415ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.341+0000 7f9763fff640 1 -- 172.21.15.72:0/2598308211 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f97540039b0 msgr2=0x7f9764149570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.341+0000 7f9763fff640 1 --2- 172.21.15.72:0/2598308211 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f97540039b0 0x7f9764149570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.341+0000 7f9763fff640 1 -- 172.21.15.72:0/2598308211 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f976415b750 con 0x7f97641f81a0 2024-04-15T15:15:39.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.341+0000 7f9763fff640 1 --2- 172.21.15.72:0/2598308211 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97641f81a0 0x7f97641f8620 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f97500017e0 tx=0x7f975000eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:39.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.341+0000 7f97617fa640 1 -- 172.21.15.72:0/2598308211 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f975000c660 con 0x7f97641f81a0 2024-04-15T15:15:39.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.341+0000 7f97617fa640 1 -- 172.21.15.72:0/2598308211 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9750018070 con 0x7f97641f81a0 2024-04-15T15:15:39.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.341+0000 7f976b131640 1 -- 172.21.15.72:0/2598308211 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f976415ba30 con 0x7f97641f81a0 2024-04-15T15:15:39.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.341+0000 7f97617fa640 1 -- 172.21.15.72:0/2598308211 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f975000ca00 con 0x7f97641f81a0 2024-04-15T15:15:39.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.341+0000 7f976b131640 1 -- 172.21.15.72:0/2598308211 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f976415bff0 con 0x7f97641f81a0 2024-04-15T15:15:39.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.342+0000 7f976b131640 1 -- 172.21.15.72:0/2598308211 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9764075c50 con 0x7f97641f81a0 2024-04-15T15:15:39.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.342+0000 7f97617fa640 1 -- 172.21.15.72:0/2598308211 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f975000cba0 con 0x7f97641f81a0 2024-04-15T15:15:39.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.343+0000 7f97617fa640 1 --2- 172.21.15.72:0/2598308211 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f974806c8f0 0x7f974806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.343+0000 7f97617fa640 1 -- 172.21.15.72:0/2598308211 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f975008bc90 con 0x7f97641f81a0 2024-04-15T15:15:39.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.343+0000 7f9768ecd640 1 --2- 172.21.15.72:0/2598308211 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f974806c8f0 0x7f974806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.343+0000 7f9768ecd640 1 --2- 172.21.15.72:0/2598308211 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f974806c8f0 0x7f974806edb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9758033000 tx=0x7f9758009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:39.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.347+0000 7f97617fa640 1 -- 172.21.15.72:0/2598308211 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9750059cc0 con 0x7f97641f81a0 2024-04-15T15:15:39.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.523+0000 7f976b131640 1 -- 172.21.15.72:0/2598308211 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f976406d610 con 0x7f97641f81a0 2024-04-15T15:15:39.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.523+0000 7f97617fa640 1 -- 172.21.15.72:0/2598308211 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f97500595c0 con 0x7f97641f81a0 2024-04-15T15:15:39.525 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-15T15:15:39.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.525+0000 7f976b131640 1 -- 172.21.15.72:0/2598308211 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f974806c8f0 msgr2=0x7f974806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.525+0000 7f976b131640 1 --2- 172.21.15.72:0/2598308211 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f974806c8f0 0x7f974806edb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9758033000 tx=0x7f9758009030 comp rx=0 tx=0).stop 2024-04-15T15:15:39.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.525+0000 7f976b131640 1 -- 172.21.15.72:0/2598308211 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97641f81a0 msgr2=0x7f97641f8620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.525+0000 7f976b131640 1 --2- 172.21.15.72:0/2598308211 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97641f81a0 0x7f97641f8620 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f97500017e0 tx=0x7f975000eb90 comp rx=0 tx=0).stop 2024-04-15T15:15:39.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.525+0000 7f976b131640 1 -- 172.21.15.72:0/2598308211 shutdown_connections 2024-04-15T15:15:39.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.525+0000 7f976b131640 1 --2- 172.21.15.72:0/2598308211 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f974806c8f0 0x7f974806edb0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.525+0000 7f976b131640 1 --2- 172.21.15.72:0/2598308211 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f976415a990 0x7f976415ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.525+0000 7f976b131640 1 --2- 172.21.15.72:0/2598308211 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f97641f81a0 0x7f97641f8620 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.525+0000 7f976b131640 1 --2- 172.21.15.72:0/2598308211 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f97540039b0 0x7f9764149570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.525+0000 7f976b131640 1 -- 172.21.15.72:0/2598308211 >> 172.21.15.72:0/2598308211 conn(0x7f97640d0a50 msgr2=0x7f97640ad8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:39.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.525+0000 7f976b131640 1 -- 172.21.15.72:0/2598308211 shutdown_connections 2024-04-15T15:15:39.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.525+0000 7f976b131640 1 -- 172.21.15.72:0/2598308211 wait complete. 2024-04-15T15:15:39.538 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:15:39.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.643+0000 7f2eca04a640 1 Processor -- start 2024-04-15T15:15:39.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.643+0000 7f2eca04a640 1 -- start start 2024-04-15T15:15:39.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.644+0000 7f2eca04a640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2ec40b0050 0x7f2ec40cf960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.644+0000 7f2eca04a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ec40d07c0 con 0x7f2ec40b5410 2024-04-15T15:15:39.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.644+0000 7f2eca04a640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ec40b3aa0 con 0x7f2ec40afae0 2024-04-15T15:15:39.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.644+0000 7f2eca04a640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f2ec40d0330 con 0x7f2ec40b0050 2024-04-15T15:15:39.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.644+0000 7f2ec37fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f2ec40b5410 0x7f2ec40b57f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50110/0 (socket says 172.21.15.72:50110) 2024-04-15T15:15:39.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.644+0000 7f2ec37fe640 1 -- 172.21.15.72:0/3255120004 learned_addr learned my addr 172.21.15.72:0/3255120004 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:39.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.644+0000 7f2ec2ffd640 1 --2- 172.21.15.72:0/3255120004 >> v2:172.21.15.175:3300/0 conn(0x7f2ec40b0050 0x7f2ec40cf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.644+0000 7f2ec27fc640 1 -- 172.21.15.72:0/3255120004 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 636727489 0 0) 0x7f2ec40d07c0 con 0x7f2ec40b5410 2024-04-15T15:15:39.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.644+0000 7f2ec27fc640 1 -- 172.21.15.72:0/3255120004 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2eac003520 con 0x7f2ec40b5410 2024-04-15T15:15:39.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.644+0000 7f2ec27fc640 1 -- 172.21.15.72:0/3255120004 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2677526046 0 0) 0x7f2eac003520 con 0x7f2ec40b5410 2024-04-15T15:15:39.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.644+0000 7f2ec27fc640 1 -- 172.21.15.72:0/3255120004 >> v2:172.21.15.175:3300/0 conn(0x7f2ec40b0050 msgr2=0x7f2ec40cf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:39.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.644+0000 7f2ec27fc640 1 --2- 172.21.15.72:0/3255120004 >> v2:172.21.15.175:3300/0 conn(0x7f2ec40b0050 0x7f2ec40cf960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.644+0000 7f2ec27fc640 1 -- 172.21.15.72:0/3255120004 >> v1:172.21.15.175:6789/0 conn(0x7f2ec40afae0 legacy=0x7f2ec406c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.644+0000 7f2ec27fc640 1 -- 172.21.15.72:0/3255120004 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ec40b3d80 con 0x7f2ec40b5410 2024-04-15T15:15:39.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.644+0000 7f2ec27fc640 1 -- 172.21.15.72:0/3255120004 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f2eb4001670 con 0x7f2ec40b5410 2024-04-15T15:15:39.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.644+0000 7f2ec27fc640 1 -- 172.21.15.72:0/3255120004 >> v1:172.21.15.72:6789/0 conn(0x7f2ec40b5410 legacy=0x7f2ec40b57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.644+0000 7f2ec27fc640 1 --2- 172.21.15.72:0/3255120004 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2eac0039b0 0x7f2eac003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.644+0000 7f2ec27fc640 1 --2- 172.21.15.72:0/3255120004 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2eac004800 0x7f2eac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.644+0000 7f2ec27fc640 1 --2- 172.21.15.72:0/3255120004 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2eac0051c0 0x7f2eac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.644+0000 7f2ec27fc640 1 -- 172.21.15.72:0/3255120004 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec40d0330 con 0x7f2eac004800 2024-04-15T15:15:39.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.644+0000 7f2ec27fc640 1 -- 172.21.15.72:0/3255120004 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f2eac00c160 con 0x7f2eac0051c0 2024-04-15T15:15:39.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.644+0000 7f2ec27fc640 1 -- 172.21.15.72:0/3255120004 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f2eac00c360 con 0x7f2eac0039b0 2024-04-15T15:15:39.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.645+0000 7f2ec2ffd640 1 --2- 172.21.15.72:0/3255120004 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2eac004800 0x7f2eac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.645+0000 7f2ec37fe640 1 --2- 172.21.15.72:0/3255120004 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2eac0039b0 0x7f2eac003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.645+0000 7f2ec3fff640 1 --2- 172.21.15.72:0/3255120004 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2eac0051c0 0x7f2eac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.645+0000 7f2ec2ffd640 1 -- 172.21.15.72:0/3255120004 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2eac0039b0 msgr2=0x7f2eac003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:39.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.645+0000 7f2ec2ffd640 1 --2- 172.21.15.72:0/3255120004 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2eac0039b0 0x7f2eac003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.645+0000 7f2ec2ffd640 1 -- 172.21.15.72:0/3255120004 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2eac0051c0 msgr2=0x7f2eac00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:39.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.645+0000 7f2ec2ffd640 1 --2- 172.21.15.72:0/3255120004 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2eac0051c0 0x7f2eac00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.645+0000 7f2ec2ffd640 1 -- 172.21.15.72:0/3255120004 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eac003520 con 0x7f2eac004800 2024-04-15T15:15:39.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.645+0000 7f2ec3fff640 1 -- 172.21.15.72:0/3255120004 reap_dead start 2024-04-15T15:15:39.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.646+0000 7f2ec2ffd640 1 --2- 172.21.15.72:0/3255120004 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2eac004800 0x7f2eac004c80 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8009ac0 tx=0x7f2eac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=675ca6aaca131f0d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:39.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.646+0000 7f2ec27fc640 1 -- 172.21.15.72:0/3255120004 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb8039070 con 0x7f2eac004800 2024-04-15T15:15:39.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.646+0000 7f2ec27fc640 1 -- 172.21.15.72:0/3255120004 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2eb802c6b0 con 0x7f2eac004800 2024-04-15T15:15:39.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.646+0000 7f2ec27fc640 1 -- 172.21.15.72:0/3255120004 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb8009690 con 0x7f2eac004800 2024-04-15T15:15:39.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.646+0000 7f2eca04a640 1 -- 172.21.15.72:0/3255120004 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2eac004800 msgr2=0x7f2eac004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.646+0000 7f2eca04a640 1 --2- 172.21.15.72:0/3255120004 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2eac004800 0x7f2eac004c80 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8009ac0 tx=0x7f2eac005830 comp rx=0 tx=0).stop 2024-04-15T15:15:39.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.646+0000 7f2eca04a640 1 -- 172.21.15.72:0/3255120004 shutdown_connections 2024-04-15T15:15:39.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.646+0000 7f2eca04a640 1 --2- 172.21.15.72:0/3255120004 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2eac004800 0x7f2eac004c80 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.646+0000 7f2eca04a640 1 -- 172.21.15.72:0/3255120004 >> 172.21.15.72:0/3255120004 conn(0x7f2ec40d0a50 msgr2=0x7f2ec40d0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:39.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.646+0000 7f2eca04a640 1 -- 172.21.15.72:0/3255120004 shutdown_connections 2024-04-15T15:15:39.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.646+0000 7f2eca04a640 1 -- 172.21.15.72:0/3255120004 wait complete. 2024-04-15T15:15:39.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.647+0000 7f2eca04a640 1 Processor -- start 2024-04-15T15:15:39.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.648+0000 7f2eca04a640 1 -- start start 2024-04-15T15:15:39.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.648+0000 7f2eca04a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2eac004800 0x7f2ec4146730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.648+0000 7f2eca04a640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2ec41e7300 0x7f2ec4147e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.648+0000 7f2eca04a640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2ec41487c0 0x7f2ec4148c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.648+0000 7f2eca04a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec406db40 con 0x7f2eac004800 2024-04-15T15:15:39.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.648+0000 7f2eca04a640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f2ec40d18f0 con 0x7f2ec41487c0 2024-04-15T15:15:39.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.648+0000 7f2eca04a640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f2ec406dcc0 con 0x7f2ec41e7300 2024-04-15T15:15:39.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.648+0000 7f2ec37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2eac004800 0x7f2ec4146730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.648+0000 7f2ec37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2eac004800 0x7f2ec4146730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55072/0 (socket says 172.21.15.72:55072) 2024-04-15T15:15:39.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.648+0000 7f2ec37fe640 1 -- 172.21.15.72:0/154270966 learned_addr learned my addr 172.21.15.72:0/154270966 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:39.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.649+0000 7f2ec2ffd640 1 --2- 172.21.15.72:0/154270966 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2ec41e7300 0x7f2ec4147e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.649+0000 7f2ec3fff640 1 --2- 172.21.15.72:0/154270966 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2ec41487c0 0x7f2ec4148c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.649+0000 7f2ec37fe640 1 -- 172.21.15.72:0/154270966 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2ec41e7300 msgr2=0x7f2ec4147e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.649+0000 7f2ec37fe640 1 --2- 172.21.15.72:0/154270966 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2ec41e7300 0x7f2ec4147e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.649+0000 7f2ec37fe640 1 -- 172.21.15.72:0/154270966 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2ec41487c0 msgr2=0x7f2ec4148c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.649+0000 7f2ec37fe640 1 --2- 172.21.15.72:0/154270966 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2ec41487c0 0x7f2ec4148c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.649+0000 7f2ec37fe640 1 -- 172.21.15.72:0/154270966 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ec4146dc0 con 0x7f2eac004800 2024-04-15T15:15:39.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.649+0000 7f2ec2ffd640 1 --2- 172.21.15.72:0/154270966 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2ec41e7300 0x7f2ec4147e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:39.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.649+0000 7f2ec37fe640 1 --2- 172.21.15.72:0/154270966 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2eac004800 0x7f2ec4146730 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4007340 tx=0x7f2eb4007740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:39.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.649+0000 7f2eabfff640 1 -- 172.21.15.72:0/154270966 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb400f940 con 0x7f2eac004800 2024-04-15T15:15:39.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.649+0000 7f2eabfff640 1 -- 172.21.15.72:0/154270966 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2eb4018070 con 0x7f2eac004800 2024-04-15T15:15:39.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.649+0000 7f2eca04a640 1 -- 172.21.15.72:0/154270966 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ec41470a0 con 0x7f2eac004800 2024-04-15T15:15:39.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.649+0000 7f2eabfff640 1 -- 172.21.15.72:0/154270966 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb400fd60 con 0x7f2eac004800 2024-04-15T15:15:39.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.649+0000 7f2eca04a640 1 -- 172.21.15.72:0/154270966 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ec4147660 con 0x7f2eac004800 2024-04-15T15:15:39.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.650+0000 7f2eca04a640 1 -- 172.21.15.72:0/154270966 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ec406f430 con 0x7f2eac004800 2024-04-15T15:15:39.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.650+0000 7f2eabfff640 1 -- 172.21.15.72:0/154270966 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2eb401d460 con 0x7f2eac004800 2024-04-15T15:15:39.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.651+0000 7f2eabfff640 1 --2- 172.21.15.72:0/154270966 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2eb006c8f0 0x7f2eb006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.651+0000 7f2eabfff640 1 -- 172.21.15.72:0/154270966 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f2eb408c150 con 0x7f2eac004800 2024-04-15T15:15:39.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.651+0000 7f2ec2ffd640 1 --2- 172.21.15.72:0/154270966 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2eb006c8f0 0x7f2eb006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.651+0000 7f2ec2ffd640 1 --2- 172.21.15.72:0/154270966 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2eb006c8f0 0x7f2eb006edb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8036000 tx=0x7f2eb8030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:39.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.655+0000 7f2eabfff640 1 -- 172.21.15.72:0/154270966 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2ec406f430 con 0x7f2eac004800 2024-04-15T15:15:39.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.832+0000 7f2eca04a640 1 -- 172.21.15.72:0/154270966 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2ec4075c50 con 0x7f2eac004800 2024-04-15T15:15:39.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.833+0000 7f2eabfff640 1 -- 172.21.15.72:0/154270966 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f2eb405a180 con 0x7f2eac004800 2024-04-15T15:15:39.834 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:39.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.835+0000 7f2eca04a640 1 -- 172.21.15.72:0/154270966 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2eb006c8f0 msgr2=0x7f2eb006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.835+0000 7f2eca04a640 1 --2- 172.21.15.72:0/154270966 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2eb006c8f0 0x7f2eb006edb0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8036000 tx=0x7f2eb8030000 comp rx=0 tx=0).stop 2024-04-15T15:15:39.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.835+0000 7f2eca04a640 1 -- 172.21.15.72:0/154270966 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2eac004800 msgr2=0x7f2ec4146730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.835+0000 7f2eca04a640 1 --2- 172.21.15.72:0/154270966 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2eac004800 0x7f2ec4146730 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4007340 tx=0x7f2eb4007740 comp rx=0 tx=0).stop 2024-04-15T15:15:39.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.835+0000 7f2eca04a640 1 -- 172.21.15.72:0/154270966 shutdown_connections 2024-04-15T15:15:39.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.835+0000 7f2eca04a640 1 --2- 172.21.15.72:0/154270966 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2eb006c8f0 0x7f2eb006edb0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.835+0000 7f2eca04a640 1 --2- 172.21.15.72:0/154270966 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2ec41487c0 0x7f2ec4148c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.835+0000 7f2eca04a640 1 --2- 172.21.15.72:0/154270966 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2ec41e7300 0x7f2ec4147e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.835+0000 7f2eca04a640 1 --2- 172.21.15.72:0/154270966 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2eac004800 0x7f2ec4146730 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.835+0000 7f2eca04a640 1 -- 172.21.15.72:0/154270966 >> 172.21.15.72:0/154270966 conn(0x7f2ec40d0a50 msgr2=0x7f2ec406f630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:39.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.835+0000 7f2eca04a640 1 -- 172.21.15.72:0/154270966 shutdown_connections 2024-04-15T15:15:39.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.835+0000 7f2eca04a640 1 -- 172.21.15.72:0/154270966 wait complete. 2024-04-15T15:15:39.848 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:15:39.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.952+0000 7f7e96189640 1 Processor -- start 2024-04-15T15:15:39.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.952+0000 7f7e96189640 1 -- start start 2024-04-15T15:15:39.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.953+0000 7f7e96189640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7e90160980 0x7f7e9006c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.953+0000 7f7e96189640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e9006d000 con 0x7f7e90145220 2024-04-15T15:15:39.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.953+0000 7f7e96189640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e9006d2e0 con 0x7f7e9006c810 2024-04-15T15:15:39.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.953+0000 7f7e96189640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f7e900cdef0 con 0x7f7e90160980 2024-04-15T15:15:39.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.953+0000 7f7e86ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f7e90160980 0x7f7e9006c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.953+0000 7f7e8f7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f7e90145220 0x7f7e90160270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50118/0 (socket says 172.21.15.72:50118) 2024-04-15T15:15:39.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.953+0000 7f7e8f7fe640 1 -- 172.21.15.72:0/499233517 learned_addr learned my addr 172.21.15.72:0/499233517 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:39.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.954+0000 7f7e86ffd640 1 -- 172.21.15.72:0/499233517 >> v1:172.21.15.175:6789/0 conn(0x7f7e9006c810 legacy=0x7f7e900ce380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:39.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.954+0000 7f7e86ffd640 1 -- 172.21.15.72:0/499233517 >> v1:172.21.15.72:6789/0 conn(0x7f7e90145220 legacy=0x7f7e90160270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.954+0000 7f7e86ffd640 1 -- 172.21.15.72:0/499233517 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e9006d5c0 con 0x7f7e90160980 2024-04-15T15:15:39.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.954+0000 7f7e8effd640 1 -- 172.21.15.72:0/499233517 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3344691735 0 0) 0x7f7e9006d000 con 0x7f7e90145220 2024-04-15T15:15:39.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.954+0000 7f7e86ffd640 1 --2- 172.21.15.72:0/499233517 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e90160980 0x7f7e9006c2d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f7e780097a0 tx=0x7f7e7802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=253c4a1e9d5bb1cf server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:39.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.955+0000 7f7e8effd640 1 -- 172.21.15.72:0/499233517 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e780315a0 con 0x7f7e90160980 2024-04-15T15:15:39.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.955+0000 7f7e8effd640 1 -- 172.21.15.72:0/499233517 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e78037070 con 0x7f7e90160980 2024-04-15T15:15:39.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.955+0000 7f7e8effd640 1 -- 172.21.15.72:0/499233517 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e78031960 con 0x7f7e90160980 2024-04-15T15:15:39.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.955+0000 7f7e96189640 1 -- 172.21.15.72:0/499233517 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e90160980 msgr2=0x7f7e9006c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.955+0000 7f7e96189640 1 --2- 172.21.15.72:0/499233517 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e90160980 0x7f7e9006c2d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f7e780097a0 tx=0x7f7e7802b840 comp rx=0 tx=0).stop 2024-04-15T15:15:39.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.955+0000 7f7e96189640 1 -- 172.21.15.72:0/499233517 shutdown_connections 2024-04-15T15:15:39.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.955+0000 7f7e96189640 1 --2- 172.21.15.72:0/499233517 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e90160980 0x7f7e9006c2d0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.955+0000 7f7e96189640 1 -- 172.21.15.72:0/499233517 >> 172.21.15.72:0/499233517 conn(0x7f7e900703a0 msgr2=0x7f7e900707b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:39.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.955+0000 7f7e96189640 1 -- 172.21.15.72:0/499233517 shutdown_connections 2024-04-15T15:15:39.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.955+0000 7f7e96189640 1 -- 172.21.15.72:0/499233517 wait complete. 2024-04-15T15:15:39.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.956+0000 7f7e96189640 1 Processor -- start 2024-04-15T15:15:39.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.957+0000 7f7e96189640 1 -- start start 2024-04-15T15:15:39.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.957+0000 7f7e96189640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7e9006c810 0x7f7e900ac590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.957+0000 7f7e96189640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e90145220 0x7f7e900adfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.957+0000 7f7e96189640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7e90160980 0x7f7e900ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.957+0000 7f7e96189640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f7e9006cc50 con 0x7f7e90145220 2024-04-15T15:15:39.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.957+0000 7f7e96189640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f7e90145600 con 0x7f7e90160980 2024-04-15T15:15:39.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.957+0000 7f7e96189640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f7e900aeb80 con 0x7f7e9006c810 2024-04-15T15:15:39.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.957+0000 7f7e86ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e90145220 0x7f7e900adfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.957+0000 7f7e86ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e90145220 0x7f7e900adfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55076/0 (socket says 172.21.15.72:55076) 2024-04-15T15:15:39.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.957+0000 7f7e86ffd640 1 -- 172.21.15.72:0/1471709179 learned_addr learned my addr 172.21.15.72:0/1471709179 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:39.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.958+0000 7f7e8f7fe640 1 --2- 172.21.15.72:0/1471709179 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7e9006c810 0x7f7e900ac590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.958+0000 7f7e8ffff640 1 --2- 172.21.15.72:0/1471709179 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7e90160980 0x7f7e900ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.958+0000 7f7e86ffd640 1 -- 172.21.15.72:0/1471709179 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7e9006c810 msgr2=0x7f7e900ac590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.958+0000 7f7e86ffd640 1 --2- 172.21.15.72:0/1471709179 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7e9006c810 0x7f7e900ac590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.958+0000 7f7e86ffd640 1 -- 172.21.15.72:0/1471709179 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7e90160980 msgr2=0x7f7e900ae520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.958+0000 7f7e86ffd640 1 --2- 172.21.15.72:0/1471709179 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7e90160980 0x7f7e900ae520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.958+0000 7f7e86ffd640 1 -- 172.21.15.72:0/1471709179 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e900aee60 con 0x7f7e90145220 2024-04-15T15:15:39.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.958+0000 7f7e8ffff640 1 --2- 172.21.15.72:0/1471709179 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7e90160980 0x7f7e900ae520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:39.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.958+0000 7f7e8f7fe640 1 --2- 172.21.15.72:0/1471709179 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7e9006c810 0x7f7e900ac590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:39.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.958+0000 7f7e86ffd640 1 --2- 172.21.15.72:0/1471709179 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e90145220 0x7f7e900adfe0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7e780097a0 tx=0x7f7e780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:39.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.958+0000 7f7e8cff9640 1 -- 172.21.15.72:0/1471709179 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e78032840 con 0x7f7e90145220 2024-04-15T15:15:39.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.958+0000 7f7e8cff9640 1 -- 172.21.15.72:0/1471709179 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e780329e0 con 0x7f7e90145220 2024-04-15T15:15:39.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.958+0000 7f7e96189640 1 -- 172.21.15.72:0/1471709179 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e900af140 con 0x7f7e90145220 2024-04-15T15:15:39.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.958+0000 7f7e8cff9640 1 -- 172.21.15.72:0/1471709179 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e78032dd0 con 0x7f7e90145220 2024-04-15T15:15:39.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.958+0000 7f7e96189640 1 -- 172.21.15.72:0/1471709179 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e900af700 con 0x7f7e90145220 2024-04-15T15:15:39.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.959+0000 7f7e96189640 1 -- 172.21.15.72:0/1471709179 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e90075c50 con 0x7f7e90145220 2024-04-15T15:15:39.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.959+0000 7f7e8cff9640 1 -- 172.21.15.72:0/1471709179 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7e7803b3f0 con 0x7f7e90145220 2024-04-15T15:15:39.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.960+0000 7f7e8cff9640 1 --2- 172.21.15.72:0/1471709179 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7e7006c920 0x7f7e7006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.960+0000 7f7e8cff9640 1 -- 172.21.15.72:0/1471709179 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f7e7802f030 con 0x7f7e90145220 2024-04-15T15:15:39.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.960+0000 7f7e8f7fe640 1 --2- 172.21.15.72:0/1471709179 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7e7006c920 0x7f7e7006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.960+0000 7f7e8f7fe640 1 --2- 172.21.15.72:0/1471709179 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7e7006c920 0x7f7e7006ede0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7e80000f60 tx=0x7f7e80009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:39.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:39.964+0000 7f7e8cff9640 1 -- 172.21.15.72:0/1471709179 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7e780b1030 con 0x7f7e90145220 2024-04-15T15:15:40.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.143+0000 7f7e96189640 1 -- 172.21.15.72:0/1471709179 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7e900b0c30 con 0x7f7e90145220 2024-04-15T15:15:40.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.143+0000 7f7e8cff9640 1 -- 172.21.15.72:0/1471709179 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f7e78040070 con 0x7f7e90145220 2024-04-15T15:15:40.145 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-04-15T15:15:40.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.145+0000 7f7e96189640 1 -- 172.21.15.72:0/1471709179 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7e7006c920 msgr2=0x7f7e7006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:40.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.145+0000 7f7e96189640 1 --2- 172.21.15.72:0/1471709179 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7e7006c920 0x7f7e7006ede0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7e80000f60 tx=0x7f7e80009040 comp rx=0 tx=0).stop 2024-04-15T15:15:40.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.145+0000 7f7e96189640 1 -- 172.21.15.72:0/1471709179 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e90145220 msgr2=0x7f7e900adfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:40.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.145+0000 7f7e96189640 1 --2- 172.21.15.72:0/1471709179 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e90145220 0x7f7e900adfe0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7e780097a0 tx=0x7f7e780046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:40.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.145+0000 7f7e96189640 1 -- 172.21.15.72:0/1471709179 shutdown_connections 2024-04-15T15:15:40.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.145+0000 7f7e96189640 1 --2- 172.21.15.72:0/1471709179 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7e7006c920 0x7f7e7006ede0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:40.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.145+0000 7f7e96189640 1 --2- 172.21.15.72:0/1471709179 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7e90160980 0x7f7e900ae520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:40.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.145+0000 7f7e96189640 1 --2- 172.21.15.72:0/1471709179 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e90145220 0x7f7e900adfe0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:40.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.145+0000 7f7e96189640 1 --2- 172.21.15.72:0/1471709179 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7e9006c810 0x7f7e900ac590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:40.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.145+0000 7f7e96189640 1 -- 172.21.15.72:0/1471709179 >> 172.21.15.72:0/1471709179 conn(0x7f7e900703a0 msgr2=0x7f7e901444a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:40.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.145+0000 7f7e96189640 1 -- 172.21.15.72:0/1471709179 shutdown_connections 2024-04-15T15:15:40.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.145+0000 7f7e96189640 1 -- 172.21.15.72:0/1471709179 wait complete. 2024-04-15T15:15:40.158 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:15:40.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.262+0000 7f239300d640 1 Processor -- start 2024-04-15T15:15:40.263 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.262+0000 7f239300d640 1 -- start start 2024-04-15T15:15:40.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.263+0000 7f239300d640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f238c156cf0 0x7f238c06ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:40.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.263+0000 7f239300d640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f238c0b0270 0x7f238c06d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:40.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.263+0000 7f239300d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f238c15ce00 con 0x7f238c0d1570 2024-04-15T15:15:40.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.263+0000 7f239300d640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f238c06d8b0 con 0x7f238c0b0270 2024-04-15T15:15:40.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.263+0000 7f239300d640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f238c15d0e0 con 0x7f238c156cf0 2024-04-15T15:15:40.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.263+0000 7f23915aa640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f238c0d1570 0x7f238c160250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50128/0 (socket says 172.21.15.72:50128) 2024-04-15T15:15:40.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.263+0000 7f23915aa640 1 -- 172.21.15.72:0/4173091707 learned_addr learned my addr 172.21.15.72:0/4173091707 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:40.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.263+0000 7f2390da9640 1 --2- 172.21.15.72:0/4173091707 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f238c156cf0 0x7f238c06ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:40.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.263+0000 7f238b7fe640 1 -- 172.21.15.72:0/4173091707 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2507119917 0 0) 0x7f238c15ce00 con 0x7f238c0d1570 2024-04-15T15:15:40.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.263+0000 7f238b7fe640 1 -- 172.21.15.72:0/4173091707 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f237c003520 con 0x7f238c0d1570 2024-04-15T15:15:40.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.263+0000 7f238bfff640 1 --2- 172.21.15.72:0/4173091707 >> v2:172.21.15.175:3300/0 conn(0x7f238c0b0270 0x7f238c06d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:40.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.263+0000 7f238b7fe640 1 -- 172.21.15.72:0/4173091707 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1099280931 0 0) 0x7f237c003520 con 0x7f238c0d1570 2024-04-15T15:15:40.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.263+0000 7f238b7fe640 1 -- 172.21.15.72:0/4173091707 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f238c156cf0 msgr2=0x7f238c06ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:40.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.263+0000 7f238b7fe640 1 --2- 172.21.15.72:0/4173091707 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f238c156cf0 0x7f238c06ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:40.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.263+0000 7f238b7fe640 1 -- 172.21.15.72:0/4173091707 >> v2:172.21.15.175:3300/0 conn(0x7f238c0b0270 msgr2=0x7f238c06d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:40.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.263+0000 7f238b7fe640 1 --2- 172.21.15.72:0/4173091707 >> v2:172.21.15.175:3300/0 conn(0x7f238c0b0270 0x7f238c06d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:40.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.263+0000 7f238b7fe640 1 -- 172.21.15.72:0/4173091707 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f238c15d3c0 con 0x7f238c0d1570 2024-04-15T15:15:40.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.263+0000 7f238b7fe640 1 -- 172.21.15.72:0/4173091707 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f23840019f0 con 0x7f238c0d1570 2024-04-15T15:15:40.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.263+0000 7f238b7fe640 1 -- 172.21.15.72:0/4173091707 >> v1:172.21.15.72:6789/0 conn(0x7f238c0d1570 legacy=0x7f238c160250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:40.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.263+0000 7f238b7fe640 1 --2- 172.21.15.72:0/4173091707 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f238c156cf0 0x7f237c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:40.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.263+0000 7f238b7fe640 1 --2- 172.21.15.72:0/4173091707 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f237c003f30 0x7f237c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:40.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.263+0000 7f238b7fe640 1 --2- 172.21.15.72:0/4173091707 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f237c004d40 0x7f237c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:40.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.263+0000 7f238b7fe640 1 -- 172.21.15.72:0/4173091707 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f238c06d8b0 con 0x7f237c003f30 2024-04-15T15:15:40.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.263+0000 7f238b7fe640 1 -- 172.21.15.72:0/4173091707 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f238c15d0e0 con 0x7f237c004d40 2024-04-15T15:15:40.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.263+0000 7f238b7fe640 1 -- 172.21.15.72:0/4173091707 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f237c00ba10 con 0x7f238c156cf0 2024-04-15T15:15:40.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.263+0000 7f238bfff640 1 --2- 172.21.15.72:0/4173091707 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f237c003f30 0x7f237c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:40.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.264+0000 7f2390da9640 1 --2- 172.21.15.72:0/4173091707 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f238c156cf0 0x7f237c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:40.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.264+0000 7f23915aa640 1 --2- 172.21.15.72:0/4173091707 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f237c004d40 0x7f237c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:40.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.264+0000 7f238bfff640 1 -- 172.21.15.72:0/4173091707 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f238c156cf0 msgr2=0x7f237c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:40.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.264+0000 7f238bfff640 1 --2- 172.21.15.72:0/4173091707 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f238c156cf0 0x7f237c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:40.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.264+0000 7f238bfff640 1 -- 172.21.15.72:0/4173091707 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f237c004d40 msgr2=0x7f237c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:40.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.264+0000 7f238bfff640 1 --2- 172.21.15.72:0/4173091707 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f237c004d40 0x7f237c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:40.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.264+0000 7f238bfff640 1 -- 172.21.15.72:0/4173091707 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f237c003520 con 0x7f237c003f30 2024-04-15T15:15:40.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.264+0000 7f23915aa640 1 --2- 172.21.15.72:0/4173091707 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f237c004d40 0x7f237c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:40.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.264+0000 7f238bfff640 1 --2- 172.21.15.72:0/4173091707 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f237c003f30 0x7f237c004800 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f2378009af0 tx=0x7f237c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=858eee693f52f9e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:40.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.265+0000 7f238b7fe640 1 -- 172.21.15.72:0/4173091707 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2378039070 con 0x7f237c003f30 2024-04-15T15:15:40.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.265+0000 7f238b7fe640 1 -- 172.21.15.72:0/4173091707 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f237802c710 con 0x7f237c003f30 2024-04-15T15:15:40.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.265+0000 7f238b7fe640 1 -- 172.21.15.72:0/4173091707 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2378009690 con 0x7f237c003f30 2024-04-15T15:15:40.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.265+0000 7f239300d640 1 -- 172.21.15.72:0/4173091707 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f237c003f30 msgr2=0x7f237c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:40.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.265+0000 7f239300d640 1 --2- 172.21.15.72:0/4173091707 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f237c003f30 0x7f237c004800 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f2378009af0 tx=0x7f237c005830 comp rx=0 tx=0).stop 2024-04-15T15:15:40.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.265+0000 7f23915aa640 1 -- 172.21.15.72:0/4173091707 reap_dead start 2024-04-15T15:15:40.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.265+0000 7f239300d640 1 -- 172.21.15.72:0/4173091707 shutdown_connections 2024-04-15T15:15:40.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.265+0000 7f239300d640 1 -- 172.21.15.72:0/4173091707 >> 172.21.15.72:0/4173091707 conn(0x7f238c0d0a50 msgr2=0x7f238c0d0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:40.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.265+0000 7f239300d640 1 -- 172.21.15.72:0/4173091707 shutdown_connections 2024-04-15T15:15:40.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.265+0000 7f239300d640 1 -- 172.21.15.72:0/4173091707 wait complete. 2024-04-15T15:15:40.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.266+0000 7f239300d640 1 Processor -- start 2024-04-15T15:15:40.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.266+0000 7f239300d640 1 -- start start 2024-04-15T15:15:40.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.267+0000 7f239300d640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f237c003f30 0x7f238c1f1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:40.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.267+0000 7f239300d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f238c1516e0 0x7f238c1e6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:40.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.267+0000 7f239300d640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f238c154190 0x7f238c154640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:40.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.267+0000 7f239300d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f238c1570d0 con 0x7f238c1516e0 2024-04-15T15:15:40.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.267+0000 7f239300d640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f238c15d980 con 0x7f237c003f30 2024-04-15T15:15:40.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.267+0000 7f239300d640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f238c06dbb0 con 0x7f238c154190 2024-04-15T15:15:40.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.267+0000 7f238bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f238c1516e0 0x7f238c1e6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:40.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.267+0000 7f238bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f238c1516e0 0x7f238c1e6970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55090/0 (socket says 172.21.15.72:55090) 2024-04-15T15:15:40.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.267+0000 7f238bfff640 1 -- 172.21.15.72:0/1658530879 learned_addr learned my addr 172.21.15.72:0/1658530879 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:40.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.267+0000 7f2390da9640 1 --2- 172.21.15.72:0/1658530879 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f237c003f30 0x7f238c1f1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:40.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.267+0000 7f23915aa640 1 --2- 172.21.15.72:0/1658530879 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f238c154190 0x7f238c154640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:40.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.268+0000 7f238bfff640 1 -- 172.21.15.72:0/1658530879 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f238c154190 msgr2=0x7f238c154640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:40.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.268+0000 7f238bfff640 1 --2- 172.21.15.72:0/1658530879 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f238c154190 0x7f238c154640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:40.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.268+0000 7f238bfff640 1 -- 172.21.15.72:0/1658530879 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f237c003f30 msgr2=0x7f238c1f1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:40.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.268+0000 7f238bfff640 1 --2- 172.21.15.72:0/1658530879 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f237c003f30 0x7f238c1f1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:40.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.268+0000 7f238bfff640 1 -- 172.21.15.72:0/1658530879 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f238c1e72c0 con 0x7f238c1516e0 2024-04-15T15:15:40.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.268+0000 7f2390da9640 1 --2- 172.21.15.72:0/1658530879 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f237c003f30 0x7f238c1f1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:40.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.268+0000 7f238bfff640 1 --2- 172.21.15.72:0/1658530879 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f238c1516e0 0x7f238c1e6970 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f2378037040 tx=0x7f2378004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:40.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.268+0000 7f23897fa640 1 -- 172.21.15.72:0/1658530879 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2378039050 con 0x7f238c1516e0 2024-04-15T15:15:40.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.268+0000 7f23897fa640 1 -- 172.21.15.72:0/1658530879 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2378045070 con 0x7f238c1516e0 2024-04-15T15:15:40.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.268+0000 7f239300d640 1 -- 172.21.15.72:0/1658530879 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f238c1e75a0 con 0x7f238c1516e0 2024-04-15T15:15:40.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.268+0000 7f23897fa640 1 -- 172.21.15.72:0/1658530879 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23780044e0 con 0x7f238c1516e0 2024-04-15T15:15:40.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.268+0000 7f239300d640 1 -- 172.21.15.72:0/1658530879 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f238c1e7b60 con 0x7f238c1516e0 2024-04-15T15:15:40.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.269+0000 7f239300d640 1 -- 172.21.15.72:0/1658530879 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f238c075c50 con 0x7f238c1516e0 2024-04-15T15:15:40.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.269+0000 7f23897fa640 1 -- 172.21.15.72:0/1658530879 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2378003a50 con 0x7f238c1516e0 2024-04-15T15:15:40.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.270+0000 7f23897fa640 1 --2- 172.21.15.72:0/1658530879 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f237006c920 0x7f237006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:40.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.270+0000 7f23897fa640 1 -- 172.21.15.72:0/1658530879 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f23780ada80 con 0x7f238c1516e0 2024-04-15T15:15:40.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.270+0000 7f2390da9640 1 --2- 172.21.15.72:0/1658530879 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f237006c920 0x7f237006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:40.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.270+0000 7f2390da9640 1 --2- 172.21.15.72:0/1658530879 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f237006c920 0x7f237006ede0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2380009ae0 tx=0x7f2380008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:40.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.274+0000 7f23897fa640 1 -- 172.21.15.72:0/1658530879 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f237804e020 con 0x7f238c1516e0 2024-04-15T15:15:40.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.450+0000 7f239300d640 1 -- 172.21.15.72:0/1658530879 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f238c0af8d0 con 0x7f238c1516e0 2024-04-15T15:15:40.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.451+0000 7f23897fa640 1 -- 172.21.15.72:0/1658530879 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f237807bab0 con 0x7f238c1516e0 2024-04-15T15:15:40.452 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:40.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.453+0000 7f239300d640 1 -- 172.21.15.72:0/1658530879 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f237006c920 msgr2=0x7f237006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:40.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.453+0000 7f239300d640 1 --2- 172.21.15.72:0/1658530879 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f237006c920 0x7f237006ede0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2380009ae0 tx=0x7f2380008040 comp rx=0 tx=0).stop 2024-04-15T15:15:40.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.453+0000 7f239300d640 1 -- 172.21.15.72:0/1658530879 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f238c1516e0 msgr2=0x7f238c1e6970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:40.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.453+0000 7f239300d640 1 --2- 172.21.15.72:0/1658530879 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f238c1516e0 0x7f238c1e6970 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f2378037040 tx=0x7f2378004730 comp rx=0 tx=0).stop 2024-04-15T15:15:40.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.453+0000 7f239300d640 1 -- 172.21.15.72:0/1658530879 shutdown_connections 2024-04-15T15:15:40.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.453+0000 7f239300d640 1 --2- 172.21.15.72:0/1658530879 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f237006c920 0x7f237006ede0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:40.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.453+0000 7f239300d640 1 --2- 172.21.15.72:0/1658530879 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f238c154190 0x7f238c154640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:40.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.453+0000 7f239300d640 1 --2- 172.21.15.72:0/1658530879 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f238c1516e0 0x7f238c1e6970 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:40.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.453+0000 7f239300d640 1 --2- 172.21.15.72:0/1658530879 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f237c003f30 0x7f238c1f1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:40.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.453+0000 7f239300d640 1 -- 172.21.15.72:0/1658530879 >> 172.21.15.72:0/1658530879 conn(0x7f238c0d0a50 msgr2=0x7f238c15dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:40.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.453+0000 7f239300d640 1 -- 172.21.15.72:0/1658530879 shutdown_connections 2024-04-15T15:15:40.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.453+0000 7f239300d640 1 -- 172.21.15.72:0/1658530879 wait complete. 2024-04-15T15:15:40.467 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:15:40.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.570+0000 7f099a359640 1 Processor -- start 2024-04-15T15:15:40.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.570+0000 7f099a359640 1 -- start start 2024-04-15T15:15:40.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.571+0000 7f099a359640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f099415e9f0 0x7f09940d3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:40.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.571+0000 7f099a359640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f099406c400 0x7f099406c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:40.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.571+0000 7f099a359640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09940d2320 con 0x7f099415e610 2024-04-15T15:15:40.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.571+0000 7f099a359640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f09940d31d0 con 0x7f099406c400 2024-04-15T15:15:40.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.571+0000 7f099a359640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f09940d25c0 con 0x7f099415e9f0 2024-04-15T15:15:40.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.571+0000 7f09988f6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f099406c400 0x7f099406c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:40.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.571+0000 7f0993fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f099415e610 0x7f09940d0690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50132/0 (socket says 172.21.15.72:50132) 2024-04-15T15:15:40.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.571+0000 7f0993fff640 1 -- 172.21.15.72:0/2486000867 learned_addr learned my addr 172.21.15.72:0/2486000867 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:40.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.571+0000 7f09937fe640 1 --2- 172.21.15.72:0/2486000867 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f099415e9f0 0x7f09940d3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:40.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.571+0000 7f09988f6640 1 -- 172.21.15.72:0/2486000867 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f099415e9f0 msgr2=0x7f09940d3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:40.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.571+0000 7f09988f6640 1 --2- 172.21.15.72:0/2486000867 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f099415e9f0 0x7f09940d3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:40.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.571+0000 7f09988f6640 1 -- 172.21.15.72:0/2486000867 >> v1:172.21.15.72:6789/0 conn(0x7f099415e610 legacy=0x7f09940d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:40.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.571+0000 7f09988f6640 1 -- 172.21.15.72:0/2486000867 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09940d28a0 con 0x7f099406c400 2024-04-15T15:15:40.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.572+0000 7f09988f6640 1 --2- 172.21.15.72:0/2486000867 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f099406c400 0x7f099406c890 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f098c009d90 tx=0x7f098c02bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=359c501d7c2efb51 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:40.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.572+0000 7f0992ffd640 1 -- 172.21.15.72:0/2486000867 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f098c039070 con 0x7f099406c400 2024-04-15T15:15:40.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.572+0000 7f0992ffd640 1 -- 172.21.15.72:0/2486000867 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f098c02cce0 con 0x7f099406c400 2024-04-15T15:15:40.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.572+0000 7f0992ffd640 1 -- 172.21.15.72:0/2486000867 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f098c034660 con 0x7f099406c400 2024-04-15T15:15:40.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.573+0000 7f099a359640 1 -- 172.21.15.72:0/2486000867 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f099406c400 msgr2=0x7f099406c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:40.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.573+0000 7f099a359640 1 --2- 172.21.15.72:0/2486000867 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f099406c400 0x7f099406c890 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f098c009d90 tx=0x7f098c02bf30 comp rx=0 tx=0).stop 2024-04-15T15:15:40.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.573+0000 7f099a359640 1 -- 172.21.15.72:0/2486000867 shutdown_connections 2024-04-15T15:15:40.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.573+0000 7f099a359640 1 --2- 172.21.15.72:0/2486000867 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f099406c400 0x7f099406c890 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:40.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.573+0000 7f099a359640 1 --2- 172.21.15.72:0/2486000867 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f099415e9f0 0x7f09940d3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:40.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.573+0000 7f099a359640 1 -- 172.21.15.72:0/2486000867 >> 172.21.15.72:0/2486000867 conn(0x7f09940d4750 msgr2=0x7f09940d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:40.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.573+0000 7f099a359640 1 -- 172.21.15.72:0/2486000867 shutdown_connections 2024-04-15T15:15:40.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.573+0000 7f099a359640 1 -- 172.21.15.72:0/2486000867 wait complete. 2024-04-15T15:15:40.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.574+0000 7f099a359640 1 Processor -- start 2024-04-15T15:15:40.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.574+0000 7f099a359640 1 -- start start 2024-04-15T15:15:40.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.575+0000 7f099a359640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f099406c400 0x7f099415c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:40.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.575+0000 7f099a359640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f099415e610 0x7f0994161750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:40.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.575+0000 7f099a359640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f099415e9f0 0x7f09940ccdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:40.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.575+0000 7f099a359640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f099406d130 con 0x7f099406c400 2024-04-15T15:15:40.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.575+0000 7f099a359640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f09940d2e60 con 0x7f099415e9f0 2024-04-15T15:15:40.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.575+0000 7f099a359640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f09940d3ba0 con 0x7f099415e610 2024-04-15T15:15:40.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.575+0000 7f0993fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f099406c400 0x7f099415c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:40.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.575+0000 7f0993fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f099406c400 0x7f099415c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55102/0 (socket says 172.21.15.72:55102) 2024-04-15T15:15:40.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.575+0000 7f0993fff640 1 -- 172.21.15.72:0/3908100228 learned_addr learned my addr 172.21.15.72:0/3908100228 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:40.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.575+0000 7f09937fe640 1 --2- 172.21.15.72:0/3908100228 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f099415e610 0x7f0994161750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:40.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.575+0000 7f09988f6640 1 --2- 172.21.15.72:0/3908100228 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f099415e9f0 0x7f09940ccdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:40.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.575+0000 7f0993fff640 1 -- 172.21.15.72:0/3908100228 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f099415e610 msgr2=0x7f0994161750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:40.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.575+0000 7f0993fff640 1 --2- 172.21.15.72:0/3908100228 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f099415e610 0x7f0994161750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:40.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.575+0000 7f0993fff640 1 -- 172.21.15.72:0/3908100228 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f099415e9f0 msgr2=0x7f09940ccdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:40.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.575+0000 7f0993fff640 1 --2- 172.21.15.72:0/3908100228 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f099415e9f0 0x7f09940ccdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:40.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.575+0000 7f0993fff640 1 -- 172.21.15.72:0/3908100228 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09940cbd50 con 0x7f099406c400 2024-04-15T15:15:40.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.575+0000 7f09988f6640 1 --2- 172.21.15.72:0/3908100228 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f099415e9f0 0x7f09940ccdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:40.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.575+0000 7f0993fff640 1 --2- 172.21.15.72:0/3908100228 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f099406c400 0x7f099415c800 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f098400c400 tx=0x7f098400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:40.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.575+0000 7f0990ff9640 1 -- 172.21.15.72:0/3908100228 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0984017070 con 0x7f099406c400 2024-04-15T15:15:40.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.575+0000 7f0990ff9640 1 -- 172.21.15.72:0/3908100228 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0984005ce0 con 0x7f099406c400 2024-04-15T15:15:40.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.575+0000 7f099a359640 1 -- 172.21.15.72:0/3908100228 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09940cc030 con 0x7f099406c400 2024-04-15T15:15:40.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.576+0000 7f0990ff9640 1 -- 172.21.15.72:0/3908100228 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09840072f0 con 0x7f099406c400 2024-04-15T15:15:40.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.576+0000 7f099a359640 1 -- 172.21.15.72:0/3908100228 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09940cc5f0 con 0x7f099406c400 2024-04-15T15:15:40.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.576+0000 7f099a359640 1 -- 172.21.15.72:0/3908100228 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0994075c50 con 0x7f099406c400 2024-04-15T15:15:40.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.577+0000 7f0990ff9640 1 -- 172.21.15.72:0/3908100228 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f098401a070 con 0x7f099406c400 2024-04-15T15:15:40.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.577+0000 7f0990ff9640 1 --2- 172.21.15.72:0/3908100228 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f098006c8f0 0x7f098006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:40.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.577+0000 7f0990ff9640 1 -- 172.21.15.72:0/3908100228 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f098408b340 con 0x7f099406c400 2024-04-15T15:15:40.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.578+0000 7f09937fe640 1 --2- 172.21.15.72:0/3908100228 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f098006c8f0 0x7f098006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:40.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.578+0000 7f09937fe640 1 --2- 172.21.15.72:0/3908100228 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f098006c8f0 0x7f098006edb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0988006fd0 tx=0x7f0988008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:40.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.582+0000 7f0990ff9640 1 -- 172.21.15.72:0/3908100228 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0984059370 con 0x7f099406c400 2024-04-15T15:15:40.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.767+0000 7f099a359640 1 -- 172.21.15.72:0/3908100228 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f099406bc40 con 0x7f099406c400 2024-04-15T15:15:40.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.767+0000 7f0990ff9640 1 -- 172.21.15.72:0/3908100228 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7f098405f070 con 0x7f099406c400 2024-04-15T15:15:40.769 INFO:teuthology.orchestra.run.smithi072.stderr:listed 0 entries 2024-04-15T15:15:40.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.769+0000 7f099a359640 1 -- 172.21.15.72:0/3908100228 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f098006c8f0 msgr2=0x7f098006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:40.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.769+0000 7f099a359640 1 --2- 172.21.15.72:0/3908100228 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f098006c8f0 0x7f098006edb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0988006fd0 tx=0x7f0988008040 comp rx=0 tx=0).stop 2024-04-15T15:15:40.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.769+0000 7f099a359640 1 -- 172.21.15.72:0/3908100228 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f099406c400 msgr2=0x7f099415c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:40.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.769+0000 7f099a359640 1 --2- 172.21.15.72:0/3908100228 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f099406c400 0x7f099415c800 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f098400c400 tx=0x7f098400c8c0 comp rx=0 tx=0).stop 2024-04-15T15:15:40.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.769+0000 7f099a359640 1 -- 172.21.15.72:0/3908100228 shutdown_connections 2024-04-15T15:15:40.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.769+0000 7f099a359640 1 --2- 172.21.15.72:0/3908100228 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f098006c8f0 0x7f098006edb0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:40.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.769+0000 7f099a359640 1 --2- 172.21.15.72:0/3908100228 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f099415e9f0 0x7f09940ccdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:40.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.769+0000 7f099a359640 1 --2- 172.21.15.72:0/3908100228 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f099415e610 0x7f0994161750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:40.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.769+0000 7f099a359640 1 --2- 172.21.15.72:0/3908100228 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f099406c400 0x7f099415c800 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:40.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.769+0000 7f099a359640 1 -- 172.21.15.72:0/3908100228 >> 172.21.15.72:0/3908100228 conn(0x7f09940d4750 msgr2=0x7f09940b42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:40.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.769+0000 7f099a359640 1 -- 172.21.15.72:0/3908100228 shutdown_connections 2024-04-15T15:15:40.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:40.769+0000 7f099a359640 1 -- 172.21.15.72:0/3908100228 wait complete. 2024-04-15T15:15:40.782 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi072.front.sepia.ceph.com... 2024-04-15T15:15:40.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:40.782 DEBUG:teuthology.orchestra.run.smithi072:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-15T15:15:40.811 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:15:40.812 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi072.front.sepia.ceph.com 2024-04-15T15:15:40.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:40.812 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-15T15:15:40.868 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:15:40.872 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:15:40.892 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:15:40.892 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:15:40.952 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:41.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.053+0000 7f1d97196640 1 Processor -- start 2024-04-15T15:15:41.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.053+0000 7f1d97196640 1 -- start start 2024-04-15T15:15:41.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.053+0000 7f1d97196640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1d900d12b0 0x7f1d900d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.053+0000 7f1d97196640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d90162220 con 0x7f1d900d2ea0 2024-04-15T15:15:41.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.053+0000 7f1d97196640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d900d1820 con 0x7f1d90161e40 2024-04-15T15:15:41.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.053+0000 7f1d97196640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f1d900b3f00 con 0x7f1d900d12b0 2024-04-15T15:15:41.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.054+0000 7f1d8ffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1d900d12b0 0x7f1d900d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.054+0000 7f1d95733640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f1d900d2ea0 0x7f1d900d45b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50136/0 (socket says 172.21.15.72:50136) 2024-04-15T15:15:41.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.054+0000 7f1d95733640 1 -- 172.21.15.72:0/553346119 learned_addr learned my addr 172.21.15.72:0/553346119 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:41.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.054+0000 7f1d8ffff640 1 -- 172.21.15.72:0/553346119 >> v1:172.21.15.175:6789/0 conn(0x7f1d90161e40 legacy=0x7f1d900b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:41.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.054+0000 7f1d8ffff640 1 -- 172.21.15.72:0/553346119 >> v1:172.21.15.72:6789/0 conn(0x7f1d900d2ea0 legacy=0x7f1d900d45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:41.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.054+0000 7f1d8ffff640 1 -- 172.21.15.72:0/553346119 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d900d1b00 con 0x7f1d900d12b0 2024-04-15T15:15:41.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.054+0000 7f1d8f7fe640 1 -- 172.21.15.72:0/553346119 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2297647723 0 0) 0x7f1d90162220 con 0x7f1d900d2ea0 2024-04-15T15:15:41.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.055+0000 7f1d8ffff640 1 --2- 172.21.15.72:0/553346119 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d900d12b0 0x7f1d900d2960 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c0097d0 tx=0x7f1d7c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85cc2524a406bed6 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:41.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.055+0000 7f1d8f7fe640 1 -- 172.21.15.72:0/553346119 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d7c0315a0 con 0x7f1d900d12b0 2024-04-15T15:15:41.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.055+0000 7f1d8f7fe640 1 -- 172.21.15.72:0/553346119 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d7c037070 con 0x7f1d900d12b0 2024-04-15T15:15:41.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.055+0000 7f1d8f7fe640 1 -- 172.21.15.72:0/553346119 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d7c031960 con 0x7f1d900d12b0 2024-04-15T15:15:41.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.055+0000 7f1d97196640 1 -- 172.21.15.72:0/553346119 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d900d12b0 msgr2=0x7f1d900d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:41.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.055+0000 7f1d97196640 1 --2- 172.21.15.72:0/553346119 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d900d12b0 0x7f1d900d2960 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c0097d0 tx=0x7f1d7c02b840 comp rx=0 tx=0).stop 2024-04-15T15:15:41.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.055+0000 7f1d97196640 1 -- 172.21.15.72:0/553346119 shutdown_connections 2024-04-15T15:15:41.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.055+0000 7f1d97196640 1 --2- 172.21.15.72:0/553346119 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d900d12b0 0x7f1d900d2960 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.055+0000 7f1d97196640 1 -- 172.21.15.72:0/553346119 >> 172.21.15.72:0/553346119 conn(0x7f1d9009d970 msgr2=0x7f1d900b95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:41.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.055+0000 7f1d97196640 1 -- 172.21.15.72:0/553346119 shutdown_connections 2024-04-15T15:15:41.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.055+0000 7f1d97196640 1 -- 172.21.15.72:0/553346119 wait complete. 2024-04-15T15:15:41.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.056+0000 7f1d97196640 1 Processor -- start 2024-04-15T15:15:41.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.057+0000 7f1d97196640 1 -- start start 2024-04-15T15:15:41.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.057+0000 7f1d97196640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d900d12b0 0x7f1d900c8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.057+0000 7f1d97196640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d90161e40 0x7f1d900ca9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.057+0000 7f1d97196640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1d9006dc20 0x7f1d900b55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.057+0000 7f1d97196640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f1d900d2440 con 0x7f1d90161e40 2024-04-15T15:15:41.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.057+0000 7f1d97196640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f1d900d21f0 con 0x7f1d900d12b0 2024-04-15T15:15:41.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.057+0000 7f1d97196640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f1d900d4cc0 con 0x7f1d9006dc20 2024-04-15T15:15:41.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.058+0000 7f1d8ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d90161e40 0x7f1d900ca9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.058+0000 7f1d8ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d90161e40 0x7f1d900ca9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55118/0 (socket says 172.21.15.72:55118) 2024-04-15T15:15:41.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.058+0000 7f1d8ffff640 1 -- 172.21.15.72:0/3815564935 learned_addr learned my addr 172.21.15.72:0/3815564935 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:41.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.058+0000 7f1d94f32640 1 --2- 172.21.15.72:0/3815564935 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d900d12b0 0x7f1d900c8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.058+0000 7f1d95733640 1 --2- 172.21.15.72:0/3815564935 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1d9006dc20 0x7f1d900b55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.058+0000 7f1d8ffff640 1 -- 172.21.15.72:0/3815564935 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1d9006dc20 msgr2=0x7f1d900b55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:41.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.058+0000 7f1d8ffff640 1 --2- 172.21.15.72:0/3815564935 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1d9006dc20 0x7f1d900b55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.058+0000 7f1d8ffff640 1 -- 172.21.15.72:0/3815564935 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d900d12b0 msgr2=0x7f1d900c8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:41.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.058+0000 7f1d8ffff640 1 --2- 172.21.15.72:0/3815564935 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d900d12b0 0x7f1d900c8a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.058+0000 7f1d8ffff640 1 -- 172.21.15.72:0/3815564935 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d900b50c0 con 0x7f1d90161e40 2024-04-15T15:15:41.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.058+0000 7f1d94f32640 1 --2- 172.21.15.72:0/3815564935 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d900d12b0 0x7f1d900c8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:41.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.058+0000 7f1d8ffff640 1 --2- 172.21.15.72:0/3815564935 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d90161e40 0x7f1d900ca9f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c009720 tx=0x7f1d7c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:41.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.058+0000 7f1d8d7fa640 1 -- 172.21.15.72:0/3815564935 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d7c032840 con 0x7f1d90161e40 2024-04-15T15:15:41.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.058+0000 7f1d8d7fa640 1 -- 172.21.15.72:0/3815564935 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d7c0329e0 con 0x7f1d90161e40 2024-04-15T15:15:41.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.058+0000 7f1d97196640 1 -- 172.21.15.72:0/3815564935 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d900c9170 con 0x7f1d90161e40 2024-04-15T15:15:41.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.058+0000 7f1d8d7fa640 1 -- 172.21.15.72:0/3815564935 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d7c032dd0 con 0x7f1d90161e40 2024-04-15T15:15:41.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.058+0000 7f1d97196640 1 -- 172.21.15.72:0/3815564935 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d900c9730 con 0x7f1d90161e40 2024-04-15T15:15:41.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.059+0000 7f1d97196640 1 -- 172.21.15.72:0/3815564935 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d900b5290 con 0x7f1d90161e40 2024-04-15T15:15:41.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.060+0000 7f1d8d7fa640 1 -- 172.21.15.72:0/3815564935 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1d7c03b3f0 con 0x7f1d90161e40 2024-04-15T15:15:41.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.060+0000 7f1d8d7fa640 1 --2- 172.21.15.72:0/3815564935 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1d7406c8f0 0x7f1d7406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.060+0000 7f1d8d7fa640 1 -- 172.21.15.72:0/3815564935 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f1d7c02f030 con 0x7f1d90161e40 2024-04-15T15:15:41.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.060+0000 7f1d94f32640 1 --2- 172.21.15.72:0/3815564935 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1d7406c8f0 0x7f1d7406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.061+0000 7f1d94f32640 1 --2- 172.21.15.72:0/3815564935 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1d7406c8f0 0x7f1d7406edb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1d8400a830 tx=0x7f1d84008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:41.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.065+0000 7f1d8d7fa640 1 -- 172.21.15.72:0/3815564935 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1d7c040070 con 0x7f1d90161e40 2024-04-15T15:15:41.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.240+0000 7f1d97196640 1 -- 172.21.15.72:0/3815564935 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1d900689e0 con 0x7f1d90161e40 2024-04-15T15:15:41.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.241+0000 7f1d8d7fa640 1 -- 172.21.15.72:0/3815564935 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18903 (secure 0 0 0) 0x7f1d7c07bf40 con 0x7f1d90161e40 2024-04-15T15:15:41.243 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:41.243 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":26,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:15:32.421728+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:14:49.438366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:14:50.368379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:14:54.081157+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":1}]},{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:41.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.243+0000 7f1d97196640 1 -- 172.21.15.72:0/3815564935 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1d7406c8f0 msgr2=0x7f1d7406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:41.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.243+0000 7f1d97196640 1 --2- 172.21.15.72:0/3815564935 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1d7406c8f0 0x7f1d7406edb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1d8400a830 tx=0x7f1d84008040 comp rx=0 tx=0).stop 2024-04-15T15:15:41.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.243+0000 7f1d97196640 1 -- 172.21.15.72:0/3815564935 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d90161e40 msgr2=0x7f1d900ca9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:41.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.243+0000 7f1d97196640 1 --2- 172.21.15.72:0/3815564935 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d90161e40 0x7f1d900ca9f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c009720 tx=0x7f1d7c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.243+0000 7f1d97196640 1 -- 172.21.15.72:0/3815564935 shutdown_connections 2024-04-15T15:15:41.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.243+0000 7f1d97196640 1 --2- 172.21.15.72:0/3815564935 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1d7406c8f0 0x7f1d7406edb0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.243+0000 7f1d97196640 1 --2- 172.21.15.72:0/3815564935 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1d9006dc20 0x7f1d900b55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.243+0000 7f1d97196640 1 --2- 172.21.15.72:0/3815564935 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1d90161e40 0x7f1d900ca9f0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.243+0000 7f1d97196640 1 --2- 172.21.15.72:0/3815564935 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1d900d12b0 0x7f1d900c8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.243+0000 7f1d97196640 1 -- 172.21.15.72:0/3815564935 >> 172.21.15.72:0/3815564935 conn(0x7f1d9009d970 msgr2=0x7f1d900b95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:41.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.243+0000 7f1d97196640 1 -- 172.21.15.72:0/3815564935 shutdown_connections 2024-04-15T15:15:41.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.244+0000 7f1d97196640 1 -- 172.21.15.72:0/3815564935 wait complete. 2024-04-15T15:15:41.256 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:39.359102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:14:49.438366+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:14:50.368379+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:14:54.081157+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:15:41.257 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:41.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.363+0000 7f95164a5640 1 Processor -- start 2024-04-15T15:15:41.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.363+0000 7f95164a5640 1 -- start start 2024-04-15T15:15:41.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.363+0000 7f95164a5640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f951015e610 0x7f95100d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.363+0000 7f95164a5640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f951015e9f0 0x7f95100d3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.363+0000 7f95164a5640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95100d2320 con 0x7f951006c400 2024-04-15T15:15:41.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.363+0000 7f95164a5640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f95100d31d0 con 0x7f951015e9f0 2024-04-15T15:15:41.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.363+0000 7f95164a5640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f95100d25c0 con 0x7f951015e610 2024-04-15T15:15:41.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.364+0000 7f9514a42640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f951006c400 0x7f95100b7940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:50148/0 (socket says 172.21.15.72:50148) 2024-04-15T15:15:41.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.364+0000 7f9514a42640 1 -- 172.21.15.72:0/3725679344 learned_addr learned my addr 172.21.15.72:0/3725679344 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:41.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.364+0000 7f950ffff640 1 --2- 172.21.15.72:0/3725679344 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f951015e610 0x7f95100d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.364+0000 7f950f7fe640 1 --2- 172.21.15.72:0/3725679344 >> v2:172.21.15.175:3300/0 conn(0x7f951015e9f0 0x7f95100d3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.364+0000 7f950effd640 1 -- 172.21.15.72:0/3725679344 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4276243890 0 0) 0x7f95100d2320 con 0x7f951006c400 2024-04-15T15:15:41.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.364+0000 7f950effd640 1 -- 172.21.15.72:0/3725679344 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f94f8003520 con 0x7f951006c400 2024-04-15T15:15:41.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.364+0000 7f950effd640 1 -- 172.21.15.72:0/3725679344 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2685828068 0 0) 0x7f94f8003520 con 0x7f951006c400 2024-04-15T15:15:41.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.364+0000 7f950effd640 1 -- 172.21.15.72:0/3725679344 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f951015e610 msgr2=0x7f95100d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:41.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.364+0000 7f950effd640 1 --2- 172.21.15.72:0/3725679344 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f951015e610 0x7f95100d0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.364+0000 7f950effd640 1 -- 172.21.15.72:0/3725679344 >> v2:172.21.15.175:3300/0 conn(0x7f951015e9f0 msgr2=0x7f95100d3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:41.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.364+0000 7f950effd640 1 --2- 172.21.15.72:0/3725679344 >> v2:172.21.15.175:3300/0 conn(0x7f951015e9f0 0x7f95100d3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.364+0000 7f950effd640 1 -- 172.21.15.72:0/3725679344 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95100d28a0 con 0x7f951006c400 2024-04-15T15:15:41.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.364+0000 7f950effd640 1 -- 172.21.15.72:0/3725679344 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f95080019f0 con 0x7f951006c400 2024-04-15T15:15:41.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.364+0000 7f950effd640 1 -- 172.21.15.72:0/3725679344 >> v1:172.21.15.72:6789/0 conn(0x7f951006c400 legacy=0x7f95100b7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:41.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.364+0000 7f950effd640 1 --2- 172.21.15.72:0/3725679344 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94f80039b0 0x7f94f8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.364+0000 7f950effd640 1 --2- 172.21.15.72:0/3725679344 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f951015e610 0x7f94f8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.364+0000 7f950effd640 1 --2- 172.21.15.72:0/3725679344 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94f8004d40 0x7f94f80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.364+0000 7f950effd640 1 -- 172.21.15.72:0/3725679344 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f95100d31d0 con 0x7f94f8004d40 2024-04-15T15:15:41.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.364+0000 7f950effd640 1 -- 172.21.15.72:0/3725679344 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f95100d25c0 con 0x7f94f80039b0 2024-04-15T15:15:41.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.364+0000 7f950effd640 1 -- 172.21.15.72:0/3725679344 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f94f800b9c0 con 0x7f951015e610 2024-04-15T15:15:41.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.364+0000 7f9514a42640 1 --2- 172.21.15.72:0/3725679344 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94f8004d40 0x7f94f80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.364+0000 7f950ffff640 1 --2- 172.21.15.72:0/3725679344 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94f80039b0 0x7f94f8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.364+0000 7f950f7fe640 1 --2- 172.21.15.72:0/3725679344 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f951015e610 0x7f94f8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.365+0000 7f9514a42640 1 -- 172.21.15.72:0/3725679344 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f951015e610 msgr2=0x7f94f8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:41.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.365+0000 7f9514a42640 1 --2- 172.21.15.72:0/3725679344 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f951015e610 0x7f94f8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.365+0000 7f9514a42640 1 -- 172.21.15.72:0/3725679344 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94f80039b0 msgr2=0x7f94f8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:41.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.365+0000 7f9514a42640 1 --2- 172.21.15.72:0/3725679344 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94f80039b0 0x7f94f8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.365+0000 7f9514a42640 1 -- 172.21.15.72:0/3725679344 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94f8003520 con 0x7f94f8004d40 2024-04-15T15:15:41.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.365+0000 7f950ffff640 1 --2- 172.21.15.72:0/3725679344 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94f80039b0 0x7f94f8003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:41.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.365+0000 7f950f7fe640 1 --2- 172.21.15.72:0/3725679344 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f951015e610 0x7f94f8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:41.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.365+0000 7f9514a42640 1 --2- 172.21.15.72:0/3725679344 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94f8004d40 0x7f94f80051c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f950800a880 tx=0x7f94f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70dbd8eb9f8ef295 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:41.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.365+0000 7f950effd640 1 -- 172.21.15.72:0/3725679344 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f950800a4a0 con 0x7f94f8004d40 2024-04-15T15:15:41.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.365+0000 7f950effd640 1 -- 172.21.15.72:0/3725679344 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9508034650 con 0x7f94f8004d40 2024-04-15T15:15:41.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.365+0000 7f950effd640 1 -- 172.21.15.72:0/3725679344 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9508034a40 con 0x7f94f8004d40 2024-04-15T15:15:41.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.366+0000 7f95164a5640 1 -- 172.21.15.72:0/3725679344 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94f8004d40 msgr2=0x7f94f80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:41.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.366+0000 7f95164a5640 1 --2- 172.21.15.72:0/3725679344 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94f8004d40 0x7f94f80051c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f950800a880 tx=0x7f94f8005830 comp rx=0 tx=0).stop 2024-04-15T15:15:41.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.366+0000 7f9514a42640 1 -- 172.21.15.72:0/3725679344 reap_dead start 2024-04-15T15:15:41.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.366+0000 7f95164a5640 1 -- 172.21.15.72:0/3725679344 shutdown_connections 2024-04-15T15:15:41.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.366+0000 7f95164a5640 1 -- 172.21.15.72:0/3725679344 >> 172.21.15.72:0/3725679344 conn(0x7f95100d4750 msgr2=0x7f95100d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:41.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.366+0000 7f95164a5640 1 -- 172.21.15.72:0/3725679344 shutdown_connections 2024-04-15T15:15:41.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.366+0000 7f95164a5640 1 -- 172.21.15.72:0/3725679344 wait complete. 2024-04-15T15:15:41.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.367+0000 7f95164a5640 1 Processor -- start 2024-04-15T15:15:41.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.367+0000 7f95164a5640 1 -- start start 2024-04-15T15:15:41.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.368+0000 7f95164a5640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f95101526b0 0x7f9510145460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.368+0000 7f95164a5640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9510161890 0x7f9510159860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.368+0000 7f95164a5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9510159da0 0x7f9510142830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.368+0000 7f95164a5640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f951006d130 con 0x7f9510159da0 2024-04-15T15:15:41.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.368+0000 7f95164a5640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f95100d2e60 con 0x7f95101526b0 2024-04-15T15:15:41.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.368+0000 7f95164a5640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f95100d3ba0 con 0x7f9510161890 2024-04-15T15:15:41.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.368+0000 7f9514a42640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9510159da0 0x7f9510142830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.368+0000 7f9514a42640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9510159da0 0x7f9510142830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55122/0 (socket says 172.21.15.72:55122) 2024-04-15T15:15:41.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.368+0000 7f9514a42640 1 -- 172.21.15.72:0/1780828271 learned_addr learned my addr 172.21.15.72:0/1780828271 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:41.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.368+0000 7f950ffff640 1 --2- 172.21.15.72:0/1780828271 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f95101526b0 0x7f9510145460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.368+0000 7f950f7fe640 1 --2- 172.21.15.72:0/1780828271 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9510161890 0x7f9510159860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.368+0000 7f9514a42640 1 -- 172.21.15.72:0/1780828271 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9510161890 msgr2=0x7f9510159860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:41.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.368+0000 7f9514a42640 1 --2- 172.21.15.72:0/1780828271 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9510161890 0x7f9510159860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.368+0000 7f9514a42640 1 -- 172.21.15.72:0/1780828271 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f95101526b0 msgr2=0x7f9510145460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:41.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.368+0000 7f9514a42640 1 --2- 172.21.15.72:0/1780828271 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f95101526b0 0x7f9510145460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.368+0000 7f9514a42640 1 -- 172.21.15.72:0/1780828271 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95101587c0 con 0x7f9510159da0 2024-04-15T15:15:41.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.368+0000 7f950ffff640 1 --2- 172.21.15.72:0/1780828271 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f95101526b0 0x7f9510145460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:41.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.368+0000 7f950f7fe640 1 --2- 172.21.15.72:0/1780828271 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9510161890 0x7f9510159860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:41.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.369+0000 7f9514a42640 1 --2- 172.21.15.72:0/1780828271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9510159da0 0x7f9510142830 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f950800a970 tx=0x7f95100d3140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:41.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.369+0000 7f950cff9640 1 -- 172.21.15.72:0/1780828271 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9508045070 con 0x7f9510159da0 2024-04-15T15:15:41.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.369+0000 7f950cff9640 1 -- 172.21.15.72:0/1780828271 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f950803a040 con 0x7f9510159da0 2024-04-15T15:15:41.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.369+0000 7f95164a5640 1 -- 172.21.15.72:0/1780828271 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9510158aa0 con 0x7f9510159da0 2024-04-15T15:15:41.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.369+0000 7f950cff9640 1 -- 172.21.15.72:0/1780828271 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f950800a4a0 con 0x7f9510159da0 2024-04-15T15:15:41.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.369+0000 7f95164a5640 1 -- 172.21.15.72:0/1780828271 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9510159060 con 0x7f9510159da0 2024-04-15T15:15:41.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.370+0000 7f95164a5640 1 -- 172.21.15.72:0/1780828271 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9510075c50 con 0x7f9510159da0 2024-04-15T15:15:41.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.370+0000 7f950cff9640 1 -- 172.21.15.72:0/1780828271 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9508048070 con 0x7f9510159da0 2024-04-15T15:15:41.379 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.371+0000 7f950cff9640 1 --2- 172.21.15.72:0/1780828271 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f94fc06c8f0 0x7f94fc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.379 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.371+0000 7f950cff9640 1 -- 172.21.15.72:0/1780828271 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f95080ada70 con 0x7f9510159da0 2024-04-15T15:15:41.379 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.371+0000 7f950ffff640 1 --2- 172.21.15.72:0/1780828271 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f94fc06c8f0 0x7f94fc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.379 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.371+0000 7f950ffff640 1 --2- 172.21.15.72:0/1780828271 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f94fc06c8f0 0x7f94fc06edb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9500009990 tx=0x7f9500008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:41.379 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.375+0000 7f950cff9640 1 -- 172.21.15.72:0/1780828271 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f950807b3c0 con 0x7f9510159da0 2024-04-15T15:15:41.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.552+0000 7f95164a5640 1 -- 172.21.15.72:0/1780828271 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f951015dfb0 con 0x7f9510159da0 2024-04-15T15:15:41.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.553+0000 7f950cff9640 1 -- 172.21.15.72:0/1780828271 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f950807f6c0 con 0x7f9510159da0 2024-04-15T15:15:41.554 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-15T15:15:41.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.554+0000 7f95164a5640 1 -- 172.21.15.72:0/1780828271 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f94fc06c8f0 msgr2=0x7f94fc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:41.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.554+0000 7f95164a5640 1 --2- 172.21.15.72:0/1780828271 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f94fc06c8f0 0x7f94fc06edb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9500009990 tx=0x7f9500008040 comp rx=0 tx=0).stop 2024-04-15T15:15:41.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.554+0000 7f95164a5640 1 -- 172.21.15.72:0/1780828271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9510159da0 msgr2=0x7f9510142830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:41.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.554+0000 7f95164a5640 1 --2- 172.21.15.72:0/1780828271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9510159da0 0x7f9510142830 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f950800a970 tx=0x7f95100d3140 comp rx=0 tx=0).stop 2024-04-15T15:15:41.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.555+0000 7f95164a5640 1 -- 172.21.15.72:0/1780828271 shutdown_connections 2024-04-15T15:15:41.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.555+0000 7f95164a5640 1 --2- 172.21.15.72:0/1780828271 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f94fc06c8f0 0x7f94fc06edb0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.555+0000 7f95164a5640 1 --2- 172.21.15.72:0/1780828271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9510159da0 0x7f9510142830 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.555+0000 7f95164a5640 1 --2- 172.21.15.72:0/1780828271 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9510161890 0x7f9510159860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.555+0000 7f95164a5640 1 --2- 172.21.15.72:0/1780828271 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f95101526b0 0x7f9510145460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.555+0000 7f95164a5640 1 -- 172.21.15.72:0/1780828271 >> 172.21.15.72:0/1780828271 conn(0x7f95100d4750 msgr2=0x7f95100b4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:41.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.555+0000 7f95164a5640 1 -- 172.21.15.72:0/1780828271 shutdown_connections 2024-04-15T15:15:41.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.555+0000 7f95164a5640 1 -- 172.21.15.72:0/1780828271 wait complete. 2024-04-15T15:15:41.567 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:15:41.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.671+0000 7f42e262a640 1 Processor -- start 2024-04-15T15:15:41.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.671+0000 7f42e262a640 1 -- start start 2024-04-15T15:15:41.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.672+0000 7f42e262a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f42dc0d1810 0x7f42dc0b6750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.672+0000 7f42e262a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42dc160ff0 con 0x7f42dc15c5f0 2024-04-15T15:15:41.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.672+0000 7f42e262a640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42dc0b5840 con 0x7f42dc0ce190 2024-04-15T15:15:41.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.672+0000 7f42e262a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f42dc160da0 con 0x7f42dc0d1810 2024-04-15T15:15:41.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.672+0000 7f42e0bc7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f42dc0d1810 0x7f42dc0b6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.672+0000 7f42dbfff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f42dc15c5f0 0x7f42dc0d1100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:59990/0 (socket says 172.21.15.72:59990) 2024-04-15T15:15:41.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.672+0000 7f42e0bc7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f42dc0d1810 0x7f42dc0b6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46026/0 (socket says 172.21.15.72:46026) 2024-04-15T15:15:41.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.672+0000 7f42dbfff640 1 -- 172.21.15.72:0/1548034918 learned_addr learned my addr 172.21.15.72:0/1548034918 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:41.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.672+0000 7f42e0bc7640 1 -- 172.21.15.72:0/1548034918 >> v1:172.21.15.175:6789/0 conn(0x7f42dc0ce190 legacy=0x7f42dc0b6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:41.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.672+0000 7f42e0bc7640 1 -- 172.21.15.72:0/1548034918 >> v1:172.21.15.72:6789/0 conn(0x7f42dc15c5f0 legacy=0x7f42dc0d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:41.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.672+0000 7f42e0bc7640 1 -- 172.21.15.72:0/1548034918 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42dc0d2150 con 0x7f42dc0d1810 2024-04-15T15:15:41.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.673+0000 7f42e0bc7640 1 --2- 172.21.15.72:0/1548034918 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42dc0d1810 0x7f42dc0b6750 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f42d4009d90 tx=0x7f42d402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8573ecace5a103cb server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:41.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.673+0000 7f42daffd640 1 -- 172.21.15.72:0/1548034918 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d4039070 con 0x7f42dc0d1810 2024-04-15T15:15:41.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.673+0000 7f42daffd640 1 -- 172.21.15.72:0/1548034918 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42d402cbd0 con 0x7f42dc0d1810 2024-04-15T15:15:41.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.673+0000 7f42daffd640 1 -- 172.21.15.72:0/1548034918 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d4034440 con 0x7f42dc0d1810 2024-04-15T15:15:41.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.674+0000 7f42e262a640 1 -- 172.21.15.72:0/1548034918 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42dc0d1810 msgr2=0x7f42dc0b6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:41.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.674+0000 7f42e262a640 1 --2- 172.21.15.72:0/1548034918 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42dc0d1810 0x7f42dc0b6750 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f42d4009d90 tx=0x7f42d402be20 comp rx=0 tx=0).stop 2024-04-15T15:15:41.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.674+0000 7f42e262a640 1 -- 172.21.15.72:0/1548034918 shutdown_connections 2024-04-15T15:15:41.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.674+0000 7f42e262a640 1 --2- 172.21.15.72:0/1548034918 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42dc0d1810 0x7f42dc0b6750 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.674+0000 7f42e262a640 1 -- 172.21.15.72:0/1548034918 >> 172.21.15.72:0/1548034918 conn(0x7f42dc0b4010 msgr2=0x7f42dc0b4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:41.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.674+0000 7f42e262a640 1 -- 172.21.15.72:0/1548034918 shutdown_connections 2024-04-15T15:15:41.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.674+0000 7f42e262a640 1 -- 172.21.15.72:0/1548034918 wait complete. 2024-04-15T15:15:41.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.675+0000 7f42e262a640 1 Processor -- start 2024-04-15T15:15:41.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.675+0000 7f42e262a640 1 -- start start 2024-04-15T15:15:41.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.676+0000 7f42e262a640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f42dc0d1810 0x7f42dc156ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.676+0000 7f42e262a640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f42dc15c5f0 0x7f42dc154b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.676+0000 7f42e262a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42dc157560 0x7f42dc1579f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.676+0000 7f42e262a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f42dc1612d0 con 0x7f42dc157560 2024-04-15T15:15:41.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.676+0000 7f42e262a640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f42dc0b5bb0 con 0x7f42dc15c5f0 2024-04-15T15:15:41.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.676+0000 7f42e262a640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f42dc06f510 con 0x7f42dc0d1810 2024-04-15T15:15:41.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.676+0000 7f42e0bc7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42dc157560 0x7f42dc1579f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.676+0000 7f42e0bc7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42dc157560 0x7f42dc1579f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46028/0 (socket says 172.21.15.72:46028) 2024-04-15T15:15:41.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.676+0000 7f42e0bc7640 1 -- 172.21.15.72:0/920193782 learned_addr learned my addr 172.21.15.72:0/920193782 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:41.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.676+0000 7f42dbfff640 1 --2- 172.21.15.72:0/920193782 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f42dc0d1810 0x7f42dc156ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.676+0000 7f42db7fe640 1 --2- 172.21.15.72:0/920193782 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f42dc15c5f0 0x7f42dc154b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.676+0000 7f42e0bc7640 1 -- 172.21.15.72:0/920193782 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f42dc0d1810 msgr2=0x7f42dc156ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:41.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.676+0000 7f42e0bc7640 1 --2- 172.21.15.72:0/920193782 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f42dc0d1810 0x7f42dc156ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.676+0000 7f42e0bc7640 1 -- 172.21.15.72:0/920193782 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f42dc15c5f0 msgr2=0x7f42dc154b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:41.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.676+0000 7f42e0bc7640 1 --2- 172.21.15.72:0/920193782 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f42dc15c5f0 0x7f42dc154b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.676+0000 7f42e0bc7640 1 -- 172.21.15.72:0/920193782 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42dc156ff0 con 0x7f42dc157560 2024-04-15T15:15:41.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.676+0000 7f42e0bc7640 1 --2- 172.21.15.72:0/920193782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42dc157560 0x7f42dc1579f0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f42d4009d90 tx=0x7f42d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:41.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.677+0000 7f42d8ff9640 1 -- 172.21.15.72:0/920193782 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d4039050 con 0x7f42dc157560 2024-04-15T15:15:41.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.677+0000 7f42d8ff9640 1 -- 172.21.15.72:0/920193782 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42d403a040 con 0x7f42dc157560 2024-04-15T15:15:41.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.677+0000 7f42d8ff9640 1 -- 172.21.15.72:0/920193782 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d403e610 con 0x7f42dc157560 2024-04-15T15:15:41.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.677+0000 7f42e262a640 1 -- 172.21.15.72:0/920193782 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42dc152bb0 con 0x7f42dc157560 2024-04-15T15:15:41.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.677+0000 7f42e262a640 1 -- 172.21.15.72:0/920193782 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42dc153170 con 0x7f42dc157560 2024-04-15T15:15:41.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.678+0000 7f42e262a640 1 -- 172.21.15.72:0/920193782 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42dc0b48e0 con 0x7f42dc157560 2024-04-15T15:15:41.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.678+0000 7f42d8ff9640 1 -- 172.21.15.72:0/920193782 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f42d4047020 con 0x7f42dc157560 2024-04-15T15:15:41.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.678+0000 7f42d8ff9640 1 --2- 172.21.15.72:0/920193782 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f42c806c8f0 0x7f42c806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.679+0000 7f42d8ff9640 1 -- 172.21.15.72:0/920193782 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f42d40ae3b0 con 0x7f42dc157560 2024-04-15T15:15:41.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.679+0000 7f42dbfff640 1 --2- 172.21.15.72:0/920193782 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f42c806c8f0 0x7f42c806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.679+0000 7f42dbfff640 1 --2- 172.21.15.72:0/920193782 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f42c806c8f0 0x7f42c806edb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f42cc000f60 tx=0x7f42cc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:41.684 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.683+0000 7f42d8ff9640 1 -- 172.21.15.72:0/920193782 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f42dc0b48e0 con 0x7f42dc157560 2024-04-15T15:15:41.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.857+0000 7f42e262a640 1 -- 172.21.15.72:0/920193782 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f42dc075c50 con 0x7f42dc157560 2024-04-15T15:15:41.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.858+0000 7f42d8ff9640 1 -- 172.21.15.72:0/920193782 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f42d404e020 con 0x7f42dc157560 2024-04-15T15:15:41.860 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:41.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.859+0000 7f42e262a640 1 -- 172.21.15.72:0/920193782 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f42c806c8f0 msgr2=0x7f42c806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:41.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.859+0000 7f42e262a640 1 --2- 172.21.15.72:0/920193782 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f42c806c8f0 0x7f42c806edb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f42cc000f60 tx=0x7f42cc009040 comp rx=0 tx=0).stop 2024-04-15T15:15:41.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.859+0000 7f42e262a640 1 -- 172.21.15.72:0/920193782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42dc157560 msgr2=0x7f42dc1579f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:41.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.859+0000 7f42e262a640 1 --2- 172.21.15.72:0/920193782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42dc157560 0x7f42dc1579f0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f42d4009d90 tx=0x7f42d4004a40 comp rx=0 tx=0).stop 2024-04-15T15:15:41.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.859+0000 7f42e262a640 1 -- 172.21.15.72:0/920193782 shutdown_connections 2024-04-15T15:15:41.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.859+0000 7f42e262a640 1 --2- 172.21.15.72:0/920193782 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f42c806c8f0 0x7f42c806edb0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.859+0000 7f42e262a640 1 --2- 172.21.15.72:0/920193782 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f42dc157560 0x7f42dc1579f0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.859+0000 7f42e262a640 1 --2- 172.21.15.72:0/920193782 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f42dc15c5f0 0x7f42dc154b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.859+0000 7f42e262a640 1 --2- 172.21.15.72:0/920193782 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f42dc0d1810 0x7f42dc156ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.860+0000 7f42e262a640 1 -- 172.21.15.72:0/920193782 >> 172.21.15.72:0/920193782 conn(0x7f42dc0b4010 msgr2=0x7f42dc070f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:41.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.860+0000 7f42e262a640 1 -- 172.21.15.72:0/920193782 shutdown_connections 2024-04-15T15:15:41.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.860+0000 7f42e262a640 1 -- 172.21.15.72:0/920193782 wait complete. 2024-04-15T15:15:41.872 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:15:41.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.977+0000 7f86a913f640 1 Processor -- start 2024-04-15T15:15:41.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.977+0000 7f86a913f640 1 -- start start 2024-04-15T15:15:41.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.977+0000 7f86a913f640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f86a40ce2f0 0x7f86a415c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.977+0000 7f86a913f640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86a40d08f0 0x7f86a40cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.977+0000 7f86a913f640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86a406bbb0 con 0x7f86a40ce6d0 2024-04-15T15:15:41.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.978+0000 7f86a913f640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f86a406b960 con 0x7f86a40ce2f0 2024-04-15T15:15:41.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.978+0000 7f86a913f640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f86a406be90 con 0x7f86a40d08f0 2024-04-15T15:15:41.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.978+0000 7f86a359e640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f86a40ce6d0 0x7f86a4155600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60002/0 (socket says 172.21.15.72:60002) 2024-04-15T15:15:41.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.978+0000 7f86a359e640 1 -- 172.21.15.72:0/60201389 learned_addr learned my addr 172.21.15.72:0/60201389 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:41.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.978+0000 7f86a2d9d640 1 --2- 172.21.15.72:0/60201389 >> v2:172.21.15.175:3300/0 conn(0x7f86a40ce2f0 0x7f86a415c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.978+0000 7f869a59c640 1 --2- 172.21.15.72:0/60201389 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86a40d08f0 0x7f86a40cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.978+0000 7f86a259c640 1 -- 172.21.15.72:0/60201389 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3345169117 0 0) 0x7f86a406bbb0 con 0x7f86a40ce6d0 2024-04-15T15:15:41.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.978+0000 7f86a259c640 1 -- 172.21.15.72:0/60201389 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8690003520 con 0x7f86a40ce6d0 2024-04-15T15:15:41.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.978+0000 7f86a259c640 1 -- 172.21.15.72:0/60201389 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2890797203 0 0) 0x7f8690003520 con 0x7f86a40ce6d0 2024-04-15T15:15:41.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.978+0000 7f86a259c640 1 -- 172.21.15.72:0/60201389 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86a40d08f0 msgr2=0x7f86a40cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:41.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.978+0000 7f86a259c640 1 --2- 172.21.15.72:0/60201389 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86a40d08f0 0x7f86a40cc0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.978+0000 7f86a259c640 1 -- 172.21.15.72:0/60201389 >> v2:172.21.15.175:3300/0 conn(0x7f86a40ce2f0 msgr2=0x7f86a415c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:41.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.978+0000 7f86a259c640 1 --2- 172.21.15.72:0/60201389 >> v2:172.21.15.175:3300/0 conn(0x7f86a40ce2f0 0x7f86a415c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.978+0000 7f86a259c640 1 -- 172.21.15.72:0/60201389 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a406c170 con 0x7f86a40ce6d0 2024-04-15T15:15:41.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.978+0000 7f86a259c640 1 -- 172.21.15.72:0/60201389 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f869c0019f0 con 0x7f86a40ce6d0 2024-04-15T15:15:41.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.978+0000 7f86a259c640 1 -- 172.21.15.72:0/60201389 >> v1:172.21.15.72:6789/0 conn(0x7f86a40ce6d0 legacy=0x7f86a4155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:41.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.978+0000 7f86a259c640 1 --2- 172.21.15.72:0/60201389 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86900039b0 0x7f8690003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.978+0000 7f86a259c640 1 --2- 172.21.15.72:0/60201389 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8690004800 0x7f8690004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.978+0000 7f86a259c640 1 --2- 172.21.15.72:0/60201389 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86a40d08f0 0x7f86900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.978+0000 7f86a259c640 1 -- 172.21.15.72:0/60201389 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f86a406b960 con 0x7f86900039b0 2024-04-15T15:15:41.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.978+0000 7f86a259c640 1 -- 172.21.15.72:0/60201389 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f86a406be90 con 0x7f8690004800 2024-04-15T15:15:41.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.978+0000 7f86a259c640 1 -- 172.21.15.72:0/60201389 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f869000b9c0 con 0x7f86a40d08f0 2024-04-15T15:15:41.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.978+0000 7f86a2d9d640 1 --2- 172.21.15.72:0/60201389 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86900039b0 0x7f8690003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.978+0000 7f869a59c640 1 --2- 172.21.15.72:0/60201389 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8690004800 0x7f8690004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.979+0000 7f86a359e640 1 --2- 172.21.15.72:0/60201389 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86a40d08f0 0x7f86900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.979+0000 7f86a2d9d640 1 -- 172.21.15.72:0/60201389 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86a40d08f0 msgr2=0x7f86900051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:41.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.979+0000 7f86a2d9d640 1 --2- 172.21.15.72:0/60201389 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86a40d08f0 0x7f86900051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.979+0000 7f86a2d9d640 1 -- 172.21.15.72:0/60201389 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8690004800 msgr2=0x7f8690004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:41.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.979+0000 7f86a2d9d640 1 --2- 172.21.15.72:0/60201389 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8690004800 0x7f8690004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.979+0000 7f86a2d9d640 1 -- 172.21.15.72:0/60201389 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8690003520 con 0x7f86900039b0 2024-04-15T15:15:41.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.979+0000 7f86a359e640 1 --2- 172.21.15.72:0/60201389 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86a40d08f0 0x7f86900051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:41.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.979+0000 7f869a59c640 1 --2- 172.21.15.72:0/60201389 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8690004800 0x7f8690004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:41.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.979+0000 7f86a2d9d640 1 --2- 172.21.15.72:0/60201389 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86900039b0 0x7f8690003dd0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f8694009ac0 tx=0x7f8690005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d87f13359badc65 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:41.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.979+0000 7f86a259c640 1 -- 172.21.15.72:0/60201389 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8694039070 con 0x7f86900039b0 2024-04-15T15:15:41.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.979+0000 7f86a259c640 1 -- 172.21.15.72:0/60201389 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f869402c710 con 0x7f86900039b0 2024-04-15T15:15:41.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.979+0000 7f86a259c640 1 -- 172.21.15.72:0/60201389 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8694009690 con 0x7f86900039b0 2024-04-15T15:15:41.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.980+0000 7f86a913f640 1 -- 172.21.15.72:0/60201389 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86900039b0 msgr2=0x7f8690003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:41.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.980+0000 7f86a913f640 1 --2- 172.21.15.72:0/60201389 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86900039b0 0x7f8690003dd0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f8694009ac0 tx=0x7f8690005830 comp rx=0 tx=0).stop 2024-04-15T15:15:41.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.980+0000 7f86a359e640 1 -- 172.21.15.72:0/60201389 reap_dead start 2024-04-15T15:15:41.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.980+0000 7f86a913f640 1 -- 172.21.15.72:0/60201389 shutdown_connections 2024-04-15T15:15:41.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.980+0000 7f86a913f640 1 -- 172.21.15.72:0/60201389 >> 172.21.15.72:0/60201389 conn(0x7f86a40b31d0 msgr2=0x7f86a40b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:41.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.980+0000 7f86a913f640 1 -- 172.21.15.72:0/60201389 shutdown_connections 2024-04-15T15:15:41.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.980+0000 7f86a913f640 1 -- 172.21.15.72:0/60201389 wait complete. 2024-04-15T15:15:41.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.981+0000 7f86a913f640 1 Processor -- start 2024-04-15T15:15:41.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.982+0000 7f86a913f640 1 -- start start 2024-04-15T15:15:41.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.982+0000 7f86a913f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86900039b0 0x7f86a414ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.982+0000 7f86a913f640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f86a41583f0 0x7f86a4158850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.982+0000 7f86a913f640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86a40afe40 0x7f86a40b02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.982+0000 7f86a913f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f86a4155d10 con 0x7f86900039b0 2024-04-15T15:15:41.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.982+0000 7f86a913f640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f86a415ca70 con 0x7f86a41583f0 2024-04-15T15:15:41.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.982+0000 7f86a913f640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f86a4155e90 con 0x7f86a40afe40 2024-04-15T15:15:41.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.982+0000 7f86a2d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86900039b0 0x7f86a414ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.982+0000 7f86a2d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86900039b0 0x7f86a414ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46048/0 (socket says 172.21.15.72:46048) 2024-04-15T15:15:41.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.982+0000 7f86a2d9d640 1 -- 172.21.15.72:0/378661951 learned_addr learned my addr 172.21.15.72:0/378661951 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:41.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.983+0000 7f869a59c640 1 --2- 172.21.15.72:0/378661951 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f86a41583f0 0x7f86a4158850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.983+0000 7f86a359e640 1 --2- 172.21.15.72:0/378661951 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86a40afe40 0x7f86a40b02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.983+0000 7f86a2d9d640 1 -- 172.21.15.72:0/378661951 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86a40afe40 msgr2=0x7f86a40b02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:41.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.983+0000 7f86a2d9d640 1 --2- 172.21.15.72:0/378661951 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86a40afe40 0x7f86a40b02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.983+0000 7f86a2d9d640 1 -- 172.21.15.72:0/378661951 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f86a41583f0 msgr2=0x7f86a4158850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:41.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.983+0000 7f86a2d9d640 1 --2- 172.21.15.72:0/378661951 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f86a41583f0 0x7f86a4158850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.983+0000 7f86a2d9d640 1 -- 172.21.15.72:0/378661951 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86a40b0bc0 con 0x7f86900039b0 2024-04-15T15:15:41.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.983+0000 7f869a59c640 1 --2- 172.21.15.72:0/378661951 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f86a41583f0 0x7f86a4158850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:41.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.983+0000 7f86a359e640 1 --2- 172.21.15.72:0/378661951 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86a40afe40 0x7f86a40b02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:41.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.983+0000 7f86a2d9d640 1 --2- 172.21.15.72:0/378661951 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86900039b0 0x7f86a414ac70 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f8694037040 tx=0x7f8694004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:41.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.983+0000 7f869bfff640 1 -- 172.21.15.72:0/378661951 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8694042020 con 0x7f86900039b0 2024-04-15T15:15:41.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.983+0000 7f869bfff640 1 -- 172.21.15.72:0/378661951 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f869403a040 con 0x7f86900039b0 2024-04-15T15:15:41.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.983+0000 7f86a913f640 1 -- 172.21.15.72:0/378661951 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86a40b0ea0 con 0x7f86900039b0 2024-04-15T15:15:41.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.983+0000 7f869bfff640 1 -- 172.21.15.72:0/378661951 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8694039070 con 0x7f86900039b0 2024-04-15T15:15:41.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.983+0000 7f86a913f640 1 -- 172.21.15.72:0/378661951 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86a40b1460 con 0x7f86900039b0 2024-04-15T15:15:41.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.984+0000 7f86a913f640 1 -- 172.21.15.72:0/378661951 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86a4075c50 con 0x7f86900039b0 2024-04-15T15:15:41.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.984+0000 7f869bfff640 1 -- 172.21.15.72:0/378661951 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8694045020 con 0x7f86900039b0 2024-04-15T15:15:41.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.985+0000 7f869bfff640 1 --2- 172.21.15.72:0/378661951 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f868406c920 0x7f868406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.985+0000 7f869a59c640 1 --2- 172.21.15.72:0/378661951 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f868406c920 0x7f868406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.985+0000 7f869bfff640 1 -- 172.21.15.72:0/378661951 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f86940ad440 con 0x7f86900039b0 2024-04-15T15:15:41.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.986+0000 7f869a59c640 1 --2- 172.21.15.72:0/378661951 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f868406c920 0x7f868406ede0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f86a4154050 tx=0x7f868c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:41.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:41.990+0000 7f869bfff640 1 -- 172.21.15.72:0/378661951 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f869404c070 con 0x7f86900039b0 2024-04-15T15:15:42.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.165+0000 7f86a913f640 1 -- 172.21.15.72:0/378661951 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f86a415ff10 con 0x7f86900039b0 2024-04-15T15:15:42.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.165+0000 7f869bfff640 1 -- 172.21.15.72:0/378661951 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f869407b3e0 con 0x7f86900039b0 2024-04-15T15:15:42.167 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-04-15T15:15:42.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.167+0000 7f86a913f640 1 -- 172.21.15.72:0/378661951 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f868406c920 msgr2=0x7f868406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:42.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.167+0000 7f86a913f640 1 --2- 172.21.15.72:0/378661951 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f868406c920 0x7f868406ede0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f86a4154050 tx=0x7f868c008040 comp rx=0 tx=0).stop 2024-04-15T15:15:42.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.167+0000 7f86a913f640 1 -- 172.21.15.72:0/378661951 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86900039b0 msgr2=0x7f86a414ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:42.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.167+0000 7f86a913f640 1 --2- 172.21.15.72:0/378661951 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86900039b0 0x7f86a414ac70 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f8694037040 tx=0x7f8694004730 comp rx=0 tx=0).stop 2024-04-15T15:15:42.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.167+0000 7f86a913f640 1 -- 172.21.15.72:0/378661951 shutdown_connections 2024-04-15T15:15:42.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.167+0000 7f86a913f640 1 --2- 172.21.15.72:0/378661951 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f868406c920 0x7f868406ede0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.167+0000 7f86a913f640 1 --2- 172.21.15.72:0/378661951 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86a40afe40 0x7f86a40b02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.170 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.167+0000 7f86a913f640 1 --2- 172.21.15.72:0/378661951 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f86a41583f0 0x7f86a4158850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.167+0000 7f86a913f640 1 --2- 172.21.15.72:0/378661951 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86900039b0 0x7f86a414ac70 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.167+0000 7f86a913f640 1 -- 172.21.15.72:0/378661951 >> 172.21.15.72:0/378661951 conn(0x7f86a40b31d0 msgr2=0x7f86a41540f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:42.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.167+0000 7f86a913f640 1 -- 172.21.15.72:0/378661951 shutdown_connections 2024-04-15T15:15:42.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.167+0000 7f86a913f640 1 -- 172.21.15.72:0/378661951 wait complete. 2024-04-15T15:15:42.180 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:15:42.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.284+0000 7fec12441640 1 Processor -- start 2024-04-15T15:15:42.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.284+0000 7fec12441640 1 -- start start 2024-04-15T15:15:42.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.285+0000 7fec12441640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fec0c0ce2f0 0x7fec0c15c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:42.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.285+0000 7fec12441640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fec0c0d08f0 0x7fec0c0cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:42.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.285+0000 7fec12441640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fec0c0ce6d0 0x7fec0c0cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:42.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.285+0000 7fec12441640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fec0c06b960 con 0x7fec0c0d08f0 2024-04-15T15:15:42.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.285+0000 7fec12441640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fec0c06bbb0 con 0x7fec0c0ce2f0 2024-04-15T15:15:42.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.285+0000 7fec12441640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fec0c06be90 con 0x7fec0c0ce6d0 2024-04-15T15:15:42.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.285+0000 7fec03fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fec0c0d08f0 0x7fec0c0cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:42.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.285+0000 7fec03fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fec0c0d08f0 0x7fec0c0cc0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46058/0 (socket says 172.21.15.72:46058) 2024-04-15T15:15:42.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.285+0000 7fec03fff640 1 -- 172.21.15.72:0/65462034 learned_addr learned my addr 172.21.15.72:0/65462034 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:42.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.285+0000 7fec109de640 1 --2- 172.21.15.72:0/65462034 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fec0c0ce6d0 0x7fec0c0cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:42.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.285+0000 7fec0bfff640 1 --2- 172.21.15.72:0/65462034 >> v2:172.21.15.175:3300/0 conn(0x7fec0c0ce2f0 0x7fec0c15c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:42.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.285+0000 7fec03fff640 1 -- 172.21.15.72:0/65462034 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fec0c0ce6d0 msgr2=0x7fec0c0cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:42.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.285+0000 7fec03fff640 1 --2- 172.21.15.72:0/65462034 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fec0c0ce6d0 0x7fec0c0cc5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.285+0000 7fec03fff640 1 -- 172.21.15.72:0/65462034 >> v2:172.21.15.175:3300/0 conn(0x7fec0c0ce2f0 msgr2=0x7fec0c15c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:42.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.285+0000 7fec03fff640 1 --2- 172.21.15.72:0/65462034 >> v2:172.21.15.175:3300/0 conn(0x7fec0c0ce2f0 0x7fec0c15c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.285+0000 7fec03fff640 1 -- 172.21.15.72:0/65462034 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec0c06c170 con 0x7fec0c0d08f0 2024-04-15T15:15:42.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.286+0000 7fec03fff640 1 --2- 172.21.15.72:0/65462034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec0c0d08f0 0x7fec0c0cc0b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fec0c0cdf10 tx=0x7febf402db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccce2abbb379af44 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:42.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.286+0000 7fec0b7fe640 1 -- 172.21.15.72:0/65462034 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf402f8e0 con 0x7fec0c0d08f0 2024-04-15T15:15:42.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.286+0000 7fec0b7fe640 1 -- 172.21.15.72:0/65462034 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febf402fa80 con 0x7fec0c0d08f0 2024-04-15T15:15:42.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.286+0000 7fec0b7fe640 1 -- 172.21.15.72:0/65462034 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf402fe50 con 0x7fec0c0d08f0 2024-04-15T15:15:42.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.287+0000 7fec12441640 1 -- 172.21.15.72:0/65462034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec0c0d08f0 msgr2=0x7fec0c0cc0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:42.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.287+0000 7fec12441640 1 --2- 172.21.15.72:0/65462034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec0c0d08f0 0x7fec0c0cc0b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fec0c0cdf10 tx=0x7febf402db20 comp rx=0 tx=0).stop 2024-04-15T15:15:42.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.287+0000 7fec12441640 1 -- 172.21.15.72:0/65462034 shutdown_connections 2024-04-15T15:15:42.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.287+0000 7fec12441640 1 --2- 172.21.15.72:0/65462034 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fec0c0ce6d0 0x7fec0c0cc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.287+0000 7fec12441640 1 --2- 172.21.15.72:0/65462034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec0c0d08f0 0x7fec0c0cc0b0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.287+0000 7fec12441640 1 --2- 172.21.15.72:0/65462034 >> v2:172.21.15.175:3300/0 conn(0x7fec0c0ce2f0 0x7fec0c15c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.287+0000 7fec12441640 1 -- 172.21.15.72:0/65462034 >> 172.21.15.72:0/65462034 conn(0x7fec0c0b31d0 msgr2=0x7fec0c0b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:42.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.287+0000 7fec12441640 1 -- 172.21.15.72:0/65462034 shutdown_connections 2024-04-15T15:15:42.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.287+0000 7fec12441640 1 -- 172.21.15.72:0/65462034 wait complete. 2024-04-15T15:15:42.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.288+0000 7fec12441640 1 Processor -- start 2024-04-15T15:15:42.291 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.288+0000 7fec12441640 1 -- start start 2024-04-15T15:15:42.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.289+0000 7fec12441640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fec0c0ce2f0 0x7fec0c156860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:42.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.289+0000 7fec12441640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fec0c0ce6d0 0x7fec0c0b1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:42.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.289+0000 7fec12441640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec0c0d08f0 0x7fec0c14d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:42.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.289+0000 7fec12441640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fec0c155600 con 0x7fec0c0d08f0 2024-04-15T15:15:42.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.289+0000 7fec12441640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fec0c15ca70 con 0x7fec0c0ce2f0 2024-04-15T15:15:42.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.289+0000 7fec12441640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fec0c155780 con 0x7fec0c0ce6d0 2024-04-15T15:15:42.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.289+0000 7fec109de640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec0c0d08f0 0x7fec0c14d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:42.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.289+0000 7fec109de640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec0c0d08f0 0x7fec0c14d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46074/0 (socket says 172.21.15.72:46074) 2024-04-15T15:15:42.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.289+0000 7fec109de640 1 -- 172.21.15.72:0/1055841101 learned_addr learned my addr 172.21.15.72:0/1055841101 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:42.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.289+0000 7fec03fff640 1 --2- 172.21.15.72:0/1055841101 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fec0c0ce6d0 0x7fec0c0b1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:42.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.289+0000 7fec0bfff640 1 --2- 172.21.15.72:0/1055841101 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fec0c0ce2f0 0x7fec0c156860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:42.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.289+0000 7fec109de640 1 -- 172.21.15.72:0/1055841101 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fec0c0ce6d0 msgr2=0x7fec0c0b1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:42.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.289+0000 7fec109de640 1 --2- 172.21.15.72:0/1055841101 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fec0c0ce6d0 0x7fec0c0b1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.289+0000 7fec109de640 1 -- 172.21.15.72:0/1055841101 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fec0c0ce2f0 msgr2=0x7fec0c156860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:42.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.289+0000 7fec109de640 1 --2- 172.21.15.72:0/1055841101 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fec0c0ce2f0 0x7fec0c156860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.289+0000 7fec109de640 1 -- 172.21.15.72:0/1055841101 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec0c0b26e0 con 0x7fec0c0d08f0 2024-04-15T15:15:42.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.289+0000 7fec109de640 1 --2- 172.21.15.72:0/1055841101 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec0c0d08f0 0x7fec0c14d800 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fec04007e40 tx=0x7fec0400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:42.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.290+0000 7fec097fa640 1 -- 172.21.15.72:0/1055841101 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec04014620 con 0x7fec0c0d08f0 2024-04-15T15:15:42.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.290+0000 7fec097fa640 1 -- 172.21.15.72:0/1055841101 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec04019070 con 0x7fec0c0d08f0 2024-04-15T15:15:42.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.290+0000 7fec097fa640 1 -- 172.21.15.72:0/1055841101 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec04014a20 con 0x7fec0c0d08f0 2024-04-15T15:15:42.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.290+0000 7fec12441640 1 -- 172.21.15.72:0/1055841101 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec0c0b29c0 con 0x7fec0c0d08f0 2024-04-15T15:15:42.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.290+0000 7fec12441640 1 -- 172.21.15.72:0/1055841101 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec0c0b01c0 con 0x7fec0c0d08f0 2024-04-15T15:15:42.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.291+0000 7fec12441640 1 -- 172.21.15.72:0/1055841101 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec0c075c50 con 0x7fec0c0d08f0 2024-04-15T15:15:42.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.291+0000 7fec097fa640 1 -- 172.21.15.72:0/1055841101 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fec04014bc0 con 0x7fec0c0d08f0 2024-04-15T15:15:42.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.291+0000 7fec097fa640 1 --2- 172.21.15.72:0/1055841101 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7febec06c8f0 0x7febec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:42.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.292+0000 7fec097fa640 1 -- 172.21.15.72:0/1055841101 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fec0408c790 con 0x7fec0c0d08f0 2024-04-15T15:15:42.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.292+0000 7fec0bfff640 1 --2- 172.21.15.72:0/1055841101 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7febec06c8f0 0x7febec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:42.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.292+0000 7fec0bfff640 1 --2- 172.21.15.72:0/1055841101 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7febec06c8f0 0x7febec06edb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7febfc006fd0 tx=0x7febfc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:42.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.296+0000 7fec097fa640 1 -- 172.21.15.72:0/1055841101 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fec0405a170 con 0x7fec0c0d08f0 2024-04-15T15:15:42.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.470+0000 7fec12441640 1 -- 172.21.15.72:0/1055841101 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fec0c06cab0 con 0x7fec0c0d08f0 2024-04-15T15:15:42.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.470+0000 7fec097fa640 1 -- 172.21.15.72:0/1055841101 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7fec0405e470 con 0x7fec0c0d08f0 2024-04-15T15:15:42.473 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:42.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.472+0000 7fec12441640 1 -- 172.21.15.72:0/1055841101 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7febec06c8f0 msgr2=0x7febec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:42.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.472+0000 7fec12441640 1 --2- 172.21.15.72:0/1055841101 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7febec06c8f0 0x7febec06edb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7febfc006fd0 tx=0x7febfc008040 comp rx=0 tx=0).stop 2024-04-15T15:15:42.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.472+0000 7fec12441640 1 -- 172.21.15.72:0/1055841101 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec0c0d08f0 msgr2=0x7fec0c14d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:42.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.472+0000 7fec12441640 1 --2- 172.21.15.72:0/1055841101 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec0c0d08f0 0x7fec0c14d800 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fec04007e40 tx=0x7fec0400e4b0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.472+0000 7fec12441640 1 -- 172.21.15.72:0/1055841101 shutdown_connections 2024-04-15T15:15:42.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.472+0000 7fec12441640 1 --2- 172.21.15.72:0/1055841101 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7febec06c8f0 0x7febec06edb0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.472+0000 7fec12441640 1 --2- 172.21.15.72:0/1055841101 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec0c0d08f0 0x7fec0c14d800 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.472+0000 7fec12441640 1 --2- 172.21.15.72:0/1055841101 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fec0c0ce6d0 0x7fec0c0b1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.472+0000 7fec12441640 1 --2- 172.21.15.72:0/1055841101 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fec0c0ce2f0 0x7fec0c156860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.472+0000 7fec12441640 1 -- 172.21.15.72:0/1055841101 >> 172.21.15.72:0/1055841101 conn(0x7fec0c0b31d0 msgr2=0x7fec0c15ae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:42.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.473+0000 7fec12441640 1 -- 172.21.15.72:0/1055841101 shutdown_connections 2024-04-15T15:15:42.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.473+0000 7fec12441640 1 -- 172.21.15.72:0/1055841101 wait complete. 2024-04-15T15:15:42.486 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:15:42.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.589+0000 7f9ff218d640 1 Processor -- start 2024-04-15T15:15:42.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.589+0000 7f9ff218d640 1 -- start start 2024-04-15T15:15:42.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.590+0000 7f9ff218d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9fec0b36f0 0x7f9fec0b5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:42.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.590+0000 7f9ff218d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fec0d02d0 con 0x7f9fec0b57d0 2024-04-15T15:15:42.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.590+0000 7f9ff218d640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fec15ca30 con 0x7f9fec15c650 2024-04-15T15:15:42.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.590+0000 7f9ff218d640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f9fec0b2450 con 0x7f9fec0b36f0 2024-04-15T15:15:42.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.590+0000 7f9feaffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9fec0b36f0 0x7f9fec0b5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:42.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.590+0000 7f9febfff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f9fec0b57d0 0x7f9fec0d0a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60018/0 (socket says 172.21.15.72:60018) 2024-04-15T15:15:42.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.590+0000 7f9febfff640 1 -- 172.21.15.72:0/1389230684 learned_addr learned my addr 172.21.15.72:0/1389230684 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:42.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.590+0000 7f9feaffd640 1 -- 172.21.15.72:0/1389230684 >> v1:172.21.15.175:6789/0 conn(0x7f9fec15c650 legacy=0x7f9fec0b1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:42.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.590+0000 7f9feaffd640 1 -- 172.21.15.72:0/1389230684 >> v1:172.21.15.72:6789/0 conn(0x7f9fec0b57d0 legacy=0x7f9fec0d0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:42.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.590+0000 7f9fea7fc640 1 -- 172.21.15.72:0/1389230684 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 199132308 0 0) 0x7f9fec0d02d0 con 0x7f9fec0b57d0 2024-04-15T15:15:42.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.590+0000 7f9feaffd640 1 -- 172.21.15.72:0/1389230684 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fec0b26b0 con 0x7f9fec0b36f0 2024-04-15T15:15:42.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.591+0000 7f9feaffd640 1 --2- 172.21.15.72:0/1389230684 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fec0b36f0 0x7f9fec0b5290 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f9fe00097a0 tx=0x7f9fe002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9f3c6e9ff45102a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:42.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.591+0000 7f9fea7fc640 1 -- 172.21.15.72:0/1389230684 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe00315a0 con 0x7f9fec0b36f0 2024-04-15T15:15:42.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.591+0000 7f9fea7fc640 1 -- 172.21.15.72:0/1389230684 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fe0037070 con 0x7f9fec0b36f0 2024-04-15T15:15:42.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.591+0000 7f9fea7fc640 1 -- 172.21.15.72:0/1389230684 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe0031960 con 0x7f9fec0b36f0 2024-04-15T15:15:42.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.591+0000 7f9ff218d640 1 -- 172.21.15.72:0/1389230684 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fec0b36f0 msgr2=0x7f9fec0b5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:42.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.591+0000 7f9ff218d640 1 --2- 172.21.15.72:0/1389230684 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fec0b36f0 0x7f9fec0b5290 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f9fe00097a0 tx=0x7f9fe002b840 comp rx=0 tx=0).stop 2024-04-15T15:15:42.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.591+0000 7f9ff218d640 1 -- 172.21.15.72:0/1389230684 shutdown_connections 2024-04-15T15:15:42.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.591+0000 7f9ff218d640 1 --2- 172.21.15.72:0/1389230684 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fec0b36f0 0x7f9fec0b5290 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.591+0000 7f9ff218d640 1 -- 172.21.15.72:0/1389230684 >> 172.21.15.72:0/1389230684 conn(0x7f9fec0ceec0 msgr2=0x7f9fec0cf2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:42.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.591+0000 7f9ff218d640 1 -- 172.21.15.72:0/1389230684 shutdown_connections 2024-04-15T15:15:42.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.591+0000 7f9ff218d640 1 -- 172.21.15.72:0/1389230684 wait complete. 2024-04-15T15:15:42.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.593+0000 7f9ff218d640 1 Processor -- start 2024-04-15T15:15:42.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.593+0000 7f9ff218d640 1 -- start start 2024-04-15T15:15:42.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.594+0000 7f9ff218d640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9fec0b36f0 0x7f9fec1f6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:42.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.594+0000 7f9ff218d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fec0b57d0 0x7f9fec0b6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:42.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.594+0000 7f9ff218d640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9fec15c650 0x7f9fec0b7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:42.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.594+0000 7f9ff218d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f9fec0b2c70 con 0x7f9fec0b57d0 2024-04-15T15:15:42.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.594+0000 7f9ff218d640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f9fec0d0540 con 0x7f9fec0b36f0 2024-04-15T15:15:42.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.594+0000 7f9ff218d640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f9fec0d1170 con 0x7f9fec15c650 2024-04-15T15:15:42.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.594+0000 7f9feaffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fec0b57d0 0x7f9fec0b6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:42.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.594+0000 7f9feaffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fec0b57d0 0x7f9fec0b6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46086/0 (socket says 172.21.15.72:46086) 2024-04-15T15:15:42.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.594+0000 7f9feaffd640 1 -- 172.21.15.72:0/2262190334 learned_addr learned my addr 172.21.15.72:0/2262190334 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:42.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.594+0000 7f9feaffd640 1 -- 172.21.15.72:0/2262190334 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9fec15c650 msgr2=0x7f9fec0b7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:42.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.594+0000 7f9febfff640 1 --2- 172.21.15.72:0/2262190334 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9fec15c650 0x7f9fec0b7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:42.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.594+0000 7f9feb7fe640 1 --2- 172.21.15.72:0/2262190334 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9fec0b36f0 0x7f9fec1f6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:42.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.594+0000 7f9feaffd640 1 --2- 172.21.15.72:0/2262190334 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9fec15c650 0x7f9fec0b7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.594+0000 7f9feaffd640 1 -- 172.21.15.72:0/2262190334 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9fec0b36f0 msgr2=0x7f9fec1f6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:42.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.594+0000 7f9feaffd640 1 --2- 172.21.15.72:0/2262190334 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9fec0b36f0 0x7f9fec1f6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.594+0000 7f9feaffd640 1 -- 172.21.15.72:0/2262190334 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fec1f6790 con 0x7f9fec0b57d0 2024-04-15T15:15:42.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.594+0000 7f9feb7fe640 1 --2- 172.21.15.72:0/2262190334 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9fec0b36f0 0x7f9fec1f6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:42.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.594+0000 7f9febfff640 1 --2- 172.21.15.72:0/2262190334 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9fec15c650 0x7f9fec0b7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:42.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.594+0000 7f9feaffd640 1 --2- 172.21.15.72:0/2262190334 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fec0b57d0 0x7f9fec0b6e10 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f9fe00098c0 tx=0x7f9fe00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:42.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.594+0000 7f9fd3fff640 1 -- 172.21.15.72:0/2262190334 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe0032840 con 0x7f9fec0b57d0 2024-04-15T15:15:42.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.594+0000 7f9fd3fff640 1 -- 172.21.15.72:0/2262190334 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fe00329e0 con 0x7f9fec0b57d0 2024-04-15T15:15:42.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.594+0000 7f9ff218d640 1 -- 172.21.15.72:0/2262190334 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fec0c5c30 con 0x7f9fec0b57d0 2024-04-15T15:15:42.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.594+0000 7f9fd3fff640 1 -- 172.21.15.72:0/2262190334 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fe0032dd0 con 0x7f9fec0b57d0 2024-04-15T15:15:42.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.594+0000 7f9ff218d640 1 -- 172.21.15.72:0/2262190334 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fec160650 con 0x7f9fec0b57d0 2024-04-15T15:15:42.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.595+0000 7f9ff218d640 1 -- 172.21.15.72:0/2262190334 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fec075c50 con 0x7f9fec0b57d0 2024-04-15T15:15:42.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.596+0000 7f9fd3fff640 1 -- 172.21.15.72:0/2262190334 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9fe003b3f0 con 0x7f9fec0b57d0 2024-04-15T15:15:42.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.596+0000 7f9fd3fff640 1 --2- 172.21.15.72:0/2262190334 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f9fd806c8f0 0x7f9fd806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:42.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.596+0000 7f9feb7fe640 1 --2- 172.21.15.72:0/2262190334 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f9fd806c8f0 0x7f9fd806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:42.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.597+0000 7f9fd3fff640 1 -- 172.21.15.72:0/2262190334 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f9fe002f030 con 0x7f9fec0b57d0 2024-04-15T15:15:42.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.597+0000 7f9feb7fe640 1 --2- 172.21.15.72:0/2262190334 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f9fd806c8f0 0x7f9fd806edb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc00a830 tx=0x7f9fdc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:42.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.601+0000 7f9fd3fff640 1 -- 172.21.15.72:0/2262190334 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9fe0040070 con 0x7f9fec0b57d0 2024-04-15T15:15:42.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.787+0000 7f9ff218d640 1 -- 172.21.15.72:0/2262190334 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f9fec0d0010 con 0x7f9fec0b57d0 2024-04-15T15:15:42.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.787+0000 7f9fd3fff640 1 -- 172.21.15.72:0/2262190334 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7f9fe007bff0 con 0x7f9fec0b57d0 2024-04-15T15:15:42.789 INFO:teuthology.orchestra.run.smithi072.stderr:listed 0 entries 2024-04-15T15:15:42.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.789+0000 7f9ff218d640 1 -- 172.21.15.72:0/2262190334 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f9fd806c8f0 msgr2=0x7f9fd806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:42.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.789+0000 7f9ff218d640 1 --2- 172.21.15.72:0/2262190334 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f9fd806c8f0 0x7f9fd806edb0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc00a830 tx=0x7f9fdc008040 comp rx=0 tx=0).stop 2024-04-15T15:15:42.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.789+0000 7f9ff218d640 1 -- 172.21.15.72:0/2262190334 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fec0b57d0 msgr2=0x7f9fec0b6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:42.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.789+0000 7f9ff218d640 1 --2- 172.21.15.72:0/2262190334 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fec0b57d0 0x7f9fec0b6e10 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f9fe00098c0 tx=0x7f9fe00046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.789+0000 7f9ff218d640 1 -- 172.21.15.72:0/2262190334 shutdown_connections 2024-04-15T15:15:42.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.789+0000 7f9ff218d640 1 --2- 172.21.15.72:0/2262190334 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f9fd806c8f0 0x7f9fd806edb0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.789+0000 7f9ff218d640 1 --2- 172.21.15.72:0/2262190334 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9fec15c650 0x7f9fec0b7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.789+0000 7f9ff218d640 1 --2- 172.21.15.72:0/2262190334 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9fec0b57d0 0x7f9fec0b6e10 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.789+0000 7f9ff218d640 1 --2- 172.21.15.72:0/2262190334 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9fec0b36f0 0x7f9fec1f6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.789+0000 7f9ff218d640 1 -- 172.21.15.72:0/2262190334 >> 172.21.15.72:0/2262190334 conn(0x7f9fec0ceec0 msgr2=0x7f9fec0d0a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:42.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.789+0000 7f9ff218d640 1 -- 172.21.15.72:0/2262190334 shutdown_connections 2024-04-15T15:15:42.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.789+0000 7f9ff218d640 1 -- 172.21.15.72:0/2262190334 wait complete. 2024-04-15T15:15:42.802 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:15:42.802 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:15:42.822 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:42.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.924+0000 7f14de035640 1 Processor -- start 2024-04-15T15:15:42.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.924+0000 7f14de035640 1 -- start start 2024-04-15T15:15:42.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.925+0000 7f14de035640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f14d815c120 0x7f14d815f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:42.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.925+0000 7f14de035640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f14d80cc0b0 0x7f14d80c68d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:42.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.925+0000 7f14de035640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14d81624e0 con 0x7f14d815c500 2024-04-15T15:15:42.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.925+0000 7f14de035640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f14d81627c0 con 0x7f14d80cc0b0 2024-04-15T15:15:42.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.925+0000 7f14de035640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f14d8162aa0 con 0x7f14d815c120 2024-04-15T15:15:42.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.925+0000 7f14d77fe640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f14d815c120 0x7f14d815f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:42.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.925+0000 7f14d7fff640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f14d80cc0b0 0x7f14d80c68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:42.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.925+0000 7f14d77fe640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f14d815c120 0x7f14d815f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3301/0 says I am v2:172.21.15.72:43458/0 (socket says 172.21.15.72:43458) 2024-04-15T15:15:42.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.925+0000 7f14d77fe640 1 -- 172.21.15.72:0/3464289619 learned_addr learned my addr 172.21.15.72:0/3464289619 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:42.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.926+0000 7f14d77fe640 1 -- 172.21.15.72:0/3464289619 >> v2:172.21.15.175:3300/0 conn(0x7f14d80cc0b0 msgr2=0x7f14d80c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:42.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.926+0000 7f14d77fe640 1 --2- 172.21.15.72:0/3464289619 >> v2:172.21.15.175:3300/0 conn(0x7f14d80cc0b0 0x7f14d80c68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.926+0000 7f14d77fe640 1 -- 172.21.15.72:0/3464289619 >> v1:172.21.15.175:6789/0 conn(0x7f14d815c500 legacy=0x7f14d80cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:42.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.926+0000 7f14d77fe640 1 -- 172.21.15.72:0/3464289619 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14d8162d80 con 0x7f14d815c120 2024-04-15T15:15:42.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.926+0000 7f14d7fff640 1 --2- 172.21.15.72:0/3464289619 >> v2:172.21.15.175:3300/0 conn(0x7f14d80cc0b0 0x7f14d80c68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:42.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.926+0000 7f14d77fe640 1 --2- 172.21.15.72:0/3464289619 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f14d815c120 0x7f14d815f7f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f14c8009e50 tx=0x7f14c802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f4e516680506d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:42.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.927+0000 7f14d67fc640 1 -- 172.21.15.72:0/3464289619 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c802cb00 con 0x7f14d815c120 2024-04-15T15:15:42.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.927+0000 7f14d67fc640 1 -- 172.21.15.72:0/3464289619 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14c802cca0 con 0x7f14d815c120 2024-04-15T15:15:42.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.927+0000 7f14d67fc640 1 -- 172.21.15.72:0/3464289619 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c8034610 con 0x7f14d815c120 2024-04-15T15:15:42.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.927+0000 7f14de035640 1 -- 172.21.15.72:0/3464289619 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f14d815c120 msgr2=0x7f14d815f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:42.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.927+0000 7f14de035640 1 --2- 172.21.15.72:0/3464289619 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f14d815c120 0x7f14d815f7f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f14c8009e50 tx=0x7f14c802bf60 comp rx=0 tx=0).stop 2024-04-15T15:15:42.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.927+0000 7f14de035640 1 -- 172.21.15.72:0/3464289619 shutdown_connections 2024-04-15T15:15:42.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.927+0000 7f14de035640 1 --2- 172.21.15.72:0/3464289619 >> v2:172.21.15.175:3300/0 conn(0x7f14d80cc0b0 0x7f14d80c68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.927+0000 7f14de035640 1 --2- 172.21.15.72:0/3464289619 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f14d815c120 0x7f14d815f7f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.927+0000 7f14de035640 1 -- 172.21.15.72:0/3464289619 >> 172.21.15.72:0/3464289619 conn(0x7f14d80cd4a0 msgr2=0x7f14d80cd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:42.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.927+0000 7f14de035640 1 -- 172.21.15.72:0/3464289619 shutdown_connections 2024-04-15T15:15:42.930 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.927+0000 7f14de035640 1 -- 172.21.15.72:0/3464289619 wait complete. 2024-04-15T15:15:42.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.928+0000 7f14de035640 1 Processor -- start 2024-04-15T15:15:42.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.929+0000 7f14de035640 1 -- start start 2024-04-15T15:15:42.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.929+0000 7f14de035640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14d80cc0b0 0x7f14d814dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:42.932 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.929+0000 7f14de035640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f14d815c120 0x7f14d80af2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:42.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.929+0000 7f14de035640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f14d815c500 0x7f14d814a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:42.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.929+0000 7f14de035640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f14d80708e0 con 0x7f14d80cc0b0 2024-04-15T15:15:42.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.929+0000 7f14de035640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f14d8070760 con 0x7f14d815c500 2024-04-15T15:15:42.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.929+0000 7f14de035640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f14d8070a60 con 0x7f14d815c120 2024-04-15T15:15:42.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.929+0000 7f14d77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14d80cc0b0 0x7f14d814dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:42.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.929+0000 7f14d77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14d80cc0b0 0x7f14d814dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46090/0 (socket says 172.21.15.72:46090) 2024-04-15T15:15:42.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.929+0000 7f14d77fe640 1 -- 172.21.15.72:0/828951150 learned_addr learned my addr 172.21.15.72:0/828951150 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:42.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.930+0000 7f14d7fff640 1 --2- 172.21.15.72:0/828951150 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f14d815c500 0x7f14d814a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:42.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.930+0000 7f14d6ffd640 1 --2- 172.21.15.72:0/828951150 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f14d815c120 0x7f14d80af2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:42.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.930+0000 7f14d77fe640 1 -- 172.21.15.72:0/828951150 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f14d815c120 msgr2=0x7f14d80af2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:42.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.930+0000 7f14d77fe640 1 --2- 172.21.15.72:0/828951150 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f14d815c120 0x7f14d80af2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.930+0000 7f14d77fe640 1 -- 172.21.15.72:0/828951150 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f14d815c500 msgr2=0x7f14d814a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:42.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.930+0000 7f14d77fe640 1 --2- 172.21.15.72:0/828951150 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f14d815c500 0x7f14d814a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:42.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.930+0000 7f14d77fe640 1 -- 172.21.15.72:0/828951150 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14d80c97f0 con 0x7f14d80cc0b0 2024-04-15T15:15:42.933 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.930+0000 7f14d77fe640 1 --2- 172.21.15.72:0/828951150 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14d80cc0b0 0x7f14d814dae0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f14c8009f70 tx=0x7f14c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:42.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.930+0000 7f14bffff640 1 -- 172.21.15.72:0/828951150 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c8043020 con 0x7f14d80cc0b0 2024-04-15T15:15:42.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.930+0000 7f14bffff640 1 -- 172.21.15.72:0/828951150 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14c802ce50 con 0x7f14d80cc0b0 2024-04-15T15:15:42.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.930+0000 7f14de035640 1 -- 172.21.15.72:0/828951150 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14d80af830 con 0x7f14d80cc0b0 2024-04-15T15:15:42.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.930+0000 7f14bffff640 1 -- 172.21.15.72:0/828951150 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14c8009510 con 0x7f14d80cc0b0 2024-04-15T15:15:42.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.930+0000 7f14de035640 1 -- 172.21.15.72:0/828951150 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14d80c7dd0 con 0x7f14d80cc0b0 2024-04-15T15:15:42.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.931+0000 7f14de035640 1 -- 172.21.15.72:0/828951150 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14d80689e0 con 0x7f14d80cc0b0 2024-04-15T15:15:42.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.931+0000 7f14bffff640 1 -- 172.21.15.72:0/828951150 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f14c8046020 con 0x7f14d80cc0b0 2024-04-15T15:15:42.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.932+0000 7f14bffff640 1 --2- 172.21.15.72:0/828951150 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f14c406c920 0x7f14c406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:42.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.932+0000 7f14bffff640 1 -- 172.21.15.72:0/828951150 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f14c80ae520 con 0x7f14d80cc0b0 2024-04-15T15:15:42.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.932+0000 7f14d6ffd640 1 --2- 172.21.15.72:0/828951150 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f14c406c920 0x7f14c406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:42.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.932+0000 7f14d6ffd640 1 --2- 172.21.15.72:0/828951150 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f14c406c920 0x7f14c406ede0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f14d80af1f0 tx=0x7f14cc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:42.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:42.936+0000 7f14bffff640 1 -- 172.21.15.72:0/828951150 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f14c804d030 con 0x7f14d80cc0b0 2024-04-15T15:15:43.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.112+0000 7f14de035640 1 -- 172.21.15.72:0/828951150 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f14d80cec90 con 0x7f14d80cc0b0 2024-04-15T15:15:43.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.113+0000 7f14bffff640 1 -- 172.21.15.72:0/828951150 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18903 (secure 0 0 0) 0x7f14c807c550 con 0x7f14d80cc0b0 2024-04-15T15:15:43.115 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:43.115 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":26,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:15:32.421728+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:14:49.438366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:14:50.368379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:14:54.081157+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":1}]},{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:43.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.115+0000 7f14de035640 1 -- 172.21.15.72:0/828951150 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f14c406c920 msgr2=0x7f14c406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:43.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.115+0000 7f14de035640 1 --2- 172.21.15.72:0/828951150 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f14c406c920 0x7f14c406ede0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f14d80af1f0 tx=0x7f14cc009040 comp rx=0 tx=0).stop 2024-04-15T15:15:43.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.115+0000 7f14de035640 1 -- 172.21.15.72:0/828951150 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14d80cc0b0 msgr2=0x7f14d814dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:43.117 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.115+0000 7f14de035640 1 --2- 172.21.15.72:0/828951150 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14d80cc0b0 0x7f14d814dae0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f14c8009f70 tx=0x7f14c8004a40 comp rx=0 tx=0).stop 2024-04-15T15:15:43.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.115+0000 7f14de035640 1 -- 172.21.15.72:0/828951150 shutdown_connections 2024-04-15T15:15:43.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.115+0000 7f14de035640 1 --2- 172.21.15.72:0/828951150 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f14c406c920 0x7f14c406ede0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.115+0000 7f14de035640 1 --2- 172.21.15.72:0/828951150 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f14d815c500 0x7f14d814a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.115+0000 7f14de035640 1 --2- 172.21.15.72:0/828951150 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f14d815c120 0x7f14d80af2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.115+0000 7f14de035640 1 --2- 172.21.15.72:0/828951150 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14d80cc0b0 0x7f14d814dae0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.115+0000 7f14de035640 1 -- 172.21.15.72:0/828951150 >> 172.21.15.72:0/828951150 conn(0x7f14d80cd4a0 msgr2=0x7f14d80b1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:43.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.115+0000 7f14de035640 1 -- 172.21.15.72:0/828951150 shutdown_connections 2024-04-15T15:15:43.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.115+0000 7f14de035640 1 -- 172.21.15.72:0/828951150 wait complete. 2024-04-15T15:15:43.128 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:39.359102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:14:49.438366+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:14:50.368379+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:14:54.081157+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:15:43.129 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:43.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.232+0000 7fb49c596640 1 Processor -- start 2024-04-15T15:15:43.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.233+0000 7fb49c596640 1 -- start start 2024-04-15T15:15:43.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.233+0000 7fb49c596640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb49415c250 0x7fb49406d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.233+0000 7fb49c596640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb49406df30 con 0x7fb4940d0010 2024-04-15T15:15:43.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.233+0000 7fb49c596640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb49406e210 con 0x7fb49415f9f0 2024-04-15T15:15:43.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.233+0000 7fb49c596640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fb4940cd1a0 con 0x7fb49415c250 2024-04-15T15:15:43.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.234+0000 7fb49ab33640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fb4940d0010 0x7fb4940b3e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60024/0 (socket says 172.21.15.72:60024) 2024-04-15T15:15:43.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.234+0000 7fb49ab33640 1 -- 172.21.15.72:0/997028906 learned_addr learned my addr 172.21.15.72:0/997028906 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:43.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.234+0000 7fb499330640 1 -- 172.21.15.72:0/997028906 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 854069303 0 0) 0x7fb49406df30 con 0x7fb4940d0010 2024-04-15T15:15:43.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.234+0000 7fb499330640 1 -- 172.21.15.72:0/997028906 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb480003520 con 0x7fb4940d0010 2024-04-15T15:15:43.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.234+0000 7fb49a332640 1 --2- 172.21.15.72:0/997028906 >> v2:172.21.15.175:3300/0 conn(0x7fb49415c250 0x7fb49406d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.234+0000 7fb499330640 1 -- 172.21.15.72:0/997028906 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2521314106 0 0) 0x7fb480003520 con 0x7fb4940d0010 2024-04-15T15:15:43.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.234+0000 7fb499330640 1 -- 172.21.15.72:0/997028906 >> v2:172.21.15.175:3300/0 conn(0x7fb49415c250 msgr2=0x7fb49406d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:43.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.234+0000 7fb499330640 1 --2- 172.21.15.72:0/997028906 >> v2:172.21.15.175:3300/0 conn(0x7fb49415c250 0x7fb49406d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.234+0000 7fb499330640 1 -- 172.21.15.72:0/997028906 >> v1:172.21.15.175:6789/0 conn(0x7fb49415f9f0 legacy=0x7fb4940cca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:43.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.234+0000 7fb499330640 1 -- 172.21.15.72:0/997028906 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb49406e4f0 con 0x7fb4940d0010 2024-04-15T15:15:43.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.234+0000 7fb499330640 1 -- 172.21.15.72:0/997028906 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7fb48c0019f0 con 0x7fb4940d0010 2024-04-15T15:15:43.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.234+0000 7fb499330640 1 -- 172.21.15.72:0/997028906 >> v1:172.21.15.72:6789/0 conn(0x7fb4940d0010 legacy=0x7fb4940b3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:43.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.234+0000 7fb499330640 1 --2- 172.21.15.72:0/997028906 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb4800039b0 0x7fb480003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.234+0000 7fb499330640 1 --2- 172.21.15.72:0/997028906 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb480004800 0x7fb480004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.234+0000 7fb499330640 1 --2- 172.21.15.72:0/997028906 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb4800052e0 0x7fb48000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.234+0000 7fb499330640 1 -- 172.21.15.72:0/997028906 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb4940cd1a0 con 0x7fb4800052e0 2024-04-15T15:15:43.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.234+0000 7fb499330640 1 -- 172.21.15.72:0/997028906 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fb48000c1b0 con 0x7fb480004800 2024-04-15T15:15:43.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.234+0000 7fb499330640 1 -- 172.21.15.72:0/997028906 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fb48000c3b0 con 0x7fb4800039b0 2024-04-15T15:15:43.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.234+0000 7fb49ab33640 1 --2- 172.21.15.72:0/997028906 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb4800052e0 0x7fb48000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.234+0000 7fb499b31640 1 --2- 172.21.15.72:0/997028906 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb480004800 0x7fb480004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.234+0000 7fb49a332640 1 --2- 172.21.15.72:0/997028906 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb4800039b0 0x7fb480003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.235+0000 7fb49ab33640 1 -- 172.21.15.72:0/997028906 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb4800039b0 msgr2=0x7fb480003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:43.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.235+0000 7fb49ab33640 1 --2- 172.21.15.72:0/997028906 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb4800039b0 0x7fb480003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.235+0000 7fb49ab33640 1 -- 172.21.15.72:0/997028906 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb480004800 msgr2=0x7fb480004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:43.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.235+0000 7fb49ab33640 1 --2- 172.21.15.72:0/997028906 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb480004800 0x7fb480004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.235+0000 7fb49ab33640 1 -- 172.21.15.72:0/997028906 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb480003520 con 0x7fb4800052e0 2024-04-15T15:15:43.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.235+0000 7fb49ab33640 1 -- 172.21.15.72:0/997028906 reap_dead start 2024-04-15T15:15:43.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.235+0000 7fb49ab33640 1 --2- 172.21.15.72:0/997028906 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb4800052e0 0x7fb48000b9c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb48c00a810 tx=0x7fb480005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6cd39a6dbd6ee91 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:43.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.235+0000 7fb499330640 1 -- 172.21.15.72:0/997028906 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb48c00a460 con 0x7fb4800052e0 2024-04-15T15:15:43.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.236+0000 7fb499330640 1 -- 172.21.15.72:0/997028906 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb48c034650 con 0x7fb4800052e0 2024-04-15T15:15:43.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.236+0000 7fb499330640 1 -- 172.21.15.72:0/997028906 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb48c034a40 con 0x7fb4800052e0 2024-04-15T15:15:43.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.236+0000 7fb49c596640 1 -- 172.21.15.72:0/997028906 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb4800052e0 msgr2=0x7fb48000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:43.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.236+0000 7fb49c596640 1 --2- 172.21.15.72:0/997028906 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb4800052e0 0x7fb48000b9c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb48c00a810 tx=0x7fb480005830 comp rx=0 tx=0).stop 2024-04-15T15:15:43.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.236+0000 7fb49c596640 1 -- 172.21.15.72:0/997028906 shutdown_connections 2024-04-15T15:15:43.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.236+0000 7fb49c596640 1 --2- 172.21.15.72:0/997028906 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb4800052e0 0x7fb48000b9c0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.236+0000 7fb49c596640 1 -- 172.21.15.72:0/997028906 >> 172.21.15.72:0/997028906 conn(0x7fb4940ceec0 msgr2=0x7fb4940cf2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:43.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.236+0000 7fb49c596640 1 -- 172.21.15.72:0/997028906 shutdown_connections 2024-04-15T15:15:43.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.236+0000 7fb49c596640 1 -- 172.21.15.72:0/997028906 wait complete. 2024-04-15T15:15:43.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.237+0000 7fb49c596640 1 Processor -- start 2024-04-15T15:15:43.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.237+0000 7fb49c596640 1 -- start start 2024-04-15T15:15:43.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.238+0000 7fb49c596640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb4800052e0 0x7fb4941f7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.238+0000 7fb49c596640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb4941516b0 0x7fb494159a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.238+0000 7fb49c596640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb494159f90 0x7fb4940c6bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.238+0000 7fb49c596640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb49415c630 con 0x7fb494159f90 2024-04-15T15:15:43.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.238+0000 7fb49c596640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fb4940b38f0 con 0x7fb4941516b0 2024-04-15T15:15:43.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.238+0000 7fb49c596640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fb49415c7b0 con 0x7fb4800052e0 2024-04-15T15:15:43.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.238+0000 7fb49ab33640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb494159f90 0x7fb4940c6bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.238+0000 7fb49ab33640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb494159f90 0x7fb4940c6bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46104/0 (socket says 172.21.15.72:46104) 2024-04-15T15:15:43.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.238+0000 7fb49ab33640 1 -- 172.21.15.72:0/429149441 learned_addr learned my addr 172.21.15.72:0/429149441 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:43.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.238+0000 7fb49a332640 1 --2- 172.21.15.72:0/429149441 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb4800052e0 0x7fb4941f7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.238+0000 7fb499b31640 1 --2- 172.21.15.72:0/429149441 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb4941516b0 0x7fb494159a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.238+0000 7fb49ab33640 1 -- 172.21.15.72:0/429149441 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb4800052e0 msgr2=0x7fb4941f7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:43.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.238+0000 7fb49ab33640 1 --2- 172.21.15.72:0/429149441 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb4800052e0 0x7fb4941f7260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.238+0000 7fb49ab33640 1 -- 172.21.15.72:0/429149441 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb4941516b0 msgr2=0x7fb494159a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:43.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.238+0000 7fb49ab33640 1 --2- 172.21.15.72:0/429149441 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb4941516b0 0x7fb494159a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.238+0000 7fb49ab33640 1 -- 172.21.15.72:0/429149441 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb49415f070 con 0x7fb494159f90 2024-04-15T15:15:43.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.238+0000 7fb499b31640 1 --2- 172.21.15.72:0/429149441 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb4941516b0 0x7fb494159a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:43.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.239+0000 7fb49ab33640 1 --2- 172.21.15.72:0/429149441 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb494159f90 0x7fb4940c6bf0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb48c00a810 tx=0x7fb49406e100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:43.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.239+0000 7fb486ffd640 1 -- 172.21.15.72:0/429149441 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb48c043070 con 0x7fb494159f90 2024-04-15T15:15:43.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.239+0000 7fb486ffd640 1 -- 172.21.15.72:0/429149441 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb48c0019b0 con 0x7fb494159f90 2024-04-15T15:15:43.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.239+0000 7fb49c596640 1 -- 172.21.15.72:0/429149441 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4940c6110 con 0x7fb494159f90 2024-04-15T15:15:43.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.239+0000 7fb486ffd640 1 -- 172.21.15.72:0/429149441 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb48c039040 con 0x7fb494159f90 2024-04-15T15:15:43.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.239+0000 7fb49c596640 1 -- 172.21.15.72:0/429149441 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4940c66d0 con 0x7fb494159f90 2024-04-15T15:15:43.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.240+0000 7fb49c596640 1 -- 172.21.15.72:0/429149441 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4940b2f30 con 0x7fb494159f90 2024-04-15T15:15:43.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.240+0000 7fb486ffd640 1 -- 172.21.15.72:0/429149441 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb48c046070 con 0x7fb494159f90 2024-04-15T15:15:43.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.241+0000 7fb486ffd640 1 --2- 172.21.15.72:0/429149441 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb47806c920 0x7fb47806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.241+0000 7fb49a332640 1 --2- 172.21.15.72:0/429149441 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb47806c920 0x7fb47806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.241+0000 7fb486ffd640 1 -- 172.21.15.72:0/429149441 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fb48c0ad360 con 0x7fb494159f90 2024-04-15T15:15:43.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.241+0000 7fb49a332640 1 --2- 172.21.15.72:0/429149441 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb47806c920 0x7fb47806ede0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb490006fd0 tx=0x7fb490008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:43.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.245+0000 7fb486ffd640 1 -- 172.21.15.72:0/429149441 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb4940b2f30 con 0x7fb494159f90 2024-04-15T15:15:43.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.418+0000 7fb49c596640 1 -- 172.21.15.72:0/429149441 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb494075c50 con 0x7fb494159f90 2024-04-15T15:15:43.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.418+0000 7fb486ffd640 1 -- 172.21.15.72:0/429149441 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7fb48c07acb0 con 0x7fb494159f90 2024-04-15T15:15:43.420 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-15T15:15:43.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.420+0000 7fb49c596640 1 -- 172.21.15.72:0/429149441 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb47806c920 msgr2=0x7fb47806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:43.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.420+0000 7fb49c596640 1 --2- 172.21.15.72:0/429149441 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb47806c920 0x7fb47806ede0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb490006fd0 tx=0x7fb490008040 comp rx=0 tx=0).stop 2024-04-15T15:15:43.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.420+0000 7fb49c596640 1 -- 172.21.15.72:0/429149441 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb494159f90 msgr2=0x7fb4940c6bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:43.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.420+0000 7fb49c596640 1 --2- 172.21.15.72:0/429149441 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb494159f90 0x7fb4940c6bf0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb48c00a810 tx=0x7fb49406e100 comp rx=0 tx=0).stop 2024-04-15T15:15:43.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.420+0000 7fb49c596640 1 -- 172.21.15.72:0/429149441 shutdown_connections 2024-04-15T15:15:43.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.420+0000 7fb49c596640 1 --2- 172.21.15.72:0/429149441 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb47806c920 0x7fb47806ede0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.420+0000 7fb49c596640 1 --2- 172.21.15.72:0/429149441 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb494159f90 0x7fb4940c6bf0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.420+0000 7fb49c596640 1 --2- 172.21.15.72:0/429149441 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb4941516b0 0x7fb494159a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.420+0000 7fb49c596640 1 --2- 172.21.15.72:0/429149441 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb4800052e0 0x7fb4941f7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.420+0000 7fb49c596640 1 -- 172.21.15.72:0/429149441 >> 172.21.15.72:0/429149441 conn(0x7fb4940ceec0 msgr2=0x7fb49406fed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:43.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.420+0000 7fb49c596640 1 -- 172.21.15.72:0/429149441 shutdown_connections 2024-04-15T15:15:43.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.420+0000 7fb49c596640 1 -- 172.21.15.72:0/429149441 wait complete. 2024-04-15T15:15:43.433 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:15:43.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.537+0000 7f9877e06640 1 Processor -- start 2024-04-15T15:15:43.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.537+0000 7f9877e06640 1 -- start start 2024-04-15T15:15:43.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.538+0000 7f9877e06640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f98700ce2f0 0x7f987015c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.538+0000 7f9877e06640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f98700d08f0 0x7f98700cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.538+0000 7f9877e06640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f987006bbb0 con 0x7f98700ce6d0 2024-04-15T15:15:43.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.538+0000 7f9877e06640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f987006b960 con 0x7f98700d08f0 2024-04-15T15:15:43.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.538+0000 7f9877e06640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f987006be90 con 0x7f98700ce2f0 2024-04-15T15:15:43.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.538+0000 7f98763a3640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f98700ce6d0 0x7f9870155600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60038/0 (socket says 172.21.15.72:60038) 2024-04-15T15:15:43.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.538+0000 7f98763a3640 1 -- 172.21.15.72:0/961338166 learned_addr learned my addr 172.21.15.72:0/961338166 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:43.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.539+0000 7f9874ba0640 1 -- 172.21.15.72:0/961338166 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1100603075 0 0) 0x7f987006bbb0 con 0x7f98700ce6d0 2024-04-15T15:15:43.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.539+0000 7f9874ba0640 1 -- 172.21.15.72:0/961338166 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9858003520 con 0x7f98700ce6d0 2024-04-15T15:15:43.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.539+0000 7f98753a1640 1 --2- 172.21.15.72:0/961338166 >> v2:172.21.15.175:3300/0 conn(0x7f98700d08f0 0x7f98700cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.539+0000 7f9875ba2640 1 --2- 172.21.15.72:0/961338166 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f98700ce2f0 0x7f987015c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.539+0000 7f9874ba0640 1 -- 172.21.15.72:0/961338166 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3882058805 0 0) 0x7f9858003520 con 0x7f98700ce6d0 2024-04-15T15:15:43.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.539+0000 7f9874ba0640 1 -- 172.21.15.72:0/961338166 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f98700ce2f0 msgr2=0x7f987015c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:43.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.539+0000 7f9874ba0640 1 --2- 172.21.15.72:0/961338166 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f98700ce2f0 0x7f987015c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.539+0000 7f9874ba0640 1 -- 172.21.15.72:0/961338166 >> v2:172.21.15.175:3300/0 conn(0x7f98700d08f0 msgr2=0x7f98700cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:43.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.539+0000 7f9874ba0640 1 --2- 172.21.15.72:0/961338166 >> v2:172.21.15.175:3300/0 conn(0x7f98700d08f0 0x7f98700cc0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.539+0000 7f9874ba0640 1 -- 172.21.15.72:0/961338166 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f987006c170 con 0x7f98700ce6d0 2024-04-15T15:15:43.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.539+0000 7f9874ba0640 1 -- 172.21.15.72:0/961338166 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f98680019f0 con 0x7f98700ce6d0 2024-04-15T15:15:43.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.539+0000 7f9874ba0640 1 -- 172.21.15.72:0/961338166 >> v1:172.21.15.72:6789/0 conn(0x7f98700ce6d0 legacy=0x7f9870155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:43.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.539+0000 7f9874ba0640 1 --2- 172.21.15.72:0/961338166 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f98700ce2f0 0x7f98580039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.539+0000 7f9874ba0640 1 --2- 172.21.15.72:0/961338166 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9858003f30 0x7f9858004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.539+0000 7f9874ba0640 1 --2- 172.21.15.72:0/961338166 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9858004d40 0x7f98580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.539+0000 7f9874ba0640 1 -- 172.21.15.72:0/961338166 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f987006b960 con 0x7f9858003f30 2024-04-15T15:15:43.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.539+0000 7f9874ba0640 1 -- 172.21.15.72:0/961338166 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f987006be90 con 0x7f9858004d40 2024-04-15T15:15:43.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.539+0000 7f9874ba0640 1 -- 172.21.15.72:0/961338166 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f985800b9c0 con 0x7f98700ce2f0 2024-04-15T15:15:43.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.539+0000 7f98753a1640 1 --2- 172.21.15.72:0/961338166 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9858003f30 0x7f9858004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.539+0000 7f98763a3640 1 --2- 172.21.15.72:0/961338166 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9858004d40 0x7f98580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.539+0000 7f9875ba2640 1 --2- 172.21.15.72:0/961338166 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f98700ce2f0 0x7f98580039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.539+0000 7f98753a1640 1 -- 172.21.15.72:0/961338166 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f98700ce2f0 msgr2=0x7f98580039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:43.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.539+0000 7f98753a1640 1 --2- 172.21.15.72:0/961338166 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f98700ce2f0 0x7f98580039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.539+0000 7f98753a1640 1 -- 172.21.15.72:0/961338166 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9858004d40 msgr2=0x7f98580051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:43.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.539+0000 7f98753a1640 1 --2- 172.21.15.72:0/961338166 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9858004d40 0x7f98580051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.539+0000 7f98753a1640 1 -- 172.21.15.72:0/961338166 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9858003520 con 0x7f9858003f30 2024-04-15T15:15:43.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.539+0000 7f98763a3640 1 --2- 172.21.15.72:0/961338166 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9858004d40 0x7f98580051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:43.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.540+0000 7f98753a1640 1 --2- 172.21.15.72:0/961338166 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9858003f30 0x7f9858004800 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f9864009ac0 tx=0x7f9858005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a65852f9c8006070 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:43.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.540+0000 7f9874ba0640 1 -- 172.21.15.72:0/961338166 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9864039070 con 0x7f9858003f30 2024-04-15T15:15:43.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.540+0000 7f9874ba0640 1 -- 172.21.15.72:0/961338166 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f986402c6d0 con 0x7f9858003f30 2024-04-15T15:15:43.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.540+0000 7f9874ba0640 1 -- 172.21.15.72:0/961338166 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9864009690 con 0x7f9858003f30 2024-04-15T15:15:43.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.541+0000 7f9877e06640 1 -- 172.21.15.72:0/961338166 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9858003f30 msgr2=0x7f9858004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:43.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.541+0000 7f9877e06640 1 --2- 172.21.15.72:0/961338166 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9858003f30 0x7f9858004800 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f9864009ac0 tx=0x7f9858005830 comp rx=0 tx=0).stop 2024-04-15T15:15:43.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.541+0000 7f98763a3640 1 -- 172.21.15.72:0/961338166 reap_dead start 2024-04-15T15:15:43.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.541+0000 7f9877e06640 1 -- 172.21.15.72:0/961338166 shutdown_connections 2024-04-15T15:15:43.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.541+0000 7f9877e06640 1 -- 172.21.15.72:0/961338166 >> 172.21.15.72:0/961338166 conn(0x7f98700b31d0 msgr2=0x7f98700b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:43.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.541+0000 7f9877e06640 1 -- 172.21.15.72:0/961338166 shutdown_connections 2024-04-15T15:15:43.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.541+0000 7f9877e06640 1 -- 172.21.15.72:0/961338166 wait complete. 2024-04-15T15:15:43.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.542+0000 7f9877e06640 1 Processor -- start 2024-04-15T15:15:43.549 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.542+0000 7f9877e06640 1 -- start start 2024-04-15T15:15:43.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.543+0000 7f9877e06640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9858003f30 0x7f98701f4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.543+0000 7f9877e06640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9870073330 0x7f98700737b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.543+0000 7f9877e06640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f98700ae570 0x7f98700ae9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.543+0000 7f9877e06640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f9870155e90 con 0x7f9870073330 2024-04-15T15:15:43.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.543+0000 7f9877e06640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f9870155d10 con 0x7f9858003f30 2024-04-15T15:15:43.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.543+0000 7f9877e06640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f9870156010 con 0x7f98700ae570 2024-04-15T15:15:43.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.543+0000 7f98753a1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9870073330 0x7f98700737b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.543+0000 7f98753a1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9870073330 0x7f98700737b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46116/0 (socket says 172.21.15.72:46116) 2024-04-15T15:15:43.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.543+0000 7f98753a1640 1 -- 172.21.15.72:0/1891472958 learned_addr learned my addr 172.21.15.72:0/1891472958 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:43.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.543+0000 7f98753a1640 1 -- 172.21.15.72:0/1891472958 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f98700ae570 msgr2=0x7f98700ae9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:43.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.543+0000 7f98763a3640 1 --2- 172.21.15.72:0/1891472958 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f98700ae570 0x7f98700ae9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.543+0000 7f9875ba2640 1 --2- 172.21.15.72:0/1891472958 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9858003f30 0x7f98701f4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.543+0000 7f98753a1640 1 --2- 172.21.15.72:0/1891472958 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f98700ae570 0x7f98700ae9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.543+0000 7f98753a1640 1 -- 172.21.15.72:0/1891472958 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9858003f30 msgr2=0x7f98701f4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:43.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.543+0000 7f98753a1640 1 --2- 172.21.15.72:0/1891472958 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9858003f30 0x7f98701f4120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.543+0000 7f98753a1640 1 -- 172.21.15.72:0/1891472958 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98700aef30 con 0x7f9870073330 2024-04-15T15:15:43.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.543+0000 7f9875ba2640 1 --2- 172.21.15.72:0/1891472958 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9858003f30 0x7f98701f4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:43.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.543+0000 7f98753a1640 1 --2- 172.21.15.72:0/1891472958 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9870073330 0x7f98700737b0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f9864000c00 tx=0x7f9864004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:43.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.543+0000 7f986e7fc640 1 -- 172.21.15.72:0/1891472958 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9864039050 con 0x7f9870073330 2024-04-15T15:15:43.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.543+0000 7f986e7fc640 1 -- 172.21.15.72:0/1891472958 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9864045070 con 0x7f9870073330 2024-04-15T15:15:43.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.543+0000 7f9877e06640 1 -- 172.21.15.72:0/1891472958 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98700afec0 con 0x7f9870073330 2024-04-15T15:15:43.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.543+0000 7f986e7fc640 1 -- 172.21.15.72:0/1891472958 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98640044e0 con 0x7f9870073330 2024-04-15T15:15:43.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.543+0000 7f9877e06640 1 -- 172.21.15.72:0/1891472958 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98700b0180 con 0x7f9870073330 2024-04-15T15:15:43.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.544+0000 7f9877e06640 1 -- 172.21.15.72:0/1891472958 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9870075c50 con 0x7f9870073330 2024-04-15T15:15:43.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.545+0000 7f986e7fc640 1 -- 172.21.15.72:0/1891472958 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9864003a50 con 0x7f9870073330 2024-04-15T15:15:43.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.545+0000 7f986e7fc640 1 --2- 172.21.15.72:0/1891472958 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f985c06c8f0 0x7f985c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.545+0000 7f986e7fc640 1 -- 172.21.15.72:0/1891472958 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f98640ae1c0 con 0x7f9870073330 2024-04-15T15:15:43.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.545+0000 7f9875ba2640 1 --2- 172.21.15.72:0/1891472958 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f985c06c8f0 0x7f985c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.546+0000 7f9875ba2640 1 --2- 172.21.15.72:0/1891472958 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f985c06c8f0 0x7f985c06edb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f98600099e0 tx=0x7f9860008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:43.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.549+0000 7f986e7fc640 1 -- 172.21.15.72:0/1891472958 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f986407c1f0 con 0x7f9870073330 2024-04-15T15:15:43.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.725+0000 7f9877e06640 1 -- 172.21.15.72:0/1891472958 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f987006cab0 con 0x7f9870073330 2024-04-15T15:15:43.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.725+0000 7f986e7fc640 1 -- 172.21.15.72:0/1891472958 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f986407baf0 con 0x7f9870073330 2024-04-15T15:15:43.727 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:43.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.727+0000 7f9877e06640 1 -- 172.21.15.72:0/1891472958 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f985c06c8f0 msgr2=0x7f985c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:43.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.727+0000 7f9877e06640 1 --2- 172.21.15.72:0/1891472958 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f985c06c8f0 0x7f985c06edb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f98600099e0 tx=0x7f9860008040 comp rx=0 tx=0).stop 2024-04-15T15:15:43.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.727+0000 7f9877e06640 1 -- 172.21.15.72:0/1891472958 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9870073330 msgr2=0x7f98700737b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:43.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.727+0000 7f9877e06640 1 --2- 172.21.15.72:0/1891472958 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9870073330 0x7f98700737b0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f9864000c00 tx=0x7f9864004730 comp rx=0 tx=0).stop 2024-04-15T15:15:43.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.727+0000 7f9877e06640 1 -- 172.21.15.72:0/1891472958 shutdown_connections 2024-04-15T15:15:43.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.727+0000 7f9877e06640 1 --2- 172.21.15.72:0/1891472958 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f985c06c8f0 0x7f985c06edb0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.727+0000 7f9877e06640 1 --2- 172.21.15.72:0/1891472958 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f98700ae570 0x7f98700ae9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.727+0000 7f9877e06640 1 --2- 172.21.15.72:0/1891472958 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9870073330 0x7f98700737b0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.727+0000 7f9877e06640 1 --2- 172.21.15.72:0/1891472958 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9858003f30 0x7f98701f4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.727+0000 7f9877e06640 1 -- 172.21.15.72:0/1891472958 >> 172.21.15.72:0/1891472958 conn(0x7f98700b31d0 msgr2=0x7f9870155570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:43.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.727+0000 7f9877e06640 1 -- 172.21.15.72:0/1891472958 shutdown_connections 2024-04-15T15:15:43.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.727+0000 7f9877e06640 1 -- 172.21.15.72:0/1891472958 wait complete. 2024-04-15T15:15:43.740 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:15:43.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.844+0000 7efe40be9640 1 Processor -- start 2024-04-15T15:15:43.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.844+0000 7efe40be9640 1 -- start start 2024-04-15T15:15:43.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.845+0000 7efe40be9640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efe3c145f10 0x7efe3c0d0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.845+0000 7efe40be9640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7efe3c06c1f0 0x7efe3c06ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.845+0000 7efe40be9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe3c06d720 con 0x7efe3c06f1a0 2024-04-15T15:15:43.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.845+0000 7efe40be9640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7efe3c1462f0 con 0x7efe3c06c1f0 2024-04-15T15:15:43.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.845+0000 7efe40be9640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7efe3c06d290 con 0x7efe3c145f10 2024-04-15T15:15:43.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.845+0000 7efe3ad9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7efe3c06f1a0 0x7efe3c160240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60054/0 (socket says 172.21.15.72:60054) 2024-04-15T15:15:43.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.845+0000 7efe3ad9d640 1 -- 172.21.15.72:0/1336155518 learned_addr learned my addr 172.21.15.72:0/1336155518 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:43.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.846+0000 7efe39d9b640 1 --2- 172.21.15.72:0/1336155518 >> v2:172.21.15.175:3300/0 conn(0x7efe3c06c1f0 0x7efe3c06ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.846+0000 7efe3959a640 1 -- 172.21.15.72:0/1336155518 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1453933711 0 0) 0x7efe3c06d720 con 0x7efe3c06f1a0 2024-04-15T15:15:43.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.846+0000 7efe3a59c640 1 --2- 172.21.15.72:0/1336155518 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efe3c145f10 0x7efe3c0d0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.846+0000 7efe3959a640 1 -- 172.21.15.72:0/1336155518 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe24003520 con 0x7efe3c06f1a0 2024-04-15T15:15:43.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.846+0000 7efe3959a640 1 -- 172.21.15.72:0/1336155518 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2524290185 0 0) 0x7efe24003520 con 0x7efe3c06f1a0 2024-04-15T15:15:43.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.846+0000 7efe3959a640 1 -- 172.21.15.72:0/1336155518 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efe3c145f10 msgr2=0x7efe3c0d0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:43.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.846+0000 7efe3959a640 1 --2- 172.21.15.72:0/1336155518 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efe3c145f10 0x7efe3c0d0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.846+0000 7efe3959a640 1 -- 172.21.15.72:0/1336155518 >> v2:172.21.15.175:3300/0 conn(0x7efe3c06c1f0 msgr2=0x7efe3c06ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:43.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.846+0000 7efe3959a640 1 --2- 172.21.15.72:0/1336155518 >> v2:172.21.15.175:3300/0 conn(0x7efe3c06c1f0 0x7efe3c06ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.846+0000 7efe3959a640 1 -- 172.21.15.72:0/1336155518 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe3c06c680 con 0x7efe3c06f1a0 2024-04-15T15:15:43.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.846+0000 7efe3959a640 1 -- 172.21.15.72:0/1336155518 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7efe34001670 con 0x7efe3c06f1a0 2024-04-15T15:15:43.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.846+0000 7efe3959a640 1 -- 172.21.15.72:0/1336155518 >> v1:172.21.15.72:6789/0 conn(0x7efe3c06f1a0 legacy=0x7efe3c160240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:43.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.846+0000 7efe3959a640 1 --2- 172.21.15.72:0/1336155518 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efe3c145f10 0x7efe240039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.846+0000 7efe3959a640 1 --2- 172.21.15.72:0/1336155518 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efe24003f30 0x7efe24004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.846+0000 7efe3959a640 1 --2- 172.21.15.72:0/1336155518 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efe24004d40 0x7efe240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.846+0000 7efe3959a640 1 -- 172.21.15.72:0/1336155518 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7efe3c1462f0 con 0x7efe24004d40 2024-04-15T15:15:43.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.846+0000 7efe3959a640 1 -- 172.21.15.72:0/1336155518 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7efe3c06d290 con 0x7efe24003f30 2024-04-15T15:15:43.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.846+0000 7efe3959a640 1 -- 172.21.15.72:0/1336155518 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7efe2400b9c0 con 0x7efe3c145f10 2024-04-15T15:15:43.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.846+0000 7efe3ad9d640 1 --2- 172.21.15.72:0/1336155518 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efe24004d40 0x7efe240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.846+0000 7efe39d9b640 1 --2- 172.21.15.72:0/1336155518 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efe24003f30 0x7efe24004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.846+0000 7efe3a59c640 1 --2- 172.21.15.72:0/1336155518 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efe3c145f10 0x7efe240039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.846+0000 7efe3ad9d640 1 -- 172.21.15.72:0/1336155518 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efe3c145f10 msgr2=0x7efe240039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:43.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.846+0000 7efe3ad9d640 1 --2- 172.21.15.72:0/1336155518 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efe3c145f10 0x7efe240039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.846+0000 7efe3ad9d640 1 -- 172.21.15.72:0/1336155518 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efe24003f30 msgr2=0x7efe24004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:43.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.846+0000 7efe3ad9d640 1 --2- 172.21.15.72:0/1336155518 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efe24003f30 0x7efe24004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.846+0000 7efe3ad9d640 1 -- 172.21.15.72:0/1336155518 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe24003520 con 0x7efe24004d40 2024-04-15T15:15:43.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.846+0000 7efe39d9b640 1 --2- 172.21.15.72:0/1336155518 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efe24003f30 0x7efe24004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:43.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.847+0000 7efe3ad9d640 1 --2- 172.21.15.72:0/1336155518 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efe24004d40 0x7efe240051c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7efe34001050 tx=0x7efe24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55087248e6edc1ec server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:43.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.847+0000 7efe3959a640 1 -- 172.21.15.72:0/1336155518 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe34007d70 con 0x7efe24004d40 2024-04-15T15:15:43.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.847+0000 7efe3959a640 1 -- 172.21.15.72:0/1336155518 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe34030070 con 0x7efe24004d40 2024-04-15T15:15:43.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.847+0000 7efe3959a640 1 -- 172.21.15.72:0/1336155518 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe3402cd70 con 0x7efe24004d40 2024-04-15T15:15:43.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.848+0000 7efe40be9640 1 -- 172.21.15.72:0/1336155518 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efe24004d40 msgr2=0x7efe240051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:43.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.848+0000 7efe40be9640 1 --2- 172.21.15.72:0/1336155518 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efe24004d40 0x7efe240051c0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7efe34001050 tx=0x7efe24005830 comp rx=0 tx=0).stop 2024-04-15T15:15:43.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.848+0000 7efe3ad9d640 1 -- 172.21.15.72:0/1336155518 reap_dead start 2024-04-15T15:15:43.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.848+0000 7efe40be9640 1 -- 172.21.15.72:0/1336155518 shutdown_connections 2024-04-15T15:15:43.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.848+0000 7efe40be9640 1 -- 172.21.15.72:0/1336155518 >> 172.21.15.72:0/1336155518 conn(0x7efe3c0d13a0 msgr2=0x7efe3c0d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:43.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.848+0000 7efe40be9640 1 -- 172.21.15.72:0/1336155518 shutdown_connections 2024-04-15T15:15:43.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.848+0000 7efe40be9640 1 -- 172.21.15.72:0/1336155518 wait complete. 2024-04-15T15:15:43.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.849+0000 7efe40be9640 1 Processor -- start 2024-04-15T15:15:43.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.849+0000 7efe40be9640 1 -- start start 2024-04-15T15:15:43.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.850+0000 7efe40be9640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efe24004d40 0x7efe3c0b0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.850+0000 7efe40be9640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efe3c15baa0 0x7efe3c15bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.850+0000 7efe40be9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efe3c0b4ec0 0x7efe3c0b5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.850+0000 7efe40be9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7efe3c06cf20 con 0x7efe3c0b4ec0 2024-04-15T15:15:43.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.850+0000 7efe40be9640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7efe3c06cc40 con 0x7efe3c15baa0 2024-04-15T15:15:43.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.850+0000 7efe40be9640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7efe3c0cec20 con 0x7efe24004d40 2024-04-15T15:15:43.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.850+0000 7efe3ad9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efe3c0b4ec0 0x7efe3c0b5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.850+0000 7efe3ad9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efe3c0b4ec0 0x7efe3c0b5320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46136/0 (socket says 172.21.15.72:46136) 2024-04-15T15:15:43.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.850+0000 7efe3ad9d640 1 -- 172.21.15.72:0/2892621154 learned_addr learned my addr 172.21.15.72:0/2892621154 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:43.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.850+0000 7efe39d9b640 1 --2- 172.21.15.72:0/2892621154 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efe3c15baa0 0x7efe3c15bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.850+0000 7efe3a59c640 1 --2- 172.21.15.72:0/2892621154 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efe24004d40 0x7efe3c0b0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.850+0000 7efe3ad9d640 1 -- 172.21.15.72:0/2892621154 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efe24004d40 msgr2=0x7efe3c0b0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:43.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.850+0000 7efe3ad9d640 1 --2- 172.21.15.72:0/2892621154 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efe24004d40 0x7efe3c0b0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.850+0000 7efe3ad9d640 1 -- 172.21.15.72:0/2892621154 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efe3c15baa0 msgr2=0x7efe3c15bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:43.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.850+0000 7efe3ad9d640 1 --2- 172.21.15.72:0/2892621154 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efe3c15baa0 0x7efe3c15bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.850+0000 7efe3ad9d640 1 -- 172.21.15.72:0/2892621154 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe3c1441c0 con 0x7efe3c0b4ec0 2024-04-15T15:15:43.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.851+0000 7efe3ad9d640 1 --2- 172.21.15.72:0/2892621154 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efe3c0b4ec0 0x7efe3c0b5320 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7efe3400c300 tx=0x7efe34001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:43.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.851+0000 7efe2affd640 1 -- 172.21.15.72:0/2892621154 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe30009690 con 0x7efe3c0b4ec0 2024-04-15T15:15:43.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.851+0000 7efe2affd640 1 -- 172.21.15.72:0/2892621154 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe34004bf0 con 0x7efe3c0b4ec0 2024-04-15T15:15:43.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.851+0000 7efe40be9640 1 -- 172.21.15.72:0/2892621154 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe3c0b5aa0 con 0x7efe3c0b4ec0 2024-04-15T15:15:43.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.851+0000 7efe2affd640 1 -- 172.21.15.72:0/2892621154 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe34007d50 con 0x7efe3c0b4ec0 2024-04-15T15:15:43.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.851+0000 7efe40be9640 1 -- 172.21.15.72:0/2892621154 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe3c0adc20 con 0x7efe3c0b4ec0 2024-04-15T15:15:43.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.852+0000 7efe40be9640 1 -- 172.21.15.72:0/2892621154 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe3c075c50 con 0x7efe3c0b4ec0 2024-04-15T15:15:43.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.852+0000 7efe2affd640 1 -- 172.21.15.72:0/2892621154 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efe34007070 con 0x7efe3c0b4ec0 2024-04-15T15:15:43.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.853+0000 7efe2affd640 1 --2- 172.21.15.72:0/2892621154 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7efe1c06c920 0x7efe1c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.853+0000 7efe2affd640 1 -- 172.21.15.72:0/2892621154 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7efe340ad100 con 0x7efe3c0b4ec0 2024-04-15T15:15:43.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.853+0000 7efe3a59c640 1 --2- 172.21.15.72:0/2892621154 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7efe1c06c920 0x7efe1c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.853+0000 7efe3a59c640 1 --2- 172.21.15.72:0/2892621154 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7efe1c06c920 0x7efe1c06ede0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7efe2c014510 tx=0x7efe2c00b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:43.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:43.857+0000 7efe2affd640 1 -- 172.21.15.72:0/2892621154 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efe3c1441c0 con 0x7efe3c0b4ec0 2024-04-15T15:15:44.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.030+0000 7efe40be9640 1 -- 172.21.15.72:0/2892621154 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7efe3c0d2130 con 0x7efe3c0b4ec0 2024-04-15T15:15:44.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.031+0000 7efe2affd640 1 -- 172.21.15.72:0/2892621154 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7efe3407aae0 con 0x7efe3c0b4ec0 2024-04-15T15:15:44.032 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-04-15T15:15:44.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.033+0000 7efe40be9640 1 -- 172.21.15.72:0/2892621154 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7efe1c06c920 msgr2=0x7efe1c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:44.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.033+0000 7efe40be9640 1 --2- 172.21.15.72:0/2892621154 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7efe1c06c920 0x7efe1c06ede0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7efe2c014510 tx=0x7efe2c00b040 comp rx=0 tx=0).stop 2024-04-15T15:15:44.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.033+0000 7efe40be9640 1 -- 172.21.15.72:0/2892621154 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efe3c0b4ec0 msgr2=0x7efe3c0b5320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:44.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.033+0000 7efe40be9640 1 --2- 172.21.15.72:0/2892621154 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efe3c0b4ec0 0x7efe3c0b5320 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7efe3400c300 tx=0x7efe34001a40 comp rx=0 tx=0).stop 2024-04-15T15:15:44.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.033+0000 7efe40be9640 1 -- 172.21.15.72:0/2892621154 shutdown_connections 2024-04-15T15:15:44.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.033+0000 7efe40be9640 1 --2- 172.21.15.72:0/2892621154 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7efe1c06c920 0x7efe1c06ede0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.033+0000 7efe40be9640 1 --2- 172.21.15.72:0/2892621154 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efe3c0b4ec0 0x7efe3c0b5320 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.033+0000 7efe40be9640 1 --2- 172.21.15.72:0/2892621154 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efe3c15baa0 0x7efe3c15bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.033+0000 7efe40be9640 1 --2- 172.21.15.72:0/2892621154 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efe24004d40 0x7efe3c0b0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.033+0000 7efe40be9640 1 -- 172.21.15.72:0/2892621154 >> 172.21.15.72:0/2892621154 conn(0x7efe3c0d13a0 msgr2=0x7efe3c0cf7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:44.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.033+0000 7efe40be9640 1 -- 172.21.15.72:0/2892621154 shutdown_connections 2024-04-15T15:15:44.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.033+0000 7efe40be9640 1 -- 172.21.15.72:0/2892621154 wait complete. 2024-04-15T15:15:44.045 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:15:44.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.151+0000 7f6072f0e640 1 Processor -- start 2024-04-15T15:15:44.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.151+0000 7f6072f0e640 1 -- start start 2024-04-15T15:15:44.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.152+0000 7f6072f0e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f606c0cc7c0 0x7f606c155600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:44.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.152+0000 7f6072f0e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f606c06bbb0 con 0x7f606c0d08f0 2024-04-15T15:15:44.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.152+0000 7f6072f0e640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f606c06be90 con 0x7f606c0ce2f0 2024-04-15T15:15:44.154 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.152+0000 7f6072f0e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f606c06b960 con 0x7f606c0cc7c0 2024-04-15T15:15:44.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.152+0000 7f60714ab640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f606c0cc7c0 0x7f606c155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:44.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.152+0000 7f606bfff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f606c0d08f0 0x7f606c0cc0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60066/0 (socket says 172.21.15.72:60066) 2024-04-15T15:15:44.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.152+0000 7f60714ab640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f606c0cc7c0 0x7f606c155600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46148/0 (socket says 172.21.15.72:46148) 2024-04-15T15:15:44.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.152+0000 7f606bfff640 1 -- 172.21.15.72:0/2502463429 learned_addr learned my addr 172.21.15.72:0/2502463429 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:44.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.153+0000 7f60714ab640 1 -- 172.21.15.72:0/2502463429 >> v1:172.21.15.175:6789/0 conn(0x7f606c0ce2f0 legacy=0x7f606c15c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:44.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.153+0000 7f60714ab640 1 -- 172.21.15.72:0/2502463429 >> v1:172.21.15.72:6789/0 conn(0x7f606c0d08f0 legacy=0x7f606c0cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:44.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.153+0000 7f60714ab640 1 -- 172.21.15.72:0/2502463429 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f606c06c170 con 0x7f606c0cc7c0 2024-04-15T15:15:44.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.153+0000 7f606b7fe640 1 -- 172.21.15.72:0/2502463429 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2336131940 0 0) 0x7f606c06bbb0 con 0x7f606c0d08f0 2024-04-15T15:15:44.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.153+0000 7f60714ab640 1 --2- 172.21.15.72:0/2502463429 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f606c0cc7c0 0x7f606c155600 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f606c06bae0 tx=0x7f606402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16d54f8341599f96 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:44.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.153+0000 7f606b7fe640 1 -- 172.21.15.72:0/2502463429 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6064039070 con 0x7f606c0cc7c0 2024-04-15T15:15:44.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.154+0000 7f606b7fe640 1 -- 172.21.15.72:0/2502463429 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f606402cc50 con 0x7f606c0cc7c0 2024-04-15T15:15:44.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.154+0000 7f606b7fe640 1 -- 172.21.15.72:0/2502463429 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6064034440 con 0x7f606c0cc7c0 2024-04-15T15:15:44.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.154+0000 7f6072f0e640 1 -- 172.21.15.72:0/2502463429 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f606c0cc7c0 msgr2=0x7f606c155600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:44.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.154+0000 7f6072f0e640 1 --2- 172.21.15.72:0/2502463429 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f606c0cc7c0 0x7f606c155600 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f606c06bae0 tx=0x7f606402bea0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.154+0000 7f6072f0e640 1 -- 172.21.15.72:0/2502463429 shutdown_connections 2024-04-15T15:15:44.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.154+0000 7f6072f0e640 1 --2- 172.21.15.72:0/2502463429 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f606c0cc7c0 0x7f606c155600 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.154+0000 7f6072f0e640 1 -- 172.21.15.72:0/2502463429 >> 172.21.15.72:0/2502463429 conn(0x7f606c0b31d0 msgr2=0x7f606c0b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:44.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.154+0000 7f6072f0e640 1 -- 172.21.15.72:0/2502463429 shutdown_connections 2024-04-15T15:15:44.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.154+0000 7f6072f0e640 1 -- 172.21.15.72:0/2502463429 wait complete. 2024-04-15T15:15:44.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.155+0000 7f6072f0e640 1 Processor -- start 2024-04-15T15:15:44.157 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.155+0000 7f6072f0e640 1 -- start start 2024-04-15T15:15:44.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.156+0000 7f6072f0e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f606c0cc7c0 0x7f606c153090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:44.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.156+0000 7f6072f0e640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f606c0ce2f0 0x7f606c0b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:44.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.156+0000 7f6072f0e640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f606c0d08f0 0x7f606c1527c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:44.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.156+0000 7f6072f0e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f606c06ef50 con 0x7f606c0cc7c0 2024-04-15T15:15:44.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.156+0000 7f6072f0e640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f606c06edd0 con 0x7f606c0d08f0 2024-04-15T15:15:44.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.156+0000 7f6072f0e640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f606c06f0d0 con 0x7f606c0ce2f0 2024-04-15T15:15:44.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.156+0000 7f6070caa640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f606c0cc7c0 0x7f606c153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:44.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.156+0000 7f6070caa640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f606c0cc7c0 0x7f606c153090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46152/0 (socket says 172.21.15.72:46152) 2024-04-15T15:15:44.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.156+0000 7f6070caa640 1 -- 172.21.15.72:0/2003332664 learned_addr learned my addr 172.21.15.72:0/2003332664 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:44.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.156+0000 7f60714ab640 1 --2- 172.21.15.72:0/2003332664 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f606c0d08f0 0x7f606c1527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:44.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.156+0000 7f606bfff640 1 --2- 172.21.15.72:0/2003332664 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f606c0ce2f0 0x7f606c0b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:44.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.157+0000 7f6070caa640 1 -- 172.21.15.72:0/2003332664 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f606c0ce2f0 msgr2=0x7f606c0b2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:44.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.157+0000 7f6070caa640 1 --2- 172.21.15.72:0/2003332664 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f606c0ce2f0 0x7f606c0b2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.157+0000 7f6070caa640 1 -- 172.21.15.72:0/2003332664 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f606c0d08f0 msgr2=0x7f606c1527c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:44.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.157+0000 7f6070caa640 1 --2- 172.21.15.72:0/2003332664 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f606c0d08f0 0x7f606c1527c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.157+0000 7f6070caa640 1 -- 172.21.15.72:0/2003332664 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f606c154700 con 0x7f606c0cc7c0 2024-04-15T15:15:44.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.157+0000 7f60714ab640 1 --2- 172.21.15.72:0/2003332664 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f606c0d08f0 0x7f606c1527c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:44.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.157+0000 7f6070caa640 1 --2- 172.21.15.72:0/2003332664 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f606c0cc7c0 0x7f606c153090 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f605c00c430 tx=0x7f605c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:44.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.157+0000 7f60697fa640 1 -- 172.21.15.72:0/2003332664 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f605c011790 con 0x7f606c0cc7c0 2024-04-15T15:15:44.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.157+0000 7f60697fa640 1 -- 172.21.15.72:0/2003332664 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f605c017070 con 0x7f606c0cc7c0 2024-04-15T15:15:44.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.157+0000 7f6072f0e640 1 -- 172.21.15.72:0/2003332664 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f606c153670 con 0x7f606c0cc7c0 2024-04-15T15:15:44.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.157+0000 7f60697fa640 1 -- 172.21.15.72:0/2003332664 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f605c011bb0 con 0x7f606c0cc7c0 2024-04-15T15:15:44.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.157+0000 7f6072f0e640 1 -- 172.21.15.72:0/2003332664 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f606c153c30 con 0x7f606c0cc7c0 2024-04-15T15:15:44.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.158+0000 7f6072f0e640 1 -- 172.21.15.72:0/2003332664 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f606c1548d0 con 0x7f606c0cc7c0 2024-04-15T15:15:44.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.158+0000 7f60697fa640 1 -- 172.21.15.72:0/2003332664 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f605c01c460 con 0x7f606c0cc7c0 2024-04-15T15:15:44.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.159+0000 7f60697fa640 1 --2- 172.21.15.72:0/2003332664 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f605806c8f0 0x7f605806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:44.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.159+0000 7f60697fa640 1 -- 172.21.15.72:0/2003332664 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f605c08be60 con 0x7f606c0cc7c0 2024-04-15T15:15:44.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.159+0000 7f606bfff640 1 --2- 172.21.15.72:0/2003332664 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f605806c8f0 0x7f605806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:44.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.159+0000 7f606bfff640 1 --2- 172.21.15.72:0/2003332664 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f605806c8f0 0x7f605806edb0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f606c0cc750 tx=0x7f6060007c00 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:44.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.163+0000 7f60697fa640 1 -- 172.21.15.72:0/2003332664 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f605c059ea0 con 0x7f606c0cc7c0 2024-04-15T15:15:44.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.339+0000 7f6072f0e640 1 -- 172.21.15.72:0/2003332664 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f606c075c50 con 0x7f606c0cc7c0 2024-04-15T15:15:44.341 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.339+0000 7f60697fa640 1 -- 172.21.15.72:0/2003332664 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f605c0597a0 con 0x7f606c0cc7c0 2024-04-15T15:15:44.341 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:44.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.341+0000 7f6072f0e640 1 -- 172.21.15.72:0/2003332664 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f605806c8f0 msgr2=0x7f605806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:44.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.341+0000 7f6072f0e640 1 --2- 172.21.15.72:0/2003332664 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f605806c8f0 0x7f605806edb0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f606c0cc750 tx=0x7f6060007c00 comp rx=0 tx=0).stop 2024-04-15T15:15:44.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.341+0000 7f6072f0e640 1 -- 172.21.15.72:0/2003332664 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f606c0cc7c0 msgr2=0x7f606c153090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:44.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.341+0000 7f6072f0e640 1 --2- 172.21.15.72:0/2003332664 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f606c0cc7c0 0x7f606c153090 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f605c00c430 tx=0x7f605c00c8f0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.341+0000 7f6072f0e640 1 -- 172.21.15.72:0/2003332664 shutdown_connections 2024-04-15T15:15:44.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.341+0000 7f6072f0e640 1 --2- 172.21.15.72:0/2003332664 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f605806c8f0 0x7f605806edb0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.341+0000 7f6072f0e640 1 --2- 172.21.15.72:0/2003332664 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f606c0d08f0 0x7f606c1527c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.341+0000 7f6072f0e640 1 --2- 172.21.15.72:0/2003332664 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f606c0ce2f0 0x7f606c0b2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.341+0000 7f6072f0e640 1 --2- 172.21.15.72:0/2003332664 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f606c0cc7c0 0x7f606c153090 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.341+0000 7f6072f0e640 1 -- 172.21.15.72:0/2003332664 >> 172.21.15.72:0/2003332664 conn(0x7f606c0b31d0 msgr2=0x7f606c15e660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:44.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.341+0000 7f6072f0e640 1 -- 172.21.15.72:0/2003332664 shutdown_connections 2024-04-15T15:15:44.344 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.341+0000 7f6072f0e640 1 -- 172.21.15.72:0/2003332664 wait complete. 2024-04-15T15:15:44.354 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:15:44.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.456+0000 7f6d826c7640 1 Processor -- start 2024-04-15T15:15:44.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.457+0000 7f6d826c7640 1 -- start start 2024-04-15T15:15:44.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.457+0000 7f6d826c7640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f6d7c15e9f0 0x7f6d7c0d3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:44.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.457+0000 7f6d826c7640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6d7c06c400 0x7f6d7c06c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:44.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.457+0000 7f6d826c7640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d7c0d2320 con 0x7f6d7c15e610 2024-04-15T15:15:44.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.457+0000 7f6d826c7640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f6d7c0d31d0 con 0x7f6d7c15e9f0 2024-04-15T15:15:44.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.457+0000 7f6d826c7640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f6d7c0d25c0 con 0x7f6d7c06c400 2024-04-15T15:15:44.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.457+0000 7f6d7b7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f6d7c15e9f0 0x7f6d7c0d3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:44.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.458+0000 7f6d7b7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f6d7c15e9f0 0x7f6d7c0d3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46158/0 (socket says 172.21.15.72:46158) 2024-04-15T15:15:44.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.458+0000 7f6d7b7fe640 1 -- 172.21.15.72:0/1142007333 learned_addr learned my addr 172.21.15.72:0/1142007333 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:44.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.458+0000 7f6d80c64640 1 --2- 172.21.15.72:0/1142007333 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6d7c06c400 0x7f6d7c06c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:44.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.458+0000 7f6d7b7fe640 1 -- 172.21.15.72:0/1142007333 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6d7c06c400 msgr2=0x7f6d7c06c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:44.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.458+0000 7f6d7b7fe640 1 --2- 172.21.15.72:0/1142007333 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6d7c06c400 0x7f6d7c06c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.458+0000 7f6d7b7fe640 1 -- 172.21.15.72:0/1142007333 >> v1:172.21.15.175:6789/0 conn(0x7f6d7c15e610 legacy=0x7f6d7c0d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:44.470 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.458+0000 7f6d7b7fe640 1 -- 172.21.15.72:0/1142007333 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d7c0d28a0 con 0x7f6d7c15e9f0 2024-04-15T15:15:44.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.459+0000 7f6d7b7fe640 1 --2- 172.21.15.72:0/1142007333 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d7c15e9f0 0x7f6d7c0d3660 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f6d70009870 tx=0x7f6d7002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0fa01fd974fa5a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:44.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.459+0000 7f6d7affd640 1 -- 172.21.15.72:0/1142007333 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d7002f840 con 0x7f6d7c15e9f0 2024-04-15T15:15:44.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.459+0000 7f6d7affd640 1 -- 172.21.15.72:0/1142007333 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d7002f9e0 con 0x7f6d7c15e9f0 2024-04-15T15:15:44.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.459+0000 7f6d7affd640 1 -- 172.21.15.72:0/1142007333 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d7002fdb0 con 0x7f6d7c15e9f0 2024-04-15T15:15:44.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.459+0000 7f6d826c7640 1 -- 172.21.15.72:0/1142007333 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d7c15e9f0 msgr2=0x7f6d7c0d3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:44.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.459+0000 7f6d826c7640 1 --2- 172.21.15.72:0/1142007333 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d7c15e9f0 0x7f6d7c0d3660 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f6d70009870 tx=0x7f6d7002ba20 comp rx=0 tx=0).stop 2024-04-15T15:15:44.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.459+0000 7f6d826c7640 1 -- 172.21.15.72:0/1142007333 shutdown_connections 2024-04-15T15:15:44.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.459+0000 7f6d826c7640 1 --2- 172.21.15.72:0/1142007333 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6d7c06c400 0x7f6d7c06c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.459+0000 7f6d826c7640 1 --2- 172.21.15.72:0/1142007333 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d7c15e9f0 0x7f6d7c0d3660 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.459+0000 7f6d826c7640 1 -- 172.21.15.72:0/1142007333 >> 172.21.15.72:0/1142007333 conn(0x7f6d7c0d4750 msgr2=0x7f6d7c0d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:44.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.459+0000 7f6d826c7640 1 -- 172.21.15.72:0/1142007333 shutdown_connections 2024-04-15T15:15:44.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.459+0000 7f6d826c7640 1 -- 172.21.15.72:0/1142007333 wait complete. 2024-04-15T15:15:44.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.460+0000 7f6d826c7640 1 Processor -- start 2024-04-15T15:15:44.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.461+0000 7f6d826c7640 1 -- start start 2024-04-15T15:15:44.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.461+0000 7f6d826c7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d7c06c400 0x7f6d7c158df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:44.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.461+0000 7f6d826c7640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6d7c15e610 0x7f6d7c0ca830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:44.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.461+0000 7f6d826c7640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6d7c15e9f0 0x7f6d7c161060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:44.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.461+0000 7f6d826c7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6d7c06d130 con 0x7f6d7c06c400 2024-04-15T15:15:44.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.461+0000 7f6d826c7640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f6d7c0d2e60 con 0x7f6d7c15e9f0 2024-04-15T15:15:44.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.461+0000 7f6d826c7640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f6d7c0d3ba0 con 0x7f6d7c15e610 2024-04-15T15:15:44.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.461+0000 7f6d7bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d7c06c400 0x7f6d7c158df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:44.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.461+0000 7f6d7bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d7c06c400 0x7f6d7c158df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46160/0 (socket says 172.21.15.72:46160) 2024-04-15T15:15:44.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.461+0000 7f6d7bfff640 1 -- 172.21.15.72:0/2147714941 learned_addr learned my addr 172.21.15.72:0/2147714941 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:44.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.462+0000 7f6d80c64640 1 --2- 172.21.15.72:0/2147714941 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6d7c15e9f0 0x7f6d7c161060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:44.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.462+0000 7f6d7b7fe640 1 --2- 172.21.15.72:0/2147714941 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6d7c15e610 0x7f6d7c0ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:44.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.462+0000 7f6d7bfff640 1 -- 172.21.15.72:0/2147714941 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6d7c15e610 msgr2=0x7f6d7c0ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:44.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.462+0000 7f6d7bfff640 1 --2- 172.21.15.72:0/2147714941 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6d7c15e610 0x7f6d7c0ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.462+0000 7f6d7bfff640 1 -- 172.21.15.72:0/2147714941 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6d7c15e9f0 msgr2=0x7f6d7c161060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:44.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.462+0000 7f6d7bfff640 1 --2- 172.21.15.72:0/2147714941 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6d7c15e9f0 0x7f6d7c161060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.462+0000 7f6d7bfff640 1 -- 172.21.15.72:0/2147714941 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d7c159330 con 0x7f6d7c06c400 2024-04-15T15:15:44.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.462+0000 7f6d7bfff640 1 --2- 172.21.15.72:0/2147714941 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d7c06c400 0x7f6d7c158df0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c00c400 tx=0x7f6d6c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:44.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.462+0000 7f6d78ff9640 1 -- 172.21.15.72:0/2147714941 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d6c0052d0 con 0x7f6d7c06c400 2024-04-15T15:15:44.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.462+0000 7f6d78ff9640 1 -- 172.21.15.72:0/2147714941 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d6c005490 con 0x7f6d7c06c400 2024-04-15T15:15:44.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.462+0000 7f6d826c7640 1 -- 172.21.15.72:0/2147714941 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d7c1561d0 con 0x7f6d7c06c400 2024-04-15T15:15:44.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.462+0000 7f6d78ff9640 1 -- 172.21.15.72:0/2147714941 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d6c007260 con 0x7f6d7c06c400 2024-04-15T15:15:44.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.462+0000 7f6d826c7640 1 -- 172.21.15.72:0/2147714941 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d7c159b90 con 0x7f6d7c06c400 2024-04-15T15:15:44.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.463+0000 7f6d826c7640 1 -- 172.21.15.72:0/2147714941 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d7c075c50 con 0x7f6d7c06c400 2024-04-15T15:15:44.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.464+0000 7f6d78ff9640 1 -- 172.21.15.72:0/2147714941 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6d6c007400 con 0x7f6d7c06c400 2024-04-15T15:15:44.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.464+0000 7f6d78ff9640 1 --2- 172.21.15.72:0/2147714941 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6d6806c8f0 0x7f6d6806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:44.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.464+0000 7f6d78ff9640 1 -- 172.21.15.72:0/2147714941 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f6d6c08b3a0 con 0x7f6d7c06c400 2024-04-15T15:15:44.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.464+0000 7f6d7b7fe640 1 --2- 172.21.15.72:0/2147714941 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6d6806c8f0 0x7f6d6806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:44.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.465+0000 7f6d7b7fe640 1 --2- 172.21.15.72:0/2147714941 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6d6806c8f0 0x7f6d6806edb0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6d70036000 tx=0x7f6d70008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:44.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.469+0000 7f6d78ff9640 1 -- 172.21.15.72:0/2147714941 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6d6c058e10 con 0x7f6d7c06c400 2024-04-15T15:15:44.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.656+0000 7f6d826c7640 1 -- 172.21.15.72:0/2147714941 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f6d7c0cf740 con 0x7f6d7c06c400 2024-04-15T15:15:44.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.657+0000 7f6d78ff9640 1 -- 172.21.15.72:0/2147714941 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7f6d6c05d110 con 0x7f6d7c06c400 2024-04-15T15:15:44.658 INFO:teuthology.orchestra.run.smithi072.stderr:listed 0 entries 2024-04-15T15:15:44.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.659+0000 7f6d826c7640 1 -- 172.21.15.72:0/2147714941 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6d6806c8f0 msgr2=0x7f6d6806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:44.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.659+0000 7f6d826c7640 1 --2- 172.21.15.72:0/2147714941 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6d6806c8f0 0x7f6d6806edb0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6d70036000 tx=0x7f6d70008030 comp rx=0 tx=0).stop 2024-04-15T15:15:44.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.659+0000 7f6d826c7640 1 -- 172.21.15.72:0/2147714941 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d7c06c400 msgr2=0x7f6d7c158df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:44.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.659+0000 7f6d826c7640 1 --2- 172.21.15.72:0/2147714941 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d7c06c400 0x7f6d7c158df0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c00c400 tx=0x7f6d6c00c8c0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.659+0000 7f6d826c7640 1 -- 172.21.15.72:0/2147714941 shutdown_connections 2024-04-15T15:15:44.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.659+0000 7f6d826c7640 1 --2- 172.21.15.72:0/2147714941 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6d6806c8f0 0x7f6d6806edb0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.659+0000 7f6d826c7640 1 --2- 172.21.15.72:0/2147714941 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6d7c15e9f0 0x7f6d7c161060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.659+0000 7f6d826c7640 1 --2- 172.21.15.72:0/2147714941 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6d7c15e610 0x7f6d7c0ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.659+0000 7f6d826c7640 1 --2- 172.21.15.72:0/2147714941 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6d7c06c400 0x7f6d7c158df0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:44.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.659+0000 7f6d826c7640 1 -- 172.21.15.72:0/2147714941 >> 172.21.15.72:0/2147714941 conn(0x7f6d7c0d4750 msgr2=0x7f6d7c0d4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:44.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.659+0000 7f6d826c7640 1 -- 172.21.15.72:0/2147714941 shutdown_connections 2024-04-15T15:15:44.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:44.659+0000 7f6d826c7640 1 -- 172.21.15.72:0/2147714941 wait complete. 2024-04-15T15:15:44.671 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi175.front.sepia.ceph.com... 2024-04-15T15:15:44.671 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:44.672 DEBUG:teuthology.orchestra.run.smithi175:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-04-15T15:15:44.707 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:15:44.707 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi175.front.sepia.ceph.com 2024-04-15T15:15:44.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:44.707 DEBUG:teuthology.orchestra.run.smithi175:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-15T15:15:44.859 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:15:44.859 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:15:44.875 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:15:44.876 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:15:44.934 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:45.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.076+0000 7fb0b382c640 1 Processor -- start 2024-04-15T15:15:45.078 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.076+0000 7fb0b382c640 1 -- start start 2024-04-15T15:15:45.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.077+0000 7fb0b382c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb0ac15c7c0 0x7fb0ac06e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.077+0000 7fb0b382c640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb0ac0af700 0x7fb0ac06e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.077+0000 7fb0b382c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0ac0cffd0 con 0x7fb0ac0ccb70 2024-04-15T15:15:45.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.077+0000 7fb0b382c640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fb0ac15cba0 con 0x7fb0ac15c7c0 2024-04-15T15:15:45.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.077+0000 7fb0b382c640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fb0ac06dc10 con 0x7fb0ac0af700 2024-04-15T15:15:45.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.077+0000 7fb0b15c8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb0ac15c7c0 0x7fb0ac06e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.077+0000 7fb0b15c8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb0ac15c7c0 0x7fb0ac06e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46176/0 (socket says 172.21.15.72:46176) 2024-04-15T15:15:45.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.077+0000 7fb0b15c8640 1 -- 172.21.15.72:0/1496165742 learned_addr learned my addr 172.21.15.72:0/1496165742 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:45.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.077+0000 7fb0b1dc9640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fb0ac0ccb70 0x7fb0ac0ccfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60076/0 (socket says 172.21.15.72:60076) 2024-04-15T15:15:45.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.077+0000 7fb0b0dc7640 1 --2- 172.21.15.72:0/1496165742 >> v2:172.21.15.175:3300/0 conn(0x7fb0ac0af700 0x7fb0ac06e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.077+0000 7fb0b15c8640 1 -- 172.21.15.72:0/1496165742 >> v2:172.21.15.175:3300/0 conn(0x7fb0ac0af700 msgr2=0x7fb0ac06e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:45.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.077+0000 7fb0b15c8640 1 --2- 172.21.15.72:0/1496165742 >> v2:172.21.15.175:3300/0 conn(0x7fb0ac0af700 0x7fb0ac06e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.077+0000 7fb0b15c8640 1 -- 172.21.15.72:0/1496165742 >> v1:172.21.15.72:6789/0 conn(0x7fb0ac0ccb70 legacy=0x7fb0ac0ccfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.077+0000 7fb0b15c8640 1 -- 172.21.15.72:0/1496165742 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0ac06de60 con 0x7fb0ac15c7c0 2024-04-15T15:15:45.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.078+0000 7fb0b15c8640 1 --2- 172.21.15.72:0/1496165742 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0ac15c7c0 0x7fb0ac06e420 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac04ff20 tx=0x7fb0a802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc133d5e35faed85 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:45.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.078+0000 7fb0a3fff640 1 -- 172.21.15.72:0/1496165742 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8034070 con 0x7fb0ac15c7c0 2024-04-15T15:15:45.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.078+0000 7fb0a3fff640 1 -- 172.21.15.72:0/1496165742 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0a8030860 con 0x7fb0ac15c7c0 2024-04-15T15:15:45.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.078+0000 7fb0a3fff640 1 -- 172.21.15.72:0/1496165742 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8030c30 con 0x7fb0ac15c7c0 2024-04-15T15:15:45.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.079+0000 7fb0b382c640 1 -- 172.21.15.72:0/1496165742 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0ac15c7c0 msgr2=0x7fb0ac06e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.079+0000 7fb0b382c640 1 --2- 172.21.15.72:0/1496165742 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0ac15c7c0 0x7fb0ac06e420 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb0ac04ff20 tx=0x7fb0a802b9a0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.079+0000 7fb0b382c640 1 -- 172.21.15.72:0/1496165742 shutdown_connections 2024-04-15T15:15:45.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.079+0000 7fb0b382c640 1 --2- 172.21.15.72:0/1496165742 >> v2:172.21.15.175:3300/0 conn(0x7fb0ac0af700 0x7fb0ac06e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.079+0000 7fb0b382c640 1 --2- 172.21.15.72:0/1496165742 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0ac15c7c0 0x7fb0ac06e420 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.079+0000 7fb0b382c640 1 -- 172.21.15.72:0/1496165742 >> 172.21.15.72:0/1496165742 conn(0x7fb0ac0b1dd0 msgr2=0x7fb0ac0b21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:45.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.079+0000 7fb0b382c640 1 -- 172.21.15.72:0/1496165742 shutdown_connections 2024-04-15T15:15:45.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.079+0000 7fb0b382c640 1 -- 172.21.15.72:0/1496165742 wait complete. 2024-04-15T15:15:45.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.080+0000 7fb0b382c640 1 Processor -- start 2024-04-15T15:15:45.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.080+0000 7fb0b382c640 1 -- start start 2024-04-15T15:15:45.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.081+0000 7fb0b382c640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb0ac0af700 0x7fb0ac0cd580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.081+0000 7fb0b382c640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb0ac0ccb70 0x7fb0ac14f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.081+0000 7fb0b382c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0ac15c7c0 0x7fb0ac1f3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.081+0000 7fb0b382c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb0ac0cfab0 con 0x7fb0ac15c7c0 2024-04-15T15:15:45.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.081+0000 7fb0b382c640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fb0ac0cf7d0 con 0x7fb0ac0ccb70 2024-04-15T15:15:45.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.081+0000 7fb0b382c640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fb0ac0cdc60 con 0x7fb0ac0af700 2024-04-15T15:15:45.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.081+0000 7fb0b1dc9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0ac15c7c0 0x7fb0ac1f3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.081+0000 7fb0b1dc9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0ac15c7c0 0x7fb0ac1f3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46184/0 (socket says 172.21.15.72:46184) 2024-04-15T15:15:45.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.081+0000 7fb0b1dc9640 1 -- 172.21.15.72:0/64146017 learned_addr learned my addr 172.21.15.72:0/64146017 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:45.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.081+0000 7fb0b0dc7640 1 --2- 172.21.15.72:0/64146017 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb0ac0ccb70 0x7fb0ac14f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.081+0000 7fb0b15c8640 1 --2- 172.21.15.72:0/64146017 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb0ac0af700 0x7fb0ac0cd580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.081+0000 7fb0b1dc9640 1 -- 172.21.15.72:0/64146017 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb0ac0af700 msgr2=0x7fb0ac0cd580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.081+0000 7fb0b1dc9640 1 --2- 172.21.15.72:0/64146017 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb0ac0af700 0x7fb0ac0cd580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.081+0000 7fb0b1dc9640 1 -- 172.21.15.72:0/64146017 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb0ac0ccb70 msgr2=0x7fb0ac14f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.081+0000 7fb0b1dc9640 1 --2- 172.21.15.72:0/64146017 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb0ac0ccb70 0x7fb0ac14f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.081+0000 7fb0b1dc9640 1 -- 172.21.15.72:0/64146017 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0ac1f4580 con 0x7fb0ac15c7c0 2024-04-15T15:15:45.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.081+0000 7fb0b15c8640 1 --2- 172.21.15.72:0/64146017 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb0ac0af700 0x7fb0ac0cd580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:45.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.081+0000 7fb0b1dc9640 1 --2- 172.21.15.72:0/64146017 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0ac15c7c0 0x7fb0ac1f3c30 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fb0a400c760 tx=0x7fb0a400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:45.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.081+0000 7fb0a1ffb640 1 -- 172.21.15.72:0/64146017 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a4017070 con 0x7fb0ac15c7c0 2024-04-15T15:15:45.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.081+0000 7fb0a1ffb640 1 -- 172.21.15.72:0/64146017 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0a400a640 con 0x7fb0ac15c7c0 2024-04-15T15:15:45.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.081+0000 7fb0b382c640 1 -- 172.21.15.72:0/64146017 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0ac1f4860 con 0x7fb0ac15c7c0 2024-04-15T15:15:45.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.081+0000 7fb0a1ffb640 1 -- 172.21.15.72:0/64146017 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a400a9f0 con 0x7fb0ac15c7c0 2024-04-15T15:15:45.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.081+0000 7fb0b382c640 1 -- 172.21.15.72:0/64146017 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0ac1f4e20 con 0x7fb0ac15c7c0 2024-04-15T15:15:45.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.083+0000 7fb0b382c640 1 -- 172.21.15.72:0/64146017 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0ac0689e0 con 0x7fb0ac15c7c0 2024-04-15T15:15:45.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.083+0000 7fb0a1ffb640 1 -- 172.21.15.72:0/64146017 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb0a401a070 con 0x7fb0ac15c7c0 2024-04-15T15:15:45.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.083+0000 7fb0a1ffb640 1 --2- 172.21.15.72:0/64146017 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb09006c8f0 0x7fb09006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.083+0000 7fb0a1ffb640 1 -- 172.21.15.72:0/64146017 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fb0a408b420 con 0x7fb0ac15c7c0 2024-04-15T15:15:45.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.083+0000 7fb0b15c8640 1 --2- 172.21.15.72:0/64146017 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb09006c8f0 0x7fb09006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.084+0000 7fb0b15c8640 1 --2- 172.21.15.72:0/64146017 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb09006c8f0 0x7fb09006edb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8036000 tx=0x7fb0a803b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:45.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.088+0000 7fb0a1ffb640 1 -- 172.21.15.72:0/64146017 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb0a4058e00 con 0x7fb0ac15c7c0 2024-04-15T15:15:45.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.266+0000 7fb0b382c640 1 -- 172.21.15.72:0/64146017 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb0ac0cc600 con 0x7fb0ac15c7c0 2024-04-15T15:15:45.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.267+0000 7fb0a1ffb640 1 -- 172.21.15.72:0/64146017 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18903 (secure 0 0 0) 0x7fb0a405d100 con 0x7fb0ac15c7c0 2024-04-15T15:15:45.269 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:45.269 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":26,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:15:32.421728+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:14:49.438366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:14:50.368379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:14:54.081157+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":1}]},{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:45.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.269+0000 7fb0b382c640 1 -- 172.21.15.72:0/64146017 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb09006c8f0 msgr2=0x7fb09006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.269+0000 7fb0b382c640 1 --2- 172.21.15.72:0/64146017 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb09006c8f0 0x7fb09006edb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8036000 tx=0x7fb0a803b040 comp rx=0 tx=0).stop 2024-04-15T15:15:45.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.269+0000 7fb0b382c640 1 -- 172.21.15.72:0/64146017 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0ac15c7c0 msgr2=0x7fb0ac1f3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.269+0000 7fb0b382c640 1 --2- 172.21.15.72:0/64146017 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0ac15c7c0 0x7fb0ac1f3c30 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fb0a400c760 tx=0x7fb0a400cc20 comp rx=0 tx=0).stop 2024-04-15T15:15:45.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.270+0000 7fb0b382c640 1 -- 172.21.15.72:0/64146017 shutdown_connections 2024-04-15T15:15:45.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.270+0000 7fb0b382c640 1 --2- 172.21.15.72:0/64146017 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb09006c8f0 0x7fb09006edb0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.270+0000 7fb0b382c640 1 --2- 172.21.15.72:0/64146017 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb0ac15c7c0 0x7fb0ac1f3c30 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.270+0000 7fb0b382c640 1 --2- 172.21.15.72:0/64146017 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb0ac0ccb70 0x7fb0ac14f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.270+0000 7fb0b382c640 1 --2- 172.21.15.72:0/64146017 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb0ac0af700 0x7fb0ac0cd580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.270+0000 7fb0b382c640 1 -- 172.21.15.72:0/64146017 >> 172.21.15.72:0/64146017 conn(0x7fb0ac0b1dd0 msgr2=0x7fb0ac0adbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:45.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.270+0000 7fb0b382c640 1 -- 172.21.15.72:0/64146017 shutdown_connections 2024-04-15T15:15:45.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.270+0000 7fb0b382c640 1 -- 172.21.15.72:0/64146017 wait complete. 2024-04-15T15:15:45.283 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:39.359102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:14:49.438366+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:14:50.368379+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:14:54.081157+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:15:45.283 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:45.389 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.388+0000 7fea5dd04640 1 Processor -- start 2024-04-15T15:15:45.390 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.388+0000 7fea5dd04640 1 -- start start 2024-04-15T15:15:45.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.389+0000 7fea5dd04640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fea580ce2f0 0x7fea5815c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.389+0000 7fea5dd04640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fea580d08f0 0x7fea580cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.389+0000 7fea5dd04640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea5806bbb0 con 0x7fea580ce6d0 2024-04-15T15:15:45.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.389+0000 7fea5dd04640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fea5806b960 con 0x7fea580ce2f0 2024-04-15T15:15:45.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.389+0000 7fea5dd04640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fea5806be90 con 0x7fea580d08f0 2024-04-15T15:15:45.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.389+0000 7fea577fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fea580ce2f0 0x7fea5815c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.389+0000 7fea577fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fea580ce2f0 0x7fea5815c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46192/0 (socket says 172.21.15.72:46192) 2024-04-15T15:15:45.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.389+0000 7fea577fe640 1 -- 172.21.15.72:0/1457610125 learned_addr learned my addr 172.21.15.72:0/1457610125 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:45.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.389+0000 7fea4ffff640 1 --2- 172.21.15.72:0/1457610125 >> v2:172.21.15.175:3300/0 conn(0x7fea580d08f0 0x7fea580cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.389+0000 7fea577fe640 1 -- 172.21.15.72:0/1457610125 >> v2:172.21.15.175:3300/0 conn(0x7fea580d08f0 msgr2=0x7fea580cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:45.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.389+0000 7fea577fe640 1 --2- 172.21.15.72:0/1457610125 >> v2:172.21.15.175:3300/0 conn(0x7fea580d08f0 0x7fea580cc0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.389+0000 7fea577fe640 1 -- 172.21.15.72:0/1457610125 >> v1:172.21.15.175:6789/0 conn(0x7fea580ce6d0 legacy=0x7fea58155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:45.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.389+0000 7fea577fe640 1 -- 172.21.15.72:0/1457610125 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea5806c170 con 0x7fea580ce2f0 2024-04-15T15:15:45.392 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.390+0000 7fea577fe640 1 --2- 172.21.15.72:0/1457610125 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea580ce2f0 0x7fea5815c530 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fea580b56b0 tx=0x7fea4802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d816fdf43ac2236 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:45.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.390+0000 7fea56ffd640 1 -- 172.21.15.72:0/1457610125 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea48035020 con 0x7fea580ce2f0 2024-04-15T15:15:45.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.390+0000 7fea56ffd640 1 -- 172.21.15.72:0/1457610125 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea48031870 con 0x7fea580ce2f0 2024-04-15T15:15:45.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.390+0000 7fea56ffd640 1 -- 172.21.15.72:0/1457610125 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea48031c40 con 0x7fea580ce2f0 2024-04-15T15:15:45.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.391+0000 7fea5dd04640 1 -- 172.21.15.72:0/1457610125 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea580ce2f0 msgr2=0x7fea5815c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.391+0000 7fea5dd04640 1 --2- 172.21.15.72:0/1457610125 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea580ce2f0 0x7fea5815c530 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fea580b56b0 tx=0x7fea4802b910 comp rx=0 tx=0).stop 2024-04-15T15:15:45.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.391+0000 7fea5dd04640 1 -- 172.21.15.72:0/1457610125 shutdown_connections 2024-04-15T15:15:45.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.391+0000 7fea5dd04640 1 --2- 172.21.15.72:0/1457610125 >> v2:172.21.15.175:3300/0 conn(0x7fea580d08f0 0x7fea580cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.391+0000 7fea5dd04640 1 --2- 172.21.15.72:0/1457610125 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea580ce2f0 0x7fea5815c530 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.391+0000 7fea5dd04640 1 -- 172.21.15.72:0/1457610125 >> 172.21.15.72:0/1457610125 conn(0x7fea580b31d0 msgr2=0x7fea580b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:45.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.391+0000 7fea5dd04640 1 -- 172.21.15.72:0/1457610125 shutdown_connections 2024-04-15T15:15:45.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.391+0000 7fea5dd04640 1 -- 172.21.15.72:0/1457610125 wait complete. 2024-04-15T15:15:45.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.392+0000 7fea5dd04640 1 Processor -- start 2024-04-15T15:15:45.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.392+0000 7fea5dd04640 1 -- start start 2024-04-15T15:15:45.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.393+0000 7fea5dd04640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fea580ce2f0 0x7fea580ae850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.393+0000 7fea5dd04640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fea580ce6d0 0x7fea5814c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.393+0000 7fea5dd04640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea580d08f0 0x7fea58154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.393+0000 7fea5dd04640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fea58155d10 con 0x7fea580d08f0 2024-04-15T15:15:45.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.393+0000 7fea5dd04640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fea5815ca70 con 0x7fea580ce2f0 2024-04-15T15:15:45.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.393+0000 7fea5dd04640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fea58155e90 con 0x7fea580ce6d0 2024-04-15T15:15:45.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.393+0000 7fea57fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea580d08f0 0x7fea58154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.393+0000 7fea57fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea580d08f0 0x7fea58154ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46196/0 (socket says 172.21.15.72:46196) 2024-04-15T15:15:45.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.393+0000 7fea57fff640 1 -- 172.21.15.72:0/1195918935 learned_addr learned my addr 172.21.15.72:0/1195918935 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:45.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.393+0000 7fea577fe640 1 --2- 172.21.15.72:0/1195918935 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fea580ce2f0 0x7fea580ae850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.393+0000 7fea4ffff640 1 --2- 172.21.15.72:0/1195918935 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fea580ce6d0 0x7fea5814c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.393+0000 7fea57fff640 1 -- 172.21.15.72:0/1195918935 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fea580ce6d0 msgr2=0x7fea5814c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.393+0000 7fea57fff640 1 --2- 172.21.15.72:0/1195918935 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fea580ce6d0 0x7fea5814c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.393+0000 7fea57fff640 1 -- 172.21.15.72:0/1195918935 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fea580ce2f0 msgr2=0x7fea580ae850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.393+0000 7fea57fff640 1 --2- 172.21.15.72:0/1195918935 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fea580ce2f0 0x7fea580ae850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.393+0000 7fea57fff640 1 -- 172.21.15.72:0/1195918935 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea5814ea90 con 0x7fea580d08f0 2024-04-15T15:15:45.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.393+0000 7fea57fff640 1 --2- 172.21.15.72:0/1195918935 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea580d08f0 0x7fea58154ff0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fea5000c7b0 tx=0x7fea5000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:45.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.393+0000 7fea54ff9640 1 -- 172.21.15.72:0/1195918935 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea50004b60 con 0x7fea580d08f0 2024-04-15T15:15:45.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.393+0000 7fea54ff9640 1 -- 172.21.15.72:0/1195918935 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea50004d00 con 0x7fea580d08f0 2024-04-15T15:15:45.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.394+0000 7fea5dd04640 1 -- 172.21.15.72:0/1195918935 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea5814ac90 con 0x7fea580d08f0 2024-04-15T15:15:45.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.394+0000 7fea54ff9640 1 -- 172.21.15.72:0/1195918935 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea5000aa90 con 0x7fea580d08f0 2024-04-15T15:15:45.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.394+0000 7fea5dd04640 1 -- 172.21.15.72:0/1195918935 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea5814ed30 con 0x7fea580d08f0 2024-04-15T15:15:45.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.395+0000 7fea5dd04640 1 -- 172.21.15.72:0/1195918935 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea58075c50 con 0x7fea580d08f0 2024-04-15T15:15:45.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.395+0000 7fea54ff9640 1 -- 172.21.15.72:0/1195918935 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fea5000ac30 con 0x7fea580d08f0 2024-04-15T15:15:45.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.395+0000 7fea54ff9640 1 --2- 172.21.15.72:0/1195918935 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fea3806c920 0x7fea3806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.395+0000 7fea54ff9640 1 -- 172.21.15.72:0/1195918935 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fea5008c460 con 0x7fea580d08f0 2024-04-15T15:15:45.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.395+0000 7fea577fe640 1 --2- 172.21.15.72:0/1195918935 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fea3806c920 0x7fea3806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.396+0000 7fea577fe640 1 --2- 172.21.15.72:0/1195918935 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fea3806c920 0x7fea3806ede0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fea48036000 tx=0x7fea4803c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:45.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.400+0000 7fea54ff9640 1 -- 172.21.15.72:0/1195918935 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fea5005a490 con 0x7fea580d08f0 2024-04-15T15:15:45.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.576+0000 7fea5dd04640 1 -- 172.21.15.72:0/1195918935 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fea5806cab0 con 0x7fea580d08f0 2024-04-15T15:15:45.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.576+0000 7fea54ff9640 1 -- 172.21.15.72:0/1195918935 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7fea50059d90 con 0x7fea580d08f0 2024-04-15T15:15:45.578 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-15T15:15:45.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.578+0000 7fea5dd04640 1 -- 172.21.15.72:0/1195918935 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fea3806c920 msgr2=0x7fea3806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.578+0000 7fea5dd04640 1 --2- 172.21.15.72:0/1195918935 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fea3806c920 0x7fea3806ede0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fea48036000 tx=0x7fea4803c040 comp rx=0 tx=0).stop 2024-04-15T15:15:45.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.578+0000 7fea5dd04640 1 -- 172.21.15.72:0/1195918935 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea580d08f0 msgr2=0x7fea58154ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.578+0000 7fea5dd04640 1 --2- 172.21.15.72:0/1195918935 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea580d08f0 0x7fea58154ff0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fea5000c7b0 tx=0x7fea5000cc70 comp rx=0 tx=0).stop 2024-04-15T15:15:45.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.578+0000 7fea5dd04640 1 -- 172.21.15.72:0/1195918935 shutdown_connections 2024-04-15T15:15:45.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.578+0000 7fea5dd04640 1 --2- 172.21.15.72:0/1195918935 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fea3806c920 0x7fea3806ede0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.578+0000 7fea5dd04640 1 --2- 172.21.15.72:0/1195918935 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea580d08f0 0x7fea58154ff0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.578+0000 7fea5dd04640 1 --2- 172.21.15.72:0/1195918935 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fea580ce6d0 0x7fea5814c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.578+0000 7fea5dd04640 1 --2- 172.21.15.72:0/1195918935 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fea580ce2f0 0x7fea580ae850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.578+0000 7fea5dd04640 1 -- 172.21.15.72:0/1195918935 >> 172.21.15.72:0/1195918935 conn(0x7fea580b31d0 msgr2=0x7fea580cd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:45.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.578+0000 7fea5dd04640 1 -- 172.21.15.72:0/1195918935 shutdown_connections 2024-04-15T15:15:45.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.579+0000 7fea5dd04640 1 -- 172.21.15.72:0/1195918935 wait complete. 2024-04-15T15:15:45.591 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:15:45.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.695+0000 7fac1c4ba640 1 Processor -- start 2024-04-15T15:15:45.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.695+0000 7fac1c4ba640 1 -- start start 2024-04-15T15:15:45.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.696+0000 7fac1c4ba640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fac1406c400 0x7fac1406c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.696+0000 7fac1c4ba640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac140d2320 con 0x7fac1415e610 2024-04-15T15:15:45.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.696+0000 7fac1c4ba640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac140d25c0 con 0x7fac1415e9f0 2024-04-15T15:15:45.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.696+0000 7fac1c4ba640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fac140d31d0 con 0x7fac1406c400 2024-04-15T15:15:45.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.697+0000 7fac1a256640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fac1415e610 0x7fac140d0690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60080/0 (socket says 172.21.15.72:60080) 2024-04-15T15:15:45.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.697+0000 7fac1a256640 1 -- 172.21.15.72:0/1337289109 learned_addr learned my addr 172.21.15.72:0/1337289109 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:45.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.697+0000 7fac19254640 1 -- 172.21.15.72:0/1337289109 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3156279550 0 0) 0x7fac140d2320 con 0x7fac1415e610 2024-04-15T15:15:45.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.697+0000 7fac19254640 1 -- 172.21.15.72:0/1337289109 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac00003520 con 0x7fac1415e610 2024-04-15T15:15:45.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.697+0000 7fac1aa57640 1 --2- 172.21.15.72:0/1337289109 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fac1406c400 0x7fac1406c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.697+0000 7fac19254640 1 -- 172.21.15.72:0/1337289109 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2627221452 0 0) 0x7fac00003520 con 0x7fac1415e610 2024-04-15T15:15:45.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.697+0000 7fac19254640 1 -- 172.21.15.72:0/1337289109 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fac1406c400 msgr2=0x7fac1406c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:45.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.697+0000 7fac19254640 1 --2- 172.21.15.72:0/1337289109 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fac1406c400 0x7fac1406c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.697+0000 7fac19254640 1 -- 172.21.15.72:0/1337289109 >> v1:172.21.15.175:6789/0 conn(0x7fac1415e9f0 legacy=0x7fac140d3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.697+0000 7fac19254640 1 -- 172.21.15.72:0/1337289109 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac140d28a0 con 0x7fac1415e610 2024-04-15T15:15:45.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.697+0000 7fac19254640 1 -- 172.21.15.72:0/1337289109 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7fac10001670 con 0x7fac1415e610 2024-04-15T15:15:45.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.697+0000 7fac19254640 1 -- 172.21.15.72:0/1337289109 >> v1:172.21.15.72:6789/0 conn(0x7fac1415e610 legacy=0x7fac140d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.697+0000 7fac19254640 1 --2- 172.21.15.72:0/1337289109 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac000039e0 0x7fac00003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.697+0000 7fac19254640 1 --2- 172.21.15.72:0/1337289109 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac00004800 0x7fac00004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.697+0000 7fac19254640 1 --2- 172.21.15.72:0/1337289109 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fac1406c400 0x7fac000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.697+0000 7fac19254640 1 -- 172.21.15.72:0/1337289109 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fac140d31d0 con 0x7fac000039e0 2024-04-15T15:15:45.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.697+0000 7fac19254640 1 -- 172.21.15.72:0/1337289109 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fac0000bb30 con 0x7fac00004800 2024-04-15T15:15:45.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.697+0000 7fac19254640 1 -- 172.21.15.72:0/1337289109 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fac0000bd30 con 0x7fac1406c400 2024-04-15T15:15:45.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.697+0000 7fac1a256640 1 --2- 172.21.15.72:0/1337289109 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac000039e0 0x7fac00003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.697+0000 7fac19a55640 1 --2- 172.21.15.72:0/1337289109 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac00004800 0x7fac00004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.697+0000 7fac1aa57640 1 --2- 172.21.15.72:0/1337289109 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fac1406c400 0x7fac000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.698+0000 7fac1a256640 1 -- 172.21.15.72:0/1337289109 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fac1406c400 msgr2=0x7fac000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:45.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.698+0000 7fac1a256640 1 --2- 172.21.15.72:0/1337289109 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fac1406c400 0x7fac000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.698+0000 7fac1a256640 1 -- 172.21.15.72:0/1337289109 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac00004800 msgr2=0x7fac00004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:45.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.698+0000 7fac1a256640 1 --2- 172.21.15.72:0/1337289109 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac00004800 0x7fac00004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.698+0000 7fac1a256640 1 -- 172.21.15.72:0/1337289109 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac00003520 con 0x7fac000039e0 2024-04-15T15:15:45.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.698+0000 7fac1aa57640 1 --2- 172.21.15.72:0/1337289109 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fac1406c400 0x7fac000051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:45.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.698+0000 7fac1a256640 1 --2- 172.21.15.72:0/1337289109 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac000039e0 0x7fac00003e20 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fac1000c210 tx=0x7fac00005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97c124ffb3c61935 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:45.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.698+0000 7fac19254640 1 -- 172.21.15.72:0/1337289109 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac10007d10 con 0x7fac000039e0 2024-04-15T15:15:45.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.698+0000 7fac19254640 1 -- 172.21.15.72:0/1337289109 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac10030070 con 0x7fac000039e0 2024-04-15T15:15:45.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.698+0000 7fac19254640 1 -- 172.21.15.72:0/1337289109 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac10005d10 con 0x7fac000039e0 2024-04-15T15:15:45.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.699+0000 7fac1c4ba640 1 -- 172.21.15.72:0/1337289109 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac000039e0 msgr2=0x7fac00003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.699+0000 7fac1c4ba640 1 --2- 172.21.15.72:0/1337289109 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac000039e0 0x7fac00003e20 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fac1000c210 tx=0x7fac00005830 comp rx=0 tx=0).stop 2024-04-15T15:15:45.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.699+0000 7fac1aa57640 1 -- 172.21.15.72:0/1337289109 reap_dead start 2024-04-15T15:15:45.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.699+0000 7fac1c4ba640 1 -- 172.21.15.72:0/1337289109 shutdown_connections 2024-04-15T15:15:45.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.699+0000 7fac1c4ba640 1 -- 172.21.15.72:0/1337289109 >> 172.21.15.72:0/1337289109 conn(0x7fac140d4750 msgr2=0x7fac140d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:45.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.699+0000 7fac1c4ba640 1 -- 172.21.15.72:0/1337289109 shutdown_connections 2024-04-15T15:15:45.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.699+0000 7fac1c4ba640 1 -- 172.21.15.72:0/1337289109 wait complete. 2024-04-15T15:15:45.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.700+0000 7fac1c4ba640 1 Processor -- start 2024-04-15T15:15:45.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.700+0000 7fac1c4ba640 1 -- start start 2024-04-15T15:15:45.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.701+0000 7fac1c4ba640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac000039e0 0x7fac1415ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.701+0000 7fac1c4ba640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fac14152570 0x7fac14151f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.701+0000 7fac1c4ba640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac1415ac20 0x7fac1415b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.701+0000 7fac1c4ba640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fac1406d130 con 0x7fac1415ac20 2024-04-15T15:15:45.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.701+0000 7fac1c4ba640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fac140d2e60 con 0x7fac000039e0 2024-04-15T15:15:45.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.701+0000 7fac1c4ba640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fac1406c890 con 0x7fac14152570 2024-04-15T15:15:45.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.701+0000 7fac1aa57640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac1415ac20 0x7fac1415b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.701+0000 7fac1aa57640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac1415ac20 0x7fac1415b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46218/0 (socket says 172.21.15.72:46218) 2024-04-15T15:15:45.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.701+0000 7fac1aa57640 1 -- 172.21.15.72:0/1063819167 learned_addr learned my addr 172.21.15.72:0/1063819167 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:45.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.701+0000 7fac1a256640 1 --2- 172.21.15.72:0/1063819167 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac000039e0 0x7fac1415ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.701+0000 7fac19a55640 1 --2- 172.21.15.72:0/1063819167 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fac14152570 0x7fac14151f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.701+0000 7fac1aa57640 1 -- 172.21.15.72:0/1063819167 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fac14152570 msgr2=0x7fac14151f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.701+0000 7fac1aa57640 1 --2- 172.21.15.72:0/1063819167 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fac14152570 0x7fac14151f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.701+0000 7fac1aa57640 1 -- 172.21.15.72:0/1063819167 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac000039e0 msgr2=0x7fac1415ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.701+0000 7fac1aa57640 1 --2- 172.21.15.72:0/1063819167 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac000039e0 0x7fac1415ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.701+0000 7fac1aa57640 1 -- 172.21.15.72:0/1063819167 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac1415f470 con 0x7fac1415ac20 2024-04-15T15:15:45.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.701+0000 7fac19a55640 1 --2- 172.21.15.72:0/1063819167 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fac14152570 0x7fac14151f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:45.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.701+0000 7fac1aa57640 1 --2- 172.21.15.72:0/1063819167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac1415ac20 0x7fac1415b0a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fac140b7750 tx=0x7fac0c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:45.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.702+0000 7fac0affd640 1 -- 172.21.15.72:0/1063819167 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac0c01a030 con 0x7fac1415ac20 2024-04-15T15:15:45.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.702+0000 7fac0affd640 1 -- 172.21.15.72:0/1063819167 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac0c010df0 con 0x7fac1415ac20 2024-04-15T15:15:45.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.702+0000 7fac1c4ba640 1 -- 172.21.15.72:0/1063819167 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac1415f750 con 0x7fac1415ac20 2024-04-15T15:15:45.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.702+0000 7fac0affd640 1 -- 172.21.15.72:0/1063819167 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac0c003bd0 con 0x7fac1415ac20 2024-04-15T15:15:45.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.702+0000 7fac1c4ba640 1 -- 172.21.15.72:0/1063819167 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac1415fd10 con 0x7fac1415ac20 2024-04-15T15:15:45.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.703+0000 7fac1c4ba640 1 -- 172.21.15.72:0/1063819167 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac14075c50 con 0x7fac1415ac20 2024-04-15T15:15:45.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.703+0000 7fac0affd640 1 -- 172.21.15.72:0/1063819167 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fac0c01d070 con 0x7fac1415ac20 2024-04-15T15:15:45.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.704+0000 7fac0affd640 1 --2- 172.21.15.72:0/1063819167 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fabf806c920 0x7fabf806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:45.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.704+0000 7fac0affd640 1 -- 172.21.15.72:0/1063819167 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fac0c08c4c0 con 0x7fac1415ac20 2024-04-15T15:15:45.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.704+0000 7fac1a256640 1 --2- 172.21.15.72:0/1063819167 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fabf806c920 0x7fabf806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:45.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.704+0000 7fac1a256640 1 --2- 172.21.15.72:0/1063819167 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fabf806c920 0x7fabf806ede0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fac10034000 tx=0x7fac10009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:45.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.708+0000 7fac0affd640 1 -- 172.21.15.72:0/1063819167 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fac0c05a580 con 0x7fac1415ac20 2024-04-15T15:15:45.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.884+0000 7fac1c4ba640 1 -- 172.21.15.72:0/1063819167 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fac140d0af0 con 0x7fac1415ac20 2024-04-15T15:15:45.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.885+0000 7fac0affd640 1 -- 172.21.15.72:0/1063819167 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7fac0c059e80 con 0x7fac1415ac20 2024-04-15T15:15:45.886 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:45.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.886+0000 7fac1c4ba640 1 -- 172.21.15.72:0/1063819167 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fabf806c920 msgr2=0x7fabf806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.886+0000 7fac1c4ba640 1 --2- 172.21.15.72:0/1063819167 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fabf806c920 0x7fabf806ede0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fac10034000 tx=0x7fac10009030 comp rx=0 tx=0).stop 2024-04-15T15:15:45.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.886+0000 7fac1c4ba640 1 -- 172.21.15.72:0/1063819167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac1415ac20 msgr2=0x7fac1415b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:45.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.886+0000 7fac1c4ba640 1 --2- 172.21.15.72:0/1063819167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac1415ac20 0x7fac1415b0a0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fac140b7750 tx=0x7fac0c009f90 comp rx=0 tx=0).stop 2024-04-15T15:15:45.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.886+0000 7fac1c4ba640 1 -- 172.21.15.72:0/1063819167 shutdown_connections 2024-04-15T15:15:45.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.886+0000 7fac1c4ba640 1 --2- 172.21.15.72:0/1063819167 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fabf806c920 0x7fabf806ede0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.886+0000 7fac1c4ba640 1 --2- 172.21.15.72:0/1063819167 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac1415ac20 0x7fac1415b0a0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.886+0000 7fac1c4ba640 1 --2- 172.21.15.72:0/1063819167 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fac14152570 0x7fac14151f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.886+0000 7fac1c4ba640 1 --2- 172.21.15.72:0/1063819167 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac000039e0 0x7fac1415ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:45.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.887+0000 7fac1c4ba640 1 -- 172.21.15.72:0/1063819167 >> 172.21.15.72:0/1063819167 conn(0x7fac140d4750 msgr2=0x7fac1415db90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:45.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.887+0000 7fac1c4ba640 1 -- 172.21.15.72:0/1063819167 shutdown_connections 2024-04-15T15:15:45.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:45.887+0000 7fac1c4ba640 1 -- 172.21.15.72:0/1063819167 wait complete. 2024-04-15T15:15:45.899 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:15:46.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.003+0000 7fd355a47640 1 Processor -- start 2024-04-15T15:15:46.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.003+0000 7fd355a47640 1 -- start start 2024-04-15T15:15:46.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.004+0000 7fd355a47640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd350161ee0 0x7fd3500c95f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.004+0000 7fd355a47640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd3500d1810 0x7fd3500ce080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.004+0000 7fd355a47640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3500ccfe0 con 0x7fd3500b67c0 2024-04-15T15:15:46.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.004+0000 7fd355a47640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fd3501622c0 con 0x7fd3500d1810 2024-04-15T15:15:46.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.004+0000 7fd355a47640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fd3500cd2c0 con 0x7fd350161ee0 2024-04-15T15:15:46.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.004+0000 7fd34e7fc640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fd3500b67c0 0x7fd3500d1100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60088/0 (socket says 172.21.15.72:60088) 2024-04-15T15:15:46.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.004+0000 7fd34e7fc640 1 -- 172.21.15.72:0/2379091091 learned_addr learned my addr 172.21.15.72:0/2379091091 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:46.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.004+0000 7fd34effd640 1 --2- 172.21.15.72:0/2379091091 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd350161ee0 0x7fd3500c95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.004+0000 7fd34f7fe640 1 --2- 172.21.15.72:0/2379091091 >> v2:172.21.15.175:3300/0 conn(0x7fd3500d1810 0x7fd3500ce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.004+0000 7fd34dffb640 1 -- 172.21.15.72:0/2379091091 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 932567672 0 0) 0x7fd3500ccfe0 con 0x7fd3500b67c0 2024-04-15T15:15:46.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.004+0000 7fd34dffb640 1 -- 172.21.15.72:0/2379091091 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd338003520 con 0x7fd3500b67c0 2024-04-15T15:15:46.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.005+0000 7fd34effd640 1 -- 172.21.15.72:0/2379091091 >> v2:172.21.15.175:3300/0 conn(0x7fd3500d1810 msgr2=0x7fd3500ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:46.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.005+0000 7fd34effd640 1 --2- 172.21.15.72:0/2379091091 >> v2:172.21.15.175:3300/0 conn(0x7fd3500d1810 0x7fd3500ce080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.005+0000 7fd34effd640 1 -- 172.21.15.72:0/2379091091 >> v1:172.21.15.72:6789/0 conn(0x7fd3500b67c0 legacy=0x7fd3500d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.005+0000 7fd34effd640 1 -- 172.21.15.72:0/2379091091 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3500cd5a0 con 0x7fd350161ee0 2024-04-15T15:15:46.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.005+0000 7fd34f7fe640 1 --2- 172.21.15.72:0/2379091091 >> v2:172.21.15.175:3300/0 conn(0x7fd3500d1810 0x7fd3500ce080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:46.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.005+0000 7fd34effd640 1 --2- 172.21.15.72:0/2379091091 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd350161ee0 0x7fd3500c95f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd3400097a0 tx=0x7fd34002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da0421d92287857e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:46.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.006+0000 7fd34dffb640 1 -- 172.21.15.72:0/2379091091 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd340035020 con 0x7fd350161ee0 2024-04-15T15:15:46.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.006+0000 7fd34dffb640 1 -- 172.21.15.72:0/2379091091 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd340031830 con 0x7fd350161ee0 2024-04-15T15:15:46.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.006+0000 7fd34dffb640 1 -- 172.21.15.72:0/2379091091 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd340031be0 con 0x7fd350161ee0 2024-04-15T15:15:46.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.006+0000 7fd355a47640 1 -- 172.21.15.72:0/2379091091 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd350161ee0 msgr2=0x7fd3500c95f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.006+0000 7fd355a47640 1 --2- 172.21.15.72:0/2379091091 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd350161ee0 0x7fd3500c95f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd3400097a0 tx=0x7fd34002b8d0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.006+0000 7fd355a47640 1 -- 172.21.15.72:0/2379091091 shutdown_connections 2024-04-15T15:15:46.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.006+0000 7fd355a47640 1 --2- 172.21.15.72:0/2379091091 >> v2:172.21.15.175:3300/0 conn(0x7fd3500d1810 0x7fd3500ce080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.006+0000 7fd355a47640 1 --2- 172.21.15.72:0/2379091091 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd350161ee0 0x7fd3500c95f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.006+0000 7fd355a47640 1 -- 172.21.15.72:0/2379091091 >> 172.21.15.72:0/2379091091 conn(0x7fd3500b4010 msgr2=0x7fd3500b4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:46.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.006+0000 7fd355a47640 1 -- 172.21.15.72:0/2379091091 shutdown_connections 2024-04-15T15:15:46.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.006+0000 7fd355a47640 1 -- 172.21.15.72:0/2379091091 wait complete. 2024-04-15T15:15:46.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.007+0000 7fd355a47640 1 Processor -- start 2024-04-15T15:15:46.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.008+0000 7fd355a47640 1 -- start start 2024-04-15T15:15:46.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.008+0000 7fd355a47640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3500b67c0 0x7fd350158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.008+0000 7fd355a47640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd3500d1810 0x7fd3500b29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.008+0000 7fd355a47640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd350161ee0 0x7fd3500b2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.008+0000 7fd355a47640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd3500ccb70 con 0x7fd350161ee0 2024-04-15T15:15:46.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.008+0000 7fd355a47640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fd3500c9b30 con 0x7fd3500b67c0 2024-04-15T15:15:46.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.008+0000 7fd355a47640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fd3500cccf0 con 0x7fd3500d1810 2024-04-15T15:15:46.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.008+0000 7fd34f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd350161ee0 0x7fd3500b2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.008+0000 7fd34f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd350161ee0 0x7fd3500b2ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46232/0 (socket says 172.21.15.72:46232) 2024-04-15T15:15:46.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.008+0000 7fd34f7fe640 1 -- 172.21.15.72:0/1697231262 learned_addr learned my addr 172.21.15.72:0/1697231262 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:46.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.009+0000 7fd34effd640 1 --2- 172.21.15.72:0/1697231262 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3500b67c0 0x7fd350158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.009+0000 7fd34e7fc640 1 --2- 172.21.15.72:0/1697231262 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd3500d1810 0x7fd3500b29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.009+0000 7fd34f7fe640 1 -- 172.21.15.72:0/1697231262 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd3500d1810 msgr2=0x7fd3500b29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.009+0000 7fd34f7fe640 1 --2- 172.21.15.72:0/1697231262 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd3500d1810 0x7fd3500b29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.009+0000 7fd34f7fe640 1 -- 172.21.15.72:0/1697231262 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3500b67c0 msgr2=0x7fd350158ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.009+0000 7fd34f7fe640 1 --2- 172.21.15.72:0/1697231262 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3500b67c0 0x7fd350158ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.009+0000 7fd34f7fe640 1 -- 172.21.15.72:0/1697231262 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3500b16d0 con 0x7fd350161ee0 2024-04-15T15:15:46.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.009+0000 7fd34effd640 1 --2- 172.21.15.72:0/1697231262 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3500b67c0 0x7fd350158ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:46.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.009+0000 7fd34e7fc640 1 --2- 172.21.15.72:0/1697231262 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd3500d1810 0x7fd3500b29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:46.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.009+0000 7fd34f7fe640 1 --2- 172.21.15.72:0/1697231262 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd350161ee0 0x7fd3500b2ef0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fd34800efc0 tx=0x7fd348009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:46.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.009+0000 7fd337fff640 1 -- 172.21.15.72:0/1697231262 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd348018030 con 0x7fd350161ee0 2024-04-15T15:15:46.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.009+0000 7fd337fff640 1 -- 172.21.15.72:0/1697231262 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd34800cde0 con 0x7fd350161ee0 2024-04-15T15:15:46.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.009+0000 7fd355a47640 1 -- 172.21.15.72:0/1697231262 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3500c5d00 con 0x7fd350161ee0 2024-04-15T15:15:46.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.009+0000 7fd337fff640 1 -- 172.21.15.72:0/1697231262 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd348003bb0 con 0x7fd350161ee0 2024-04-15T15:15:46.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.009+0000 7fd355a47640 1 -- 172.21.15.72:0/1697231262 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3500c62c0 con 0x7fd350161ee0 2024-04-15T15:15:46.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.010+0000 7fd355a47640 1 -- 172.21.15.72:0/1697231262 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd350075c50 con 0x7fd350161ee0 2024-04-15T15:15:46.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.010+0000 7fd337fff640 1 -- 172.21.15.72:0/1697231262 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd348005d10 con 0x7fd350161ee0 2024-04-15T15:15:46.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.011+0000 7fd337fff640 1 --2- 172.21.15.72:0/1697231262 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd33c06c920 0x7fd33c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.011+0000 7fd337fff640 1 -- 172.21.15.72:0/1697231262 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fd34808c8f0 con 0x7fd350161ee0 2024-04-15T15:15:46.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.011+0000 7fd34effd640 1 --2- 172.21.15.72:0/1697231262 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd33c06c920 0x7fd33c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.012+0000 7fd34effd640 1 --2- 172.21.15.72:0/1697231262 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd33c06c920 0x7fd33c06ede0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd340036000 tx=0x7fd34003c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:46.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.015+0000 7fd337fff640 1 -- 172.21.15.72:0/1697231262 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd348022070 con 0x7fd350161ee0 2024-04-15T15:15:46.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.194+0000 7fd355a47640 1 -- 172.21.15.72:0/1697231262 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd3500d1e50 con 0x7fd350161ee0 2024-04-15T15:15:46.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.194+0000 7fd337fff640 1 -- 172.21.15.72:0/1697231262 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7fd348010040 con 0x7fd350161ee0 2024-04-15T15:15:46.197 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-04-15T15:15:46.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.196+0000 7fd355a47640 1 -- 172.21.15.72:0/1697231262 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd33c06c920 msgr2=0x7fd33c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.196+0000 7fd355a47640 1 --2- 172.21.15.72:0/1697231262 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd33c06c920 0x7fd33c06ede0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd340036000 tx=0x7fd34003c040 comp rx=0 tx=0).stop 2024-04-15T15:15:46.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.196+0000 7fd355a47640 1 -- 172.21.15.72:0/1697231262 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd350161ee0 msgr2=0x7fd3500b2ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.196+0000 7fd355a47640 1 --2- 172.21.15.72:0/1697231262 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd350161ee0 0x7fd3500b2ef0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fd34800efc0 tx=0x7fd348009f90 comp rx=0 tx=0).stop 2024-04-15T15:15:46.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.196+0000 7fd355a47640 1 -- 172.21.15.72:0/1697231262 shutdown_connections 2024-04-15T15:15:46.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.196+0000 7fd355a47640 1 --2- 172.21.15.72:0/1697231262 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd33c06c920 0x7fd33c06ede0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.196+0000 7fd355a47640 1 --2- 172.21.15.72:0/1697231262 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd350161ee0 0x7fd3500b2ef0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.196+0000 7fd355a47640 1 --2- 172.21.15.72:0/1697231262 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd3500d1810 0x7fd3500b29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.196+0000 7fd355a47640 1 --2- 172.21.15.72:0/1697231262 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd3500b67c0 0x7fd350158ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.196+0000 7fd355a47640 1 -- 172.21.15.72:0/1697231262 >> 172.21.15.72:0/1697231262 conn(0x7fd3500b4010 msgr2=0x7fd35015f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:46.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.196+0000 7fd355a47640 1 -- 172.21.15.72:0/1697231262 shutdown_connections 2024-04-15T15:15:46.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.196+0000 7fd355a47640 1 -- 172.21.15.72:0/1697231262 wait complete. 2024-04-15T15:15:46.208 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:15:46.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.314+0000 7f75199db640 1 Processor -- start 2024-04-15T15:15:46.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.314+0000 7f75199db640 1 -- start start 2024-04-15T15:15:46.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.314+0000 7f75199db640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f751415e610 0x7f75140d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.314+0000 7f75199db640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75140d2320 con 0x7f751406c400 2024-04-15T15:15:46.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.314+0000 7f75199db640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75140d25c0 con 0x7f751415e9f0 2024-04-15T15:15:46.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.314+0000 7f75199db640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f75140d31d0 con 0x7f751415e610 2024-04-15T15:15:46.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.315+0000 7f75137fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f751406c400 0x7f75140b7940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60094/0 (socket says 172.21.15.72:60094) 2024-04-15T15:15:46.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.315+0000 7f75137fe640 1 -- 172.21.15.72:0/3260389133 learned_addr learned my addr 172.21.15.72:0/3260389133 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:46.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.315+0000 7f7512ffd640 1 --2- 172.21.15.72:0/3260389133 >> v2:172.21.15.175:3300/0 conn(0x7f751415e610 0x7f75140d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.315+0000 7f7511ffb640 1 -- 172.21.15.72:0/3260389133 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3591766679 0 0) 0x7f75140d2320 con 0x7f751406c400 2024-04-15T15:15:46.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.315+0000 7f7511ffb640 1 -- 172.21.15.72:0/3260389133 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f74fc003520 con 0x7f751406c400 2024-04-15T15:15:46.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.315+0000 7f7511ffb640 1 -- 172.21.15.72:0/3260389133 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 717751762 0 0) 0x7f74fc003520 con 0x7f751406c400 2024-04-15T15:15:46.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.315+0000 7f7512ffd640 1 -- 172.21.15.72:0/3260389133 >> v1:172.21.15.175:6789/0 conn(0x7f751415e9f0 legacy=0x7f75140d3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.315+0000 7f7512ffd640 1 -- 172.21.15.72:0/3260389133 >> v1:172.21.15.72:6789/0 conn(0x7f751406c400 legacy=0x7f75140b7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.315+0000 7f7512ffd640 1 -- 172.21.15.72:0/3260389133 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75140d28a0 con 0x7f751415e610 2024-04-15T15:15:46.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.316+0000 7f7512ffd640 1 --2- 172.21.15.72:0/3260389133 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f751415e610 0x7f75140d0690 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f75040097a0 tx=0x7f750402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a6a9341cc2acad5 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:46.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.316+0000 7f7511ffb640 1 -- 172.21.15.72:0/3260389133 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f750c001a10 con 0x7f751415e610 2024-04-15T15:15:46.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.316+0000 7f7511ffb640 1 -- 172.21.15.72:0/3260389133 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7504037070 con 0x7f751415e610 2024-04-15T15:15:46.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.316+0000 7f7511ffb640 1 -- 172.21.15.72:0/3260389133 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7504031790 con 0x7f751415e610 2024-04-15T15:15:46.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.317+0000 7f75199db640 1 -- 172.21.15.72:0/3260389133 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f751415e610 msgr2=0x7f75140d0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.317+0000 7f75199db640 1 --2- 172.21.15.72:0/3260389133 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f751415e610 0x7f75140d0690 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f75040097a0 tx=0x7f750402b8b0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.317+0000 7f75199db640 1 -- 172.21.15.72:0/3260389133 shutdown_connections 2024-04-15T15:15:46.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.317+0000 7f75199db640 1 --2- 172.21.15.72:0/3260389133 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f751415e610 0x7f75140d0690 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.317+0000 7f75199db640 1 -- 172.21.15.72:0/3260389133 >> 172.21.15.72:0/3260389133 conn(0x7f75140d4750 msgr2=0x7f75140d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:46.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.317+0000 7f75199db640 1 -- 172.21.15.72:0/3260389133 shutdown_connections 2024-04-15T15:15:46.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.317+0000 7f75199db640 1 -- 172.21.15.72:0/3260389133 wait complete. 2024-04-15T15:15:46.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.318+0000 7f75199db640 1 Processor -- start 2024-04-15T15:15:46.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.318+0000 7f75199db640 1 -- start start 2024-04-15T15:15:46.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.319+0000 7f75199db640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f751406c400 0x7f75140ce340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.319+0000 7f75199db640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f751415e610 0x7f7514151df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.319+0000 7f75199db640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f751415e9f0 0x7f751414c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.319+0000 7f75199db640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f751406d130 con 0x7f751406c400 2024-04-15T15:15:46.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.319+0000 7f75199db640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f75140d2e60 con 0x7f751415e610 2024-04-15T15:15:46.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.319+0000 7f75199db640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f75140d0bd0 con 0x7f751415e9f0 2024-04-15T15:15:46.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.319+0000 7f7512ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f751406c400 0x7f75140ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.319+0000 7f7512ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f751406c400 0x7f75140ce340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46246/0 (socket says 172.21.15.72:46246) 2024-04-15T15:15:46.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.319+0000 7f7512ffd640 1 -- 172.21.15.72:0/2120790878 learned_addr learned my addr 172.21.15.72:0/2120790878 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:46.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.319+0000 7f75137fe640 1 --2- 172.21.15.72:0/2120790878 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f751415e9f0 0x7f751414c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.319+0000 7f75127fc640 1 --2- 172.21.15.72:0/2120790878 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f751415e610 0x7f7514151df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.319+0000 7f7512ffd640 1 -- 172.21.15.72:0/2120790878 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f751415e9f0 msgr2=0x7f751414c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.319+0000 7f7512ffd640 1 --2- 172.21.15.72:0/2120790878 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f751415e9f0 0x7f751414c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.319+0000 7f7512ffd640 1 -- 172.21.15.72:0/2120790878 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f751415e610 msgr2=0x7f7514151df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.319+0000 7f7512ffd640 1 --2- 172.21.15.72:0/2120790878 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f751415e610 0x7f7514151df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.319+0000 7f7512ffd640 1 -- 172.21.15.72:0/2120790878 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f751415f370 con 0x7f751406c400 2024-04-15T15:15:46.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.319+0000 7f75127fc640 1 --2- 172.21.15.72:0/2120790878 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f751415e610 0x7f7514151df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:46.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.319+0000 7f7512ffd640 1 --2- 172.21.15.72:0/2120790878 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f751406c400 0x7f75140ce340 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f7504000c00 tx=0x7f75040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:46.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.320+0000 7f74fbfff640 1 -- 172.21.15.72:0/2120790878 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7504032620 con 0x7f751406c400 2024-04-15T15:15:46.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.320+0000 7f74fbfff640 1 -- 172.21.15.72:0/2120790878 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75040327c0 con 0x7f751406c400 2024-04-15T15:15:46.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.320+0000 7f75199db640 1 -- 172.21.15.72:0/2120790878 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f751415f650 con 0x7f751406c400 2024-04-15T15:15:46.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.320+0000 7f74fbfff640 1 -- 172.21.15.72:0/2120790878 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7504032bb0 con 0x7f751406c400 2024-04-15T15:15:46.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.320+0000 7f75199db640 1 -- 172.21.15.72:0/2120790878 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f751415fc10 con 0x7f751406c400 2024-04-15T15:15:46.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.321+0000 7f75199db640 1 -- 172.21.15.72:0/2120790878 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7514075c50 con 0x7f751406c400 2024-04-15T15:15:46.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.321+0000 7f74fbfff640 1 -- 172.21.15.72:0/2120790878 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f750404c060 con 0x7f751406c400 2024-04-15T15:15:46.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.321+0000 7f74fbfff640 1 --2- 172.21.15.72:0/2120790878 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f750006c8f0 0x7f750006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.322+0000 7f74fbfff640 1 -- 172.21.15.72:0/2120790878 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f75040ad950 con 0x7f751406c400 2024-04-15T15:15:46.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.322+0000 7f75127fc640 1 --2- 172.21.15.72:0/2120790878 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f750006c8f0 0x7f750006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.322+0000 7f75127fc640 1 --2- 172.21.15.72:0/2120790878 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f750006c8f0 0x7f750006edb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7508007fd0 tx=0x7f7508009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:46.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.326+0000 7f74fbfff640 1 -- 172.21.15.72:0/2120790878 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7504040070 con 0x7f751406c400 2024-04-15T15:15:46.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.501+0000 7f75199db640 1 -- 172.21.15.72:0/2120790878 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f75140b6240 con 0x7f751406c400 2024-04-15T15:15:46.503 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.501+0000 7f74fbfff640 1 -- 172.21.15.72:0/2120790878 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f750407bab0 con 0x7f751406c400 2024-04-15T15:15:46.503 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:46.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.503+0000 7f75199db640 1 -- 172.21.15.72:0/2120790878 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f750006c8f0 msgr2=0x7f750006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.503+0000 7f75199db640 1 --2- 172.21.15.72:0/2120790878 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f750006c8f0 0x7f750006edb0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7508007fd0 tx=0x7f7508009040 comp rx=0 tx=0).stop 2024-04-15T15:15:46.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.503+0000 7f75199db640 1 -- 172.21.15.72:0/2120790878 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f751406c400 msgr2=0x7f75140ce340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.503+0000 7f75199db640 1 --2- 172.21.15.72:0/2120790878 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f751406c400 0x7f75140ce340 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f7504000c00 tx=0x7f75040046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.503+0000 7f75199db640 1 -- 172.21.15.72:0/2120790878 shutdown_connections 2024-04-15T15:15:46.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.503+0000 7f75199db640 1 --2- 172.21.15.72:0/2120790878 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f750006c8f0 0x7f750006edb0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.503+0000 7f75199db640 1 --2- 172.21.15.72:0/2120790878 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f751415e9f0 0x7f751414c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.503+0000 7f75199db640 1 --2- 172.21.15.72:0/2120790878 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f751415e610 0x7f7514151df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.503+0000 7f75199db640 1 --2- 172.21.15.72:0/2120790878 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f751406c400 0x7f75140ce340 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.503+0000 7f75199db640 1 -- 172.21.15.72:0/2120790878 >> 172.21.15.72:0/2120790878 conn(0x7f75140d4750 msgr2=0x7f75140b5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:46.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.503+0000 7f75199db640 1 -- 172.21.15.72:0/2120790878 shutdown_connections 2024-04-15T15:15:46.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.503+0000 7f75199db640 1 -- 172.21.15.72:0/2120790878 wait complete. 2024-04-15T15:15:46.516 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:15:46.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.619+0000 7f01e54b2640 1 Processor -- start 2024-04-15T15:15:46.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.619+0000 7f01e54b2640 1 -- start start 2024-04-15T15:15:46.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.619+0000 7f01e54b2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f01e015e9f0 0x7f01e00d3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.619+0000 7f01e54b2640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01e00d2320 con 0x7f01e015e610 2024-04-15T15:15:46.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.619+0000 7f01e54b2640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01e00d25c0 con 0x7f01e006c400 2024-04-15T15:15:46.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.619+0000 7f01e54b2640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f01e00d31d0 con 0x7f01e015e9f0 2024-04-15T15:15:46.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.620+0000 7f01d67fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f01e015e9f0 0x7f01e00d3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.620+0000 7f01deffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f01e015e610 0x7f01e00d0690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60102/0 (socket says 172.21.15.72:60102) 2024-04-15T15:15:46.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.620+0000 7f01deffd640 1 -- 172.21.15.72:0/985012173 learned_addr learned my addr 172.21.15.72:0/985012173 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:46.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.620+0000 7f01d67fc640 1 -- 172.21.15.72:0/985012173 >> v1:172.21.15.175:6789/0 conn(0x7f01e006c400 legacy=0x7f01e00b7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:46.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.620+0000 7f01d67fc640 1 -- 172.21.15.72:0/985012173 >> v1:172.21.15.72:6789/0 conn(0x7f01e015e610 legacy=0x7f01e00d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.620+0000 7f01d67fc640 1 -- 172.21.15.72:0/985012173 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01e00d28a0 con 0x7f01e015e9f0 2024-04-15T15:15:46.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.621+0000 7f01d67fc640 1 --2- 172.21.15.72:0/985012173 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01e015e9f0 0x7f01e00d3660 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f01c80097a0 tx=0x7f01c802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7a977072ee8bcc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:46.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.621+0000 7f01de7fc640 1 -- 172.21.15.72:0/985012173 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c8034070 con 0x7f01e015e9f0 2024-04-15T15:15:46.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.621+0000 7f01de7fc640 1 -- 172.21.15.72:0/985012173 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01c8030610 con 0x7f01e015e9f0 2024-04-15T15:15:46.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.621+0000 7f01de7fc640 1 -- 172.21.15.72:0/985012173 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c80309e0 con 0x7f01e015e9f0 2024-04-15T15:15:46.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.621+0000 7f01e54b2640 1 -- 172.21.15.72:0/985012173 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01e015e9f0 msgr2=0x7f01e00d3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.621+0000 7f01e54b2640 1 --2- 172.21.15.72:0/985012173 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01e015e9f0 0x7f01e00d3660 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f01c80097a0 tx=0x7f01c802b960 comp rx=0 tx=0).stop 2024-04-15T15:15:46.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.621+0000 7f01e54b2640 1 -- 172.21.15.72:0/985012173 shutdown_connections 2024-04-15T15:15:46.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.621+0000 7f01e54b2640 1 --2- 172.21.15.72:0/985012173 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01e015e9f0 0x7f01e00d3660 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.621+0000 7f01e54b2640 1 -- 172.21.15.72:0/985012173 >> 172.21.15.72:0/985012173 conn(0x7f01e00d4750 msgr2=0x7f01e00d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:46.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.621+0000 7f01e54b2640 1 -- 172.21.15.72:0/985012173 shutdown_connections 2024-04-15T15:15:46.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.621+0000 7f01e54b2640 1 -- 172.21.15.72:0/985012173 wait complete. 2024-04-15T15:15:46.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.623+0000 7f01e54b2640 1 Processor -- start 2024-04-15T15:15:46.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.623+0000 7f01e54b2640 1 -- start start 2024-04-15T15:15:46.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.623+0000 7f01e54b2640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f01e006c400 0x7f01e00cff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.623+0000 7f01e54b2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01e015e610 0x7f01e01567e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.623+0000 7f01e54b2640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f01e015e9f0 0x7f01e0156d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.623+0000 7f01e54b2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f01e006d130 con 0x7f01e015e610 2024-04-15T15:15:46.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.623+0000 7f01e54b2640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f01e00d2e60 con 0x7f01e006c400 2024-04-15T15:15:46.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.623+0000 7f01e54b2640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f01e00d3ba0 con 0x7f01e015e9f0 2024-04-15T15:15:46.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.623+0000 7f01d67fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01e015e610 0x7f01e01567e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.624+0000 7f01d67fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01e015e610 0x7f01e01567e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46262/0 (socket says 172.21.15.72:46262) 2024-04-15T15:15:46.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.624+0000 7f01d67fc640 1 -- 172.21.15.72:0/1792041104 learned_addr learned my addr 172.21.15.72:0/1792041104 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:46.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.624+0000 7f01d67fc640 1 -- 172.21.15.72:0/1792041104 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f01e015e9f0 msgr2=0x7f01e0156d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.624+0000 7f01d67fc640 1 --2- 172.21.15.72:0/1792041104 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f01e015e9f0 0x7f01e0156d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.624+0000 7f01d67fc640 1 -- 172.21.15.72:0/1792041104 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f01e006c400 msgr2=0x7f01e00cff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.624+0000 7f01d67fc640 1 --2- 172.21.15.72:0/1792041104 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f01e006c400 0x7f01e00cff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.624+0000 7f01d67fc640 1 -- 172.21.15.72:0/1792041104 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01e0157260 con 0x7f01e015e610 2024-04-15T15:15:46.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.624+0000 7f01d67fc640 1 --2- 172.21.15.72:0/1792041104 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01e015e610 0x7f01e01567e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f01c802bf70 tx=0x7f01c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:46.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.624+0000 7f01d7fff640 1 -- 172.21.15.72:0/1792041104 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c8034040 con 0x7f01e015e610 2024-04-15T15:15:46.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.624+0000 7f01d7fff640 1 -- 172.21.15.72:0/1792041104 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01c803aa20 con 0x7f01e015e610 2024-04-15T15:15:46.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.624+0000 7f01e54b2640 1 -- 172.21.15.72:0/1792041104 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01e01574f0 con 0x7f01e015e610 2024-04-15T15:15:46.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.624+0000 7f01d7fff640 1 -- 172.21.15.72:0/1792041104 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c803ae10 con 0x7f01e015e610 2024-04-15T15:15:46.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.624+0000 7f01e54b2640 1 -- 172.21.15.72:0/1792041104 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01e0157ab0 con 0x7f01e015e610 2024-04-15T15:15:46.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.625+0000 7f01e54b2640 1 -- 172.21.15.72:0/1792041104 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01e0158280 con 0x7f01e015e610 2024-04-15T15:15:46.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.625+0000 7f01d7fff640 1 -- 172.21.15.72:0/1792041104 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f01c8037070 con 0x7f01e015e610 2024-04-15T15:15:46.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.626+0000 7f01d7fff640 1 --2- 172.21.15.72:0/1792041104 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f01c006c8f0 0x7f01c006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.626+0000 7f01d7fff640 1 -- 172.21.15.72:0/1792041104 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f01c80ae020 con 0x7f01e015e610 2024-04-15T15:15:46.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.626+0000 7f01deffd640 1 --2- 172.21.15.72:0/1792041104 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f01c006c8f0 0x7f01c006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.626+0000 7f01deffd640 1 --2- 172.21.15.72:0/1792041104 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f01c006c8f0 0x7f01c006edb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f01d0005880 tx=0x7f01d0007a20 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:46.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.630+0000 7f01d7fff640 1 -- 172.21.15.72:0/1792041104 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f01c807c0e0 con 0x7f01e015e610 2024-04-15T15:15:46.818 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.817+0000 7f01e54b2640 1 -- 172.21.15.72:0/1792041104 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f01e00cf740 con 0x7f01e015e610 2024-04-15T15:15:46.819 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.817+0000 7f01d7fff640 1 -- 172.21.15.72:0/1792041104 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7f01c807b9e0 con 0x7f01e015e610 2024-04-15T15:15:46.819 INFO:teuthology.orchestra.run.smithi072.stderr:listed 0 entries 2024-04-15T15:15:46.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.819+0000 7f01e54b2640 1 -- 172.21.15.72:0/1792041104 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f01c006c8f0 msgr2=0x7f01c006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.819+0000 7f01e54b2640 1 --2- 172.21.15.72:0/1792041104 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f01c006c8f0 0x7f01c006edb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f01d0005880 tx=0x7f01d0007a20 comp rx=0 tx=0).stop 2024-04-15T15:15:46.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.819+0000 7f01e54b2640 1 -- 172.21.15.72:0/1792041104 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01e015e610 msgr2=0x7f01e01567e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.819+0000 7f01e54b2640 1 --2- 172.21.15.72:0/1792041104 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01e015e610 0x7f01e01567e0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f01c802bf70 tx=0x7f01c80046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.819+0000 7f01e54b2640 1 -- 172.21.15.72:0/1792041104 shutdown_connections 2024-04-15T15:15:46.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.819+0000 7f01e54b2640 1 --2- 172.21.15.72:0/1792041104 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f01c006c8f0 0x7f01c006edb0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.819+0000 7f01e54b2640 1 --2- 172.21.15.72:0/1792041104 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f01e015e9f0 0x7f01e0156d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.819+0000 7f01e54b2640 1 --2- 172.21.15.72:0/1792041104 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01e015e610 0x7f01e01567e0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.819+0000 7f01e54b2640 1 --2- 172.21.15.72:0/1792041104 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f01e006c400 0x7f01e00cff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.819+0000 7f01e54b2640 1 -- 172.21.15.72:0/1792041104 >> 172.21.15.72:0/1792041104 conn(0x7f01e00d4750 msgr2=0x7f01e00b8b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:46.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.819+0000 7f01e54b2640 1 -- 172.21.15.72:0/1792041104 shutdown_connections 2024-04-15T15:15:46.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.819+0000 7f01e54b2640 1 -- 172.21.15.72:0/1792041104 wait complete. 2024-04-15T15:15:46.832 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:15:46.832 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:15:46.848 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:46.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.991+0000 7ff99944a640 1 Processor -- start 2024-04-15T15:15:46.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.991+0000 7ff99944a640 1 -- start start 2024-04-15T15:15:46.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.991+0000 7ff99944a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff9940d12b0 0x7ff9940d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.992+0000 7ff99944a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff994162220 con 0x7ff994161e40 2024-04-15T15:15:46.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.992+0000 7ff992ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ff994161e40 0x7ff9940b37f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60104/0 (socket says 172.21.15.72:60104) 2024-04-15T15:15:46.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.992+0000 7ff992ffd640 1 -- 172.21.15.72:0/1509719443 learned_addr learned my addr 172.21.15.72:0/1509719443 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:46.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.992+0000 7ff9927fc640 1 --2- 172.21.15.72:0/1509719443 >> v2:172.21.15.72:3300/0 conn(0x7ff9940d12b0 0x7ff9940d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.992+0000 7ff99944a640 1 -- 172.21.15.72:0/1509719443 --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9940d1820 con 0x7ff9940d2ea0 2024-04-15T15:15:46.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.992+0000 7ff99944a640 1 -- 172.21.15.72:0/1509719443 --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7ff9940b3f00 con 0x7ff9940d12b0 2024-04-15T15:15:46.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.992+0000 7ff9927fc640 1 -- 172.21.15.72:0/1509719443 >> v1:172.21.15.175:6789/0 conn(0x7ff9940d2ea0 legacy=0x7ff9940d45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:46.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.992+0000 7ff9927fc640 1 -- 172.21.15.72:0/1509719443 >> v1:172.21.15.72:6789/0 conn(0x7ff994161e40 legacy=0x7ff9940b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.992+0000 7ff9927fc640 1 -- 172.21.15.72:0/1509719443 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9940d1b00 con 0x7ff9940d12b0 2024-04-15T15:15:46.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.992+0000 7ff991ffb640 1 -- 172.21.15.72:0/1509719443 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3886450296 0 0) 0x7ff994162220 con 0x7ff994161e40 2024-04-15T15:15:46.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.993+0000 7ff9927fc640 1 --2- 172.21.15.72:0/1509719443 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff9940d12b0 0x7ff9940d2960 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff97c0097d0 tx=0x7ff97c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeda80efdb26d5ec server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:46.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.993+0000 7ff991ffb640 1 -- 172.21.15.72:0/1509719443 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff97c0315a0 con 0x7ff9940d12b0 2024-04-15T15:15:46.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.993+0000 7ff991ffb640 1 -- 172.21.15.72:0/1509719443 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff97c037070 con 0x7ff9940d12b0 2024-04-15T15:15:46.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.993+0000 7ff991ffb640 1 -- 172.21.15.72:0/1509719443 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff97c031960 con 0x7ff9940d12b0 2024-04-15T15:15:46.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.993+0000 7ff99944a640 1 -- 172.21.15.72:0/1509719443 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff9940d12b0 msgr2=0x7ff9940d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.993+0000 7ff99944a640 1 --2- 172.21.15.72:0/1509719443 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff9940d12b0 0x7ff9940d2960 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff97c0097d0 tx=0x7ff97c02b840 comp rx=0 tx=0).stop 2024-04-15T15:15:46.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.993+0000 7ff99944a640 1 -- 172.21.15.72:0/1509719443 shutdown_connections 2024-04-15T15:15:46.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.993+0000 7ff99944a640 1 --2- 172.21.15.72:0/1509719443 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff9940d12b0 0x7ff9940d2960 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.993+0000 7ff99944a640 1 -- 172.21.15.72:0/1509719443 >> 172.21.15.72:0/1509719443 conn(0x7ff99409d970 msgr2=0x7ff9940b95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:46.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.994+0000 7ff99944a640 1 -- 172.21.15.72:0/1509719443 shutdown_connections 2024-04-15T15:15:46.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.994+0000 7ff99944a640 1 -- 172.21.15.72:0/1509719443 wait complete. 2024-04-15T15:15:46.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.995+0000 7ff99944a640 1 Processor -- start 2024-04-15T15:15:46.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.995+0000 7ff99944a640 1 -- start start 2024-04-15T15:15:46.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.995+0000 7ff99944a640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff9940d12b0 0x7ff9940cae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.995+0000 7ff99944a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff994161e40 0x7ff9940cd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.995+0000 7ff99944a640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff9941f83e0 0x7ff9941f7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.996+0000 7ff99944a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ff9940d2440 con 0x7ff994161e40 2024-04-15T15:15:46.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.996+0000 7ff99944a640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7ff9940d21f0 con 0x7ff9941f83e0 2024-04-15T15:15:46.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.996+0000 7ff99944a640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7ff9940d4cc0 con 0x7ff9940d12b0 2024-04-15T15:15:46.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.996+0000 7ff9927fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff994161e40 0x7ff9940cd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.996+0000 7ff9927fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff994161e40 0x7ff9940cd4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46274/0 (socket says 172.21.15.72:46274) 2024-04-15T15:15:46.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.996+0000 7ff9927fc640 1 -- 172.21.15.72:0/1260307694 learned_addr learned my addr 172.21.15.72:0/1260307694 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:46.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.996+0000 7ff992ffd640 1 --2- 172.21.15.72:0/1260307694 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff9940d12b0 0x7ff9940cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.996+0000 7ff9927fc640 1 -- 172.21.15.72:0/1260307694 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff9940d12b0 msgr2=0x7ff9940cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.996+0000 7ff9927fc640 1 --2- 172.21.15.72:0/1260307694 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff9940d12b0 0x7ff9940cae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.996+0000 7ff9927fc640 1 -- 172.21.15.72:0/1260307694 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff9941f83e0 msgr2=0x7ff9941f7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.996+0000 7ff9937fe640 1 --2- 172.21.15.72:0/1260307694 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff9941f83e0 0x7ff9941f7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.996+0000 7ff9927fc640 1 --2- 172.21.15.72:0/1260307694 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff9941f83e0 0x7ff9941f7820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.996+0000 7ff9927fc640 1 -- 172.21.15.72:0/1260307694 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9940cde40 con 0x7ff994161e40 2024-04-15T15:15:47.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.996+0000 7ff9927fc640 1 --2- 172.21.15.72:0/1260307694 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff994161e40 0x7ff9940cd4f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff97c009720 tx=0x7ff97c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:47.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.996+0000 7ff98b7fe640 1 -- 172.21.15.72:0/1260307694 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff97c032840 con 0x7ff994161e40 2024-04-15T15:15:47.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.996+0000 7ff98b7fe640 1 -- 172.21.15.72:0/1260307694 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff97c0329e0 con 0x7ff994161e40 2024-04-15T15:15:47.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.996+0000 7ff98b7fe640 1 -- 172.21.15.72:0/1260307694 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff97c032dd0 con 0x7ff994161e40 2024-04-15T15:15:47.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.996+0000 7ff99944a640 1 -- 172.21.15.72:0/1260307694 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9940ce120 con 0x7ff994161e40 2024-04-15T15:15:47.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.996+0000 7ff99944a640 1 -- 172.21.15.72:0/1260307694 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9940ce6e0 con 0x7ff994161e40 2024-04-15T15:15:47.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.997+0000 7ff99944a640 1 -- 172.21.15.72:0/1260307694 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9940689e0 con 0x7ff994161e40 2024-04-15T15:15:47.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.998+0000 7ff98b7fe640 1 -- 172.21.15.72:0/1260307694 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff97c03b3f0 con 0x7ff994161e40 2024-04-15T15:15:47.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.998+0000 7ff98b7fe640 1 --2- 172.21.15.72:0/1260307694 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff97406c920 0x7ff97406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.998+0000 7ff98b7fe640 1 -- 172.21.15.72:0/1260307694 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7ff97c02f030 con 0x7ff994161e40 2024-04-15T15:15:47.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.998+0000 7ff992ffd640 1 --2- 172.21.15.72:0/1260307694 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff97406c920 0x7ff97406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:46.999+0000 7ff992ffd640 1 --2- 172.21.15.72:0/1260307694 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff97406c920 0x7ff97406ede0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff9840012e0 tx=0x7ff984009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:47.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.003+0000 7ff98b7fe640 1 -- 172.21.15.72:0/1260307694 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff97c040070 con 0x7ff994161e40 2024-04-15T15:15:47.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.181+0000 7ff99944a640 1 -- 172.21.15.72:0/1260307694 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff9940cb3c0 con 0x7ff994161e40 2024-04-15T15:15:47.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.182+0000 7ff98b7fe640 1 -- 172.21.15.72:0/1260307694 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18903 (secure 0 0 0) 0x7ff97c07bed0 con 0x7ff994161e40 2024-04-15T15:15:47.184 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:47.184 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":26,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:15:32.421728+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:14:49.438366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:14:50.368379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:14:54.081157+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":1}]},{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:47.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.184+0000 7ff99944a640 1 -- 172.21.15.72:0/1260307694 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff97406c920 msgr2=0x7ff97406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.184+0000 7ff99944a640 1 --2- 172.21.15.72:0/1260307694 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff97406c920 0x7ff97406ede0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff9840012e0 tx=0x7ff984009040 comp rx=0 tx=0).stop 2024-04-15T15:15:47.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.184+0000 7ff99944a640 1 -- 172.21.15.72:0/1260307694 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff994161e40 msgr2=0x7ff9940cd4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.184+0000 7ff99944a640 1 --2- 172.21.15.72:0/1260307694 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff994161e40 0x7ff9940cd4f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff97c009720 tx=0x7ff97c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.184+0000 7ff99944a640 1 -- 172.21.15.72:0/1260307694 shutdown_connections 2024-04-15T15:15:47.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.184+0000 7ff99944a640 1 --2- 172.21.15.72:0/1260307694 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff97406c920 0x7ff97406ede0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.184+0000 7ff99944a640 1 --2- 172.21.15.72:0/1260307694 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff9941f83e0 0x7ff9941f7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.184+0000 7ff99944a640 1 --2- 172.21.15.72:0/1260307694 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff994161e40 0x7ff9940cd4f0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.184+0000 7ff99944a640 1 --2- 172.21.15.72:0/1260307694 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff9940d12b0 0x7ff9940cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.184+0000 7ff99944a640 1 -- 172.21.15.72:0/1260307694 >> 172.21.15.72:0/1260307694 conn(0x7ff99409d970 msgr2=0x7ff9940b7240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:47.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.184+0000 7ff99944a640 1 -- 172.21.15.72:0/1260307694 shutdown_connections 2024-04-15T15:15:47.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.184+0000 7ff99944a640 1 -- 172.21.15.72:0/1260307694 wait complete. 2024-04-15T15:15:47.197 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:39.359102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:14:49.438366+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:14:50.368379+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:14:54.081157+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:15:47.197 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:47.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.302+0000 7f1141ab2640 1 Processor -- start 2024-04-15T15:15:47.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.302+0000 7f1141ab2640 1 -- start start 2024-04-15T15:15:47.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.303+0000 7f1141ab2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f113c15e610 0x7f113c0d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.303+0000 7f1141ab2640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f113c06c400 0x7f113c06c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.303+0000 7f1141ab2640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f113c0d2320 con 0x7f113c15e9f0 2024-04-15T15:15:47.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.303+0000 7f1141ab2640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f113c0d31d0 con 0x7f113c15e610 2024-04-15T15:15:47.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.303+0000 7f1141ab2640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f113c0d25c0 con 0x7f113c06c400 2024-04-15T15:15:47.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.303+0000 7f113b7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f113c15e610 0x7f113c0d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.303+0000 7f1132ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f113c15e9f0 0x7f113c0d3660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60108/0 (socket says 172.21.15.72:60108) 2024-04-15T15:15:47.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.303+0000 7f1132ffd640 1 -- 172.21.15.72:0/3883433679 learned_addr learned my addr 172.21.15.72:0/3883433679 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:47.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.304+0000 7f113bfff640 1 --2- 172.21.15.72:0/3883433679 >> v2:172.21.15.175:3300/0 conn(0x7f113c06c400 0x7f113c06c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.304+0000 7f113b7fe640 1 -- 172.21.15.72:0/3883433679 >> v2:172.21.15.175:3300/0 conn(0x7f113c06c400 msgr2=0x7f113c06c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:47.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.304+0000 7f113b7fe640 1 --2- 172.21.15.72:0/3883433679 >> v2:172.21.15.175:3300/0 conn(0x7f113c06c400 0x7f113c06c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.304+0000 7f113b7fe640 1 -- 172.21.15.72:0/3883433679 >> v1:172.21.15.72:6789/0 conn(0x7f113c15e9f0 legacy=0x7f113c0d3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.304+0000 7f113b7fe640 1 -- 172.21.15.72:0/3883433679 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f113c0d28a0 con 0x7f113c15e610 2024-04-15T15:15:47.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.304+0000 7f113affd640 1 -- 172.21.15.72:0/3883433679 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4243048386 0 0) 0x7f113c0d2320 con 0x7f113c15e9f0 2024-04-15T15:15:47.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.304+0000 7f113b7fe640 1 --2- 172.21.15.72:0/3883433679 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f113c15e610 0x7f113c0d0690 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f113c0b7890 tx=0x7f112c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95b30428e0e293f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:47.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.305+0000 7f113affd640 1 -- 172.21.15.72:0/3883433679 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112c034070 con 0x7f113c15e610 2024-04-15T15:15:47.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.305+0000 7f113affd640 1 -- 172.21.15.72:0/3883433679 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f112c030610 con 0x7f113c15e610 2024-04-15T15:15:47.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.305+0000 7f113affd640 1 -- 172.21.15.72:0/3883433679 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112c0309e0 con 0x7f113c15e610 2024-04-15T15:15:47.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.305+0000 7f1141ab2640 1 -- 172.21.15.72:0/3883433679 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f113c15e610 msgr2=0x7f113c0d0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.305+0000 7f1141ab2640 1 --2- 172.21.15.72:0/3883433679 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f113c15e610 0x7f113c0d0690 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f113c0b7890 tx=0x7f112c02b960 comp rx=0 tx=0).stop 2024-04-15T15:15:47.308 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.305+0000 7f1141ab2640 1 -- 172.21.15.72:0/3883433679 shutdown_connections 2024-04-15T15:15:47.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.305+0000 7f1141ab2640 1 --2- 172.21.15.72:0/3883433679 >> v2:172.21.15.175:3300/0 conn(0x7f113c06c400 0x7f113c06c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.305+0000 7f1141ab2640 1 --2- 172.21.15.72:0/3883433679 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f113c15e610 0x7f113c0d0690 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.305+0000 7f1141ab2640 1 -- 172.21.15.72:0/3883433679 >> 172.21.15.72:0/3883433679 conn(0x7f113c0d4750 msgr2=0x7f113c0d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:47.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.305+0000 7f1141ab2640 1 -- 172.21.15.72:0/3883433679 shutdown_connections 2024-04-15T15:15:47.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.305+0000 7f1141ab2640 1 -- 172.21.15.72:0/3883433679 wait complete. 2024-04-15T15:15:47.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.306+0000 7f1141ab2640 1 Processor -- start 2024-04-15T15:15:47.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.307+0000 7f1141ab2640 1 -- start start 2024-04-15T15:15:47.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.307+0000 7f1141ab2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f113c06c400 0x7f113c1618b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.307+0000 7f1141ab2640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f113c15e610 0x7f113c1f55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.307+0000 7f1141ab2640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f113c15e9f0 0x7f113c0cded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.307+0000 7f1141ab2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f113c06d130 con 0x7f113c06c400 2024-04-15T15:15:47.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.307+0000 7f1141ab2640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f113c0d2e60 con 0x7f113c15e9f0 2024-04-15T15:15:47.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.307+0000 7f1141ab2640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f113c0d0bd0 con 0x7f113c15e610 2024-04-15T15:15:47.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.307+0000 7f113b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f113c06c400 0x7f113c1618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.307+0000 7f113b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f113c06c400 0x7f113c1618b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46288/0 (socket says 172.21.15.72:46288) 2024-04-15T15:15:47.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.307+0000 7f113b7fe640 1 -- 172.21.15.72:0/3957164250 learned_addr learned my addr 172.21.15.72:0/3957164250 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:47.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.308+0000 7f113bfff640 1 --2- 172.21.15.72:0/3957164250 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f113c15e9f0 0x7f113c0cded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.308+0000 7f1132ffd640 1 --2- 172.21.15.72:0/3957164250 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f113c15e610 0x7f113c1f55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.308+0000 7f113b7fe640 1 -- 172.21.15.72:0/3957164250 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f113c15e610 msgr2=0x7f113c1f55d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.308+0000 7f113b7fe640 1 --2- 172.21.15.72:0/3957164250 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f113c15e610 0x7f113c1f55d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.308+0000 7f113b7fe640 1 -- 172.21.15.72:0/3957164250 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f113c15e9f0 msgr2=0x7f113c0cded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.308+0000 7f113b7fe640 1 --2- 172.21.15.72:0/3957164250 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f113c15e9f0 0x7f113c0cded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.308+0000 7f113b7fe640 1 -- 172.21.15.72:0/3957164250 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f113c1597e0 con 0x7f113c06c400 2024-04-15T15:15:47.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.308+0000 7f1132ffd640 1 --2- 172.21.15.72:0/3957164250 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f113c15e610 0x7f113c1f55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:47.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.308+0000 7f113bfff640 1 --2- 172.21.15.72:0/3957164250 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f113c15e9f0 0x7f113c0cded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:47.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.308+0000 7f113b7fe640 1 --2- 172.21.15.72:0/3957164250 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f113c06c400 0x7f113c1618b0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f112c009890 tx=0x7f112c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:47.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.308+0000 7f1138ff9640 1 -- 172.21.15.72:0/3957164250 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112c046020 con 0x7f113c06c400 2024-04-15T15:15:47.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.308+0000 7f1138ff9640 1 -- 172.21.15.72:0/3957164250 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f112c039a10 con 0x7f113c06c400 2024-04-15T15:15:47.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.308+0000 7f1138ff9640 1 -- 172.21.15.72:0/3957164250 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f112c034040 con 0x7f113c06c400 2024-04-15T15:15:47.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.308+0000 7f1141ab2640 1 -- 172.21.15.72:0/3957164250 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f113c161df0 con 0x7f113c06c400 2024-04-15T15:15:47.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.308+0000 7f1141ab2640 1 -- 172.21.15.72:0/3957164250 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f113c0cc6f0 con 0x7f113c06c400 2024-04-15T15:15:47.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.309+0000 7f1141ab2640 1 -- 172.21.15.72:0/3957164250 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f113c075c50 con 0x7f113c06c400 2024-04-15T15:15:47.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.310+0000 7f1138ff9640 1 -- 172.21.15.72:0/3957164250 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f112c037070 con 0x7f113c06c400 2024-04-15T15:15:47.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.310+0000 7f1138ff9640 1 --2- 172.21.15.72:0/3957164250 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f111c06c8f0 0x7f111c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.310+0000 7f1138ff9640 1 -- 172.21.15.72:0/3957164250 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f112c0add30 con 0x7f113c06c400 2024-04-15T15:15:47.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.310+0000 7f1132ffd640 1 --2- 172.21.15.72:0/3957164250 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f111c06c8f0 0x7f111c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.311+0000 7f1132ffd640 1 --2- 172.21.15.72:0/3957164250 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f111c06c8f0 0x7f111c06edb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1124000f60 tx=0x7f1124009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:47.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.314+0000 7f1138ff9640 1 -- 172.21.15.72:0/3957164250 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f112c050020 con 0x7f113c06c400 2024-04-15T15:15:47.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.490+0000 7f1141ab2640 1 -- 172.21.15.72:0/3957164250 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f113c0ceb80 con 0x7f113c06c400 2024-04-15T15:15:47.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.490+0000 7f1138ff9640 1 -- 172.21.15.72:0/3957164250 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f113c1597e0 con 0x7f113c06c400 2024-04-15T15:15:47.492 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-15T15:15:47.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.492+0000 7f1141ab2640 1 -- 172.21.15.72:0/3957164250 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f111c06c8f0 msgr2=0x7f111c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.492+0000 7f1141ab2640 1 --2- 172.21.15.72:0/3957164250 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f111c06c8f0 0x7f111c06edb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1124000f60 tx=0x7f1124009040 comp rx=0 tx=0).stop 2024-04-15T15:15:47.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.492+0000 7f1141ab2640 1 -- 172.21.15.72:0/3957164250 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f113c06c400 msgr2=0x7f113c1618b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.492+0000 7f1141ab2640 1 --2- 172.21.15.72:0/3957164250 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f113c06c400 0x7f113c1618b0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f112c009890 tx=0x7f112c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.493+0000 7f1141ab2640 1 -- 172.21.15.72:0/3957164250 shutdown_connections 2024-04-15T15:15:47.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.493+0000 7f1141ab2640 1 --2- 172.21.15.72:0/3957164250 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f111c06c8f0 0x7f111c06edb0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.493+0000 7f1141ab2640 1 --2- 172.21.15.72:0/3957164250 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f113c15e9f0 0x7f113c0cded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.493+0000 7f1141ab2640 1 --2- 172.21.15.72:0/3957164250 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f113c15e610 0x7f113c1f55d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.493+0000 7f1141ab2640 1 --2- 172.21.15.72:0/3957164250 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f113c06c400 0x7f113c1618b0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.493+0000 7f1141ab2640 1 -- 172.21.15.72:0/3957164250 >> 172.21.15.72:0/3957164250 conn(0x7f113c0d4750 msgr2=0x7f113c15dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:47.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.493+0000 7f1141ab2640 1 -- 172.21.15.72:0/3957164250 shutdown_connections 2024-04-15T15:15:47.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.493+0000 7f1141ab2640 1 -- 172.21.15.72:0/3957164250 wait complete. 2024-04-15T15:15:47.506 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:15:47.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.610+0000 7f5d3d915640 1 Processor -- start 2024-04-15T15:15:47.612 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.610+0000 7f5d3d915640 1 -- start start 2024-04-15T15:15:47.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.611+0000 7f5d3d915640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5d3815cf40 0x7f5d3815d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.611+0000 7f5d3d915640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5d380cf280 0x7f5d38141cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.611+0000 7f5d3d915640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d380cf720 con 0x7f5d3815d860 2024-04-15T15:15:47.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.611+0000 7f5d3d915640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f5d380cfa00 con 0x7f5d380cf280 2024-04-15T15:15:47.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.611+0000 7f5d3d915640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f5d380cfce0 con 0x7f5d3815cf40 2024-04-15T15:15:47.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.611+0000 7f5d2e7fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5d380cf280 0x7f5d38141cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.611+0000 7f5d2e7fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5d380cf280 0x7f5d38141cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46294/0 (socket says 172.21.15.72:46294) 2024-04-15T15:15:47.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.611+0000 7f5d377fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f5d3815d860 0x7f5d381600e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60116/0 (socket says 172.21.15.72:60116) 2024-04-15T15:15:47.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.611+0000 7f5d2e7fc640 1 -- 172.21.15.72:0/297310570 learned_addr learned my addr 172.21.15.72:0/297310570 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:47.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.611+0000 7f5d36ffd640 1 --2- 172.21.15.72:0/297310570 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5d3815cf40 0x7f5d3815d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.614 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.611+0000 7f5d2e7fc640 1 -- 172.21.15.72:0/297310570 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5d3815cf40 msgr2=0x7f5d3815d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:47.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.611+0000 7f5d2e7fc640 1 --2- 172.21.15.72:0/297310570 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5d3815cf40 0x7f5d3815d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.611+0000 7f5d2e7fc640 1 -- 172.21.15.72:0/297310570 >> v1:172.21.15.72:6789/0 conn(0x7f5d3815d860 legacy=0x7f5d381600e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.612+0000 7f5d367fc640 1 -- 172.21.15.72:0/297310570 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3806104434 0 0) 0x7f5d380cf720 con 0x7f5d3815d860 2024-04-15T15:15:47.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.612+0000 7f5d2e7fc640 1 -- 172.21.15.72:0/297310570 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d380cffc0 con 0x7f5d380cf280 2024-04-15T15:15:47.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.612+0000 7f5d2e7fc640 1 --2- 172.21.15.72:0/297310570 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d380cf280 0x7f5d38141cb0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f5d20009870 tx=0x7f5d2002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d338e46d2b7e251 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:47.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.612+0000 7f5d367fc640 1 -- 172.21.15.72:0/297310570 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d20034070 con 0x7f5d380cf280 2024-04-15T15:15:47.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.612+0000 7f5d367fc640 1 -- 172.21.15.72:0/297310570 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d20030840 con 0x7f5d380cf280 2024-04-15T15:15:47.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.612+0000 7f5d367fc640 1 -- 172.21.15.72:0/297310570 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d20030c10 con 0x7f5d380cf280 2024-04-15T15:15:47.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.613+0000 7f5d3d915640 1 -- 172.21.15.72:0/297310570 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d380cf280 msgr2=0x7f5d38141cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.613+0000 7f5d3d915640 1 --2- 172.21.15.72:0/297310570 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d380cf280 0x7f5d38141cb0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f5d20009870 tx=0x7f5d2002b980 comp rx=0 tx=0).stop 2024-04-15T15:15:47.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.613+0000 7f5d3d915640 1 -- 172.21.15.72:0/297310570 shutdown_connections 2024-04-15T15:15:47.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.613+0000 7f5d3d915640 1 --2- 172.21.15.72:0/297310570 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d380cf280 0x7f5d38141cb0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.613+0000 7f5d3d915640 1 --2- 172.21.15.72:0/297310570 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5d3815cf40 0x7f5d3815d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.613+0000 7f5d3d915640 1 -- 172.21.15.72:0/297310570 >> 172.21.15.72:0/297310570 conn(0x7f5d380d05b0 msgr2=0x7f5d380d0990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:47.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.613+0000 7f5d3d915640 1 -- 172.21.15.72:0/297310570 shutdown_connections 2024-04-15T15:15:47.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.613+0000 7f5d3d915640 1 -- 172.21.15.72:0/297310570 wait complete. 2024-04-15T15:15:47.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.614+0000 7f5d3d915640 1 Processor -- start 2024-04-15T15:15:47.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.614+0000 7f5d3d915640 1 -- start start 2024-04-15T15:15:47.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.615+0000 7f5d3d915640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5d380cf280 0x7f5d380b3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.615+0000 7f5d3d915640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5d3815cf40 0x7f5d381f7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.615+0000 7f5d3d915640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d3815d860 0x7f5d381412b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.615+0000 7f5d3d915640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f5d3806cbb0 con 0x7f5d3815d860 2024-04-15T15:15:47.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.615+0000 7f5d3d915640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f5d38142a00 con 0x7f5d3815cf40 2024-04-15T15:15:47.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.615+0000 7f5d3d915640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f5d3806cd30 con 0x7f5d380cf280 2024-04-15T15:15:47.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.615+0000 7f5d377fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d3815d860 0x7f5d381412b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.615+0000 7f5d377fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d3815d860 0x7f5d381412b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46296/0 (socket says 172.21.15.72:46296) 2024-04-15T15:15:47.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.615+0000 7f5d377fe640 1 -- 172.21.15.72:0/512730119 learned_addr learned my addr 172.21.15.72:0/512730119 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:47.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.615+0000 7f5d36ffd640 1 --2- 172.21.15.72:0/512730119 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5d380cf280 0x7f5d380b3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.615+0000 7f5d2e7fc640 1 --2- 172.21.15.72:0/512730119 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5d3815cf40 0x7f5d381f7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.615+0000 7f5d377fe640 1 -- 172.21.15.72:0/512730119 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5d380cf280 msgr2=0x7f5d380b3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.615+0000 7f5d377fe640 1 --2- 172.21.15.72:0/512730119 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5d380cf280 0x7f5d380b3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.616+0000 7f5d377fe640 1 -- 172.21.15.72:0/512730119 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5d3815cf40 msgr2=0x7f5d381f7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.616+0000 7f5d377fe640 1 --2- 172.21.15.72:0/512730119 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5d3815cf40 0x7f5d381f7200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.616+0000 7f5d377fe640 1 -- 172.21.15.72:0/512730119 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d381417f0 con 0x7f5d3815d860 2024-04-15T15:15:47.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.616+0000 7f5d36ffd640 1 --2- 172.21.15.72:0/512730119 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5d380cf280 0x7f5d380b3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:47.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.616+0000 7f5d377fe640 1 --2- 172.21.15.72:0/512730119 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d3815d860 0x7f5d381412b0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5d3000c730 tx=0x7f5d3000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:47.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.616+0000 7f5d34ff9640 1 -- 172.21.15.72:0/512730119 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d30018070 con 0x7f5d3815d860 2024-04-15T15:15:47.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.616+0000 7f5d34ff9640 1 -- 172.21.15.72:0/512730119 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d3000ce40 con 0x7f5d3815d860 2024-04-15T15:15:47.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.616+0000 7f5d3d915640 1 -- 172.21.15.72:0/512730119 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d380b1530 con 0x7f5d3815d860 2024-04-15T15:15:47.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.616+0000 7f5d34ff9640 1 -- 172.21.15.72:0/512730119 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d3000a860 con 0x7f5d3815d860 2024-04-15T15:15:47.620 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.616+0000 7f5d3d915640 1 -- 172.21.15.72:0/512730119 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d380b1af0 con 0x7f5d3815d860 2024-04-15T15:15:47.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.617+0000 7f5d3d915640 1 -- 172.21.15.72:0/512730119 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d38075c50 con 0x7f5d3815d860 2024-04-15T15:15:47.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.617+0000 7f5d34ff9640 1 -- 172.21.15.72:0/512730119 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5d3001b070 con 0x7f5d3815d860 2024-04-15T15:15:47.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.618+0000 7f5d34ff9640 1 --2- 172.21.15.72:0/512730119 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f5d1806c8f0 0x7f5d1806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.618+0000 7f5d34ff9640 1 -- 172.21.15.72:0/512730119 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f5d3008b440 con 0x7f5d3815d860 2024-04-15T15:15:47.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.618+0000 7f5d36ffd640 1 --2- 172.21.15.72:0/512730119 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f5d1806c8f0 0x7f5d1806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.618+0000 7f5d36ffd640 1 --2- 172.21.15.72:0/512730119 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f5d1806c8f0 0x7f5d1806edb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5d2800a9d0 tx=0x7f5d28008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:47.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.622+0000 7f5d34ff9640 1 -- 172.21.15.72:0/512730119 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5d30058e20 con 0x7f5d3815d860 2024-04-15T15:15:47.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.796+0000 7f5d3d915640 1 -- 172.21.15.72:0/512730119 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5d3806e460 con 0x7f5d3815d860 2024-04-15T15:15:47.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.797+0000 7f5d34ff9640 1 -- 172.21.15.72:0/512730119 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f5d3005d120 con 0x7f5d3815d860 2024-04-15T15:15:47.798 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:47.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.799+0000 7f5d3d915640 1 -- 172.21.15.72:0/512730119 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f5d1806c8f0 msgr2=0x7f5d1806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.799+0000 7f5d3d915640 1 --2- 172.21.15.72:0/512730119 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f5d1806c8f0 0x7f5d1806edb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5d2800a9d0 tx=0x7f5d28008040 comp rx=0 tx=0).stop 2024-04-15T15:15:47.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.799+0000 7f5d3d915640 1 -- 172.21.15.72:0/512730119 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d3815d860 msgr2=0x7f5d381412b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.799+0000 7f5d3d915640 1 --2- 172.21.15.72:0/512730119 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d3815d860 0x7f5d381412b0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5d3000c730 tx=0x7f5d3000cbf0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.799+0000 7f5d3d915640 1 -- 172.21.15.72:0/512730119 shutdown_connections 2024-04-15T15:15:47.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.799+0000 7f5d3d915640 1 --2- 172.21.15.72:0/512730119 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f5d1806c8f0 0x7f5d1806edb0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.799+0000 7f5d3d915640 1 --2- 172.21.15.72:0/512730119 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5d3815d860 0x7f5d381412b0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.799+0000 7f5d3d915640 1 --2- 172.21.15.72:0/512730119 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5d3815cf40 0x7f5d381f7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.799+0000 7f5d3d915640 1 --2- 172.21.15.72:0/512730119 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5d380cf280 0x7f5d380b3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.799+0000 7f5d3d915640 1 -- 172.21.15.72:0/512730119 >> 172.21.15.72:0/512730119 conn(0x7f5d380d05b0 msgr2=0x7f5d3806c330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:47.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.799+0000 7f5d3d915640 1 -- 172.21.15.72:0/512730119 shutdown_connections 2024-04-15T15:15:47.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.799+0000 7f5d3d915640 1 -- 172.21.15.72:0/512730119 wait complete. 2024-04-15T15:15:47.812 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:15:47.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.916+0000 7f07e5460640 1 Processor -- start 2024-04-15T15:15:47.917 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.916+0000 7f07e5460640 1 -- start start 2024-04-15T15:15:47.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.916+0000 7f07e5460640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f07e015e610 0x7f07e00d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.916+0000 7f07e5460640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f07e015e9f0 0x7f07e00d3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.916+0000 7f07e5460640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07e00d2320 con 0x7f07e006c400 2024-04-15T15:15:47.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.917+0000 7f07e5460640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f07e00d31d0 con 0x7f07e015e9f0 2024-04-15T15:15:47.920 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.917+0000 7f07e5460640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f07e00d25c0 con 0x7f07e015e610 2024-04-15T15:15:47.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.917+0000 7f07d67fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f07e015e9f0 0x7f07e00d3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.917+0000 7f07d67fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f07e015e9f0 0x7f07e00d3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46306/0 (socket says 172.21.15.72:46306) 2024-04-15T15:15:47.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.917+0000 7f07d67fc640 1 -- 172.21.15.72:0/3096954317 learned_addr learned my addr 172.21.15.72:0/3096954317 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:47.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.917+0000 7f07deffd640 1 --2- 172.21.15.72:0/3096954317 >> v2:172.21.15.175:3300/0 conn(0x7f07e015e610 0x7f07e00d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.917+0000 7f07d67fc640 1 -- 172.21.15.72:0/3096954317 >> v2:172.21.15.175:3300/0 conn(0x7f07e015e610 msgr2=0x7f07e00d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:47.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.917+0000 7f07d67fc640 1 --2- 172.21.15.72:0/3096954317 >> v2:172.21.15.175:3300/0 conn(0x7f07e015e610 0x7f07e00d0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.917+0000 7f07d67fc640 1 -- 172.21.15.72:0/3096954317 >> v1:172.21.15.175:6789/0 conn(0x7f07e006c400 legacy=0x7f07e00b7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.917+0000 7f07d67fc640 1 -- 172.21.15.72:0/3096954317 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07e00d28a0 con 0x7f07e015e9f0 2024-04-15T15:15:47.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.918+0000 7f07d67fc640 1 --2- 172.21.15.72:0/3096954317 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07e015e9f0 0x7f07e00d3660 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f07e00b72a0 tx=0x7f07c802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5c32002e3cb9180 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:47.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.918+0000 7f07de7fc640 1 -- 172.21.15.72:0/3096954317 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c802f850 con 0x7f07e015e9f0 2024-04-15T15:15:47.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.918+0000 7f07de7fc640 1 -- 172.21.15.72:0/3096954317 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07c802f9f0 con 0x7f07e015e9f0 2024-04-15T15:15:47.921 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.918+0000 7f07de7fc640 1 -- 172.21.15.72:0/3096954317 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c802fdc0 con 0x7f07e015e9f0 2024-04-15T15:15:47.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.918+0000 7f07e5460640 1 -- 172.21.15.72:0/3096954317 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07e015e9f0 msgr2=0x7f07e00d3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.918+0000 7f07e5460640 1 --2- 172.21.15.72:0/3096954317 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07e015e9f0 0x7f07e00d3660 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f07e00b72a0 tx=0x7f07c802ba30 comp rx=0 tx=0).stop 2024-04-15T15:15:47.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.919+0000 7f07e5460640 1 -- 172.21.15.72:0/3096954317 shutdown_connections 2024-04-15T15:15:47.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.919+0000 7f07e5460640 1 --2- 172.21.15.72:0/3096954317 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07e015e9f0 0x7f07e00d3660 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.919+0000 7f07e5460640 1 --2- 172.21.15.72:0/3096954317 >> v2:172.21.15.175:3300/0 conn(0x7f07e015e610 0x7f07e00d0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.919+0000 7f07e5460640 1 -- 172.21.15.72:0/3096954317 >> 172.21.15.72:0/3096954317 conn(0x7f07e00d4750 msgr2=0x7f07e00d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:47.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.919+0000 7f07e5460640 1 -- 172.21.15.72:0/3096954317 shutdown_connections 2024-04-15T15:15:47.922 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.919+0000 7f07e5460640 1 -- 172.21.15.72:0/3096954317 wait complete. 2024-04-15T15:15:47.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.920+0000 7f07e5460640 1 Processor -- start 2024-04-15T15:15:47.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.920+0000 7f07e5460640 1 -- start start 2024-04-15T15:15:47.925 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.920+0000 7f07e5460640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f07e006c400 0x7f07e00b5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.921+0000 7f07e5460640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07e015e610 0x7f07e0156610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.921+0000 7f07e5460640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f07e015e9f0 0x7f07e01f7850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.921+0000 7f07e5460640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f07e006d130 con 0x7f07e015e610 2024-04-15T15:15:47.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.921+0000 7f07e5460640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f07e00d2e60 con 0x7f07e015e9f0 2024-04-15T15:15:47.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.921+0000 7f07e5460640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f07e00d0bd0 con 0x7f07e006c400 2024-04-15T15:15:47.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.921+0000 7f07d67fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07e015e610 0x7f07e0156610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.921+0000 7f07d67fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07e015e610 0x7f07e0156610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46322/0 (socket says 172.21.15.72:46322) 2024-04-15T15:15:47.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.921+0000 7f07d67fc640 1 -- 172.21.15.72:0/924274632 learned_addr learned my addr 172.21.15.72:0/924274632 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:47.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.921+0000 7f07deffd640 1 --2- 172.21.15.72:0/924274632 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f07e006c400 0x7f07e00b5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.921+0000 7f07df7fe640 1 --2- 172.21.15.72:0/924274632 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f07e015e9f0 0x7f07e01f7850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.921+0000 7f07d67fc640 1 -- 172.21.15.72:0/924274632 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f07e006c400 msgr2=0x7f07e00b5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.926 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.921+0000 7f07d67fc640 1 --2- 172.21.15.72:0/924274632 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f07e006c400 0x7f07e00b5200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.921+0000 7f07d67fc640 1 -- 172.21.15.72:0/924274632 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f07e015e9f0 msgr2=0x7f07e01f7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.921+0000 7f07d67fc640 1 --2- 172.21.15.72:0/924274632 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f07e015e9f0 0x7f07e01f7850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.921+0000 7f07d67fc640 1 -- 172.21.15.72:0/924274632 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07e0156b50 con 0x7f07e015e610 2024-04-15T15:15:47.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.921+0000 7f07d67fc640 1 --2- 172.21.15.72:0/924274632 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07e015e610 0x7f07e0156610 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f07c802ba70 tx=0x7f07c8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:47.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.921+0000 7f07d7fff640 1 -- 172.21.15.72:0/924274632 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c8047070 con 0x7f07e015e610 2024-04-15T15:15:47.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.921+0000 7f07d7fff640 1 -- 172.21.15.72:0/924274632 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07c8037d30 con 0x7f07e015e610 2024-04-15T15:15:47.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.921+0000 7f07e5460640 1 -- 172.21.15.72:0/924274632 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07e00b5830 con 0x7f07e015e610 2024-04-15T15:15:47.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.921+0000 7f07d7fff640 1 -- 172.21.15.72:0/924274632 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c8042630 con 0x7f07e015e610 2024-04-15T15:15:47.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.921+0000 7f07e5460640 1 -- 172.21.15.72:0/924274632 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07e00b5df0 con 0x7f07e015e610 2024-04-15T15:15:47.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.922+0000 7f07e5460640 1 -- 172.21.15.72:0/924274632 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07e0075c50 con 0x7f07e015e610 2024-04-15T15:15:47.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.923+0000 7f07d7fff640 1 -- 172.21.15.72:0/924274632 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f07c804a070 con 0x7f07e015e610 2024-04-15T15:15:47.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.923+0000 7f07d7fff640 1 --2- 172.21.15.72:0/924274632 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f07c006c920 0x7f07c006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.927 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.923+0000 7f07d7fff640 1 -- 172.21.15.72:0/924274632 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f07c80add80 con 0x7f07e015e610 2024-04-15T15:15:47.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.923+0000 7f07deffd640 1 --2- 172.21.15.72:0/924274632 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f07c006c920 0x7f07c006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.928 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.924+0000 7f07deffd640 1 --2- 172.21.15.72:0/924274632 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f07c006c920 0x7f07c006ede0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f07d0006fd0 tx=0x7f07d0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:47.929 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:47.927+0000 7f07d7fff640 1 -- 172.21.15.72:0/924274632 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f07e0156b50 con 0x7f07e015e610 2024-04-15T15:15:48.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.106+0000 7f07e5460640 1 -- 172.21.15.72:0/924274632 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f07e00d4eb0 con 0x7f07e015e610 2024-04-15T15:15:48.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.107+0000 7f07d7fff640 1 -- 172.21.15.72:0/924274632 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f07c8033030 con 0x7f07e015e610 2024-04-15T15:15:48.109 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-04-15T15:15:48.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.109+0000 7f07e5460640 1 -- 172.21.15.72:0/924274632 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f07c006c920 msgr2=0x7f07c006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.109+0000 7f07e5460640 1 --2- 172.21.15.72:0/924274632 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f07c006c920 0x7f07c006ede0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f07d0006fd0 tx=0x7f07d0008040 comp rx=0 tx=0).stop 2024-04-15T15:15:48.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.109+0000 7f07e5460640 1 -- 172.21.15.72:0/924274632 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07e015e610 msgr2=0x7f07e0156610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.109+0000 7f07e5460640 1 --2- 172.21.15.72:0/924274632 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07e015e610 0x7f07e0156610 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f07c802ba70 tx=0x7f07c8004730 comp rx=0 tx=0).stop 2024-04-15T15:15:48.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.109+0000 7f07e5460640 1 -- 172.21.15.72:0/924274632 shutdown_connections 2024-04-15T15:15:48.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.109+0000 7f07e5460640 1 --2- 172.21.15.72:0/924274632 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f07c006c920 0x7f07c006ede0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.109+0000 7f07e5460640 1 --2- 172.21.15.72:0/924274632 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f07e015e9f0 0x7f07e01f7850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.109+0000 7f07e5460640 1 --2- 172.21.15.72:0/924274632 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07e015e610 0x7f07e0156610 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.109+0000 7f07e5460640 1 --2- 172.21.15.72:0/924274632 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f07e006c400 0x7f07e00b5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.109+0000 7f07e5460640 1 -- 172.21.15.72:0/924274632 >> 172.21.15.72:0/924274632 conn(0x7f07e00d4750 msgr2=0x7f07e015f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:48.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.109+0000 7f07e5460640 1 -- 172.21.15.72:0/924274632 shutdown_connections 2024-04-15T15:15:48.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.109+0000 7f07e5460640 1 -- 172.21.15.72:0/924274632 wait complete. 2024-04-15T15:15:48.122 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:15:48.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.226+0000 7f1330b24640 1 Processor -- start 2024-04-15T15:15:48.228 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.226+0000 7f1330b24640 1 -- start start 2024-04-15T15:15:48.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.227+0000 7f1330b24640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f132c15e610 0x7f132c0d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:48.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.227+0000 7f1330b24640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f132c0d2320 con 0x7f132c15e9f0 2024-04-15T15:15:48.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.227+0000 7f1330b24640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f132c0d25c0 con 0x7f132c06c400 2024-04-15T15:15:48.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.227+0000 7f1330b24640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f132c0d31d0 con 0x7f132c15e610 2024-04-15T15:15:48.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.227+0000 7f132a59c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f132c15e610 0x7f132c0d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:48.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.227+0000 7f1329d9b640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f132c15e9f0 0x7f132c0d3660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60128/0 (socket says 172.21.15.72:60128) 2024-04-15T15:15:48.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.227+0000 7f1329d9b640 1 -- 172.21.15.72:0/2364944308 learned_addr learned my addr 172.21.15.72:0/2364944308 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:48.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.228+0000 7f132a59c640 1 -- 172.21.15.72:0/2364944308 >> v1:172.21.15.175:6789/0 conn(0x7f132c06c400 legacy=0x7f132c0b7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:48.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.228+0000 7f132a59c640 1 -- 172.21.15.72:0/2364944308 >> v1:172.21.15.72:6789/0 conn(0x7f132c15e9f0 legacy=0x7f132c0d3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.228+0000 7f132959a640 1 -- 172.21.15.72:0/2364944308 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3767378445 0 0) 0x7f132c0d2320 con 0x7f132c15e9f0 2024-04-15T15:15:48.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.228+0000 7f132a59c640 1 -- 172.21.15.72:0/2364944308 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f132c0d28a0 con 0x7f132c15e610 2024-04-15T15:15:48.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.228+0000 7f132a59c640 1 --2- 172.21.15.72:0/2364944308 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f132c15e610 0x7f132c0d0690 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f131c009d90 tx=0x7f131c02bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac7913a650109fbc server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:48.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.229+0000 7f132959a640 1 -- 172.21.15.72:0/2364944308 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131c039070 con 0x7f132c15e610 2024-04-15T15:15:48.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.229+0000 7f132959a640 1 -- 172.21.15.72:0/2364944308 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f131c02cd80 con 0x7f132c15e610 2024-04-15T15:15:48.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.229+0000 7f132959a640 1 -- 172.21.15.72:0/2364944308 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131c034660 con 0x7f132c15e610 2024-04-15T15:15:48.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.229+0000 7f1330b24640 1 -- 172.21.15.72:0/2364944308 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f132c15e610 msgr2=0x7f132c0d0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.229+0000 7f1330b24640 1 --2- 172.21.15.72:0/2364944308 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f132c15e610 0x7f132c0d0690 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f131c009d90 tx=0x7f131c02bfd0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.229+0000 7f1330b24640 1 -- 172.21.15.72:0/2364944308 shutdown_connections 2024-04-15T15:15:48.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.229+0000 7f1330b24640 1 --2- 172.21.15.72:0/2364944308 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f132c15e610 0x7f132c0d0690 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.229+0000 7f1330b24640 1 -- 172.21.15.72:0/2364944308 >> 172.21.15.72:0/2364944308 conn(0x7f132c0d4750 msgr2=0x7f132c0d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:48.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.229+0000 7f1330b24640 1 -- 172.21.15.72:0/2364944308 shutdown_connections 2024-04-15T15:15:48.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.229+0000 7f1330b24640 1 -- 172.21.15.72:0/2364944308 wait complete. 2024-04-15T15:15:48.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.231+0000 7f1330b24640 1 Processor -- start 2024-04-15T15:15:48.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.231+0000 7f1330b24640 1 -- start start 2024-04-15T15:15:48.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.231+0000 7f1330b24640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f132c06c400 0x7f132c0caff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:48.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.231+0000 7f1330b24640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f132c15e610 0x7f132c0b5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:48.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.231+0000 7f1330b24640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f132c15e9f0 0x7f132c0cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:48.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.231+0000 7f1330b24640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f132c06d130 con 0x7f132c06c400 2024-04-15T15:15:48.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.231+0000 7f1330b24640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f132c0d2e60 con 0x7f132c15e9f0 2024-04-15T15:15:48.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.231+0000 7f1330b24640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f132c0d0bd0 con 0x7f132c15e610 2024-04-15T15:15:48.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.231+0000 7f132a59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f132c06c400 0x7f132c0caff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:48.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.231+0000 7f132a59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f132c06c400 0x7f132c0caff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46328/0 (socket says 172.21.15.72:46328) 2024-04-15T15:15:48.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.231+0000 7f132a59c640 1 -- 172.21.15.72:0/566069561 learned_addr learned my addr 172.21.15.72:0/566069561 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:48.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.232+0000 7f1329d9b640 1 --2- 172.21.15.72:0/566069561 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f132c15e610 0x7f132c0b5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:48.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.232+0000 7f132ad9d640 1 --2- 172.21.15.72:0/566069561 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f132c15e9f0 0x7f132c0cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:48.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.232+0000 7f132a59c640 1 -- 172.21.15.72:0/566069561 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f132c15e610 msgr2=0x7f132c0b5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.232+0000 7f132a59c640 1 --2- 172.21.15.72:0/566069561 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f132c15e610 0x7f132c0b5300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.232+0000 7f132a59c640 1 -- 172.21.15.72:0/566069561 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f132c15e9f0 msgr2=0x7f132c0cdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.232+0000 7f132a59c640 1 --2- 172.21.15.72:0/566069561 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f132c15e9f0 0x7f132c0cdeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.232+0000 7f132a59c640 1 -- 172.21.15.72:0/566069561 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f132c0b5e10 con 0x7f132c06c400 2024-04-15T15:15:48.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.232+0000 7f132ad9d640 1 --2- 172.21.15.72:0/566069561 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f132c15e9f0 0x7f132c0cdeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:48.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.232+0000 7f132a59c640 1 --2- 172.21.15.72:0/566069561 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f132c06c400 0x7f132c0caff0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f131c037040 tx=0x7f131c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:48.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.232+0000 7f1322ffd640 1 -- 172.21.15.72:0/566069561 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131c039050 con 0x7f132c06c400 2024-04-15T15:15:48.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.232+0000 7f1322ffd640 1 -- 172.21.15.72:0/566069561 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f131c045070 con 0x7f132c06c400 2024-04-15T15:15:48.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.232+0000 7f1330b24640 1 -- 172.21.15.72:0/566069561 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f132c158050 con 0x7f132c06c400 2024-04-15T15:15:48.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.232+0000 7f1322ffd640 1 -- 172.21.15.72:0/566069561 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131c03ea50 con 0x7f132c06c400 2024-04-15T15:15:48.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.232+0000 7f1330b24640 1 -- 172.21.15.72:0/566069561 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f132c1585d0 con 0x7f132c06c400 2024-04-15T15:15:48.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.233+0000 7f1330b24640 1 -- 172.21.15.72:0/566069561 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f132c075c50 con 0x7f132c06c400 2024-04-15T15:15:48.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.234+0000 7f1322ffd640 1 -- 172.21.15.72:0/566069561 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f131c02cc40 con 0x7f132c06c400 2024-04-15T15:15:48.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.234+0000 7f1322ffd640 1 --2- 172.21.15.72:0/566069561 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f130c06c920 0x7f130c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:48.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.234+0000 7f1322ffd640 1 -- 172.21.15.72:0/566069561 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f131c0ae760 con 0x7f132c06c400 2024-04-15T15:15:48.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.234+0000 7f1329d9b640 1 --2- 172.21.15.72:0/566069561 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f130c06c920 0x7f130c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:48.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.235+0000 7f1329d9b640 1 --2- 172.21.15.72:0/566069561 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f130c06c920 0x7f130c06ede0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1314007fd0 tx=0x7f1314009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:48.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.238+0000 7f1322ffd640 1 -- 172.21.15.72:0/566069561 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f131c04e020 con 0x7f132c06c400 2024-04-15T15:15:48.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.414+0000 7f1330b24640 1 -- 172.21.15.72:0/566069561 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f132c0cf740 con 0x7f132c06c400 2024-04-15T15:15:48.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.414+0000 7f1322ffd640 1 -- 172.21.15.72:0/566069561 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f131c0333a0 con 0x7f132c06c400 2024-04-15T15:15:48.416 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:48.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.416+0000 7f1330b24640 1 -- 172.21.15.72:0/566069561 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f130c06c920 msgr2=0x7f130c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.416+0000 7f1330b24640 1 --2- 172.21.15.72:0/566069561 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f130c06c920 0x7f130c06ede0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1314007fd0 tx=0x7f1314009040 comp rx=0 tx=0).stop 2024-04-15T15:15:48.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.416+0000 7f1330b24640 1 -- 172.21.15.72:0/566069561 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f132c06c400 msgr2=0x7f132c0caff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.416+0000 7f1330b24640 1 --2- 172.21.15.72:0/566069561 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f132c06c400 0x7f132c0caff0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f131c037040 tx=0x7f131c004a40 comp rx=0 tx=0).stop 2024-04-15T15:15:48.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.416+0000 7f1330b24640 1 -- 172.21.15.72:0/566069561 shutdown_connections 2024-04-15T15:15:48.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.416+0000 7f1330b24640 1 --2- 172.21.15.72:0/566069561 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f130c06c920 0x7f130c06ede0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.416+0000 7f1330b24640 1 --2- 172.21.15.72:0/566069561 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f132c15e9f0 0x7f132c0cdeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.416+0000 7f1330b24640 1 --2- 172.21.15.72:0/566069561 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f132c15e610 0x7f132c0b5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.416+0000 7f1330b24640 1 --2- 172.21.15.72:0/566069561 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f132c06c400 0x7f132c0caff0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.416+0000 7f1330b24640 1 -- 172.21.15.72:0/566069561 >> 172.21.15.72:0/566069561 conn(0x7f132c0d4750 msgr2=0x7f132c0d3660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:48.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.417+0000 7f1330b24640 1 -- 172.21.15.72:0/566069561 shutdown_connections 2024-04-15T15:15:48.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.417+0000 7f1330b24640 1 -- 172.21.15.72:0/566069561 wait complete. 2024-04-15T15:15:48.429 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:15:48.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.533+0000 7fe28def2640 1 Processor -- start 2024-04-15T15:15:48.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.533+0000 7fe28def2640 1 -- start start 2024-04-15T15:15:48.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.534+0000 7fe28def2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe28815c250 0x7fe28806d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:48.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.534+0000 7fe28def2640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe28815f9f0 0x7fe2880cca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:48.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.534+0000 7fe28def2640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe28806df30 con 0x7fe2880d0010 2024-04-15T15:15:48.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.534+0000 7fe28def2640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fe28806e210 con 0x7fe28815c250 2024-04-15T15:15:48.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.534+0000 7fe28def2640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fe28806e4f0 con 0x7fe28815f9f0 2024-04-15T15:15:48.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.534+0000 7fe2877fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe28815c250 0x7fe28806d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:48.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.534+0000 7fe2877fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe28815c250 0x7fe28806d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46340/0 (socket says 172.21.15.72:46340) 2024-04-15T15:15:48.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.534+0000 7fe2877fe640 1 -- 172.21.15.72:0/2553494772 learned_addr learned my addr 172.21.15.72:0/2553494772 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:48.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.534+0000 7fe286ffd640 1 --2- 172.21.15.72:0/2553494772 >> v2:172.21.15.175:3300/0 conn(0x7fe28815f9f0 0x7fe2880cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:48.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.534+0000 7fe2877fe640 1 -- 172.21.15.72:0/2553494772 >> v2:172.21.15.175:3300/0 conn(0x7fe28815f9f0 msgr2=0x7fe2880cca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:48.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.534+0000 7fe2877fe640 1 --2- 172.21.15.72:0/2553494772 >> v2:172.21.15.175:3300/0 conn(0x7fe28815f9f0 0x7fe2880cca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.534+0000 7fe2877fe640 1 -- 172.21.15.72:0/2553494772 >> v1:172.21.15.175:6789/0 conn(0x7fe2880d0010 legacy=0x7fe2880b3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.534+0000 7fe2877fe640 1 -- 172.21.15.72:0/2553494772 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2880b3610 con 0x7fe28815c250 2024-04-15T15:15:48.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.535+0000 7fe2877fe640 1 --2- 172.21.15.72:0/2553494772 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe28815c250 0x7fe28806d6e0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fe2880b5030 tx=0x7fe27802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51d2cfeb0756b575 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:48.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.535+0000 7fe2867fc640 1 -- 172.21.15.72:0/2553494772 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe27802ca80 con 0x7fe28815c250 2024-04-15T15:15:48.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.535+0000 7fe2867fc640 1 -- 172.21.15.72:0/2553494772 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe27802cc20 con 0x7fe28815c250 2024-04-15T15:15:48.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.535+0000 7fe2867fc640 1 -- 172.21.15.72:0/2553494772 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2780343f0 con 0x7fe28815c250 2024-04-15T15:15:48.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.536+0000 7fe28def2640 1 -- 172.21.15.72:0/2553494772 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe28815c250 msgr2=0x7fe28806d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.536+0000 7fe28def2640 1 --2- 172.21.15.72:0/2553494772 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe28815c250 0x7fe28806d6e0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fe2880b5030 tx=0x7fe27802bee0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.536+0000 7fe28def2640 1 -- 172.21.15.72:0/2553494772 shutdown_connections 2024-04-15T15:15:48.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.536+0000 7fe28def2640 1 --2- 172.21.15.72:0/2553494772 >> v2:172.21.15.175:3300/0 conn(0x7fe28815f9f0 0x7fe2880cca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.536+0000 7fe28def2640 1 --2- 172.21.15.72:0/2553494772 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe28815c250 0x7fe28806d6e0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.536+0000 7fe28def2640 1 -- 172.21.15.72:0/2553494772 >> 172.21.15.72:0/2553494772 conn(0x7fe2880ceec0 msgr2=0x7fe2880cf2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:48.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.536+0000 7fe28def2640 1 -- 172.21.15.72:0/2553494772 shutdown_connections 2024-04-15T15:15:48.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.536+0000 7fe28def2640 1 -- 172.21.15.72:0/2553494772 wait complete. 2024-04-15T15:15:48.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.537+0000 7fe28def2640 1 Processor -- start 2024-04-15T15:15:48.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.537+0000 7fe28def2640 1 -- start start 2024-04-15T15:15:48.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.538+0000 7fe28def2640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe2880d0010 0x7fe2881f4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:48.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.538+0000 7fe28def2640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe28815c250 0x7fe28814dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:48.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.538+0000 7fe28def2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe28815f9f0 0x7fe28814ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:48.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.538+0000 7fe28def2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe28815c630 con 0x7fe28815f9f0 2024-04-15T15:15:48.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.538+0000 7fe28def2640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fe28806dc20 con 0x7fe2880d0010 2024-04-15T15:15:48.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.538+0000 7fe28def2640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fe28815c7b0 con 0x7fe28815c250 2024-04-15T15:15:48.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.538+0000 7fe287fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe28815f9f0 0x7fe28814ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:48.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.538+0000 7fe287fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe28815f9f0 0x7fe28814ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46354/0 (socket says 172.21.15.72:46354) 2024-04-15T15:15:48.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.538+0000 7fe287fff640 1 -- 172.21.15.72:0/2881747829 learned_addr learned my addr 172.21.15.72:0/2881747829 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:48.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.538+0000 7fe2877fe640 1 --2- 172.21.15.72:0/2881747829 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe2880d0010 0x7fe2881f4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:48.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.538+0000 7fe286ffd640 1 --2- 172.21.15.72:0/2881747829 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe28815c250 0x7fe28814dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:48.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.538+0000 7fe287fff640 1 -- 172.21.15.72:0/2881747829 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe28815c250 msgr2=0x7fe28814dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.538+0000 7fe287fff640 1 --2- 172.21.15.72:0/2881747829 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe28815c250 0x7fe28814dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.538+0000 7fe287fff640 1 -- 172.21.15.72:0/2881747829 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe2880d0010 msgr2=0x7fe2881f4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.538+0000 7fe287fff640 1 --2- 172.21.15.72:0/2881747829 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe2880d0010 0x7fe2881f4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.538+0000 7fe287fff640 1 -- 172.21.15.72:0/2881747829 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2881f5040 con 0x7fe28815f9f0 2024-04-15T15:15:48.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.538+0000 7fe286ffd640 1 --2- 172.21.15.72:0/2881747829 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe28815c250 0x7fe28814dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:48.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.538+0000 7fe287fff640 1 --2- 172.21.15.72:0/2881747829 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe28815f9f0 0x7fe28814ca10 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe280001eb0 tx=0x7fe28000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:48.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.539+0000 7fe26ffff640 1 -- 172.21.15.72:0/2881747829 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe280017070 con 0x7fe28815f9f0 2024-04-15T15:15:48.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.539+0000 7fe26ffff640 1 -- 172.21.15.72:0/2881747829 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2800052d0 con 0x7fe28815f9f0 2024-04-15T15:15:48.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.539+0000 7fe28def2640 1 -- 172.21.15.72:0/2881747829 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2881f47a0 con 0x7fe28815f9f0 2024-04-15T15:15:48.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.539+0000 7fe26ffff640 1 -- 172.21.15.72:0/2881747829 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe280003810 con 0x7fe28815f9f0 2024-04-15T15:15:48.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.539+0000 7fe28def2640 1 -- 172.21.15.72:0/2881747829 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2881f4d60 con 0x7fe28815f9f0 2024-04-15T15:15:48.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.540+0000 7fe28def2640 1 -- 172.21.15.72:0/2881747829 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe288075c50 con 0x7fe28815f9f0 2024-04-15T15:15:48.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.540+0000 7fe26ffff640 1 -- 172.21.15.72:0/2881747829 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe28001a070 con 0x7fe28815f9f0 2024-04-15T15:15:48.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.541+0000 7fe26ffff640 1 --2- 172.21.15.72:0/2881747829 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe27406c8f0 0x7fe27406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:48.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.541+0000 7fe2877fe640 1 --2- 172.21.15.72:0/2881747829 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe27406c8f0 0x7fe27406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:48.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.541+0000 7fe26ffff640 1 -- 172.21.15.72:0/2881747829 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fe28008b1c0 con 0x7fe28815f9f0 2024-04-15T15:15:48.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.541+0000 7fe2877fe640 1 --2- 172.21.15.72:0/2881747829 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe27406c8f0 0x7fe27406edb0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe278035000 tx=0x7fe278030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:48.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.545+0000 7fe26ffff640 1 -- 172.21.15.72:0/2881747829 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe280058c30 con 0x7fe28815f9f0 2024-04-15T15:15:48.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.732+0000 7fe28def2640 1 -- 172.21.15.72:0/2881747829 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe2880b83c0 con 0x7fe28815f9f0 2024-04-15T15:15:48.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.733+0000 7fe26ffff640 1 -- 172.21.15.72:0/2881747829 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7fe28005cf30 con 0x7fe28815f9f0 2024-04-15T15:15:48.737 INFO:teuthology.orchestra.run.smithi072.stderr:listed 0 entries 2024-04-15T15:15:48.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.735+0000 7fe28def2640 1 -- 172.21.15.72:0/2881747829 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe27406c8f0 msgr2=0x7fe27406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.735+0000 7fe28def2640 1 --2- 172.21.15.72:0/2881747829 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe27406c8f0 0x7fe27406edb0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe278035000 tx=0x7fe278030000 comp rx=0 tx=0).stop 2024-04-15T15:15:48.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.735+0000 7fe28def2640 1 -- 172.21.15.72:0/2881747829 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe28815f9f0 msgr2=0x7fe28814ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.735+0000 7fe28def2640 1 --2- 172.21.15.72:0/2881747829 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe28815f9f0 0x7fe28814ca10 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe280001eb0 tx=0x7fe28000c890 comp rx=0 tx=0).stop 2024-04-15T15:15:48.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.735+0000 7fe28def2640 1 -- 172.21.15.72:0/2881747829 shutdown_connections 2024-04-15T15:15:48.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.735+0000 7fe28def2640 1 --2- 172.21.15.72:0/2881747829 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe27406c8f0 0x7fe27406edb0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.735+0000 7fe28def2640 1 --2- 172.21.15.72:0/2881747829 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe28815f9f0 0x7fe28814ca10 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.735+0000 7fe28def2640 1 --2- 172.21.15.72:0/2881747829 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe28815c250 0x7fe28814dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.735+0000 7fe28def2640 1 --2- 172.21.15.72:0/2881747829 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe2880d0010 0x7fe2881f4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.735+0000 7fe28def2640 1 -- 172.21.15.72:0/2881747829 >> 172.21.15.72:0/2881747829 conn(0x7fe2880ceec0 msgr2=0x7fe2880d2170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:48.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.735+0000 7fe28def2640 1 -- 172.21.15.72:0/2881747829 shutdown_connections 2024-04-15T15:15:48.738 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:48.735+0000 7fe28def2640 1 -- 172.21.15.72:0/2881747829 wait complete. 2024-04-15T15:15:48.747 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi072.front.sepia.ceph.com... 2024-04-15T15:15:48.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:48.748 DEBUG:teuthology.orchestra.run.smithi072:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-04-15T15:15:48.776 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:15:48.776 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi072.front.sepia.ceph.com 2024-04-15T15:15:48.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:48.776 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-15T15:15:48.832 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:15:48.833 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:15:48.852 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:15:48.852 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:15:48.912 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:49.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.014+0000 7f2d33e1b640 1 Processor -- start 2024-04-15T15:15:49.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.014+0000 7f2d33e1b640 1 -- start start 2024-04-15T15:15:49.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.015+0000 7f2d33e1b640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2d2c0af700 0x7f2d2c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.015+0000 7f2d33e1b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d2c0cffd0 con 0x7f2d2c06b960 2024-04-15T15:15:49.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.015+0000 7f2d33e1b640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d2c06de60 con 0x7f2d2c15c7c0 2024-04-15T15:15:49.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.015+0000 7f2d33e1b640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f2d2c15cba0 con 0x7f2d2c0af700 2024-04-15T15:15:49.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.015+0000 7f2d323b8640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f2d2c06b960 0x7f2d2c06bdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60136/0 (socket says 172.21.15.72:60136) 2024-04-15T15:15:49.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.015+0000 7f2d323b8640 1 -- 172.21.15.72:0/1905182557 learned_addr learned my addr 172.21.15.72:0/1905182557 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:49.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.016+0000 7f2d30bb5640 1 -- 172.21.15.72:0/1905182557 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3877353515 0 0) 0x7f2d2c0cffd0 con 0x7f2d2c06b960 2024-04-15T15:15:49.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.016+0000 7f2d313b6640 1 --2- 172.21.15.72:0/1905182557 >> v2:172.21.15.175:3300/0 conn(0x7f2d2c0af700 0x7f2d2c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.016+0000 7f2d30bb5640 1 -- 172.21.15.72:0/1905182557 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d14003520 con 0x7f2d2c06b960 2024-04-15T15:15:49.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.016+0000 7f2d30bb5640 1 -- 172.21.15.72:0/1905182557 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1300463319 0 0) 0x7f2d14003520 con 0x7f2d2c06b960 2024-04-15T15:15:49.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.016+0000 7f2d30bb5640 1 -- 172.21.15.72:0/1905182557 >> v2:172.21.15.175:3300/0 conn(0x7f2d2c0af700 msgr2=0x7f2d2c06eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:49.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.016+0000 7f2d30bb5640 1 --2- 172.21.15.72:0/1905182557 >> v2:172.21.15.175:3300/0 conn(0x7f2d2c0af700 0x7f2d2c06eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.016+0000 7f2d30bb5640 1 -- 172.21.15.72:0/1905182557 >> v1:172.21.15.175:6789/0 conn(0x7f2d2c15c7c0 legacy=0x7f2d2c06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.016+0000 7f2d30bb5640 1 -- 172.21.15.72:0/1905182557 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d2c06e140 con 0x7f2d2c06b960 2024-04-15T15:15:49.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.016+0000 7f2d30bb5640 1 -- 172.21.15.72:0/1905182557 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f2d24001670 con 0x7f2d2c06b960 2024-04-15T15:15:49.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.016+0000 7f2d313b6640 1 --2- 172.21.15.72:0/1905182557 >> v2:172.21.15.175:3300/0 conn(0x7f2d2c0af700 0x7f2d2c06eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:49.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.016+0000 7f2d30bb5640 1 -- 172.21.15.72:0/1905182557 >> v1:172.21.15.72:6789/0 conn(0x7f2d2c06b960 legacy=0x7f2d2c06bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.016+0000 7f2d30bb5640 1 --2- 172.21.15.72:0/1905182557 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d140039b0 0x7f2d14003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.016+0000 7f2d30bb5640 1 --2- 172.21.15.72:0/1905182557 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2d14004800 0x7f2d14004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.016+0000 7f2d30bb5640 1 --2- 172.21.15.72:0/1905182557 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2d140051c0 0x7f2d1400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.016+0000 7f2d30bb5640 1 -- 172.21.15.72:0/1905182557 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2c15cba0 con 0x7f2d140039b0 2024-04-15T15:15:49.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.016+0000 7f2d30bb5640 1 -- 172.21.15.72:0/1905182557 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1400c110 con 0x7f2d140051c0 2024-04-15T15:15:49.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.016+0000 7f2d30bb5640 1 -- 172.21.15.72:0/1905182557 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f2d1400c310 con 0x7f2d14004800 2024-04-15T15:15:49.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.016+0000 7f2d31bb7640 1 --2- 172.21.15.72:0/1905182557 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d140039b0 0x7f2d14003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.016+0000 7f2d323b8640 1 --2- 172.21.15.72:0/1905182557 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2d140051c0 0x7f2d1400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.016+0000 7f2d313b6640 1 --2- 172.21.15.72:0/1905182557 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2d14004800 0x7f2d14004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.016+0000 7f2d31bb7640 1 -- 172.21.15.72:0/1905182557 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2d14004800 msgr2=0x7f2d14004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:49.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.016+0000 7f2d31bb7640 1 --2- 172.21.15.72:0/1905182557 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2d14004800 0x7f2d14004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.016+0000 7f2d31bb7640 1 -- 172.21.15.72:0/1905182557 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2d140051c0 msgr2=0x7f2d1400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:49.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.016+0000 7f2d31bb7640 1 --2- 172.21.15.72:0/1905182557 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2d140051c0 0x7f2d1400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.016+0000 7f2d31bb7640 1 -- 172.21.15.72:0/1905182557 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d14003520 con 0x7f2d140039b0 2024-04-15T15:15:49.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.016+0000 7f2d323b8640 1 -- 172.21.15.72:0/1905182557 reap_dead start 2024-04-15T15:15:49.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.016+0000 7f2d313b6640 1 --2- 172.21.15.72:0/1905182557 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2d14004800 0x7f2d14004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:49.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.017+0000 7f2d31bb7640 1 --2- 172.21.15.72:0/1905182557 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d140039b0 0x7f2d14003dd0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c001e50 tx=0x7f2d14005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d12ac5ad598853b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:49.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.017+0000 7f2d30bb5640 1 -- 172.21.15.72:0/1905182557 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c001a70 con 0x7f2d140039b0 2024-04-15T15:15:49.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.017+0000 7f2d30bb5640 1 -- 172.21.15.72:0/1905182557 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d1c003b70 con 0x7f2d140039b0 2024-04-15T15:15:49.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.017+0000 7f2d30bb5640 1 -- 172.21.15.72:0/1905182557 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c0050d0 con 0x7f2d140039b0 2024-04-15T15:15:49.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.018+0000 7f2d33e1b640 1 -- 172.21.15.72:0/1905182557 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d140039b0 msgr2=0x7f2d14003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.018+0000 7f2d33e1b640 1 --2- 172.21.15.72:0/1905182557 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d140039b0 0x7f2d14003dd0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c001e50 tx=0x7f2d14005830 comp rx=0 tx=0).stop 2024-04-15T15:15:49.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.018+0000 7f2d33e1b640 1 -- 172.21.15.72:0/1905182557 shutdown_connections 2024-04-15T15:15:49.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.018+0000 7f2d33e1b640 1 --2- 172.21.15.72:0/1905182557 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d140039b0 0x7f2d14003dd0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.018+0000 7f2d33e1b640 1 -- 172.21.15.72:0/1905182557 >> 172.21.15.72:0/1905182557 conn(0x7f2d2c0b1dd0 msgr2=0x7f2d2c0b21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:49.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.018+0000 7f2d33e1b640 1 -- 172.21.15.72:0/1905182557 shutdown_connections 2024-04-15T15:15:49.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.018+0000 7f2d33e1b640 1 -- 172.21.15.72:0/1905182557 wait complete. 2024-04-15T15:15:49.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.019+0000 7f2d33e1b640 1 Processor -- start 2024-04-15T15:15:49.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.019+0000 7f2d33e1b640 1 -- start start 2024-04-15T15:15:49.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.020+0000 7f2d33e1b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d140039b0 0x7f2d2c1f5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.020+0000 7f2d33e1b640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2d2c14e700 0x7f2d2c14eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.020+0000 7f2d33e1b640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2d2c1f2430 0x7f2d2c1f28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.020+0000 7f2d33e1b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2c0cfab0 con 0x7f2d140039b0 2024-04-15T15:15:49.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.020+0000 7f2d33e1b640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2c06dc10 con 0x7f2d2c14e700 2024-04-15T15:15:49.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.020+0000 7f2d33e1b640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f2d2c0cdc60 con 0x7f2d2c1f2430 2024-04-15T15:15:49.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.020+0000 7f2d31bb7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d140039b0 0x7f2d2c1f5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.020+0000 7f2d31bb7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d140039b0 0x7f2d2c1f5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46378/0 (socket says 172.21.15.72:46378) 2024-04-15T15:15:49.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.020+0000 7f2d31bb7640 1 -- 172.21.15.72:0/1144834277 learned_addr learned my addr 172.21.15.72:0/1144834277 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:49.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.020+0000 7f2d313b6640 1 --2- 172.21.15.72:0/1144834277 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2d2c14e700 0x7f2d2c14eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.020+0000 7f2d323b8640 1 --2- 172.21.15.72:0/1144834277 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2d2c1f2430 0x7f2d2c1f28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.020+0000 7f2d31bb7640 1 -- 172.21.15.72:0/1144834277 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2d2c1f2430 msgr2=0x7f2d2c1f28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.020+0000 7f2d31bb7640 1 --2- 172.21.15.72:0/1144834277 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2d2c1f2430 0x7f2d2c1f28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.020+0000 7f2d31bb7640 1 -- 172.21.15.72:0/1144834277 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2d2c14e700 msgr2=0x7f2d2c14eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.020+0000 7f2d31bb7640 1 --2- 172.21.15.72:0/1144834277 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2d2c14e700 0x7f2d2c14eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.020+0000 7f2d31bb7640 1 -- 172.21.15.72:0/1144834277 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d2c14f470 con 0x7f2d140039b0 2024-04-15T15:15:49.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.020+0000 7f2d323b8640 1 --2- 172.21.15.72:0/1144834277 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2d2c1f2430 0x7f2d2c1f28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:49.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.020+0000 7f2d31bb7640 1 --2- 172.21.15.72:0/1144834277 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d140039b0 0x7f2d2c1f5040 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c005040 tx=0x7f2d1c0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:49.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.021+0000 7f2d2a7fc640 1 -- 172.21.15.72:0/1144834277 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c006b50 con 0x7f2d140039b0 2024-04-15T15:15:49.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.021+0000 7f2d2a7fc640 1 -- 172.21.15.72:0/1144834277 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d1c006cf0 con 0x7f2d140039b0 2024-04-15T15:15:49.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.021+0000 7f2d33e1b640 1 -- 172.21.15.72:0/1144834277 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d2c14f750 con 0x7f2d140039b0 2024-04-15T15:15:49.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.021+0000 7f2d2a7fc640 1 -- 172.21.15.72:0/1144834277 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1c007270 con 0x7f2d140039b0 2024-04-15T15:15:49.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.021+0000 7f2d33e1b640 1 -- 172.21.15.72:0/1144834277 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d2c14fd10 con 0x7f2d140039b0 2024-04-15T15:15:49.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.022+0000 7f2d33e1b640 1 -- 172.21.15.72:0/1144834277 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d2c0689e0 con 0x7f2d140039b0 2024-04-15T15:15:49.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.022+0000 7f2d2a7fc640 1 -- 172.21.15.72:0/1144834277 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2d1c003a30 con 0x7f2d140039b0 2024-04-15T15:15:49.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.023+0000 7f2d2a7fc640 1 --2- 172.21.15.72:0/1144834277 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2d1806c8f0 0x7f2d1806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.023+0000 7f2d313b6640 1 --2- 172.21.15.72:0/1144834277 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2d1806c8f0 0x7f2d1806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.023+0000 7f2d2a7fc640 1 -- 172.21.15.72:0/1144834277 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f2d1c0adaf0 con 0x7f2d140039b0 2024-04-15T15:15:49.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.023+0000 7f2d313b6640 1 --2- 172.21.15.72:0/1144834277 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2d1806c8f0 0x7f2d1806edb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c06ef60 tx=0x7f2d20008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:49.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.027+0000 7f2d2a7fc640 1 -- 172.21.15.72:0/1144834277 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2d1c033020 con 0x7f2d140039b0 2024-04-15T15:15:49.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.205+0000 7f2d33e1b640 1 -- 172.21.15.72:0/1144834277 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2d2c0d0a80 con 0x7f2d140039b0 2024-04-15T15:15:49.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.206+0000 7f2d2a7fc640 1 -- 172.21.15.72:0/1144834277 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18903 (secure 0 0 0) 0x7f2d1c07bbb0 con 0x7f2d140039b0 2024-04-15T15:15:49.208 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:49.208 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":26,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:15:32.421728+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:14:49.438366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:14:50.368379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:14:54.081157+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":1}]},{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:49.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.208+0000 7f2d33e1b640 1 -- 172.21.15.72:0/1144834277 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2d1806c8f0 msgr2=0x7f2d1806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.208+0000 7f2d33e1b640 1 --2- 172.21.15.72:0/1144834277 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2d1806c8f0 0x7f2d1806edb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c06ef60 tx=0x7f2d20008040 comp rx=0 tx=0).stop 2024-04-15T15:15:49.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.208+0000 7f2d33e1b640 1 -- 172.21.15.72:0/1144834277 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d140039b0 msgr2=0x7f2d2c1f5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.208+0000 7f2d33e1b640 1 --2- 172.21.15.72:0/1144834277 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d140039b0 0x7f2d2c1f5040 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c005040 tx=0x7f2d1c0014a0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.208+0000 7f2d33e1b640 1 -- 172.21.15.72:0/1144834277 shutdown_connections 2024-04-15T15:15:49.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.208+0000 7f2d33e1b640 1 --2- 172.21.15.72:0/1144834277 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2d1806c8f0 0x7f2d1806edb0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.208+0000 7f2d33e1b640 1 --2- 172.21.15.72:0/1144834277 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2d2c1f2430 0x7f2d2c1f28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.208+0000 7f2d33e1b640 1 --2- 172.21.15.72:0/1144834277 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2d2c14e700 0x7f2d2c14eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.208+0000 7f2d33e1b640 1 --2- 172.21.15.72:0/1144834277 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2d140039b0 0x7f2d2c1f5040 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.208+0000 7f2d33e1b640 1 -- 172.21.15.72:0/1144834277 >> 172.21.15.72:0/1144834277 conn(0x7f2d2c0b1dd0 msgr2=0x7f2d2c0cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:49.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.208+0000 7f2d33e1b640 1 -- 172.21.15.72:0/1144834277 shutdown_connections 2024-04-15T15:15:49.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.208+0000 7f2d33e1b640 1 -- 172.21.15.72:0/1144834277 wait complete. 2024-04-15T15:15:49.221 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:39.359102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:14:49.438366+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:14:50.368379+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:14:54.081157+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:15:49.222 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:49.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.327+0000 7f543a1db640 1 Processor -- start 2024-04-15T15:15:49.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.327+0000 7f543a1db640 1 -- start start 2024-04-15T15:15:49.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.327+0000 7f543a1db640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f543415e610 0x7f54340d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.327+0000 7f543a1db640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f543415e9f0 0x7f54340d3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.327+0000 7f543a1db640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54340d2320 con 0x7f543406c400 2024-04-15T15:15:49.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.328+0000 7f543a1db640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f54340d31d0 con 0x7f543415e610 2024-04-15T15:15:49.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.328+0000 7f543a1db640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f54340d25c0 con 0x7f543415e9f0 2024-04-15T15:15:49.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.328+0000 7f5433fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f543406c400 0x7f54340b7940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60146/0 (socket says 172.21.15.72:60146) 2024-04-15T15:15:49.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.328+0000 7f5433fff640 1 -- 172.21.15.72:0/1628734974 learned_addr learned my addr 172.21.15.72:0/1628734974 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:49.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.328+0000 7f54337fe640 1 --2- 172.21.15.72:0/1628734974 >> v2:172.21.15.175:3300/0 conn(0x7f543415e610 0x7f54340d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.328+0000 7f5432ffd640 1 --2- 172.21.15.72:0/1628734974 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f543415e9f0 0x7f54340d3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.328+0000 7f54327fc640 1 -- 172.21.15.72:0/1628734974 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2552972125 0 0) 0x7f54340d2320 con 0x7f543406c400 2024-04-15T15:15:49.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.328+0000 7f54327fc640 1 -- 172.21.15.72:0/1628734974 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f541c003520 con 0x7f543406c400 2024-04-15T15:15:49.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.328+0000 7f54337fe640 1 -- 172.21.15.72:0/1628734974 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f543415e9f0 msgr2=0x7f54340d3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:49.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.328+0000 7f54337fe640 1 --2- 172.21.15.72:0/1628734974 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f543415e9f0 0x7f54340d3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.328+0000 7f54337fe640 1 -- 172.21.15.72:0/1628734974 >> v1:172.21.15.72:6789/0 conn(0x7f543406c400 legacy=0x7f54340b7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.328+0000 7f54337fe640 1 -- 172.21.15.72:0/1628734974 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54340d28a0 con 0x7f543415e610 2024-04-15T15:15:49.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.329+0000 7f54337fe640 1 --2- 172.21.15.72:0/1628734974 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f543415e610 0x7f54340d0690 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f54240097a0 tx=0x7f542402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=443970009cebee39 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:49.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.329+0000 7f54327fc640 1 -- 172.21.15.72:0/1628734974 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5424034070 con 0x7f543415e610 2024-04-15T15:15:49.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.329+0000 7f54327fc640 1 -- 172.21.15.72:0/1628734974 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5424030610 con 0x7f543415e610 2024-04-15T15:15:49.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.329+0000 7f54327fc640 1 -- 172.21.15.72:0/1628734974 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54240309c0 con 0x7f543415e610 2024-04-15T15:15:49.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.330+0000 7f543a1db640 1 -- 172.21.15.72:0/1628734974 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f543415e610 msgr2=0x7f54340d0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.330+0000 7f543a1db640 1 --2- 172.21.15.72:0/1628734974 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f543415e610 0x7f54340d0690 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f54240097a0 tx=0x7f542402b960 comp rx=0 tx=0).stop 2024-04-15T15:15:49.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.330+0000 7f543a1db640 1 -- 172.21.15.72:0/1628734974 shutdown_connections 2024-04-15T15:15:49.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.330+0000 7f543a1db640 1 --2- 172.21.15.72:0/1628734974 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f543415e9f0 0x7f54340d3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.330+0000 7f543a1db640 1 --2- 172.21.15.72:0/1628734974 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f543415e610 0x7f54340d0690 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.330+0000 7f543a1db640 1 -- 172.21.15.72:0/1628734974 >> 172.21.15.72:0/1628734974 conn(0x7f54340d4750 msgr2=0x7f54340d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:49.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.330+0000 7f543a1db640 1 -- 172.21.15.72:0/1628734974 shutdown_connections 2024-04-15T15:15:49.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.330+0000 7f543a1db640 1 -- 172.21.15.72:0/1628734974 wait complete. 2024-04-15T15:15:49.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.331+0000 7f543a1db640 1 Processor -- start 2024-04-15T15:15:49.333 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.331+0000 7f543a1db640 1 -- start start 2024-04-15T15:15:49.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.332+0000 7f543a1db640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f543406c400 0x7f5434156370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.332+0000 7f543a1db640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f543415e610 0x7f54341616c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.335 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.332+0000 7f543a1db640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f543415e9f0 0x7f543415ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.332+0000 7f543a1db640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f543406d130 con 0x7f543406c400 2024-04-15T15:15:49.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.332+0000 7f543a1db640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f54340d2e60 con 0x7f543415e610 2024-04-15T15:15:49.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.332+0000 7f543a1db640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f54340d0bd0 con 0x7f543415e9f0 2024-04-15T15:15:49.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.332+0000 7f54337fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f543406c400 0x7f5434156370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.332+0000 7f54337fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f543406c400 0x7f5434156370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46380/0 (socket says 172.21.15.72:46380) 2024-04-15T15:15:49.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.332+0000 7f54337fe640 1 -- 172.21.15.72:0/1533564578 learned_addr learned my addr 172.21.15.72:0/1533564578 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:49.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.332+0000 7f5432ffd640 1 --2- 172.21.15.72:0/1533564578 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f543415e610 0x7f54341616c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.332+0000 7f5433fff640 1 --2- 172.21.15.72:0/1533564578 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f543415e9f0 0x7f543415ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.332+0000 7f54337fe640 1 -- 172.21.15.72:0/1533564578 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f543415e9f0 msgr2=0x7f543415ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.332+0000 7f54337fe640 1 --2- 172.21.15.72:0/1533564578 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f543415e9f0 0x7f543415ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.332+0000 7f54337fe640 1 -- 172.21.15.72:0/1533564578 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f543415e610 msgr2=0x7f54341616c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.332+0000 7f54337fe640 1 --2- 172.21.15.72:0/1533564578 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f543415e610 0x7f54341616c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.332+0000 7f54337fe640 1 -- 172.21.15.72:0/1533564578 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5434159930 con 0x7f543406c400 2024-04-15T15:15:49.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.332+0000 7f54337fe640 1 --2- 172.21.15.72:0/1533564578 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f543406c400 0x7f5434156370 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f54240098c0 tx=0x7f54240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:49.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.333+0000 7f541bfff640 1 -- 172.21.15.72:0/1533564578 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5424034040 con 0x7f543406c400 2024-04-15T15:15:49.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.333+0000 7f541bfff640 1 -- 172.21.15.72:0/1533564578 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5424030410 con 0x7f543406c400 2024-04-15T15:15:49.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.333+0000 7f543a1db640 1 -- 172.21.15.72:0/1533564578 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5434159bd0 con 0x7f543406c400 2024-04-15T15:15:49.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.333+0000 7f541bfff640 1 -- 172.21.15.72:0/1533564578 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5424006e10 con 0x7f543406c400 2024-04-15T15:15:49.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.333+0000 7f543a1db640 1 -- 172.21.15.72:0/1533564578 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f543415a190 con 0x7f543406c400 2024-04-15T15:15:49.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.334+0000 7f543a1db640 1 -- 172.21.15.72:0/1533564578 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5434075c50 con 0x7f543406c400 2024-04-15T15:15:49.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.334+0000 7f541bfff640 1 -- 172.21.15.72:0/1533564578 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5424037070 con 0x7f543406c400 2024-04-15T15:15:49.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.335+0000 7f541bfff640 1 --2- 172.21.15.72:0/1533564578 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f542006c8f0 0x7f542006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.335+0000 7f541bfff640 1 -- 172.21.15.72:0/1533564578 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f54240adcd0 con 0x7f543406c400 2024-04-15T15:15:49.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.335+0000 7f5432ffd640 1 --2- 172.21.15.72:0/1533564578 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f542006c8f0 0x7f542006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.335+0000 7f5432ffd640 1 --2- 172.21.15.72:0/1533564578 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f542006c8f0 0x7f542006edb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f54340cac90 tx=0x7f5428009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:49.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.339+0000 7f541bfff640 1 -- 172.21.15.72:0/1533564578 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f54240305e0 con 0x7f543406c400 2024-04-15T15:15:49.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.517+0000 7f543a1db640 1 -- 172.21.15.72:0/1533564578 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f54340d3dd0 con 0x7f543406c400 2024-04-15T15:15:49.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.517+0000 7f541bfff640 1 -- 172.21.15.72:0/1533564578 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f542407b6b0 con 0x7f543406c400 2024-04-15T15:15:49.519 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-15T15:15:49.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.519+0000 7f543a1db640 1 -- 172.21.15.72:0/1533564578 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f542006c8f0 msgr2=0x7f542006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.519+0000 7f543a1db640 1 --2- 172.21.15.72:0/1533564578 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f542006c8f0 0x7f542006edb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f54340cac90 tx=0x7f5428009190 comp rx=0 tx=0).stop 2024-04-15T15:15:49.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.519+0000 7f543a1db640 1 -- 172.21.15.72:0/1533564578 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f543406c400 msgr2=0x7f5434156370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.519+0000 7f543a1db640 1 --2- 172.21.15.72:0/1533564578 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f543406c400 0x7f5434156370 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f54240098c0 tx=0x7f54240046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.519+0000 7f543a1db640 1 -- 172.21.15.72:0/1533564578 shutdown_connections 2024-04-15T15:15:49.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.519+0000 7f543a1db640 1 --2- 172.21.15.72:0/1533564578 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f542006c8f0 0x7f542006edb0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.519+0000 7f543a1db640 1 --2- 172.21.15.72:0/1533564578 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f543415e9f0 0x7f543415ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.519+0000 7f543a1db640 1 --2- 172.21.15.72:0/1533564578 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f543415e610 0x7f54341616c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.519+0000 7f543a1db640 1 --2- 172.21.15.72:0/1533564578 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f543406c400 0x7f5434156370 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.519+0000 7f543a1db640 1 -- 172.21.15.72:0/1533564578 >> 172.21.15.72:0/1533564578 conn(0x7f54340d4750 msgr2=0x7f543415d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:49.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.519+0000 7f543a1db640 1 -- 172.21.15.72:0/1533564578 shutdown_connections 2024-04-15T15:15:49.522 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.519+0000 7f543a1db640 1 -- 172.21.15.72:0/1533564578 wait complete. 2024-04-15T15:15:49.532 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:15:49.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.635+0000 7f40143cd640 1 Processor -- start 2024-04-15T15:15:49.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.635+0000 7f40143cd640 1 -- start start 2024-04-15T15:15:49.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.636+0000 7f40143cd640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f400c15c650 0x7f400c0b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.636+0000 7f40143cd640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f400c0d0a60 0x7f400c0d0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.636+0000 7f40143cd640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f400c0d02d0 con 0x7f400c0b36f0 2024-04-15T15:15:49.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.636+0000 7f40143cd640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f400c15ca30 con 0x7f400c15c650 2024-04-15T15:15:49.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.636+0000 7f40143cd640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f400c0b26b0 con 0x7f400c0d0a60 2024-04-15T15:15:49.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.637+0000 7f4012169640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f400c15c650 0x7f400c0b1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.637+0000 7f401296a640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f400c0d0a60 0x7f400c0d0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.637+0000 7f4011968640 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f400c0b36f0 0x7f400c0b5290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.72:42464/0 (socket says 172.21.15.72:42464) 2024-04-15T15:15:49.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.637+0000 7f4011968640 1 -- 172.21.15.72:0/3433160351 learned_addr learned my addr 172.21.15.72:0/3433160351 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:49.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.637+0000 7f401296a640 1 -- 172.21.15.72:0/3433160351 >> v2:172.21.15.175:3300/0 conn(0x7f400c15c650 msgr2=0x7f400c0b1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:49.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.637+0000 7f401296a640 1 --2- 172.21.15.72:0/3433160351 >> v2:172.21.15.175:3300/0 conn(0x7f400c15c650 0x7f400c0b1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.637+0000 7f401296a640 1 -- 172.21.15.72:0/3433160351 >> v1:172.21.15.175:6789/0 conn(0x7f400c0b36f0 legacy=0x7f400c0b5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.637+0000 7f401296a640 1 -- 172.21.15.72:0/3433160351 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f400c0b2990 con 0x7f400c0d0a60 2024-04-15T15:15:49.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.637+0000 7f4011167640 1 -- 172.21.15.72:0/3433160351 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1025786129 0 0) 0x7f400c0d02d0 con 0x7f400c0b36f0 2024-04-15T15:15:49.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.637+0000 7f4012169640 1 --2- 172.21.15.72:0/3433160351 >> v2:172.21.15.175:3300/0 conn(0x7f400c15c650 0x7f400c0b1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:49.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.638+0000 7f401296a640 1 --2- 172.21.15.72:0/3433160351 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f400c0d0a60 0x7f400c0d0ed0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4004009d90 tx=0x7f400402beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76cb20d9c888cf1d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:49.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.638+0000 7f4011167640 1 -- 172.21.15.72:0/3433160351 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4004039070 con 0x7f400c0d0a60 2024-04-15T15:15:49.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.638+0000 7f4011167640 1 -- 172.21.15.72:0/3433160351 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f400402cc60 con 0x7f400c0d0a60 2024-04-15T15:15:49.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.638+0000 7f4011167640 1 -- 172.21.15.72:0/3433160351 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4004034660 con 0x7f400c0d0a60 2024-04-15T15:15:49.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.639+0000 7f40143cd640 1 -- 172.21.15.72:0/3433160351 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f400c0d0a60 msgr2=0x7f400c0d0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.639+0000 7f40143cd640 1 --2- 172.21.15.72:0/3433160351 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f400c0d0a60 0x7f400c0d0ed0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4004009d90 tx=0x7f400402beb0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.639+0000 7f40143cd640 1 -- 172.21.15.72:0/3433160351 shutdown_connections 2024-04-15T15:15:49.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.639+0000 7f40143cd640 1 --2- 172.21.15.72:0/3433160351 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f400c0d0a60 0x7f400c0d0ed0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.639+0000 7f40143cd640 1 --2- 172.21.15.72:0/3433160351 >> v2:172.21.15.175:3300/0 conn(0x7f400c15c650 0x7f400c0b1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.639+0000 7f40143cd640 1 -- 172.21.15.72:0/3433160351 >> 172.21.15.72:0/3433160351 conn(0x7f400c0ceec0 msgr2=0x7f400c0cf2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:49.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.639+0000 7f40143cd640 1 -- 172.21.15.72:0/3433160351 shutdown_connections 2024-04-15T15:15:49.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.639+0000 7f40143cd640 1 -- 172.21.15.72:0/3433160351 wait complete. 2024-04-15T15:15:49.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.640+0000 7f40143cd640 1 Processor -- start 2024-04-15T15:15:49.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.640+0000 7f40143cd640 1 -- start start 2024-04-15T15:15:49.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.641+0000 7f40143cd640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f400c0b36f0 0x7f400c158c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.641+0000 7f40143cd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f400c0d0a60 0x7f400c0c6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.641+0000 7f40143cd640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f400c15c650 0x7f400c1f27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.641+0000 7f40143cd640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f400c0b59a0 con 0x7f400c0d0a60 2024-04-15T15:15:49.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.641+0000 7f40143cd640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f400c0b2c70 con 0x7f400c0b36f0 2024-04-15T15:15:49.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.641+0000 7f40143cd640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f400c0b2280 con 0x7f400c15c650 2024-04-15T15:15:49.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.641+0000 7f4011968640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f400c0d0a60 0x7f400c0c6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.641+0000 7f4011968640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f400c0d0a60 0x7f400c0c6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46392/0 (socket says 172.21.15.72:46392) 2024-04-15T15:15:49.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.641+0000 7f4011968640 1 -- 172.21.15.72:0/422971607 learned_addr learned my addr 172.21.15.72:0/422971607 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:49.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.641+0000 7f401296a640 1 --2- 172.21.15.72:0/422971607 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f400c15c650 0x7f400c1f27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.641+0000 7f4012169640 1 --2- 172.21.15.72:0/422971607 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f400c0b36f0 0x7f400c158c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.641+0000 7f4011968640 1 -- 172.21.15.72:0/422971607 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f400c15c650 msgr2=0x7f400c1f27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.641+0000 7f4011968640 1 --2- 172.21.15.72:0/422971607 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f400c15c650 0x7f400c1f27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.641+0000 7f4011968640 1 -- 172.21.15.72:0/422971607 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f400c0b36f0 msgr2=0x7f400c158c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.641+0000 7f4011968640 1 --2- 172.21.15.72:0/422971607 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f400c0b36f0 0x7f400c158c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.641+0000 7f4011968640 1 -- 172.21.15.72:0/422971607 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f400c0c6ec0 con 0x7f400c0d0a60 2024-04-15T15:15:49.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.641+0000 7f4011968640 1 --2- 172.21.15.72:0/422971607 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f400c0d0a60 0x7f400c0c6980 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f400000c430 tx=0x7f400000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:49.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.641+0000 7f400affd640 1 -- 172.21.15.72:0/422971607 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40000074e0 con 0x7f400c0d0a60 2024-04-15T15:15:49.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.641+0000 7f400affd640 1 -- 172.21.15.72:0/422971607 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40000047e0 con 0x7f400c0d0a60 2024-04-15T15:15:49.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.641+0000 7f40143cd640 1 -- 172.21.15.72:0/422971607 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f400c15f610 con 0x7f400c0d0a60 2024-04-15T15:15:49.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.642+0000 7f400affd640 1 -- 172.21.15.72:0/422971607 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4000004bd0 con 0x7f400c0d0a60 2024-04-15T15:15:49.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.642+0000 7f40143cd640 1 -- 172.21.15.72:0/422971607 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f400c15deb0 con 0x7f400c0d0a60 2024-04-15T15:15:49.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.643+0000 7f40143cd640 1 -- 172.21.15.72:0/422971607 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f400c075c50 con 0x7f400c0d0a60 2024-04-15T15:15:49.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.643+0000 7f400affd640 1 -- 172.21.15.72:0/422971607 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4000011070 con 0x7f400c0d0a60 2024-04-15T15:15:49.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.643+0000 7f400affd640 1 --2- 172.21.15.72:0/422971607 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3ff806c8f0 0x7f3ff806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.644+0000 7f4012169640 1 --2- 172.21.15.72:0/422971607 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3ff806c8f0 0x7f3ff806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.644+0000 7f400affd640 1 -- 172.21.15.72:0/422971607 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f400008b990 con 0x7f400c0d0a60 2024-04-15T15:15:49.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.644+0000 7f4012169640 1 --2- 172.21.15.72:0/422971607 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3ff806c8f0 0x7f3ff806edb0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc009560 tx=0x7f3ffc009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:49.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.648+0000 7f400affd640 1 -- 172.21.15.72:0/422971607 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4000059400 con 0x7f400c0d0a60 2024-04-15T15:15:49.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.823+0000 7f40143cd640 1 -- 172.21.15.72:0/422971607 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f400c0ceb50 con 0x7f400c0d0a60 2024-04-15T15:15:49.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.823+0000 7f400affd640 1 -- 172.21.15.72:0/422971607 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f400005d700 con 0x7f400c0d0a60 2024-04-15T15:15:49.825 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:49.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.825+0000 7f40143cd640 1 -- 172.21.15.72:0/422971607 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3ff806c8f0 msgr2=0x7f3ff806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.825+0000 7f40143cd640 1 --2- 172.21.15.72:0/422971607 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3ff806c8f0 0x7f3ff806edb0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3ffc009560 tx=0x7f3ffc009190 comp rx=0 tx=0).stop 2024-04-15T15:15:49.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.825+0000 7f40143cd640 1 -- 172.21.15.72:0/422971607 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f400c0d0a60 msgr2=0x7f400c0c6980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.825+0000 7f40143cd640 1 --2- 172.21.15.72:0/422971607 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f400c0d0a60 0x7f400c0c6980 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f400000c430 tx=0x7f400000c8f0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.825+0000 7f40143cd640 1 -- 172.21.15.72:0/422971607 shutdown_connections 2024-04-15T15:15:49.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.825+0000 7f40143cd640 1 --2- 172.21.15.72:0/422971607 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3ff806c8f0 0x7f3ff806edb0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.825+0000 7f40143cd640 1 --2- 172.21.15.72:0/422971607 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f400c15c650 0x7f400c1f27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.825+0000 7f40143cd640 1 --2- 172.21.15.72:0/422971607 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f400c0d0a60 0x7f400c0c6980 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.825+0000 7f40143cd640 1 --2- 172.21.15.72:0/422971607 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f400c0b36f0 0x7f400c158c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.825+0000 7f40143cd640 1 -- 172.21.15.72:0/422971607 >> 172.21.15.72:0/422971607 conn(0x7f400c0ceec0 msgr2=0x7f400c15b990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:49.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.825+0000 7f40143cd640 1 -- 172.21.15.72:0/422971607 shutdown_connections 2024-04-15T15:15:49.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.825+0000 7f40143cd640 1 -- 172.21.15.72:0/422971607 wait complete. 2024-04-15T15:15:49.838 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:15:49.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.943+0000 7f100610e640 1 Processor -- start 2024-04-15T15:15:49.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.943+0000 7f100610e640 1 -- start start 2024-04-15T15:15:49.954 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.944+0000 7f100610e640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f100015c630 0x7f100006cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.944+0000 7f100610e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f10000ccb90 0x7f10000ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.944+0000 7f100610e640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f100006bd50 con 0x7f100006e620 2024-04-15T15:15:49.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.944+0000 7f100610e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f100006d2f0 con 0x7f10000ccb90 2024-04-15T15:15:49.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.944+0000 7f100610e640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f10000cd530 con 0x7f100015c630 2024-04-15T15:15:49.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.944+0000 7f0ffeffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f10000ccb90 0x7f10000ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.955 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.944+0000 7f0ffeffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f10000ccb90 0x7f10000ccff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46406/0 (socket says 172.21.15.72:46406) 2024-04-15T15:15:49.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.944+0000 7f0ffeffd640 1 -- 172.21.15.72:0/964617486 learned_addr learned my addr 172.21.15.72:0/964617486 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:49.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.944+0000 7f0fff7fe640 1 --2- 172.21.15.72:0/964617486 >> v2:172.21.15.175:3300/0 conn(0x7f100015c630 0x7f100006cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.945+0000 7f0ffeffd640 1 -- 172.21.15.72:0/964617486 >> v2:172.21.15.175:3300/0 conn(0x7f100015c630 msgr2=0x7f100006cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:49.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.945+0000 7f0ffeffd640 1 --2- 172.21.15.72:0/964617486 >> v2:172.21.15.175:3300/0 conn(0x7f100015c630 0x7f100006cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.945+0000 7f0ffeffd640 1 -- 172.21.15.72:0/964617486 >> v1:172.21.15.175:6789/0 conn(0x7f100006e620 legacy=0x7f100006ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.956 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.945+0000 7f0ffeffd640 1 -- 172.21.15.72:0/964617486 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f100006bff0 con 0x7f10000ccb90 2024-04-15T15:15:49.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.945+0000 7f0ffeffd640 1 --2- 172.21.15.72:0/964617486 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10000ccb90 0x7f10000ccff0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f100006fc40 tx=0x7f0ff402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee007fd28d91709d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:49.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.945+0000 7f0ffe7fc640 1 -- 172.21.15.72:0/964617486 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff4034070 con 0x7f10000ccb90 2024-04-15T15:15:49.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.945+0000 7f0ffe7fc640 1 -- 172.21.15.72:0/964617486 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ff4030860 con 0x7f10000ccb90 2024-04-15T15:15:49.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.945+0000 7f0ffe7fc640 1 -- 172.21.15.72:0/964617486 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff4030c30 con 0x7f10000ccb90 2024-04-15T15:15:49.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.946+0000 7f100610e640 1 -- 172.21.15.72:0/964617486 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10000ccb90 msgr2=0x7f10000ccff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.957 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.946+0000 7f100610e640 1 --2- 172.21.15.72:0/964617486 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10000ccb90 0x7f10000ccff0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f100006fc40 tx=0x7f0ff402b9a0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.946+0000 7f100610e640 1 -- 172.21.15.72:0/964617486 shutdown_connections 2024-04-15T15:15:49.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.946+0000 7f100610e640 1 --2- 172.21.15.72:0/964617486 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10000ccb90 0x7f10000ccff0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.946+0000 7f100610e640 1 --2- 172.21.15.72:0/964617486 >> v2:172.21.15.175:3300/0 conn(0x7f100015c630 0x7f100006cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.946+0000 7f100610e640 1 -- 172.21.15.72:0/964617486 >> 172.21.15.72:0/964617486 conn(0x7f10000b31d0 msgr2=0x7f10000b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:49.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.946+0000 7f100610e640 1 -- 172.21.15.72:0/964617486 shutdown_connections 2024-04-15T15:15:49.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.946+0000 7f100610e640 1 -- 172.21.15.72:0/964617486 wait complete. 2024-04-15T15:15:49.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.947+0000 7f100610e640 1 Processor -- start 2024-04-15T15:15:49.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.947+0000 7f100610e640 1 -- start start 2024-04-15T15:15:49.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.948+0000 7f100610e640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f100006e620 0x7f100009b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.958 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.948+0000 7f100610e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10000ccb90 0x7f10000c5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.948+0000 7f100610e640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f100015c630 0x7f1000155d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.948+0000 7f100610e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f100006c920 con 0x7f10000ccb90 2024-04-15T15:15:49.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.948+0000 7f100610e640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f100006c5b0 con 0x7f100015c630 2024-04-15T15:15:49.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.948+0000 7f100610e640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f100015ca10 con 0x7f100006e620 2024-04-15T15:15:49.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.948+0000 7f0ffeffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10000ccb90 0x7f10000c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.948+0000 7f0ffeffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10000ccb90 0x7f10000c5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46420/0 (socket says 172.21.15.72:46420) 2024-04-15T15:15:49.959 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.948+0000 7f0ffeffd640 1 -- 172.21.15.72:0/3968591542 learned_addr learned my addr 172.21.15.72:0/3968591542 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:49.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.948+0000 7f0fff7fe640 1 --2- 172.21.15.72:0/3968591542 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f100006e620 0x7f100009b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.948+0000 7f0ffffff640 1 --2- 172.21.15.72:0/3968591542 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f100015c630 0x7f1000155d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.949+0000 7f0ffeffd640 1 -- 172.21.15.72:0/3968591542 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f100006e620 msgr2=0x7f100009b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.949+0000 7f0ffeffd640 1 --2- 172.21.15.72:0/3968591542 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f100006e620 0x7f100009b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.949+0000 7f0ffeffd640 1 -- 172.21.15.72:0/3968591542 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f100015c630 msgr2=0x7f1000155d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.949+0000 7f0ffeffd640 1 --2- 172.21.15.72:0/3968591542 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f100015c630 0x7f1000155d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.949+0000 7f0ffeffd640 1 -- 172.21.15.72:0/3968591542 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f100009c360 con 0x7f10000ccb90 2024-04-15T15:15:49.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.949+0000 7f0fff7fe640 1 --2- 172.21.15.72:0/3968591542 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f100006e620 0x7f100009b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:49.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.949+0000 7f0ffeffd640 1 --2- 172.21.15.72:0/3968591542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10000ccb90 0x7f10000c5040 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0ff402b9e0 tx=0x7f0ff4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:49.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.949+0000 7f0fe7fff640 1 -- 172.21.15.72:0/3968591542 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff4048070 con 0x7f10000ccb90 2024-04-15T15:15:49.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.949+0000 7f0fe7fff640 1 -- 172.21.15.72:0/3968591542 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ff4030660 con 0x7f10000ccb90 2024-04-15T15:15:49.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.949+0000 7f100610e640 1 -- 172.21.15.72:0/3968591542 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f100009c640 con 0x7f10000ccb90 2024-04-15T15:15:49.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.949+0000 7f0fe7fff640 1 -- 172.21.15.72:0/3968591542 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff4034040 con 0x7f10000ccb90 2024-04-15T15:15:49.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.949+0000 7f100610e640 1 -- 172.21.15.72:0/3968591542 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f100009cc00 con 0x7f10000ccb90 2024-04-15T15:15:49.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.950+0000 7f100610e640 1 -- 172.21.15.72:0/3968591542 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10000b3ae0 con 0x7f10000ccb90 2024-04-15T15:15:49.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.950+0000 7f0fe7fff640 1 -- 172.21.15.72:0/3968591542 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0ff404b070 con 0x7f10000ccb90 2024-04-15T15:15:49.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.951+0000 7f0fe7fff640 1 --2- 172.21.15.72:0/3968591542 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0fec06c8f0 0x7f0fec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.951+0000 7f0fe7fff640 1 -- 172.21.15.72:0/3968591542 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f0ff40aded0 con 0x7f10000ccb90 2024-04-15T15:15:49.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.951+0000 7f0fff7fe640 1 --2- 172.21.15.72:0/3968591542 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0fec06c8f0 0x7f0fec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.951+0000 7f0fff7fe640 1 --2- 172.21.15.72:0/3968591542 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0fec06c8f0 0x7f0fec06edb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0009fd0 tx=0x7f0ff000b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:49.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:49.955+0000 7f0fe7fff640 1 -- 172.21.15.72:0/3968591542 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f10000b3ae0 con 0x7f10000ccb90 2024-04-15T15:15:50.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.130+0000 7f100610e640 1 -- 172.21.15.72:0/3968591542 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1000075c50 con 0x7f10000ccb90 2024-04-15T15:15:50.132 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.130+0000 7f0fe7fff640 1 -- 172.21.15.72:0/3968591542 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f0ff407bf00 con 0x7f10000ccb90 2024-04-15T15:15:50.132 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-04-15T15:15:50.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.132+0000 7f100610e640 1 -- 172.21.15.72:0/3968591542 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0fec06c8f0 msgr2=0x7f0fec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.132+0000 7f100610e640 1 --2- 172.21.15.72:0/3968591542 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0fec06c8f0 0x7f0fec06edb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0009fd0 tx=0x7f0ff000b040 comp rx=0 tx=0).stop 2024-04-15T15:15:50.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.132+0000 7f100610e640 1 -- 172.21.15.72:0/3968591542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10000ccb90 msgr2=0x7f10000c5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.132+0000 7f100610e640 1 --2- 172.21.15.72:0/3968591542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10000ccb90 0x7f10000c5040 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0ff402b9e0 tx=0x7f0ff4004730 comp rx=0 tx=0).stop 2024-04-15T15:15:50.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.132+0000 7f100610e640 1 -- 172.21.15.72:0/3968591542 shutdown_connections 2024-04-15T15:15:50.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.132+0000 7f100610e640 1 --2- 172.21.15.72:0/3968591542 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0fec06c8f0 0x7f0fec06edb0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.132+0000 7f100610e640 1 --2- 172.21.15.72:0/3968591542 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f100015c630 0x7f1000155d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.132+0000 7f100610e640 1 --2- 172.21.15.72:0/3968591542 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10000ccb90 0x7f10000c5040 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.132+0000 7f100610e640 1 --2- 172.21.15.72:0/3968591542 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f100006e620 0x7f100009b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.132+0000 7f100610e640 1 -- 172.21.15.72:0/3968591542 >> 172.21.15.72:0/3968591542 conn(0x7f10000b31d0 msgr2=0x7f1000157ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:50.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.132+0000 7f100610e640 1 -- 172.21.15.72:0/3968591542 shutdown_connections 2024-04-15T15:15:50.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.132+0000 7f100610e640 1 -- 172.21.15.72:0/3968591542 wait complete. 2024-04-15T15:15:50.146 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:15:50.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.251+0000 7f3a81f8f640 1 Processor -- start 2024-04-15T15:15:50.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.251+0000 7f3a81f8f640 1 -- start start 2024-04-15T15:15:50.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.251+0000 7f3a81f8f640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3a7c15e610 0x7f3a7c0d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.251+0000 7f3a81f8f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3a7c15e9f0 0x7f3a7c0d3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.251+0000 7f3a81f8f640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a7c0d2320 con 0x7f3a7c06c400 2024-04-15T15:15:50.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.251+0000 7f3a81f8f640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f3a7c0d31d0 con 0x7f3a7c15e9f0 2024-04-15T15:15:50.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.251+0000 7f3a81f8f640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f3a7c0d25c0 con 0x7f3a7c15e610 2024-04-15T15:15:50.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.252+0000 7f3a73fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3a7c15e9f0 0x7f3a7c0d3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.252+0000 7f3a7bfff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f3a7c06c400 0x7f3a7c0b7940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60162/0 (socket says 172.21.15.72:60162) 2024-04-15T15:15:50.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.252+0000 7f3a7bfff640 1 -- 172.21.15.72:0/1681267642 learned_addr learned my addr 172.21.15.72:0/1681267642 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:50.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.252+0000 7f3a73fff640 1 --2- 172.21.15.72:0/1681267642 >> v2:172.21.15.72:3300/0 conn(0x7f3a7c15e9f0 0x7f3a7c0d3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46426/0 (socket says 172.21.15.72:46426) 2024-04-15T15:15:50.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.252+0000 7f3a7b7fe640 1 --2- 172.21.15.72:0/1681267642 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3a7c15e610 0x7f3a7c0d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.252+0000 7f3a73fff640 1 -- 172.21.15.72:0/1681267642 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3a7c15e610 msgr2=0x7f3a7c0d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:50.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.252+0000 7f3a73fff640 1 --2- 172.21.15.72:0/1681267642 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3a7c15e610 0x7f3a7c0d0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.252+0000 7f3a73fff640 1 -- 172.21.15.72:0/1681267642 >> v1:172.21.15.72:6789/0 conn(0x7f3a7c06c400 legacy=0x7f3a7c0b7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.252+0000 7f3a73fff640 1 -- 172.21.15.72:0/1681267642 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a7c0d28a0 con 0x7f3a7c15e9f0 2024-04-15T15:15:50.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.252+0000 7f3a7affd640 1 -- 172.21.15.72:0/1681267642 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2677762238 0 0) 0x7f3a7c0d2320 con 0x7f3a7c06c400 2024-04-15T15:15:50.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.252+0000 7f3a7b7fe640 1 --2- 172.21.15.72:0/1681267642 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3a7c15e610 0x7f3a7c0d0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:50.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.253+0000 7f3a73fff640 1 --2- 172.21.15.72:0/1681267642 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a7c15e9f0 0x7f3a7c0d3660 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f3a640097a0 tx=0x7f3a6402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28c011af90929854 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:50.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.253+0000 7f3a7affd640 1 -- 172.21.15.72:0/1681267642 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a64034070 con 0x7f3a7c15e9f0 2024-04-15T15:15:50.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.253+0000 7f3a7affd640 1 -- 172.21.15.72:0/1681267642 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a64030610 con 0x7f3a7c15e9f0 2024-04-15T15:15:50.255 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.253+0000 7f3a7affd640 1 -- 172.21.15.72:0/1681267642 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a640309e0 con 0x7f3a7c15e9f0 2024-04-15T15:15:50.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.253+0000 7f3a81f8f640 1 -- 172.21.15.72:0/1681267642 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a7c15e9f0 msgr2=0x7f3a7c0d3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.253+0000 7f3a81f8f640 1 --2- 172.21.15.72:0/1681267642 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a7c15e9f0 0x7f3a7c0d3660 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f3a640097a0 tx=0x7f3a6402b960 comp rx=0 tx=0).stop 2024-04-15T15:15:50.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.253+0000 7f3a81f8f640 1 -- 172.21.15.72:0/1681267642 shutdown_connections 2024-04-15T15:15:50.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.253+0000 7f3a81f8f640 1 --2- 172.21.15.72:0/1681267642 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a7c15e9f0 0x7f3a7c0d3660 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.253+0000 7f3a81f8f640 1 --2- 172.21.15.72:0/1681267642 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3a7c15e610 0x7f3a7c0d0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.254+0000 7f3a81f8f640 1 -- 172.21.15.72:0/1681267642 >> 172.21.15.72:0/1681267642 conn(0x7f3a7c0d4750 msgr2=0x7f3a7c0d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:50.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.254+0000 7f3a81f8f640 1 -- 172.21.15.72:0/1681267642 shutdown_connections 2024-04-15T15:15:50.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.254+0000 7f3a81f8f640 1 -- 172.21.15.72:0/1681267642 wait complete. 2024-04-15T15:15:50.259 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.255+0000 7f3a81f8f640 1 Processor -- start 2024-04-15T15:15:50.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.255+0000 7f3a81f8f640 1 -- start start 2024-04-15T15:15:50.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.255+0000 7f3a81f8f640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3a7c06c400 0x7f3a7c152190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.255+0000 7f3a81f8f640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a7c15e610 0x7f3a7c15f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.255+0000 7f3a81f8f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a7c15e9f0 0x7f3a7c14cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.255+0000 7f3a81f8f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7c06d130 con 0x7f3a7c15e9f0 2024-04-15T15:15:50.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.255+0000 7f3a81f8f640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7c0d2e60 con 0x7f3a7c15e610 2024-04-15T15:15:50.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.255+0000 7f3a81f8f640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f3a7c0d0bd0 con 0x7f3a7c06c400 2024-04-15T15:15:50.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.255+0000 7f3a7bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a7c15e9f0 0x7f3a7c14cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.256+0000 7f3a7bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a7c15e9f0 0x7f3a7c14cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46438/0 (socket says 172.21.15.72:46438) 2024-04-15T15:15:50.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.256+0000 7f3a7bfff640 1 -- 172.21.15.72:0/2282083225 learned_addr learned my addr 172.21.15.72:0/2282083225 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:50.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.256+0000 7f3a7bfff640 1 -- 172.21.15.72:0/2282083225 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3a7c06c400 msgr2=0x7f3a7c152190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.256+0000 7f3a7bfff640 1 --2- 172.21.15.72:0/2282083225 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3a7c06c400 0x7f3a7c152190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.256+0000 7f3a7bfff640 1 -- 172.21.15.72:0/2282083225 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a7c15e610 msgr2=0x7f3a7c15f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.256+0000 7f3a7bfff640 1 --2- 172.21.15.72:0/2282083225 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a7c15e610 0x7f3a7c15f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.260 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.256+0000 7f3a7bfff640 1 -- 172.21.15.72:0/2282083225 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a7c0cdc30 con 0x7f3a7c15e9f0 2024-04-15T15:15:50.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.256+0000 7f3a7bfff640 1 --2- 172.21.15.72:0/2282083225 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a7c15e9f0 0x7f3a7c14cd90 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f3a7400c700 tx=0x7f3a7400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:50.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.256+0000 7f3a78ff9640 1 -- 172.21.15.72:0/2282083225 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a74018070 con 0x7f3a7c15e9f0 2024-04-15T15:15:50.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.256+0000 7f3a78ff9640 1 -- 172.21.15.72:0/2282083225 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a7400ce10 con 0x7f3a7c15e9f0 2024-04-15T15:15:50.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.256+0000 7f3a81f8f640 1 -- 172.21.15.72:0/2282083225 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a7c0cdf10 con 0x7f3a7c15e9f0 2024-04-15T15:15:50.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.256+0000 7f3a78ff9640 1 -- 172.21.15.72:0/2282083225 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a7400a880 con 0x7f3a7c15e9f0 2024-04-15T15:15:50.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.256+0000 7f3a81f8f640 1 -- 172.21.15.72:0/2282083225 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a7c151c70 con 0x7f3a7c15e9f0 2024-04-15T15:15:50.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.257+0000 7f3a81f8f640 1 -- 172.21.15.72:0/2282083225 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a7c1515a0 con 0x7f3a7c15e9f0 2024-04-15T15:15:50.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.258+0000 7f3a78ff9640 1 -- 172.21.15.72:0/2282083225 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3a7401b070 con 0x7f3a7c15e9f0 2024-04-15T15:15:50.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.258+0000 7f3a78ff9640 1 --2- 172.21.15.72:0/2282083225 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3a5c06c920 0x7f3a5c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.258+0000 7f3a78ff9640 1 -- 172.21.15.72:0/2282083225 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f3a7408b4b0 con 0x7f3a7c15e9f0 2024-04-15T15:15:50.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.258+0000 7f3a7b7fe640 1 --2- 172.21.15.72:0/2282083225 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3a5c06c920 0x7f3a5c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.258+0000 7f3a7b7fe640 1 --2- 172.21.15.72:0/2282083225 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3a5c06c920 0x7f3a5c06ede0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c006fd0 tx=0x7f3a6c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:50.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.262+0000 7f3a78ff9640 1 -- 172.21.15.72:0/2282083225 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3a740594e0 con 0x7f3a7c15e9f0 2024-04-15T15:15:50.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.437+0000 7f3a81f8f640 1 -- 172.21.15.72:0/2282083225 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f3a7c075c50 con 0x7f3a7c15e9f0 2024-04-15T15:15:50.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.437+0000 7f3a78ff9640 1 -- 172.21.15.72:0/2282083225 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f3a74058de0 con 0x7f3a7c15e9f0 2024-04-15T15:15:50.439 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:50.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.439+0000 7f3a81f8f640 1 -- 172.21.15.72:0/2282083225 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3a5c06c920 msgr2=0x7f3a5c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.439+0000 7f3a81f8f640 1 --2- 172.21.15.72:0/2282083225 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3a5c06c920 0x7f3a5c06ede0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c006fd0 tx=0x7f3a6c008040 comp rx=0 tx=0).stop 2024-04-15T15:15:50.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.439+0000 7f3a81f8f640 1 -- 172.21.15.72:0/2282083225 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a7c15e9f0 msgr2=0x7f3a7c14cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.439+0000 7f3a81f8f640 1 --2- 172.21.15.72:0/2282083225 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a7c15e9f0 0x7f3a7c14cd90 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f3a7400c700 tx=0x7f3a7400cbc0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.439+0000 7f3a81f8f640 1 -- 172.21.15.72:0/2282083225 shutdown_connections 2024-04-15T15:15:50.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.439+0000 7f3a81f8f640 1 --2- 172.21.15.72:0/2282083225 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3a5c06c920 0x7f3a5c06ede0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.439+0000 7f3a81f8f640 1 --2- 172.21.15.72:0/2282083225 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3a7c15e9f0 0x7f3a7c14cd90 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.439+0000 7f3a81f8f640 1 --2- 172.21.15.72:0/2282083225 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3a7c15e610 0x7f3a7c15f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.439+0000 7f3a81f8f640 1 --2- 172.21.15.72:0/2282083225 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3a7c06c400 0x7f3a7c152190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.439+0000 7f3a81f8f640 1 -- 172.21.15.72:0/2282083225 >> 172.21.15.72:0/2282083225 conn(0x7f3a7c0d4750 msgr2=0x7f3a7c0b4770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:50.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.439+0000 7f3a81f8f640 1 -- 172.21.15.72:0/2282083225 shutdown_connections 2024-04-15T15:15:50.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.440+0000 7f3a81f8f640 1 -- 172.21.15.72:0/2282083225 wait complete. 2024-04-15T15:15:50.452 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:15:50.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.555+0000 7fd46e30e640 1 Processor -- start 2024-04-15T15:15:50.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.555+0000 7fd46e30e640 1 -- start start 2024-04-15T15:15:50.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.555+0000 7fd46e30e640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd46815e610 0x7fd4680d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.555+0000 7fd46e30e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd46806c400 0x7fd46806c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.555+0000 7fd46e30e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4680d2320 con 0x7fd46815e9f0 2024-04-15T15:15:50.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.555+0000 7fd46e30e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fd4680d31d0 con 0x7fd46806c400 2024-04-15T15:15:50.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.555+0000 7fd46e30e640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fd4680d25c0 con 0x7fd46815e610 2024-04-15T15:15:50.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.556+0000 7fd46c8ab640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd46806c400 0x7fd46806c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.556+0000 7fd45ffff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fd46815e9f0 0x7fd4680d3660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60166/0 (socket says 172.21.15.72:60166) 2024-04-15T15:15:50.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.556+0000 7fd45ffff640 1 -- 172.21.15.72:0/51057573 learned_addr learned my addr 172.21.15.72:0/51057573 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:50.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.556+0000 7fd467fff640 1 --2- 172.21.15.72:0/51057573 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd46815e610 0x7fd4680d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.556+0000 7fd46c8ab640 1 -- 172.21.15.72:0/51057573 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd46815e610 msgr2=0x7fd4680d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:50.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.556+0000 7fd46c8ab640 1 --2- 172.21.15.72:0/51057573 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd46815e610 0x7fd4680d0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.556+0000 7fd46c8ab640 1 -- 172.21.15.72:0/51057573 >> v1:172.21.15.72:6789/0 conn(0x7fd46815e9f0 legacy=0x7fd4680d3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.556+0000 7fd46c8ab640 1 -- 172.21.15.72:0/51057573 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4680d28a0 con 0x7fd46806c400 2024-04-15T15:15:50.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.556+0000 7fd4677fe640 1 -- 172.21.15.72:0/51057573 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3632485199 0 0) 0x7fd4680d2320 con 0x7fd46815e9f0 2024-04-15T15:15:50.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.557+0000 7fd46c8ab640 1 --2- 172.21.15.72:0/51057573 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd46806c400 0x7fd46806c890 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fd460009d90 tx=0x7fd46002bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e1024d713855db2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:50.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.557+0000 7fd4677fe640 1 -- 172.21.15.72:0/51057573 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd460039070 con 0x7fd46806c400 2024-04-15T15:15:50.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.557+0000 7fd4677fe640 1 -- 172.21.15.72:0/51057573 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd46002cce0 con 0x7fd46806c400 2024-04-15T15:15:50.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.557+0000 7fd4677fe640 1 -- 172.21.15.72:0/51057573 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd460034660 con 0x7fd46806c400 2024-04-15T15:15:50.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.557+0000 7fd46e30e640 1 -- 172.21.15.72:0/51057573 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd46806c400 msgr2=0x7fd46806c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.557+0000 7fd46e30e640 1 --2- 172.21.15.72:0/51057573 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd46806c400 0x7fd46806c890 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fd460009d90 tx=0x7fd46002bf30 comp rx=0 tx=0).stop 2024-04-15T15:15:50.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.557+0000 7fd46e30e640 1 -- 172.21.15.72:0/51057573 shutdown_connections 2024-04-15T15:15:50.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.557+0000 7fd46e30e640 1 --2- 172.21.15.72:0/51057573 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd46806c400 0x7fd46806c890 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.557+0000 7fd46e30e640 1 --2- 172.21.15.72:0/51057573 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd46815e610 0x7fd4680d0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.557+0000 7fd46e30e640 1 -- 172.21.15.72:0/51057573 >> 172.21.15.72:0/51057573 conn(0x7fd4680d4750 msgr2=0x7fd4680d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:50.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.557+0000 7fd46e30e640 1 -- 172.21.15.72:0/51057573 shutdown_connections 2024-04-15T15:15:50.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.558+0000 7fd46e30e640 1 -- 172.21.15.72:0/51057573 wait complete. 2024-04-15T15:15:50.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.559+0000 7fd46e30e640 1 Processor -- start 2024-04-15T15:15:50.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.559+0000 7fd46e30e640 1 -- start start 2024-04-15T15:15:50.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.559+0000 7fd46e30e640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd46806c400 0x7fd4680caaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.559+0000 7fd46e30e640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd46815e610 0x7fd468151180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.559+0000 7fd46e30e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd46815e9f0 0x7fd4680cb820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.559+0000 7fd46e30e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd46806d130 con 0x7fd46815e9f0 2024-04-15T15:15:50.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.559+0000 7fd46e30e640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fd4680d2e60 con 0x7fd46806c400 2024-04-15T15:15:50.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.559+0000 7fd46e30e640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fd4680d0bd0 con 0x7fd46815e610 2024-04-15T15:15:50.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.559+0000 7fd46c8ab640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd46815e9f0 0x7fd4680cb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.560+0000 7fd46c8ab640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd46815e9f0 0x7fd4680cb820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46458/0 (socket says 172.21.15.72:46458) 2024-04-15T15:15:50.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.560+0000 7fd46c8ab640 1 -- 172.21.15.72:0/2307025033 learned_addr learned my addr 172.21.15.72:0/2307025033 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:50.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.560+0000 7fd467fff640 1 --2- 172.21.15.72:0/2307025033 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd46806c400 0x7fd4680caaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.560+0000 7fd45ffff640 1 --2- 172.21.15.72:0/2307025033 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd46815e610 0x7fd468151180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.560+0000 7fd46c8ab640 1 -- 172.21.15.72:0/2307025033 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd46815e610 msgr2=0x7fd468151180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.560+0000 7fd46c8ab640 1 --2- 172.21.15.72:0/2307025033 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd46815e610 0x7fd468151180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.560+0000 7fd46c8ab640 1 -- 172.21.15.72:0/2307025033 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd46806c400 msgr2=0x7fd4680caaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.560+0000 7fd46c8ab640 1 --2- 172.21.15.72:0/2307025033 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd46806c400 0x7fd4680caaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.560+0000 7fd46c8ab640 1 -- 172.21.15.72:0/2307025033 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd468151c00 con 0x7fd46815e9f0 2024-04-15T15:15:50.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.560+0000 7fd46c8ab640 1 --2- 172.21.15.72:0/2307025033 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd46815e9f0 0x7fd4680cb820 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fd460009d90 tx=0x7fd460004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:50.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.560+0000 7fd4657fa640 1 -- 172.21.15.72:0/2307025033 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd460044020 con 0x7fd46815e9f0 2024-04-15T15:15:50.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.560+0000 7fd4657fa640 1 -- 172.21.15.72:0/2307025033 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd46003a040 con 0x7fd46815e9f0 2024-04-15T15:15:50.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.560+0000 7fd4657fa640 1 -- 172.21.15.72:0/2307025033 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd460039050 con 0x7fd46815e9f0 2024-04-15T15:15:50.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.560+0000 7fd46e30e640 1 -- 172.21.15.72:0/2307025033 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd468151ee0 con 0x7fd46815e9f0 2024-04-15T15:15:50.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.560+0000 7fd46e30e640 1 -- 172.21.15.72:0/2307025033 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd468150350 con 0x7fd46815e9f0 2024-04-15T15:15:50.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.561+0000 7fd46e30e640 1 -- 172.21.15.72:0/2307025033 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd468075c50 con 0x7fd46815e9f0 2024-04-15T15:15:50.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.562+0000 7fd4657fa640 1 -- 172.21.15.72:0/2307025033 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd460003bb0 con 0x7fd46815e9f0 2024-04-15T15:15:50.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.562+0000 7fd4657fa640 1 --2- 172.21.15.72:0/2307025033 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd44806c920 0x7fd44806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.562+0000 7fd467fff640 1 --2- 172.21.15.72:0/2307025033 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd44806c920 0x7fd44806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.562+0000 7fd4657fa640 1 -- 172.21.15.72:0/2307025033 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fd4600ae150 con 0x7fd46815e9f0 2024-04-15T15:15:50.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.563+0000 7fd467fff640 1 --2- 172.21.15.72:0/2307025033 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd44806c920 0x7fd44806ede0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd458009b40 tx=0x7fd458009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:50.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.567+0000 7fd4657fa640 1 -- 172.21.15.72:0/2307025033 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd46007bbc0 con 0x7fd46815e9f0 2024-04-15T15:15:50.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.753+0000 7fd46e30e640 1 -- 172.21.15.72:0/2307025033 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd4680cf740 con 0x7fd46815e9f0 2024-04-15T15:15:50.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.754+0000 7fd4657fa640 1 -- 172.21.15.72:0/2307025033 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7fd46007fec0 con 0x7fd46815e9f0 2024-04-15T15:15:50.755 INFO:teuthology.orchestra.run.smithi072.stderr:listed 0 entries 2024-04-15T15:15:50.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.756+0000 7fd46e30e640 1 -- 172.21.15.72:0/2307025033 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd44806c920 msgr2=0x7fd44806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.756+0000 7fd46e30e640 1 --2- 172.21.15.72:0/2307025033 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd44806c920 0x7fd44806ede0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd458009b40 tx=0x7fd458009190 comp rx=0 tx=0).stop 2024-04-15T15:15:50.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.756+0000 7fd46e30e640 1 -- 172.21.15.72:0/2307025033 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd46815e9f0 msgr2=0x7fd4680cb820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.756+0000 7fd46e30e640 1 --2- 172.21.15.72:0/2307025033 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd46815e9f0 0x7fd4680cb820 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fd460009d90 tx=0x7fd460004a40 comp rx=0 tx=0).stop 2024-04-15T15:15:50.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.756+0000 7fd46e30e640 1 -- 172.21.15.72:0/2307025033 shutdown_connections 2024-04-15T15:15:50.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.756+0000 7fd46e30e640 1 --2- 172.21.15.72:0/2307025033 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd44806c920 0x7fd44806ede0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.756+0000 7fd46e30e640 1 --2- 172.21.15.72:0/2307025033 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd46815e9f0 0x7fd4680cb820 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.756+0000 7fd46e30e640 1 --2- 172.21.15.72:0/2307025033 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd46815e610 0x7fd468151180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.756+0000 7fd46e30e640 1 --2- 172.21.15.72:0/2307025033 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd46806c400 0x7fd4680caaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.756+0000 7fd46e30e640 1 -- 172.21.15.72:0/2307025033 >> 172.21.15.72:0/2307025033 conn(0x7fd4680d4750 msgr2=0x7fd4680d4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:50.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.756+0000 7fd46e30e640 1 -- 172.21.15.72:0/2307025033 shutdown_connections 2024-04-15T15:15:50.758 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.756+0000 7fd46e30e640 1 -- 172.21.15.72:0/2307025033 wait complete. 2024-04-15T15:15:50.768 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:15:50.768 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:15:50.786 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:50.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.887+0000 7fcb13f75640 1 Processor -- start 2024-04-15T15:15:50.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.888+0000 7fcb13f75640 1 -- start start 2024-04-15T15:15:50.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.888+0000 7fcb13f75640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fcb0c156da0 0x7fcb0c0ccb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.888+0000 7fcb13f75640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcb0c0cd0b0 0x7fcb0c06c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.888+0000 7fcb13f75640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb0c0cbad0 con 0x7fcb0c1569c0 2024-04-15T15:15:50.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.888+0000 7fcb13f75640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fcb0c0cbdb0 con 0x7fcb0c156da0 2024-04-15T15:15:50.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.888+0000 7fcb13f75640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fcb0c0cc090 con 0x7fcb0c0cd0b0 2024-04-15T15:15:50.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.889+0000 7fcb11d11640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fcb0c1569c0 0x7fcb0c15d540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60168/0 (socket says 172.21.15.72:60168) 2024-04-15T15:15:50.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.889+0000 7fcb11d11640 1 -- 172.21.15.72:0/3266179757 learned_addr learned my addr 172.21.15.72:0/3266179757 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:50.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.889+0000 7fcb12512640 1 --2- 172.21.15.72:0/3266179757 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcb0c0cd0b0 0x7fcb0c06c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.889+0000 7fcb11510640 1 --2- 172.21.15.72:0/3266179757 >> v2:172.21.15.175:3300/0 conn(0x7fcb0c156da0 0x7fcb0c0ccb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.889+0000 7fcb10d0f640 1 -- 172.21.15.72:0/3266179757 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3539797495 0 0) 0x7fcb0c0cbad0 con 0x7fcb0c1569c0 2024-04-15T15:15:50.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.889+0000 7fcb10d0f640 1 -- 172.21.15.72:0/3266179757 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcaf4003520 con 0x7fcb0c1569c0 2024-04-15T15:15:50.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.889+0000 7fcb12512640 1 -- 172.21.15.72:0/3266179757 >> v2:172.21.15.175:3300/0 conn(0x7fcb0c156da0 msgr2=0x7fcb0c0ccb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:50.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.889+0000 7fcb12512640 1 --2- 172.21.15.72:0/3266179757 >> v2:172.21.15.175:3300/0 conn(0x7fcb0c156da0 0x7fcb0c0ccb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.889+0000 7fcb12512640 1 -- 172.21.15.72:0/3266179757 >> v1:172.21.15.72:6789/0 conn(0x7fcb0c1569c0 legacy=0x7fcb0c15d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.889+0000 7fcb10d0f640 1 -- 172.21.15.72:0/3266179757 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2297369213 0 0) 0x7fcaf4003520 con 0x7fcb0c1569c0 2024-04-15T15:15:50.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.889+0000 7fcb12512640 1 -- 172.21.15.72:0/3266179757 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb0c0cc370 con 0x7fcb0c0cd0b0 2024-04-15T15:15:50.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.890+0000 7fcb12512640 1 --2- 172.21.15.72:0/3266179757 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcb0c0cd0b0 0x7fcb0c06c370 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcafc0097a0 tx=0x7fcafc02b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4fbc65607480d48 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:50.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.890+0000 7fcb10d0f640 1 -- 172.21.15.72:0/3266179757 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcafc037070 con 0x7fcb0c0cd0b0 2024-04-15T15:15:50.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.890+0000 7fcb10d0f640 1 -- 172.21.15.72:0/3266179757 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcafc030510 con 0x7fcb0c0cd0b0 2024-04-15T15:15:50.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.890+0000 7fcb10d0f640 1 -- 172.21.15.72:0/3266179757 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb04001670 con 0x7fcb0c0cd0b0 2024-04-15T15:15:50.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.890+0000 7fcb13f75640 1 -- 172.21.15.72:0/3266179757 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcb0c0cd0b0 msgr2=0x7fcb0c06c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.890+0000 7fcb13f75640 1 --2- 172.21.15.72:0/3266179757 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcb0c0cd0b0 0x7fcb0c06c370 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcafc0097a0 tx=0x7fcafc02b790 comp rx=0 tx=0).stop 2024-04-15T15:15:50.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.891+0000 7fcb13f75640 1 -- 172.21.15.72:0/3266179757 shutdown_connections 2024-04-15T15:15:50.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.891+0000 7fcb13f75640 1 --2- 172.21.15.72:0/3266179757 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcb0c0cd0b0 0x7fcb0c06c370 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.891+0000 7fcb13f75640 1 --2- 172.21.15.72:0/3266179757 >> v2:172.21.15.175:3300/0 conn(0x7fcb0c156da0 0x7fcb0c0ccb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.891+0000 7fcb13f75640 1 -- 172.21.15.72:0/3266179757 >> 172.21.15.72:0/3266179757 conn(0x7fcb0c0cd4a0 msgr2=0x7fcb0c0cd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:50.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.891+0000 7fcb13f75640 1 -- 172.21.15.72:0/3266179757 shutdown_connections 2024-04-15T15:15:50.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.891+0000 7fcb13f75640 1 -- 172.21.15.72:0/3266179757 wait complete. 2024-04-15T15:15:50.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.892+0000 7fcb13f75640 1 Processor -- start 2024-04-15T15:15:50.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.892+0000 7fcb13f75640 1 -- start start 2024-04-15T15:15:50.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.892+0000 7fcb13f75640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb0c1569c0 0x7fcb0c0b66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.892+0000 7fcb13f75640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb0c156da0 0x7fcb0c14ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.892+0000 7fcb13f75640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcb0c070450 0x7fcb0c0ca390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.892+0000 7fcb13f75640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0c0cb750 con 0x7fcb0c156da0 2024-04-15T15:15:50.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.892+0000 7fcb13f75640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0c0b01a0 con 0x7fcb0c1569c0 2024-04-15T15:15:50.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.892+0000 7fcb13f75640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fcb0c06c8b0 con 0x7fcb0c070450 2024-04-15T15:15:50.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.893+0000 7fcb11510640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb0c156da0 0x7fcb0c14ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.893+0000 7fcb11510640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb0c156da0 0x7fcb0c14ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46470/0 (socket says 172.21.15.72:46470) 2024-04-15T15:15:50.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.893+0000 7fcb11510640 1 -- 172.21.15.72:0/1267096444 learned_addr learned my addr 172.21.15.72:0/1267096444 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:50.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.893+0000 7fcb11d11640 1 --2- 172.21.15.72:0/1267096444 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb0c1569c0 0x7fcb0c0b66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.893+0000 7fcb12512640 1 --2- 172.21.15.72:0/1267096444 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcb0c070450 0x7fcb0c0ca390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.893+0000 7fcb11510640 1 -- 172.21.15.72:0/1267096444 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcb0c070450 msgr2=0x7fcb0c0ca390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.893+0000 7fcb11510640 1 --2- 172.21.15.72:0/1267096444 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcb0c070450 0x7fcb0c0ca390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.893+0000 7fcb11510640 1 -- 172.21.15.72:0/1267096444 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb0c1569c0 msgr2=0x7fcb0c0b66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.893+0000 7fcb11510640 1 --2- 172.21.15.72:0/1267096444 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb0c1569c0 0x7fcb0c0b66c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.893+0000 7fcb11510640 1 -- 172.21.15.72:0/1267096444 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb0c0b6c00 con 0x7fcb0c156da0 2024-04-15T15:15:50.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.893+0000 7fcb11510640 1 --2- 172.21.15.72:0/1267096444 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb0c156da0 0x7fcb0c14ee20 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fcb00009d10 tx=0x7fcb0000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:50.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.893+0000 7fcb0a7fc640 1 -- 172.21.15.72:0/1267096444 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb00018020 con 0x7fcb0c156da0 2024-04-15T15:15:50.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.893+0000 7fcb0a7fc640 1 -- 172.21.15.72:0/1267096444 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb00014480 con 0x7fcb0c156da0 2024-04-15T15:15:50.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.893+0000 7fcb13f75640 1 -- 172.21.15.72:0/1267096444 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb0c14f360 con 0x7fcb0c156da0 2024-04-15T15:15:50.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.893+0000 7fcb0a7fc640 1 -- 172.21.15.72:0/1267096444 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb00014830 con 0x7fcb0c156da0 2024-04-15T15:15:50.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.893+0000 7fcb13f75640 1 -- 172.21.15.72:0/1267096444 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb0c0b3aa0 con 0x7fcb0c156da0 2024-04-15T15:15:50.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.894+0000 7fcb13f75640 1 -- 172.21.15.72:0/1267096444 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb0c0b1330 con 0x7fcb0c156da0 2024-04-15T15:15:50.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.895+0000 7fcb0a7fc640 1 -- 172.21.15.72:0/1267096444 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcb00005d10 con 0x7fcb0c156da0 2024-04-15T15:15:50.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.895+0000 7fcb0a7fc640 1 --2- 172.21.15.72:0/1267096444 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcaf806c920 0x7fcaf806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.895+0000 7fcb11d11640 1 --2- 172.21.15.72:0/1267096444 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcaf806c920 0x7fcaf806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.895+0000 7fcb0a7fc640 1 -- 172.21.15.72:0/1267096444 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fcb0008cb10 con 0x7fcb0c156da0 2024-04-15T15:15:50.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.896+0000 7fcb11d11640 1 --2- 172.21.15.72:0/1267096444 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcaf806c920 0x7fcaf806ede0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fcb040013c0 tx=0x7fcb04001350 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:50.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:50.900+0000 7fcb0a7fc640 1 -- 172.21.15.72:0/1267096444 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcb0c0b1330 con 0x7fcb0c156da0 2024-04-15T15:15:50.951 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:15:50.953 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:15:51.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.081+0000 7fcb13f75640 1 -- 172.21.15.72:0/1267096444 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcb0c0cb530 con 0x7fcb0c156da0 2024-04-15T15:15:51.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.082+0000 7fcb0a7fc640 1 -- 172.21.15.72:0/1267096444 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18903 (secure 0 0 0) 0x7fcb00022030 con 0x7fcb0c156da0 2024-04-15T15:15:51.084 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:51.084 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":26,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:15:32.421728+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:14:49.438366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:14:50.368379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:14:54.081157+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":1}]},{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:51.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.084+0000 7fcb13f75640 1 -- 172.21.15.72:0/1267096444 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcaf806c920 msgr2=0x7fcaf806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.084+0000 7fcb13f75640 1 --2- 172.21.15.72:0/1267096444 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcaf806c920 0x7fcaf806ede0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fcb040013c0 tx=0x7fcb04001350 comp rx=0 tx=0).stop 2024-04-15T15:15:51.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.084+0000 7fcb13f75640 1 -- 172.21.15.72:0/1267096444 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb0c156da0 msgr2=0x7fcb0c14ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.084+0000 7fcb13f75640 1 --2- 172.21.15.72:0/1267096444 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb0c156da0 0x7fcb0c14ee20 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fcb00009d10 tx=0x7fcb0000c490 comp rx=0 tx=0).stop 2024-04-15T15:15:51.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.084+0000 7fcb13f75640 1 -- 172.21.15.72:0/1267096444 shutdown_connections 2024-04-15T15:15:51.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.084+0000 7fcb13f75640 1 --2- 172.21.15.72:0/1267096444 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcaf806c920 0x7fcaf806ede0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.084+0000 7fcb13f75640 1 --2- 172.21.15.72:0/1267096444 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcb0c070450 0x7fcb0c0ca390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.084+0000 7fcb13f75640 1 --2- 172.21.15.72:0/1267096444 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb0c156da0 0x7fcb0c14ee20 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.084+0000 7fcb13f75640 1 --2- 172.21.15.72:0/1267096444 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb0c1569c0 0x7fcb0c0b66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.084+0000 7fcb13f75640 1 -- 172.21.15.72:0/1267096444 >> 172.21.15.72:0/1267096444 conn(0x7fcb0c0cd4a0 msgr2=0x7fcb0c0cfe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:51.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.084+0000 7fcb13f75640 1 -- 172.21.15.72:0/1267096444 shutdown_connections 2024-04-15T15:15:51.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.084+0000 7fcb13f75640 1 -- 172.21.15.72:0/1267096444 wait complete. 2024-04-15T15:15:51.097 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:39.359102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:14:49.438366+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:14:50.368379+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:14:54.081157+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:15:51.098 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:51.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.206+0000 7f949b3e6640 1 Processor -- start 2024-04-15T15:15:51.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.206+0000 7f949b3e6640 1 -- start start 2024-04-15T15:15:51.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.207+0000 7f949b3e6640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f949415e610 0x7f94940d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.207+0000 7f949b3e6640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f949415e9f0 0x7f94940d3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.207+0000 7f949b3e6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f949406c400 0x7f949406c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.207+0000 7f949b3e6640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f94940d31d0 con 0x7f949406c400 2024-04-15T15:15:51.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.207+0000 7f949b3e6640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f94940d2320 con 0x7f949415e610 2024-04-15T15:15:51.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.207+0000 7f949b3e6640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f94940d25c0 con 0x7f949415e9f0 2024-04-15T15:15:51.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.207+0000 7f9499983640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f949406c400 0x7f949406c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.207+0000 7f9499983640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f949406c400 0x7f949406c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46476/0 (socket says 172.21.15.72:46476) 2024-04-15T15:15:51.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.207+0000 7f9499983640 1 -- 172.21.15.72:0/3653701992 learned_addr learned my addr 172.21.15.72:0/3653701992 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:51.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.207+0000 7f9498981640 1 --2- 172.21.15.72:0/3653701992 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f949415e9f0 0x7f94940d3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.207+0000 7f9499182640 1 --2- 172.21.15.72:0/3653701992 >> v2:172.21.15.175:3300/0 conn(0x7f949415e610 0x7f94940d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.207+0000 7f9499983640 1 -- 172.21.15.72:0/3653701992 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f949415e9f0 msgr2=0x7f94940d3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:51.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.207+0000 7f9499983640 1 --2- 172.21.15.72:0/3653701992 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f949415e9f0 0x7f94940d3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.207+0000 7f9499983640 1 -- 172.21.15.72:0/3653701992 >> v2:172.21.15.175:3300/0 conn(0x7f949415e610 msgr2=0x7f94940d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:51.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.207+0000 7f9499983640 1 --2- 172.21.15.72:0/3653701992 >> v2:172.21.15.175:3300/0 conn(0x7f949415e610 0x7f94940d0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.207+0000 7f9499983640 1 -- 172.21.15.72:0/3653701992 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94940d28a0 con 0x7f949406c400 2024-04-15T15:15:51.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.208+0000 7f9499983640 1 --2- 172.21.15.72:0/3653701992 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f949406c400 0x7f949406c890 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f94940b72a0 tx=0x7f948c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19f13b27f100edd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:51.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.208+0000 7f9487fff640 1 -- 172.21.15.72:0/3653701992 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948c02cb00 con 0x7f949406c400 2024-04-15T15:15:51.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.208+0000 7f9487fff640 1 -- 172.21.15.72:0/3653701992 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f948c02cca0 con 0x7f949406c400 2024-04-15T15:15:51.211 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.208+0000 7f9487fff640 1 -- 172.21.15.72:0/3653701992 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948c034610 con 0x7f949406c400 2024-04-15T15:15:51.213 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.209+0000 7f949b3e6640 1 -- 172.21.15.72:0/3653701992 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f949406c400 msgr2=0x7f949406c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.209+0000 7f949b3e6640 1 --2- 172.21.15.72:0/3653701992 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f949406c400 0x7f949406c890 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f94940b72a0 tx=0x7f948c02bf60 comp rx=0 tx=0).stop 2024-04-15T15:15:51.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.209+0000 7f949b3e6640 1 -- 172.21.15.72:0/3653701992 shutdown_connections 2024-04-15T15:15:51.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.209+0000 7f949b3e6640 1 --2- 172.21.15.72:0/3653701992 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f949406c400 0x7f949406c890 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.209+0000 7f949b3e6640 1 --2- 172.21.15.72:0/3653701992 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f949415e9f0 0x7f94940d3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.209+0000 7f949b3e6640 1 --2- 172.21.15.72:0/3653701992 >> v2:172.21.15.175:3300/0 conn(0x7f949415e610 0x7f94940d0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.209+0000 7f949b3e6640 1 -- 172.21.15.72:0/3653701992 >> 172.21.15.72:0/3653701992 conn(0x7f94940d4750 msgr2=0x7f94940d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:51.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.209+0000 7f949b3e6640 1 -- 172.21.15.72:0/3653701992 shutdown_connections 2024-04-15T15:15:51.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.209+0000 7f949b3e6640 1 -- 172.21.15.72:0/3653701992 wait complete. 2024-04-15T15:15:51.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.210+0000 7f949b3e6640 1 Processor -- start 2024-04-15T15:15:51.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.210+0000 7f949b3e6640 1 -- start start 2024-04-15T15:15:51.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.211+0000 7f949b3e6640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f949406c400 0x7f949415ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.211+0000 7f949b3e6640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f949415e610 0x7f94940b4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.214 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.211+0000 7f949b3e6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f949415e9f0 0x7f949415ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.211+0000 7f949b3e6640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f949406d130 con 0x7f949415e9f0 2024-04-15T15:15:51.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.211+0000 7f949b3e6640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f94940d2e60 con 0x7f949406c400 2024-04-15T15:15:51.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.211+0000 7f949b3e6640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f94940d0bd0 con 0x7f949415e610 2024-04-15T15:15:51.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.211+0000 7f9499983640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f949415e9f0 0x7f949415ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.211+0000 7f9499983640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f949415e9f0 0x7f949415ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46488/0 (socket says 172.21.15.72:46488) 2024-04-15T15:15:51.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.211+0000 7f9499983640 1 -- 172.21.15.72:0/1150438958 learned_addr learned my addr 172.21.15.72:0/1150438958 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:51.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.211+0000 7f9499182640 1 --2- 172.21.15.72:0/1150438958 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f949406c400 0x7f949415ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.211+0000 7f9498981640 1 --2- 172.21.15.72:0/1150438958 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f949415e610 0x7f94940b4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.211+0000 7f9499983640 1 -- 172.21.15.72:0/1150438958 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f949415e610 msgr2=0x7f94940b4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.211+0000 7f9499983640 1 --2- 172.21.15.72:0/1150438958 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f949415e610 0x7f94940b4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.211+0000 7f9499983640 1 -- 172.21.15.72:0/1150438958 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f949406c400 msgr2=0x7f949415ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.211+0000 7f9499983640 1 --2- 172.21.15.72:0/1150438958 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f949406c400 0x7f949415ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.211+0000 7f9499983640 1 -- 172.21.15.72:0/1150438958 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94940b5d30 con 0x7f949415e9f0 2024-04-15T15:15:51.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.211+0000 7f9499983640 1 --2- 172.21.15.72:0/1150438958 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f949415e9f0 0x7f949415ee90 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f948c009f40 tx=0x7f948c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:51.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.212+0000 7f9485ffb640 1 -- 172.21.15.72:0/1150438958 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948c043020 con 0x7f949415e9f0 2024-04-15T15:15:51.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.212+0000 7f9485ffb640 1 -- 172.21.15.72:0/1150438958 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f948c02ce50 con 0x7f949415e9f0 2024-04-15T15:15:51.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.212+0000 7f949b3e6640 1 -- 172.21.15.72:0/1150438958 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94940b6010 con 0x7f949415e9f0 2024-04-15T15:15:51.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.212+0000 7f9485ffb640 1 -- 172.21.15.72:0/1150438958 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948c009510 con 0x7f949415e9f0 2024-04-15T15:15:51.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.212+0000 7f949b3e6640 1 -- 172.21.15.72:0/1150438958 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94940b65d0 con 0x7f949415e9f0 2024-04-15T15:15:51.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.213+0000 7f949b3e6640 1 -- 172.21.15.72:0/1150438958 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9494075c50 con 0x7f949415e9f0 2024-04-15T15:15:51.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.213+0000 7f9485ffb640 1 -- 172.21.15.72:0/1150438958 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f948c046020 con 0x7f949415e9f0 2024-04-15T15:15:51.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.214+0000 7f9485ffb640 1 --2- 172.21.15.72:0/1150438958 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f947806c8f0 0x7f947806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.214+0000 7f9499182640 1 --2- 172.21.15.72:0/1150438958 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f947806c8f0 0x7f947806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.214+0000 7f9485ffb640 1 -- 172.21.15.72:0/1150438958 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f948c0ae310 con 0x7f949415e9f0 2024-04-15T15:15:51.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.214+0000 7f9499182640 1 --2- 172.21.15.72:0/1150438958 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f947806c8f0 0x7f947806edb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9490009ac0 tx=0x7f9490009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:51.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.218+0000 7f9485ffb640 1 -- 172.21.15.72:0/1150438958 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f948c04d030 con 0x7f949415e9f0 2024-04-15T15:15:51.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.401+0000 7f949b3e6640 1 -- 172.21.15.72:0/1150438958 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f94940cf740 con 0x7f949415e9f0 2024-04-15T15:15:51.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.401+0000 7f9485ffb640 1 -- 172.21.15.72:0/1150438958 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f948c07bcf0 con 0x7f949415e9f0 2024-04-15T15:15:51.403 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-15T15:15:51.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.403+0000 7f949b3e6640 1 -- 172.21.15.72:0/1150438958 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f947806c8f0 msgr2=0x7f947806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.403+0000 7f949b3e6640 1 --2- 172.21.15.72:0/1150438958 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f947806c8f0 0x7f947806edb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9490009ac0 tx=0x7f9490009190 comp rx=0 tx=0).stop 2024-04-15T15:15:51.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.403+0000 7f949b3e6640 1 -- 172.21.15.72:0/1150438958 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f949415e9f0 msgr2=0x7f949415ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.403+0000 7f949b3e6640 1 --2- 172.21.15.72:0/1150438958 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f949415e9f0 0x7f949415ee90 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f948c009f40 tx=0x7f948c004a40 comp rx=0 tx=0).stop 2024-04-15T15:15:51.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.403+0000 7f949b3e6640 1 -- 172.21.15.72:0/1150438958 shutdown_connections 2024-04-15T15:15:51.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.403+0000 7f949b3e6640 1 --2- 172.21.15.72:0/1150438958 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f947806c8f0 0x7f947806edb0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.403+0000 7f949b3e6640 1 --2- 172.21.15.72:0/1150438958 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f949415e9f0 0x7f949415ee90 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.403+0000 7f949b3e6640 1 --2- 172.21.15.72:0/1150438958 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f949415e610 0x7f94940b4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.403+0000 7f949b3e6640 1 --2- 172.21.15.72:0/1150438958 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f949406c400 0x7f949415ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.403+0000 7f949b3e6640 1 -- 172.21.15.72:0/1150438958 >> 172.21.15.72:0/1150438958 conn(0x7f94940d4750 msgr2=0x7f94940d4b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:51.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.403+0000 7f949b3e6640 1 -- 172.21.15.72:0/1150438958 shutdown_connections 2024-04-15T15:15:51.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.403+0000 7f949b3e6640 1 -- 172.21.15.72:0/1150438958 wait complete. 2024-04-15T15:15:51.416 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:15:51.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.524+0000 7fdbb1647640 1 Processor -- start 2024-04-15T15:15:51.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.524+0000 7fdbb1647640 1 -- start start 2024-04-15T15:15:51.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.525+0000 7fdbb1647640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdbac0cd2a0 0x7fdbac06cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.525+0000 7fdbb1647640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fdbac06e620 0x7fdbac06eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.525+0000 7fdbb1647640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbac06bd50 con 0x7fdbac15c630 2024-04-15T15:15:51.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.525+0000 7fdbb1647640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fdbac06d2f0 con 0x7fdbac06e620 2024-04-15T15:15:51.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.525+0000 7fdbb1647640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fdbac06c920 con 0x7fdbac0cd2a0 2024-04-15T15:15:51.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.525+0000 7fdbaa7fc640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdbac0cd2a0 0x7fdbac06cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.525+0000 7fdbab7fe640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fdbac06e620 0x7fdbac06eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.525+0000 7fdbaaffd640 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fdbac15c630 0x7fdbac0ccb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.72:42478/0 (socket says 172.21.15.72:42478) 2024-04-15T15:15:51.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.525+0000 7fdbaaffd640 1 -- 172.21.15.72:0/4078436349 learned_addr learned my addr 172.21.15.72:0/4078436349 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:51.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.525+0000 7fdba9ffb640 1 -- 172.21.15.72:0/4078436349 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 10881298 0 0) 0x7fdbac06bd50 con 0x7fdbac15c630 2024-04-15T15:15:51.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.525+0000 7fdba9ffb640 1 -- 172.21.15.72:0/4078436349 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb98003520 con 0x7fdbac15c630 2024-04-15T15:15:51.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.525+0000 7fdbaa7fc640 1 -- 172.21.15.72:0/4078436349 >> v2:172.21.15.175:3300/0 conn(0x7fdbac06e620 msgr2=0x7fdbac06eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:51.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.525+0000 7fdbaa7fc640 1 --2- 172.21.15.72:0/4078436349 >> v2:172.21.15.175:3300/0 conn(0x7fdbac06e620 0x7fdbac06eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.525+0000 7fdbaa7fc640 1 -- 172.21.15.72:0/4078436349 >> v1:172.21.15.175:6789/0 conn(0x7fdbac15c630 legacy=0x7fdbac0ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.525+0000 7fdbaa7fc640 1 -- 172.21.15.72:0/4078436349 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbac06bff0 con 0x7fdbac0cd2a0 2024-04-15T15:15:51.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.525+0000 7fdbab7fe640 1 --2- 172.21.15.72:0/4078436349 >> v2:172.21.15.175:3300/0 conn(0x7fdbac06e620 0x7fdbac06eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:51.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.526+0000 7fdbaa7fc640 1 --2- 172.21.15.72:0/4078436349 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdbac0cd2a0 0x7fdbac06cdb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdb940097a0 tx=0x7fdb9402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20cbed51d563f2ff server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:51.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.526+0000 7fdba9ffb640 1 -- 172.21.15.72:0/4078436349 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb94035020 con 0x7fdbac0cd2a0 2024-04-15T15:15:51.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.526+0000 7fdba9ffb640 1 -- 172.21.15.72:0/4078436349 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb94031830 con 0x7fdbac0cd2a0 2024-04-15T15:15:51.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.526+0000 7fdba9ffb640 1 -- 172.21.15.72:0/4078436349 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb94031c00 con 0x7fdbac0cd2a0 2024-04-15T15:15:51.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.527+0000 7fdbb1647640 1 -- 172.21.15.72:0/4078436349 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdbac0cd2a0 msgr2=0x7fdbac06cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.527+0000 7fdbb1647640 1 --2- 172.21.15.72:0/4078436349 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdbac0cd2a0 0x7fdbac06cdb0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdb940097a0 tx=0x7fdb9402b8d0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.527+0000 7fdbb1647640 1 -- 172.21.15.72:0/4078436349 shutdown_connections 2024-04-15T15:15:51.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.527+0000 7fdbb1647640 1 --2- 172.21.15.72:0/4078436349 >> v2:172.21.15.175:3300/0 conn(0x7fdbac06e620 0x7fdbac06eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.527+0000 7fdbb1647640 1 --2- 172.21.15.72:0/4078436349 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdbac0cd2a0 0x7fdbac06cdb0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.527+0000 7fdbb1647640 1 -- 172.21.15.72:0/4078436349 >> 172.21.15.72:0/4078436349 conn(0x7fdbac0b31d0 msgr2=0x7fdbac0b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:51.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.527+0000 7fdbb1647640 1 -- 172.21.15.72:0/4078436349 shutdown_connections 2024-04-15T15:15:51.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.527+0000 7fdbb1647640 1 -- 172.21.15.72:0/4078436349 wait complete. 2024-04-15T15:15:51.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.528+0000 7fdbb1647640 1 Processor -- start 2024-04-15T15:15:51.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.529+0000 7fdbb1647640 1 -- start start 2024-04-15T15:15:51.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.529+0000 7fdbb1647640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdbac06e620 0x7fdbac0cb910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.529+0000 7fdbb1647640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdbac0cd2a0 0x7fdbac14b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.529+0000 7fdbb1647640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdbac15c630 0x7fdbac0c5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.529+0000 7fdbb1647640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fdbac15ca10 con 0x7fdbac0cd2a0 2024-04-15T15:15:51.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.529+0000 7fdbb1647640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fdbac06c5b0 con 0x7fdbac06e620 2024-04-15T15:15:51.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.529+0000 7fdbb1647640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fdbac06eff0 con 0x7fdbac15c630 2024-04-15T15:15:51.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.529+0000 7fdbaa7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdbac0cd2a0 0x7fdbac14b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.529+0000 7fdbaa7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdbac0cd2a0 0x7fdbac14b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:46504/0 (socket says 172.21.15.72:46504) 2024-04-15T15:15:51.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.529+0000 7fdbaa7fc640 1 -- 172.21.15.72:0/582535041 learned_addr learned my addr 172.21.15.72:0/582535041 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:51.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.530+0000 7fdbab7fe640 1 --2- 172.21.15.72:0/582535041 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdbac15c630 0x7fdbac0c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.530+0000 7fdbaaffd640 1 --2- 172.21.15.72:0/582535041 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdbac06e620 0x7fdbac0cb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.530+0000 7fdbaa7fc640 1 -- 172.21.15.72:0/582535041 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdbac15c630 msgr2=0x7fdbac0c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.530+0000 7fdbaa7fc640 1 --2- 172.21.15.72:0/582535041 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdbac15c630 0x7fdbac0c5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.530+0000 7fdbaa7fc640 1 -- 172.21.15.72:0/582535041 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdbac06e620 msgr2=0x7fdbac0cb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.530+0000 7fdbaa7fc640 1 --2- 172.21.15.72:0/582535041 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdbac06e620 0x7fdbac0cb910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.530+0000 7fdbaa7fc640 1 -- 172.21.15.72:0/582535041 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbac14dc10 con 0x7fdbac0cd2a0 2024-04-15T15:15:51.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.530+0000 7fdbaa7fc640 1 --2- 172.21.15.72:0/582535041 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdbac0cd2a0 0x7fdbac14b090 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fdb9402bee0 tx=0x7fdb940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:51.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.530+0000 7fdba37fe640 1 -- 172.21.15.72:0/582535041 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb94048070 con 0x7fdbac0cd2a0 2024-04-15T15:15:51.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.530+0000 7fdba37fe640 1 -- 172.21.15.72:0/582535041 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb9403a6f0 con 0x7fdbac0cd2a0 2024-04-15T15:15:51.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.530+0000 7fdbb1647640 1 -- 172.21.15.72:0/582535041 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbac14bb50 con 0x7fdbac0cd2a0 2024-04-15T15:15:51.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.530+0000 7fdba37fe640 1 -- 172.21.15.72:0/582535041 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb9403aae0 con 0x7fdbac0cd2a0 2024-04-15T15:15:51.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.530+0000 7fdbb1647640 1 -- 172.21.15.72:0/582535041 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbac14c110 con 0x7fdbac0cd2a0 2024-04-15T15:15:51.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.531+0000 7fdbb1647640 1 -- 172.21.15.72:0/582535041 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbac0cbeb0 con 0x7fdbac0cd2a0 2024-04-15T15:15:51.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.532+0000 7fdba37fe640 1 -- 172.21.15.72:0/582535041 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdb9404b070 con 0x7fdbac0cd2a0 2024-04-15T15:15:51.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.532+0000 7fdba37fe640 1 --2- 172.21.15.72:0/582535041 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fdb8c06c8f0 0x7fdb8c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.532+0000 7fdbaaffd640 1 --2- 172.21.15.72:0/582535041 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fdb8c06c8f0 0x7fdb8c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.532+0000 7fdba37fe640 1 -- 172.21.15.72:0/582535041 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fdb940ad700 con 0x7fdbac0cd2a0 2024-04-15T15:15:51.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.533+0000 7fdbaaffd640 1 --2- 172.21.15.72:0/582535041 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fdb8c06c8f0 0x7fdb8c06edb0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fdba4005c00 tx=0x7fdba4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:51.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.537+0000 7fdba37fe640 1 -- 172.21.15.72:0/582535041 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdb9407b850 con 0x7fdbac0cd2a0 2024-04-15T15:15:51.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.735+0000 7fdbb1647640 1 -- 172.21.15.72:0/582535041 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdbac06d0d0 con 0x7fdbac0cd2a0 2024-04-15T15:15:51.737 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.736+0000 7fdba37fe640 1 -- 172.21.15.72:0/582535041 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7fdb9407b150 con 0x7fdbac0cd2a0 2024-04-15T15:15:51.737 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:51.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.738+0000 7fdbb1647640 1 -- 172.21.15.72:0/582535041 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fdb8c06c8f0 msgr2=0x7fdb8c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.738+0000 7fdbb1647640 1 --2- 172.21.15.72:0/582535041 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fdb8c06c8f0 0x7fdb8c06edb0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fdba4005c00 tx=0x7fdba4009040 comp rx=0 tx=0).stop 2024-04-15T15:15:51.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.738+0000 7fdbb1647640 1 -- 172.21.15.72:0/582535041 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdbac0cd2a0 msgr2=0x7fdbac14b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.738+0000 7fdbb1647640 1 --2- 172.21.15.72:0/582535041 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdbac0cd2a0 0x7fdbac14b090 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fdb9402bee0 tx=0x7fdb940046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.738+0000 7fdbb1647640 1 -- 172.21.15.72:0/582535041 shutdown_connections 2024-04-15T15:15:51.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.738+0000 7fdbb1647640 1 --2- 172.21.15.72:0/582535041 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fdb8c06c8f0 0x7fdb8c06edb0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.738+0000 7fdbb1647640 1 --2- 172.21.15.72:0/582535041 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fdbac15c630 0x7fdbac0c5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.738+0000 7fdbb1647640 1 --2- 172.21.15.72:0/582535041 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdbac0cd2a0 0x7fdbac14b090 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.738+0000 7fdbb1647640 1 --2- 172.21.15.72:0/582535041 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fdbac06e620 0x7fdbac0cb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.738+0000 7fdbb1647640 1 -- 172.21.15.72:0/582535041 >> 172.21.15.72:0/582535041 conn(0x7fdbac0b31d0 msgr2=0x7fdbac0b0690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:51.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.738+0000 7fdbb1647640 1 -- 172.21.15.72:0/582535041 shutdown_connections 2024-04-15T15:15:51.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.738+0000 7fdbb1647640 1 -- 172.21.15.72:0/582535041 wait complete. 2024-04-15T15:15:51.751 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:15:51.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.857+0000 7feceea1b640 1 Processor -- start 2024-04-15T15:15:51.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.857+0000 7feceea1b640 1 -- start start 2024-04-15T15:15:51.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.858+0000 7feceea1b640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fece815b620 0x7fece806e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.858+0000 7feceea1b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fece806f480 0x7fece80d0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.858+0000 7feceea1b640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fece806eb30 con 0x7fece80cff00 2024-04-15T15:15:51.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.858+0000 7feceea1b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fece80d08c0 con 0x7fece806f480 2024-04-15T15:15:51.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.858+0000 7feceea1b640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fece806d300 con 0x7fece815b620 2024-04-15T15:15:51.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.858+0000 7fececfb8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fece806f480 0x7fece80d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.858+0000 7fececfb8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fece806f480 0x7fece80d0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48628/0 (socket says 172.21.15.72:48628) 2024-04-15T15:15:51.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.858+0000 7fececfb8640 1 -- 172.21.15.72:0/2948334807 learned_addr learned my addr 172.21.15.72:0/2948334807 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:51.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.858+0000 7fece7fff640 1 --2- 172.21.15.72:0/2948334807 >> v2:172.21.15.175:3300/0 conn(0x7fece815b620 0x7fece806e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.858+0000 7fececfb8640 1 -- 172.21.15.72:0/2948334807 >> v2:172.21.15.175:3300/0 conn(0x7fece815b620 msgr2=0x7fece806e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:51.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.859+0000 7fececfb8640 1 --2- 172.21.15.72:0/2948334807 >> v2:172.21.15.175:3300/0 conn(0x7fece815b620 0x7fece806e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.859+0000 7fececfb8640 1 -- 172.21.15.72:0/2948334807 >> v1:172.21.15.175:6789/0 conn(0x7fece80cff00 legacy=0x7fece806ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:51.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.859+0000 7fececfb8640 1 -- 172.21.15.72:0/2948334807 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fece815ca10 con 0x7fece806f480 2024-04-15T15:15:51.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.859+0000 7fececfb8640 1 --2- 172.21.15.72:0/2948334807 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fece806f480 0x7fece80d0380 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fece80684b0 tx=0x7fece002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d3b0cc48a9b941e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:51.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.859+0000 7fece77fe640 1 -- 172.21.15.72:0/2948334807 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece002ca80 con 0x7fece806f480 2024-04-15T15:15:51.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.860+0000 7fece77fe640 1 -- 172.21.15.72:0/2948334807 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fece002cc20 con 0x7fece806f480 2024-04-15T15:15:51.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.860+0000 7fece77fe640 1 -- 172.21.15.72:0/2948334807 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece00343f0 con 0x7fece806f480 2024-04-15T15:15:51.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.860+0000 7feceea1b640 1 -- 172.21.15.72:0/2948334807 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fece806f480 msgr2=0x7fece80d0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.860+0000 7feceea1b640 1 --2- 172.21.15.72:0/2948334807 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fece806f480 0x7fece80d0380 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fece80684b0 tx=0x7fece002bee0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.860+0000 7feceea1b640 1 -- 172.21.15.72:0/2948334807 shutdown_connections 2024-04-15T15:15:51.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.860+0000 7feceea1b640 1 --2- 172.21.15.72:0/2948334807 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fece806f480 0x7fece80d0380 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.860+0000 7feceea1b640 1 --2- 172.21.15.72:0/2948334807 >> v2:172.21.15.175:3300/0 conn(0x7fece815b620 0x7fece806e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.860+0000 7feceea1b640 1 -- 172.21.15.72:0/2948334807 >> 172.21.15.72:0/2948334807 conn(0x7fece8070540 msgr2=0x7fece8070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:51.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.860+0000 7feceea1b640 1 -- 172.21.15.72:0/2948334807 shutdown_connections 2024-04-15T15:15:51.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.860+0000 7feceea1b640 1 -- 172.21.15.72:0/2948334807 wait complete. 2024-04-15T15:15:51.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.861+0000 7feceea1b640 1 Processor -- start 2024-04-15T15:15:51.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.861+0000 7feceea1b640 1 -- start start 2024-04-15T15:15:51.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.862+0000 7feceea1b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fece806f480 0x7fece815bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.862+0000 7feceea1b640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fece80cff00 0x7fece80b41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.862+0000 7feceea1b640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fece815b620 0x7fece80cc250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.862+0000 7feceea1b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fece806db10 con 0x7fece806f480 2024-04-15T15:15:51.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.862+0000 7feceea1b640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fece806d830 con 0x7fece815b620 2024-04-15T15:15:51.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.862+0000 7feceea1b640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fece806ddf0 con 0x7fece80cff00 2024-04-15T15:15:51.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.862+0000 7fece7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fece806f480 0x7fece815bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.862+0000 7fece7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fece806f480 0x7fece815bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48640/0 (socket says 172.21.15.72:48640) 2024-04-15T15:15:51.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.862+0000 7fece7fff640 1 -- 172.21.15.72:0/3253974907 learned_addr learned my addr 172.21.15.72:0/3253974907 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:51.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.862+0000 7fececfb8640 1 --2- 172.21.15.72:0/3253974907 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fece815b620 0x7fece80cc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.862+0000 7fecdffff640 1 --2- 172.21.15.72:0/3253974907 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fece80cff00 0x7fece80b41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.863+0000 7fece7fff640 1 -- 172.21.15.72:0/3253974907 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fece80cff00 msgr2=0x7fece80b41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.863+0000 7fece7fff640 1 --2- 172.21.15.72:0/3253974907 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fece80cff00 0x7fece80b41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.863+0000 7fece7fff640 1 -- 172.21.15.72:0/3253974907 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fece815b620 msgr2=0x7fece80cc250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.863+0000 7fece7fff640 1 --2- 172.21.15.72:0/3253974907 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fece815b620 0x7fece80cc250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.863+0000 7fece7fff640 1 -- 172.21.15.72:0/3253974907 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fece80b4ca0 con 0x7fece806f480 2024-04-15T15:15:51.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.863+0000 7fececfb8640 1 --2- 172.21.15.72:0/3253974907 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fece815b620 0x7fece80cc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:51.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.863+0000 7fece7fff640 1 --2- 172.21.15.72:0/3253974907 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fece806f480 0x7fece815bc70 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fecd800d5f0 tx=0x7fecd800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:51.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.863+0000 7fece57fa640 1 -- 172.21.15.72:0/3253974907 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecd80125f0 con 0x7fece806f480 2024-04-15T15:15:51.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.863+0000 7fece57fa640 1 -- 172.21.15.72:0/3253974907 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecd8019070 con 0x7fece806f480 2024-04-15T15:15:51.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.863+0000 7feceea1b640 1 -- 172.21.15.72:0/3253974907 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fece80b4f80 con 0x7fece806f480 2024-04-15T15:15:51.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.863+0000 7fece57fa640 1 -- 172.21.15.72:0/3253974907 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecd80129d0 con 0x7fece806f480 2024-04-15T15:15:51.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.863+0000 7feceea1b640 1 -- 172.21.15.72:0/3253974907 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fece80cbd30 con 0x7fece806f480 2024-04-15T15:15:51.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.864+0000 7feceea1b640 1 -- 172.21.15.72:0/3253974907 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fece8075c50 con 0x7fece806f480 2024-04-15T15:15:51.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.864+0000 7fece57fa640 1 -- 172.21.15.72:0/3253974907 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fecd8012b70 con 0x7fece806f480 2024-04-15T15:15:51.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.865+0000 7fece57fa640 1 --2- 172.21.15.72:0/3253974907 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fecc806c8f0 0x7fecc806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.865+0000 7fece57fa640 1 -- 172.21.15.72:0/3253974907 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fecd808c350 con 0x7fece806f480 2024-04-15T15:15:51.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.865+0000 7fecdffff640 1 --2- 172.21.15.72:0/3253974907 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fecc806c8f0 0x7fecc806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.865+0000 7fecdffff640 1 --2- 172.21.15.72:0/3253974907 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fecc806c8f0 0x7fecc806edb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fece815e910 tx=0x7fecd0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:51.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:51.869+0000 7fece57fa640 1 -- 172.21.15.72:0/3253974907 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fecd805a360 con 0x7fece806f480 2024-04-15T15:15:52.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.050+0000 7feceea1b640 1 -- 172.21.15.72:0/3253974907 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fece815e450 con 0x7fece806f480 2024-04-15T15:15:52.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.050+0000 7fece57fa640 1 -- 172.21.15.72:0/3253974907 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7fecd8059c60 con 0x7fece806f480 2024-04-15T15:15:52.053 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-04-15T15:15:52.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.052+0000 7feceea1b640 1 -- 172.21.15.72:0/3253974907 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fecc806c8f0 msgr2=0x7fecc806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.052+0000 7feceea1b640 1 --2- 172.21.15.72:0/3253974907 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fecc806c8f0 0x7fecc806edb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fece815e910 tx=0x7fecd0009040 comp rx=0 tx=0).stop 2024-04-15T15:15:52.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.052+0000 7feceea1b640 1 -- 172.21.15.72:0/3253974907 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fece806f480 msgr2=0x7fece815bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.052+0000 7feceea1b640 1 --2- 172.21.15.72:0/3253974907 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fece806f480 0x7fece815bc70 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fecd800d5f0 tx=0x7fecd800dab0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.052+0000 7feceea1b640 1 -- 172.21.15.72:0/3253974907 shutdown_connections 2024-04-15T15:15:52.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.052+0000 7feceea1b640 1 --2- 172.21.15.72:0/3253974907 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fecc806c8f0 0x7fecc806edb0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.052+0000 7feceea1b640 1 --2- 172.21.15.72:0/3253974907 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fece815b620 0x7fece80cc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.052+0000 7feceea1b640 1 --2- 172.21.15.72:0/3253974907 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fece80cff00 0x7fece80b41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.052+0000 7feceea1b640 1 --2- 172.21.15.72:0/3253974907 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fece806f480 0x7fece815bc70 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.052+0000 7feceea1b640 1 -- 172.21.15.72:0/3253974907 >> 172.21.15.72:0/3253974907 conn(0x7fece8070540 msgr2=0x7fece815a520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:52.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.053+0000 7feceea1b640 1 -- 172.21.15.72:0/3253974907 shutdown_connections 2024-04-15T15:15:52.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.053+0000 7feceea1b640 1 -- 172.21.15.72:0/3253974907 wait complete. 2024-04-15T15:15:52.065 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:15:52.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.173+0000 7f90d5d15640 1 Processor -- start 2024-04-15T15:15:52.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.173+0000 7f90d5d15640 1 -- start start 2024-04-15T15:15:52.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.173+0000 7f90d5d15640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f90d015e610 0x7f90d00d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.173+0000 7f90d5d15640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f90d015e9f0 0x7f90d00d3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.173+0000 7f90d5d15640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90d00d2320 con 0x7f90d006c400 2024-04-15T15:15:52.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.173+0000 7f90d5d15640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f90d00d31d0 con 0x7f90d015e9f0 2024-04-15T15:15:52.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.173+0000 7f90d5d15640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f90d00d25c0 con 0x7f90d015e610 2024-04-15T15:15:52.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.174+0000 7f90cffff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f90d006c400 0x7f90d00b7940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:47768/0 (socket says 172.21.15.72:47768) 2024-04-15T15:15:52.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.174+0000 7f90cffff640 1 -- 172.21.15.72:0/953053828 learned_addr learned my addr 172.21.15.72:0/953053828 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:52.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.174+0000 7f90c6ffd640 1 --2- 172.21.15.72:0/953053828 >> v2:172.21.15.175:3300/0 conn(0x7f90d015e9f0 0x7f90d00d3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.174+0000 7f90cf7fe640 1 --2- 172.21.15.72:0/953053828 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f90d015e610 0x7f90d00d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.174+0000 7f90ceffd640 1 -- 172.21.15.72:0/953053828 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2363102280 0 0) 0x7f90d00d2320 con 0x7f90d006c400 2024-04-15T15:15:52.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.174+0000 7f90ceffd640 1 -- 172.21.15.72:0/953053828 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f90bc003520 con 0x7f90d006c400 2024-04-15T15:15:52.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.174+0000 7f90ceffd640 1 -- 172.21.15.72:0/953053828 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 510023804 0 0) 0x7f90bc003520 con 0x7f90d006c400 2024-04-15T15:15:52.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.174+0000 7f90c6ffd640 1 -- 172.21.15.72:0/953053828 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f90d015e610 msgr2=0x7f90d00d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:52.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.174+0000 7f90c6ffd640 1 --2- 172.21.15.72:0/953053828 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f90d015e610 0x7f90d00d0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.174+0000 7f90c6ffd640 1 -- 172.21.15.72:0/953053828 >> v1:172.21.15.72:6789/0 conn(0x7f90d006c400 legacy=0x7f90d00b7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.174+0000 7f90c6ffd640 1 -- 172.21.15.72:0/953053828 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90d00d28a0 con 0x7f90d015e9f0 2024-04-15T15:15:52.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.175+0000 7f90c6ffd640 1 --2- 172.21.15.72:0/953053828 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90d015e9f0 0x7f90d00d3660 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f90b80097a0 tx=0x7f90b802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7aae86cac56e9dc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:52.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.175+0000 7f90ceffd640 1 -- 172.21.15.72:0/953053828 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b8038070 con 0x7f90d015e9f0 2024-04-15T15:15:52.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.175+0000 7f90ceffd640 1 -- 172.21.15.72:0/953053828 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90b8031500 con 0x7f90d015e9f0 2024-04-15T15:15:52.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.175+0000 7f90ceffd640 1 -- 172.21.15.72:0/953053828 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90c80019f0 con 0x7f90d015e9f0 2024-04-15T15:15:52.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.176+0000 7f90d5d15640 1 -- 172.21.15.72:0/953053828 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90d015e9f0 msgr2=0x7f90d00d3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.176+0000 7f90d5d15640 1 --2- 172.21.15.72:0/953053828 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90d015e9f0 0x7f90d00d3660 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f90b80097a0 tx=0x7f90b802b820 comp rx=0 tx=0).stop 2024-04-15T15:15:52.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.176+0000 7f90d5d15640 1 -- 172.21.15.72:0/953053828 shutdown_connections 2024-04-15T15:15:52.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.176+0000 7f90d5d15640 1 --2- 172.21.15.72:0/953053828 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90d015e9f0 0x7f90d00d3660 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.176+0000 7f90d5d15640 1 --2- 172.21.15.72:0/953053828 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f90d015e610 0x7f90d00d0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.176+0000 7f90d5d15640 1 -- 172.21.15.72:0/953053828 >> 172.21.15.72:0/953053828 conn(0x7f90d00d4750 msgr2=0x7f90d00d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:52.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.176+0000 7f90d5d15640 1 -- 172.21.15.72:0/953053828 shutdown_connections 2024-04-15T15:15:52.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.176+0000 7f90d5d15640 1 -- 172.21.15.72:0/953053828 wait complete. 2024-04-15T15:15:52.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.177+0000 7f90d5d15640 1 Processor -- start 2024-04-15T15:15:52.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.177+0000 7f90d5d15640 1 -- start start 2024-04-15T15:15:52.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.178+0000 7f90d5d15640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90d006c400 0x7f90d00ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.178+0000 7f90d5d15640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90d015e610 0x7f90d015ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.178+0000 7f90d5d15640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f90d015e9f0 0x7f90d015f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.178+0000 7f90d5d15640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f90d006d130 con 0x7f90d015e610 2024-04-15T15:15:52.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.178+0000 7f90d5d15640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f90d00d2e60 con 0x7f90d006c400 2024-04-15T15:15:52.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.178+0000 7f90d5d15640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f90d00d0bd0 con 0x7f90d015e9f0 2024-04-15T15:15:52.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.178+0000 7f90c6ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90d015e610 0x7f90d015ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.178+0000 7f90c6ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90d015e610 0x7f90d015ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48648/0 (socket says 172.21.15.72:48648) 2024-04-15T15:15:52.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.178+0000 7f90c6ffd640 1 -- 172.21.15.72:0/422356925 learned_addr learned my addr 172.21.15.72:0/422356925 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:52.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.178+0000 7f90cf7fe640 1 --2- 172.21.15.72:0/422356925 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90d006c400 0x7f90d00ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.178+0000 7f90cffff640 1 --2- 172.21.15.72:0/422356925 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f90d015e9f0 0x7f90d015f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.178+0000 7f90c6ffd640 1 -- 172.21.15.72:0/422356925 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f90d015e9f0 msgr2=0x7f90d015f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.178+0000 7f90c6ffd640 1 --2- 172.21.15.72:0/422356925 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f90d015e9f0 0x7f90d015f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.178+0000 7f90c6ffd640 1 -- 172.21.15.72:0/422356925 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90d006c400 msgr2=0x7f90d00ca600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.178+0000 7f90c6ffd640 1 --2- 172.21.15.72:0/422356925 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90d006c400 0x7f90d00ca600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.178+0000 7f90c6ffd640 1 -- 172.21.15.72:0/422356925 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90d00b8630 con 0x7f90d015e610 2024-04-15T15:15:52.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.178+0000 7f90c6ffd640 1 --2- 172.21.15.72:0/422356925 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90d015e610 0x7f90d015ee60 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f90b8000c00 tx=0x7f90b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:52.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.179+0000 7f90ccff9640 1 -- 172.21.15.72:0/422356925 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b8042070 con 0x7f90d015e610 2024-04-15T15:15:52.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.179+0000 7f90ccff9640 1 -- 172.21.15.72:0/422356925 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90b803d870 con 0x7f90d015e610 2024-04-15T15:15:52.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.179+0000 7f90d5d15640 1 -- 172.21.15.72:0/422356925 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90d00b8910 con 0x7f90d015e610 2024-04-15T15:15:52.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.179+0000 7f90ccff9640 1 -- 172.21.15.72:0/422356925 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90b8038040 con 0x7f90d015e610 2024-04-15T15:15:52.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.179+0000 7f90d5d15640 1 -- 172.21.15.72:0/422356925 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90d00b8ed0 con 0x7f90d015e610 2024-04-15T15:15:52.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.180+0000 7f90d5d15640 1 -- 172.21.15.72:0/422356925 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90d006d2b0 con 0x7f90d015e610 2024-04-15T15:15:52.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.180+0000 7f90ccff9640 1 -- 172.21.15.72:0/422356925 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f90b8045070 con 0x7f90d015e610 2024-04-15T15:15:52.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.181+0000 7f90ccff9640 1 --2- 172.21.15.72:0/422356925 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f90b006c8f0 0x7f90b006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.181+0000 7f90ccff9640 1 -- 172.21.15.72:0/422356925 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f90b80ad930 con 0x7f90d015e610 2024-04-15T15:15:52.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.181+0000 7f90cf7fe640 1 --2- 172.21.15.72:0/422356925 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f90b006c8f0 0x7f90b006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.181+0000 7f90cf7fe640 1 --2- 172.21.15.72:0/422356925 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f90b006c8f0 0x7f90b006edb0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f90c0009b40 tx=0x7f90c0009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:52.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.185+0000 7f90ccff9640 1 -- 172.21.15.72:0/422356925 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f90d006d2b0 con 0x7f90d015e610 2024-04-15T15:15:52.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.368+0000 7f90d5d15640 1 -- 172.21.15.72:0/422356925 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f90d0075c50 con 0x7f90d015e610 2024-04-15T15:15:52.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.368+0000 7f90ccff9640 1 -- 172.21.15.72:0/422356925 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f90b8030070 con 0x7f90d015e610 2024-04-15T15:15:52.370 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:52.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.370+0000 7f90d5d15640 1 -- 172.21.15.72:0/422356925 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f90b006c8f0 msgr2=0x7f90b006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.370+0000 7f90d5d15640 1 --2- 172.21.15.72:0/422356925 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f90b006c8f0 0x7f90b006edb0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f90c0009b40 tx=0x7f90c0009190 comp rx=0 tx=0).stop 2024-04-15T15:15:52.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.370+0000 7f90d5d15640 1 -- 172.21.15.72:0/422356925 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90d015e610 msgr2=0x7f90d015ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.370+0000 7f90d5d15640 1 --2- 172.21.15.72:0/422356925 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90d015e610 0x7f90d015ee60 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f90b8000c00 tx=0x7f90b80046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.370+0000 7f90d5d15640 1 -- 172.21.15.72:0/422356925 shutdown_connections 2024-04-15T15:15:52.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.370+0000 7f90d5d15640 1 --2- 172.21.15.72:0/422356925 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f90b006c8f0 0x7f90b006edb0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.370+0000 7f90d5d15640 1 --2- 172.21.15.72:0/422356925 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f90d015e9f0 0x7f90d015f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.370+0000 7f90d5d15640 1 --2- 172.21.15.72:0/422356925 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f90d015e610 0x7f90d015ee60 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.370+0000 7f90d5d15640 1 --2- 172.21.15.72:0/422356925 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f90d006c400 0x7f90d00ca600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.370+0000 7f90d5d15640 1 -- 172.21.15.72:0/422356925 >> 172.21.15.72:0/422356925 conn(0x7f90d00d4750 msgr2=0x7f90d00cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:52.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.370+0000 7f90d5d15640 1 -- 172.21.15.72:0/422356925 shutdown_connections 2024-04-15T15:15:52.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.370+0000 7f90d5d15640 1 -- 172.21.15.72:0/422356925 wait complete. 2024-04-15T15:15:52.383 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:15:52.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.486+0000 7fd96bee4640 1 Processor -- start 2024-04-15T15:15:52.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.486+0000 7fd96bee4640 1 -- start start 2024-04-15T15:15:52.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.487+0000 7fd96bee4640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd9640d1570 0x7fd9640cf160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.487+0000 7fd96bee4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd96415ce00 con 0x7fd964156cf0 2024-04-15T15:15:52.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.487+0000 7fd96bee4640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd96415d0e0 con 0x7fd9640b0270 2024-04-15T15:15:52.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.487+0000 7fd96bee4640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fd96415d3c0 con 0x7fd9640d1570 2024-04-15T15:15:52.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.487+0000 7fd969c80640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fd964156cf0 0x7fd96406ce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:47778/0 (socket says 172.21.15.72:47778) 2024-04-15T15:15:52.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.487+0000 7fd969c80640 1 -- 172.21.15.72:0/754640714 learned_addr learned my addr 172.21.15.72:0/754640714 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:52.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.487+0000 7fd96a481640 1 --2- 172.21.15.72:0/754640714 >> v2:172.21.15.175:3300/0 conn(0x7fd9640d1570 0x7fd9640cf160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.487+0000 7fd968c7e640 1 -- 172.21.15.72:0/754640714 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2494663706 0 0) 0x7fd96415ce00 con 0x7fd964156cf0 2024-04-15T15:15:52.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.487+0000 7fd968c7e640 1 -- 172.21.15.72:0/754640714 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd950003520 con 0x7fd964156cf0 2024-04-15T15:15:52.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.488+0000 7fd968c7e640 1 -- 172.21.15.72:0/754640714 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1949818146 0 0) 0x7fd96415d0e0 con 0x7fd9640b0270 2024-04-15T15:15:52.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.488+0000 7fd968c7e640 1 -- 172.21.15.72:0/754640714 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd96415ce00 con 0x7fd9640b0270 2024-04-15T15:15:52.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.488+0000 7fd968c7e640 1 -- 172.21.15.72:0/754640714 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2763247309 0 0) 0x7fd950003520 con 0x7fd964156cf0 2024-04-15T15:15:52.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.488+0000 7fd96a481640 1 -- 172.21.15.72:0/754640714 >> v1:172.21.15.175:6789/0 conn(0x7fd9640b0270 legacy=0x7fd9640cea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.488+0000 7fd96a481640 1 -- 172.21.15.72:0/754640714 >> v1:172.21.15.72:6789/0 conn(0x7fd964156cf0 legacy=0x7fd96406ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.488+0000 7fd96a481640 1 -- 172.21.15.72:0/754640714 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd96415d6a0 con 0x7fd9640d1570 2024-04-15T15:15:52.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.489+0000 7fd96a481640 1 --2- 172.21.15.72:0/754640714 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd9640d1570 0x7fd9640cf160 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd95c009dc0 tx=0x7fd95c02bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35411011b0089521 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:52.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.489+0000 7fd968c7e640 1 -- 172.21.15.72:0/754640714 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd95c039070 con 0x7fd9640d1570 2024-04-15T15:15:52.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.489+0000 7fd968c7e640 1 -- 172.21.15.72:0/754640714 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd95c02c8a0 con 0x7fd9640d1570 2024-04-15T15:15:52.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.489+0000 7fd968c7e640 1 -- 172.21.15.72:0/754640714 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd960001670 con 0x7fd9640d1570 2024-04-15T15:15:52.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.489+0000 7fd96bee4640 1 -- 172.21.15.72:0/754640714 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd9640d1570 msgr2=0x7fd9640cf160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.489+0000 7fd96bee4640 1 --2- 172.21.15.72:0/754640714 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd9640d1570 0x7fd9640cf160 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd95c009dc0 tx=0x7fd95c02bd70 comp rx=0 tx=0).stop 2024-04-15T15:15:52.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.489+0000 7fd96bee4640 1 -- 172.21.15.72:0/754640714 shutdown_connections 2024-04-15T15:15:52.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.489+0000 7fd96bee4640 1 --2- 172.21.15.72:0/754640714 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd9640d1570 0x7fd9640cf160 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.489+0000 7fd96bee4640 1 -- 172.21.15.72:0/754640714 >> 172.21.15.72:0/754640714 conn(0x7fd9640d0a50 msgr2=0x7fd9640d0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:52.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.489+0000 7fd96bee4640 1 -- 172.21.15.72:0/754640714 shutdown_connections 2024-04-15T15:15:52.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.489+0000 7fd96bee4640 1 -- 172.21.15.72:0/754640714 wait complete. 2024-04-15T15:15:52.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.490+0000 7fd96bee4640 1 Processor -- start 2024-04-15T15:15:52.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.491+0000 7fd96bee4640 1 -- start start 2024-04-15T15:15:52.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.491+0000 7fd96bee4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9640b0270 0x7fd964158d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.491+0000 7fd96bee4640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd9640d1570 0x7fd96415a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.491+0000 7fd96bee4640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd964156cf0 0x7fd9641f59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.491+0000 7fd96bee4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd96406dbb0 con 0x7fd9640b0270 2024-04-15T15:15:52.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.491+0000 7fd96bee4640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fd9641570d0 con 0x7fd964156cf0 2024-04-15T15:15:52.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.491+0000 7fd96bee4640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fd96406d540 con 0x7fd9640d1570 2024-04-15T15:15:52.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.491+0000 7fd969c80640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9640b0270 0x7fd964158d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.491+0000 7fd969c80640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9640b0270 0x7fd964158d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48652/0 (socket says 172.21.15.72:48652) 2024-04-15T15:15:52.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.491+0000 7fd969c80640 1 -- 172.21.15.72:0/2716535498 learned_addr learned my addr 172.21.15.72:0/2716535498 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:52.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.492+0000 7fd969c80640 1 -- 172.21.15.72:0/2716535498 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd9640d1570 msgr2=0x7fd96415a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.492+0000 7fd969c80640 1 --2- 172.21.15.72:0/2716535498 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd9640d1570 0x7fd96415a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.492+0000 7fd969c80640 1 -- 172.21.15.72:0/2716535498 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd964156cf0 msgr2=0x7fd9641f59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.492+0000 7fd96a481640 1 --2- 172.21.15.72:0/2716535498 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd964156cf0 0x7fd9641f59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.492+0000 7fd969c80640 1 --2- 172.21.15.72:0/2716535498 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd964156cf0 0x7fd9641f59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.492+0000 7fd969c80640 1 -- 172.21.15.72:0/2716535498 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9641592b0 con 0x7fd9640b0270 2024-04-15T15:15:52.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.492+0000 7fd969c80640 1 --2- 172.21.15.72:0/2716535498 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9640b0270 0x7fd964158d70 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fd96000c780 tx=0x7fd96000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:52.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.492+0000 7fd95a7fc640 1 -- 172.21.15.72:0/2716535498 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd960017070 con 0x7fd9640b0270 2024-04-15T15:15:52.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.492+0000 7fd95a7fc640 1 -- 172.21.15.72:0/2716535498 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd960005ad0 con 0x7fd9640b0270 2024-04-15T15:15:52.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.492+0000 7fd96bee4640 1 -- 172.21.15.72:0/2716535498 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd96415b1c0 con 0x7fd9640b0270 2024-04-15T15:15:52.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.492+0000 7fd95a7fc640 1 -- 172.21.15.72:0/2716535498 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd960007070 con 0x7fd9640b0270 2024-04-15T15:15:52.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.492+0000 7fd96bee4640 1 -- 172.21.15.72:0/2716535498 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd96415b780 con 0x7fd9640b0270 2024-04-15T15:15:52.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.493+0000 7fd96bee4640 1 -- 172.21.15.72:0/2716535498 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd964075c50 con 0x7fd9640b0270 2024-04-15T15:15:52.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.493+0000 7fd95a7fc640 1 -- 172.21.15.72:0/2716535498 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd96001a070 con 0x7fd9640b0270 2024-04-15T15:15:52.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.494+0000 7fd95a7fc640 1 --2- 172.21.15.72:0/2716535498 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd94806c8f0 0x7fd94806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.494+0000 7fd95a7fc640 1 -- 172.21.15.72:0/2716535498 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fd96008b510 con 0x7fd9640b0270 2024-04-15T15:15:52.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.494+0000 7fd96947f640 1 --2- 172.21.15.72:0/2716535498 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd94806c8f0 0x7fd94806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.494+0000 7fd96947f640 1 --2- 172.21.15.72:0/2716535498 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd94806c8f0 0x7fd94806edb0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd9540010e0 tx=0x7fd954007950 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:52.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.498+0000 7fd95a7fc640 1 -- 172.21.15.72:0/2716535498 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd960059540 con 0x7fd9640b0270 2024-04-15T15:15:52.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.685+0000 7fd96bee4640 1 -- 172.21.15.72:0/2716535498 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd96406ce90 con 0x7fd9640b0270 2024-04-15T15:15:52.687 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.686+0000 7fd95a7fc640 1 -- 172.21.15.72:0/2716535498 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7fd960058e40 con 0x7fd9640b0270 2024-04-15T15:15:52.687 INFO:teuthology.orchestra.run.smithi072.stderr:listed 0 entries 2024-04-15T15:15:52.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.687+0000 7fd96bee4640 1 -- 172.21.15.72:0/2716535498 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd94806c8f0 msgr2=0x7fd94806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.687+0000 7fd96bee4640 1 --2- 172.21.15.72:0/2716535498 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd94806c8f0 0x7fd94806edb0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd9540010e0 tx=0x7fd954007950 comp rx=0 tx=0).stop 2024-04-15T15:15:52.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.687+0000 7fd96bee4640 1 -- 172.21.15.72:0/2716535498 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9640b0270 msgr2=0x7fd964158d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.688+0000 7fd96bee4640 1 --2- 172.21.15.72:0/2716535498 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9640b0270 0x7fd964158d70 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fd96000c780 tx=0x7fd96000cc40 comp rx=0 tx=0).stop 2024-04-15T15:15:52.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.688+0000 7fd96bee4640 1 -- 172.21.15.72:0/2716535498 shutdown_connections 2024-04-15T15:15:52.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.688+0000 7fd96bee4640 1 --2- 172.21.15.72:0/2716535498 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd94806c8f0 0x7fd94806edb0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.688+0000 7fd96bee4640 1 --2- 172.21.15.72:0/2716535498 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd964156cf0 0x7fd9641f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.688+0000 7fd96bee4640 1 --2- 172.21.15.72:0/2716535498 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd9640d1570 0x7fd96415a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.688+0000 7fd96bee4640 1 --2- 172.21.15.72:0/2716535498 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd9640b0270 0x7fd964158d70 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.688+0000 7fd96bee4640 1 -- 172.21.15.72:0/2716535498 >> 172.21.15.72:0/2716535498 conn(0x7fd9640d0a50 msgr2=0x7fd9640cf2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:52.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.688+0000 7fd96bee4640 1 -- 172.21.15.72:0/2716535498 shutdown_connections 2024-04-15T15:15:52.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.688+0000 7fd96bee4640 1 -- 172.21.15.72:0/2716535498 wait complete. 2024-04-15T15:15:52.700 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi175.front.sepia.ceph.com... 2024-04-15T15:15:52.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:52.701 DEBUG:teuthology.orchestra.run.smithi175:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-04-15T15:15:52.736 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:15:52.736 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi175.front.sepia.ceph.com 2024-04-15T15:15:52.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:52.737 DEBUG:teuthology.orchestra.run.smithi175:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-15T15:15:52.796 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-15T15:15:52.796 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:15:52.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.898+0000 7efcfd540640 1 Processor -- start 2024-04-15T15:15:52.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.898+0000 7efcfd540640 1 -- start start 2024-04-15T15:15:52.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.899+0000 7efcfd540640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efcf8161e40 0x7efcf80b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.899+0000 7efcfd540640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efcf80d45b0 0x7efcf80d4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.899+0000 7efcfd540640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcf8162220 con 0x7efcf80d12b0 2024-04-15T15:15:52.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.899+0000 7efcfd540640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7efcf80d4f60 con 0x7efcf8161e40 2024-04-15T15:15:52.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.899+0000 7efcfd540640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7efcf80d21f0 con 0x7efcf80d45b0 2024-04-15T15:15:52.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.899+0000 7efcf6ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efcf8161e40 0x7efcf80b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.899+0000 7efcf6ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7efcf8161e40 0x7efcf80b37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48654/0 (socket says 172.21.15.72:48654) 2024-04-15T15:15:52.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.899+0000 7efcf6ffd640 1 -- 172.21.15.72:0/3815103355 learned_addr learned my addr 172.21.15.72:0/3815103355 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:52.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.899+0000 7efcf6ffd640 1 -- 172.21.15.72:0/3815103355 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efcf80d45b0 msgr2=0x7efcf80d4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:52.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.899+0000 7efcf6ffd640 1 --2- 172.21.15.72:0/3815103355 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efcf80d45b0 0x7efcf80d4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.899+0000 7efcf6ffd640 1 -- 172.21.15.72:0/3815103355 >> v1:172.21.15.175:6789/0 conn(0x7efcf80d12b0 legacy=0x7efcf80d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:52.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.899+0000 7efcf6ffd640 1 -- 172.21.15.72:0/3815103355 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcf80d3070 con 0x7efcf8161e40 2024-04-15T15:15:52.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.900+0000 7efcf6ffd640 1 --2- 172.21.15.72:0/3815103355 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcf8161e40 0x7efcf80b37f0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7efce80098a0 tx=0x7efce802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dfc4d9546a24e20 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:52.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.900+0000 7efcf67fc640 1 -- 172.21.15.72:0/3815103355 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce80317c0 con 0x7efcf8161e40 2024-04-15T15:15:52.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.900+0000 7efcf67fc640 1 -- 172.21.15.72:0/3815103355 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efce8037070 con 0x7efcf8161e40 2024-04-15T15:15:52.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.900+0000 7efcf67fc640 1 -- 172.21.15.72:0/3815103355 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efce8031b80 con 0x7efcf8161e40 2024-04-15T15:15:52.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.900+0000 7efcfd540640 1 -- 172.21.15.72:0/3815103355 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcf8161e40 msgr2=0x7efcf80b37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.900+0000 7efcfd540640 1 --2- 172.21.15.72:0/3815103355 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcf8161e40 0x7efcf80b37f0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7efce80098a0 tx=0x7efce802b860 comp rx=0 tx=0).stop 2024-04-15T15:15:52.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.901+0000 7efcfd540640 1 -- 172.21.15.72:0/3815103355 shutdown_connections 2024-04-15T15:15:52.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.901+0000 7efcfd540640 1 --2- 172.21.15.72:0/3815103355 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efcf80d45b0 0x7efcf80d4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.901+0000 7efcfd540640 1 --2- 172.21.15.72:0/3815103355 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcf8161e40 0x7efcf80b37f0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.901+0000 7efcfd540640 1 -- 172.21.15.72:0/3815103355 >> 172.21.15.72:0/3815103355 conn(0x7efcf809d970 msgr2=0x7efcf80b95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:52.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.901+0000 7efcfd540640 1 -- 172.21.15.72:0/3815103355 shutdown_connections 2024-04-15T15:15:52.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.901+0000 7efcfd540640 1 -- 172.21.15.72:0/3815103355 wait complete. 2024-04-15T15:15:52.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.902+0000 7efcfd540640 1 Processor -- start 2024-04-15T15:15:52.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.902+0000 7efcfd540640 1 -- start start 2024-04-15T15:15:52.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.902+0000 7efcfd540640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efcf80d12b0 0x7efcf8151b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.903+0000 7efcfd540640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efcf80d45b0 0x7efcf81f79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.903+0000 7efcfd540640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcf8161e40 0x7efcf80b5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.903+0000 7efcfd540640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7efcf80d1de0 con 0x7efcf8161e40 2024-04-15T15:15:52.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.903+0000 7efcfd540640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7efcf80d1b00 con 0x7efcf80d12b0 2024-04-15T15:15:52.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.903+0000 7efcfd540640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7efcf80d2440 con 0x7efcf80d45b0 2024-04-15T15:15:52.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.903+0000 7efcf77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcf8161e40 0x7efcf80b5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.903+0000 7efcf77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcf8161e40 0x7efcf80b5830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48656/0 (socket says 172.21.15.72:48656) 2024-04-15T15:15:52.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.903+0000 7efcf77fe640 1 -- 172.21.15.72:0/2541874859 learned_addr learned my addr 172.21.15.72:0/2541874859 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:52.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.903+0000 7efcf6ffd640 1 --2- 172.21.15.72:0/2541874859 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efcf80d12b0 0x7efcf8151b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.903+0000 7efcee7fc640 1 --2- 172.21.15.72:0/2541874859 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efcf80d45b0 0x7efcf81f79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.903+0000 7efcf77fe640 1 -- 172.21.15.72:0/2541874859 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efcf80d45b0 msgr2=0x7efcf81f79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.903+0000 7efcf77fe640 1 --2- 172.21.15.72:0/2541874859 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efcf80d45b0 0x7efcf81f79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.903+0000 7efcf77fe640 1 -- 172.21.15.72:0/2541874859 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efcf80d12b0 msgr2=0x7efcf8151b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.903+0000 7efcf77fe640 1 --2- 172.21.15.72:0/2541874859 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efcf80d12b0 0x7efcf8151b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.903+0000 7efcf77fe640 1 -- 172.21.15.72:0/2541874859 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcf815fc60 con 0x7efcf8161e40 2024-04-15T15:15:52.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.903+0000 7efcf77fe640 1 --2- 172.21.15.72:0/2541874859 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcf8161e40 0x7efcf80b5830 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7efcf000b970 tx=0x7efcf000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:52.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.903+0000 7efceffff640 1 -- 172.21.15.72:0/2541874859 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf0019070 con 0x7efcf8161e40 2024-04-15T15:15:52.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.903+0000 7efceffff640 1 -- 172.21.15.72:0/2541874859 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcf000cb60 con 0x7efcf8161e40 2024-04-15T15:15:52.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.903+0000 7efceffff640 1 -- 172.21.15.72:0/2541874859 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf0014440 con 0x7efcf8161e40 2024-04-15T15:15:52.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.903+0000 7efcfd540640 1 -- 172.21.15.72:0/2541874859 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcf81f8210 con 0x7efcf8161e40 2024-04-15T15:15:52.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.903+0000 7efcfd540640 1 -- 172.21.15.72:0/2541874859 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcf81f87d0 con 0x7efcf8161e40 2024-04-15T15:15:52.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.904+0000 7efcfd540640 1 -- 172.21.15.72:0/2541874859 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcf80689e0 con 0x7efcf8161e40 2024-04-15T15:15:52.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.905+0000 7efceffff640 1 -- 172.21.15.72:0/2541874859 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efcf001c070 con 0x7efcf8161e40 2024-04-15T15:15:52.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.905+0000 7efceffff640 1 --2- 172.21.15.72:0/2541874859 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7efcd806c8f0 0x7efcd806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.905+0000 7efceffff640 1 -- 172.21.15.72:0/2541874859 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7efcf008bbd0 con 0x7efcf8161e40 2024-04-15T15:15:52.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.905+0000 7efcf6ffd640 1 --2- 172.21.15.72:0/2541874859 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7efcd806c8f0 0x7efcd806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.906+0000 7efcf6ffd640 1 --2- 172.21.15.72:0/2541874859 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7efcd806c8f0 0x7efcd806edb0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7efce8036000 tx=0x7efce803d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:52.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:52.910+0000 7efceffff640 1 -- 172.21.15.72:0/2541874859 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efcf00595b0 con 0x7efcf8161e40 2024-04-15T15:15:53.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.122+0000 7efcfd540640 1 -- 172.21.15.72:0/2541874859 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efcf80d2d50 con 0x7efcf8161e40 2024-04-15T15:15:53.124 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.123+0000 7efceffff640 1 -- 172.21.15.72:0/2541874859 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5183 (secure 0 0 0) 0x7efcf005d8b0 con 0x7efcf8161e40 2024-04-15T15:15:53.125 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:53.125 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4267,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6833/1659343919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1659343919},{"type":"v1","addr":"172.21.15.175:6833","nonce":1659343919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6835/4113588141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6834","nonce":4113588141},{"type":"v1","addr":"172.21.15.175:6835","nonce":4113588141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4296,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/4151548261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4151548261},{"type":"v1","addr":"172.21.15.72:6836","nonce":4151548261}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:14:51.411091+0000","modified":"2024-04-15T15:15:07.485436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/3458067804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3458067804},{"type":"v1","addr":"172.21.15.72:6837","nonce":3458067804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4299,"qdb_cluster":[4299]},"id":1}]} 2024-04-15T15:15:53.125 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 9 2024-04-15T15:15:53.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.125+0000 7efcfd540640 1 -- 172.21.15.72:0/2541874859 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7efcd806c8f0 msgr2=0x7efcd806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:53.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.125+0000 7efcfd540640 1 --2- 172.21.15.72:0/2541874859 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7efcd806c8f0 0x7efcd806edb0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7efce8036000 tx=0x7efce803d040 comp rx=0 tx=0).stop 2024-04-15T15:15:53.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.125+0000 7efcfd540640 1 -- 172.21.15.72:0/2541874859 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcf8161e40 msgr2=0x7efcf80b5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:53.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.125+0000 7efcfd540640 1 --2- 172.21.15.72:0/2541874859 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcf8161e40 0x7efcf80b5830 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7efcf000b970 tx=0x7efcf000be30 comp rx=0 tx=0).stop 2024-04-15T15:15:53.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.125+0000 7efcfd540640 1 -- 172.21.15.72:0/2541874859 shutdown_connections 2024-04-15T15:15:53.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.125+0000 7efcfd540640 1 --2- 172.21.15.72:0/2541874859 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7efcd806c8f0 0x7efcd806edb0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.125+0000 7efcfd540640 1 --2- 172.21.15.72:0/2541874859 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efcf8161e40 0x7efcf80b5830 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.125+0000 7efcfd540640 1 --2- 172.21.15.72:0/2541874859 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7efcf80d45b0 0x7efcf81f79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.125+0000 7efcfd540640 1 --2- 172.21.15.72:0/2541874859 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7efcf80d12b0 0x7efcf8151b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.125+0000 7efcfd540640 1 -- 172.21.15.72:0/2541874859 >> 172.21.15.72:0/2541874859 conn(0x7efcf809d970 msgr2=0x7efcf80b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:53.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.125+0000 7efcfd540640 1 -- 172.21.15.72:0/2541874859 shutdown_connections 2024-04-15T15:15:53.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.125+0000 7efcfd540640 1 -- 172.21.15.72:0/2541874859 wait complete. 2024-04-15T15:15:53.138 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:53.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.239+0000 7fd8c32eb640 1 Processor -- start 2024-04-15T15:15:53.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.240+0000 7fd8c32eb640 1 -- start start 2024-04-15T15:15:53.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.240+0000 7fd8c32eb640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd8bc15c710 0x7fd8bc0b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.240+0000 7fd8c32eb640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd8bc0d1790 0x7fd8bc0cd950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.240+0000 7fd8c32eb640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8bc15caf0 con 0x7fd8bc0b5790 2024-04-15T15:15:53.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.240+0000 7fd8c32eb640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fd8bc06f510 con 0x7fd8bc15c710 2024-04-15T15:15:53.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.240+0000 7fd8c32eb640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fd8bc0ccb90 con 0x7fd8bc0d1790 2024-04-15T15:15:53.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.241+0000 7fd8c0886640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fd8bc0b5790 0x7fd8bc0d1080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:47784/0 (socket says 172.21.15.72:47784) 2024-04-15T15:15:53.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.241+0000 7fd8c0886640 1 -- 172.21.15.72:0/3189184335 learned_addr learned my addr 172.21.15.72:0/3189184335 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:53.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.241+0000 7fd8c1087640 1 --2- 172.21.15.72:0/3189184335 >> v2:172.21.15.175:3300/0 conn(0x7fd8bc15c710 0x7fd8bc0b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:53.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.241+0000 7fd8c1888640 1 --2- 172.21.15.72:0/3189184335 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd8bc0d1790 0x7fd8bc0cd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:53.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.241+0000 7fd8b3fff640 1 -- 172.21.15.72:0/3189184335 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1662447393 0 0) 0x7fd8bc15caf0 con 0x7fd8bc0b5790 2024-04-15T15:15:53.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.241+0000 7fd8b3fff640 1 -- 172.21.15.72:0/3189184335 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8a8003520 con 0x7fd8bc0b5790 2024-04-15T15:15:53.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.241+0000 7fd8b3fff640 1 -- 172.21.15.72:0/3189184335 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 165827661 0 0) 0x7fd8a8003520 con 0x7fd8bc0b5790 2024-04-15T15:15:53.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.241+0000 7fd8c1087640 1 -- 172.21.15.72:0/3189184335 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd8bc0d1790 msgr2=0x7fd8bc0cd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:53.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.241+0000 7fd8c1087640 1 --2- 172.21.15.72:0/3189184335 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd8bc0d1790 0x7fd8bc0cd950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.241+0000 7fd8c1087640 1 -- 172.21.15.72:0/3189184335 >> v1:172.21.15.72:6789/0 conn(0x7fd8bc0b5790 legacy=0x7fd8bc0d1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:53.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.241+0000 7fd8c1087640 1 -- 172.21.15.72:0/3189184335 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8bc0cce70 con 0x7fd8bc15c710 2024-04-15T15:15:53.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.241+0000 7fd8c1888640 1 --2- 172.21.15.72:0/3189184335 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd8bc0d1790 0x7fd8bc0cd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:53.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.242+0000 7fd8c1087640 1 --2- 172.21.15.72:0/3189184335 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd8bc15c710 0x7fd8bc0b66b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8009d90 tx=0x7fd8b802be00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b04ca704afa256d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:53.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.242+0000 7fd8b3fff640 1 -- 172.21.15.72:0/3189184335 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b8039070 con 0x7fd8bc15c710 2024-04-15T15:15:53.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.242+0000 7fd8b3fff640 1 -- 172.21.15.72:0/3189184335 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8b802c930 con 0x7fd8bc15c710 2024-04-15T15:15:53.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.242+0000 7fd8b3fff640 1 -- 172.21.15.72:0/3189184335 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac001670 con 0x7fd8bc15c710 2024-04-15T15:15:53.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.243+0000 7fd8c32eb640 1 -- 172.21.15.72:0/3189184335 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd8bc15c710 msgr2=0x7fd8bc0b66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:53.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.243+0000 7fd8c32eb640 1 --2- 172.21.15.72:0/3189184335 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd8bc15c710 0x7fd8bc0b66b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8009d90 tx=0x7fd8b802be00 comp rx=0 tx=0).stop 2024-04-15T15:15:53.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.243+0000 7fd8c32eb640 1 -- 172.21.15.72:0/3189184335 shutdown_connections 2024-04-15T15:15:53.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.243+0000 7fd8c32eb640 1 --2- 172.21.15.72:0/3189184335 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd8bc0d1790 0x7fd8bc0cd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.243+0000 7fd8c32eb640 1 --2- 172.21.15.72:0/3189184335 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd8bc15c710 0x7fd8bc0b66b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.243+0000 7fd8c32eb640 1 -- 172.21.15.72:0/3189184335 >> 172.21.15.72:0/3189184335 conn(0x7fd8bc0b3f90 msgr2=0x7fd8bc0b43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:53.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.243+0000 7fd8c32eb640 1 -- 172.21.15.72:0/3189184335 shutdown_connections 2024-04-15T15:15:53.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.243+0000 7fd8c32eb640 1 -- 172.21.15.72:0/3189184335 wait complete. 2024-04-15T15:15:53.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.244+0000 7fd8c32eb640 1 Processor -- start 2024-04-15T15:15:53.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.244+0000 7fd8c32eb640 1 -- start start 2024-04-15T15:15:53.250 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.245+0000 7fd8c32eb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8bc0b5790 0x7fd8bc0c6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.245+0000 7fd8c32eb640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd8bc0d1790 0x7fd8bc0c6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.245+0000 7fd8c32eb640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd8bc15c710 0x7fd8bc1f15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.245+0000 7fd8c32eb640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd8bc0d1cd0 con 0x7fd8bc0b5790 2024-04-15T15:15:53.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.245+0000 7fd8c32eb640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fd8bc0cd430 con 0x7fd8bc0d1790 2024-04-15T15:15:53.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.245+0000 7fd8c32eb640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fd8bc0b6cd0 con 0x7fd8bc15c710 2024-04-15T15:15:53.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.245+0000 7fd8c1087640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8bc0b5790 0x7fd8bc0c6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:53.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.245+0000 7fd8c1087640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8bc0b5790 0x7fd8bc0c6190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48664/0 (socket says 172.21.15.72:48664) 2024-04-15T15:15:53.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.245+0000 7fd8c1087640 1 -- 172.21.15.72:0/1975177984 learned_addr learned my addr 172.21.15.72:0/1975177984 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:53.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.245+0000 7fd8c0886640 1 --2- 172.21.15.72:0/1975177984 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd8bc0d1790 0x7fd8bc0c6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:53.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.245+0000 7fd8c1888640 1 --2- 172.21.15.72:0/1975177984 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd8bc15c710 0x7fd8bc1f15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:53.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.245+0000 7fd8c1087640 1 -- 172.21.15.72:0/1975177984 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd8bc15c710 msgr2=0x7fd8bc1f15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:53.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.245+0000 7fd8c1087640 1 --2- 172.21.15.72:0/1975177984 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd8bc15c710 0x7fd8bc1f15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.245+0000 7fd8c1087640 1 -- 172.21.15.72:0/1975177984 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd8bc0d1790 msgr2=0x7fd8bc0c6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:53.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.245+0000 7fd8c1087640 1 --2- 172.21.15.72:0/1975177984 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd8bc0d1790 0x7fd8bc0c6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.245+0000 7fd8c1087640 1 -- 172.21.15.72:0/1975177984 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8bc156dc0 con 0x7fd8bc0b5790 2024-04-15T15:15:53.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.246+0000 7fd8c1087640 1 --2- 172.21.15.72:0/1975177984 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8bc0b5790 0x7fd8bc0c6190 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8037040 tx=0x7fd8b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:53.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.246+0000 7fd8b1ffb640 1 -- 172.21.15.72:0/1975177984 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b8039050 con 0x7fd8bc0b5790 2024-04-15T15:15:53.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.246+0000 7fd8b1ffb640 1 -- 172.21.15.72:0/1975177984 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8b8045070 con 0x7fd8bc0b5790 2024-04-15T15:15:53.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.246+0000 7fd8c32eb640 1 -- 172.21.15.72:0/1975177984 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8bc1f1b10 con 0x7fd8bc0b5790 2024-04-15T15:15:53.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.246+0000 7fd8b1ffb640 1 -- 172.21.15.72:0/1975177984 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8b8009450 con 0x7fd8bc0b5790 2024-04-15T15:15:53.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.246+0000 7fd8c32eb640 1 -- 172.21.15.72:0/1975177984 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8bc157310 con 0x7fd8bc0b5790 2024-04-15T15:15:53.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.247+0000 7fd8c32eb640 1 -- 172.21.15.72:0/1975177984 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8bc0c9390 con 0x7fd8bc0b5790 2024-04-15T15:15:53.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.247+0000 7fd8b1ffb640 1 -- 172.21.15.72:0/1975177984 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd8b8040690 con 0x7fd8bc0b5790 2024-04-15T15:15:53.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.248+0000 7fd8b1ffb640 1 --2- 172.21.15.72:0/1975177984 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd8a006c920 0x7fd8a006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.248+0000 7fd8b1ffb640 1 -- 172.21.15.72:0/1975177984 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fd8b80ae240 con 0x7fd8bc0b5790 2024-04-15T15:15:53.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.248+0000 7fd8c0886640 1 --2- 172.21.15.72:0/1975177984 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd8a006c920 0x7fd8a006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:53.252 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.248+0000 7fd8c0886640 1 --2- 172.21.15.72:0/1975177984 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd8a006c920 0x7fd8a006ede0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc0b1040 tx=0x7fd8ac009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:53.253 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.252+0000 7fd8b1ffb640 1 -- 172.21.15.72:0/1975177984 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd8bc0c9390 con 0x7fd8bc0b5790 2024-04-15T15:15:53.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.429+0000 7fd8c32eb640 1 -- 172.21.15.72:0/1975177984 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd8bc0689e0 con 0x7fd8bc0b5790 2024-04-15T15:15:53.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.430+0000 7fd8b1ffb640 1 -- 172.21.15.72:0/1975177984 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18903 (secure 0 0 0) 0x7fd8b804e020 con 0x7fd8bc0b5790 2024-04-15T15:15:53.432 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:53.432 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":26,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:15:32.421728+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:14:49.438366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:14:50.368379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:14:54.081157+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":1}]},{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:53.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.432+0000 7fd8c32eb640 1 -- 172.21.15.72:0/1975177984 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd8a006c920 msgr2=0x7fd8a006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:53.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.432+0000 7fd8c32eb640 1 --2- 172.21.15.72:0/1975177984 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd8a006c920 0x7fd8a006ede0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc0b1040 tx=0x7fd8ac009040 comp rx=0 tx=0).stop 2024-04-15T15:15:53.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.432+0000 7fd8c32eb640 1 -- 172.21.15.72:0/1975177984 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8bc0b5790 msgr2=0x7fd8bc0c6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:53.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.432+0000 7fd8c32eb640 1 --2- 172.21.15.72:0/1975177984 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8bc0b5790 0x7fd8bc0c6190 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8037040 tx=0x7fd8b8004a40 comp rx=0 tx=0).stop 2024-04-15T15:15:53.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.432+0000 7fd8c32eb640 1 -- 172.21.15.72:0/1975177984 shutdown_connections 2024-04-15T15:15:53.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.432+0000 7fd8c32eb640 1 --2- 172.21.15.72:0/1975177984 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd8a006c920 0x7fd8a006ede0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.434 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.432+0000 7fd8c32eb640 1 --2- 172.21.15.72:0/1975177984 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd8bc15c710 0x7fd8bc1f15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.432+0000 7fd8c32eb640 1 --2- 172.21.15.72:0/1975177984 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd8bc0d1790 0x7fd8bc0c6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.432+0000 7fd8c32eb640 1 --2- 172.21.15.72:0/1975177984 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8bc0b5790 0x7fd8bc0c6190 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.432+0000 7fd8c32eb640 1 -- 172.21.15.72:0/1975177984 >> 172.21.15.72:0/1975177984 conn(0x7fd8bc0b3f90 msgr2=0x7fd8bc070fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:53.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.432+0000 7fd8c32eb640 1 -- 172.21.15.72:0/1975177984 shutdown_connections 2024-04-15T15:15:53.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.432+0000 7fd8c32eb640 1 -- 172.21.15.72:0/1975177984 wait complete. 2024-04-15T15:15:53.445 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:39.359102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:14:49.438366+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:14:50.368379+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:14:54.081157+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:15:53.446 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:53.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.553+0000 7f104c349640 1 Processor -- start 2024-04-15T15:15:53.556 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.553+0000 7f104c349640 1 -- start start 2024-04-15T15:15:53.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.554+0000 7f104c349640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f10440b67c0 0x7f10440d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.554+0000 7f104c349640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f10440b6c20 0x7f10440d1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.554+0000 7f104c349640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10440ccfe0 con 0x7f1044161ee0 2024-04-15T15:15:53.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.554+0000 7f104c349640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f10441622c0 con 0x7f10440b6c20 2024-04-15T15:15:53.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.554+0000 7f104c349640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f10440cd2c0 con 0x7f10440b67c0 2024-04-15T15:15:53.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.554+0000 7f104a8e6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f10440b6c20 0x7f10440d1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:53.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.554+0000 7f104a8e6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f10440b6c20 0x7f10440d1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48678/0 (socket says 172.21.15.72:48678) 2024-04-15T15:15:53.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.554+0000 7f104a0e5640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f1044161ee0 0x7f10440c95f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:47798/0 (socket says 172.21.15.72:47798) 2024-04-15T15:15:53.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.554+0000 7f104a8e6640 1 -- 172.21.15.72:0/537691942 learned_addr learned my addr 172.21.15.72:0/537691942 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:53.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.555+0000 7f104a8e6640 1 -- 172.21.15.72:0/537691942 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f10440b67c0 msgr2=0x7f10440d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:53.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.555+0000 7f10498e4640 1 --2- 172.21.15.72:0/537691942 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f10440b67c0 0x7f10440d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:53.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.555+0000 7f104a8e6640 1 --2- 172.21.15.72:0/537691942 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f10440b67c0 0x7f10440d1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.555+0000 7f10490e3640 1 -- 172.21.15.72:0/537691942 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1610395437 0 0) 0x7f10440ccfe0 con 0x7f1044161ee0 2024-04-15T15:15:53.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.555+0000 7f104a8e6640 1 -- 172.21.15.72:0/537691942 >> v1:172.21.15.72:6789/0 conn(0x7f1044161ee0 legacy=0x7f10440c95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:53.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.555+0000 7f104a8e6640 1 -- 172.21.15.72:0/537691942 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10440cd5a0 con 0x7f10440b6c20 2024-04-15T15:15:53.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.555+0000 7f104a8e6640 1 --2- 172.21.15.72:0/537691942 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10440b6c20 0x7f10440d1640 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f103c0098a0 tx=0x7f103c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=874aeb47720fef7d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:53.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.555+0000 7f10490e3640 1 -- 172.21.15.72:0/537691942 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f103c034070 con 0x7f10440b6c20 2024-04-15T15:15:53.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.555+0000 7f10490e3640 1 -- 172.21.15.72:0/537691942 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f103c030840 con 0x7f10440b6c20 2024-04-15T15:15:53.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.555+0000 7f10490e3640 1 -- 172.21.15.72:0/537691942 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f103c030c10 con 0x7f10440b6c20 2024-04-15T15:15:53.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.556+0000 7f104c349640 1 -- 172.21.15.72:0/537691942 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10440b6c20 msgr2=0x7f10440d1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:53.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.556+0000 7f104c349640 1 --2- 172.21.15.72:0/537691942 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10440b6c20 0x7f10440d1640 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f103c0098a0 tx=0x7f103c02b980 comp rx=0 tx=0).stop 2024-04-15T15:15:53.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.556+0000 7f104c349640 1 -- 172.21.15.72:0/537691942 shutdown_connections 2024-04-15T15:15:53.561 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.556+0000 7f104c349640 1 --2- 172.21.15.72:0/537691942 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10440b6c20 0x7f10440d1640 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.556+0000 7f104c349640 1 --2- 172.21.15.72:0/537691942 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f10440b67c0 0x7f10440d1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.556+0000 7f104c349640 1 -- 172.21.15.72:0/537691942 >> 172.21.15.72:0/537691942 conn(0x7f10440b4010 msgr2=0x7f10440b4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:53.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.556+0000 7f104c349640 1 -- 172.21.15.72:0/537691942 shutdown_connections 2024-04-15T15:15:53.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.556+0000 7f104c349640 1 -- 172.21.15.72:0/537691942 wait complete. 2024-04-15T15:15:53.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.557+0000 7f104c349640 1 Processor -- start 2024-04-15T15:15:53.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.557+0000 7f104c349640 1 -- start start 2024-04-15T15:15:53.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.558+0000 7f104c349640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f10440b67c0 0x7f104415ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.558+0000 7f104c349640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10440b6c20 0x7f10441f54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.558+0000 7f104c349640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1044161ee0 0x7f1044070560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.558+0000 7f104c349640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f10440cccf0 con 0x7f10440b6c20 2024-04-15T15:15:53.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.558+0000 7f104c349640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f10440ccb70 con 0x7f10440b67c0 2024-04-15T15:15:53.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.558+0000 7f104c349640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f10440ce080 con 0x7f1044161ee0 2024-04-15T15:15:53.562 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.558+0000 7f10498e4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10440b6c20 0x7f10441f54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:53.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.558+0000 7f10498e4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10440b6c20 0x7f10441f54a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48690/0 (socket says 172.21.15.72:48690) 2024-04-15T15:15:53.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.558+0000 7f10498e4640 1 -- 172.21.15.72:0/1655160228 learned_addr learned my addr 172.21.15.72:0/1655160228 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:53.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.558+0000 7f104a8e6640 1 --2- 172.21.15.72:0/1655160228 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1044161ee0 0x7f1044070560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:53.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.558+0000 7f104a0e5640 1 --2- 172.21.15.72:0/1655160228 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f10440b67c0 0x7f104415ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:53.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.558+0000 7f10498e4640 1 -- 172.21.15.72:0/1655160228 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1044161ee0 msgr2=0x7f1044070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:53.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.558+0000 7f10498e4640 1 --2- 172.21.15.72:0/1655160228 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1044161ee0 0x7f1044070560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.558+0000 7f10498e4640 1 -- 172.21.15.72:0/1655160228 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f10440b67c0 msgr2=0x7f104415ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:53.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.558+0000 7f10498e4640 1 --2- 172.21.15.72:0/1655160228 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f10440b67c0 0x7f104415ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.558+0000 7f10498e4640 1 -- 172.21.15.72:0/1655160228 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f104406df50 con 0x7f10440b6c20 2024-04-15T15:15:53.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.558+0000 7f104a0e5640 1 --2- 172.21.15.72:0/1655160228 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f10440b67c0 0x7f104415ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:53.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.558+0000 7f10498e4640 1 --2- 172.21.15.72:0/1655160228 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10440b6c20 0x7f10441f54a0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f103400c820 tx=0x7f103400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:53.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.559+0000 7f103affd640 1 -- 172.21.15.72:0/1655160228 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1034015070 con 0x7f10440b6c20 2024-04-15T15:15:53.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.559+0000 7f103affd640 1 -- 172.21.15.72:0/1655160228 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1034004240 con 0x7f10440b6c20 2024-04-15T15:15:53.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.559+0000 7f104c349640 1 -- 172.21.15.72:0/1655160228 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f104406e1f0 con 0x7f10440b6c20 2024-04-15T15:15:53.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.559+0000 7f103affd640 1 -- 172.21.15.72:0/1655160228 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f103400ce10 con 0x7f10440b6c20 2024-04-15T15:15:53.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.559+0000 7f104c349640 1 -- 172.21.15.72:0/1655160228 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f104406e7b0 con 0x7f10440b6c20 2024-04-15T15:15:53.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.560+0000 7f104c349640 1 -- 172.21.15.72:0/1655160228 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1044075c50 con 0x7f10440b6c20 2024-04-15T15:15:53.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.560+0000 7f103affd640 1 -- 172.21.15.72:0/1655160228 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1034007830 con 0x7f10440b6c20 2024-04-15T15:15:53.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.561+0000 7f103affd640 1 --2- 172.21.15.72:0/1655160228 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f102806c920 0x7f102806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.561+0000 7f104a0e5640 1 --2- 172.21.15.72:0/1655160228 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f102806c920 0x7f102806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:53.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.561+0000 7f103affd640 1 -- 172.21.15.72:0/1655160228 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f103408be40 con 0x7f10440b6c20 2024-04-15T15:15:53.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.561+0000 7f104a0e5640 1 --2- 172.21.15.72:0/1655160228 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f102806c920 0x7f102806ede0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f10400012e0 tx=0x7f1040005b90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:53.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.565+0000 7f103affd640 1 -- 172.21.15.72:0/1655160228 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1034059e70 con 0x7f10440b6c20 2024-04-15T15:15:53.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.740+0000 7f104c349640 1 -- 172.21.15.72:0/1655160228 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f10440d1e50 con 0x7f10440b6c20 2024-04-15T15:15:53.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.740+0000 7f103affd640 1 -- 172.21.15.72:0/1655160228 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f1034059770 con 0x7f10440b6c20 2024-04-15T15:15:53.742 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-15T15:15:53.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.742+0000 7f104c349640 1 -- 172.21.15.72:0/1655160228 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f102806c920 msgr2=0x7f102806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:53.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.742+0000 7f104c349640 1 --2- 172.21.15.72:0/1655160228 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f102806c920 0x7f102806ede0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f10400012e0 tx=0x7f1040005b90 comp rx=0 tx=0).stop 2024-04-15T15:15:53.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.742+0000 7f104c349640 1 -- 172.21.15.72:0/1655160228 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10440b6c20 msgr2=0x7f10441f54a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:53.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.742+0000 7f104c349640 1 --2- 172.21.15.72:0/1655160228 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10440b6c20 0x7f10441f54a0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f103400c820 tx=0x7f103400cce0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.742+0000 7f104c349640 1 -- 172.21.15.72:0/1655160228 shutdown_connections 2024-04-15T15:15:53.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.742+0000 7f104c349640 1 --2- 172.21.15.72:0/1655160228 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f102806c920 0x7f102806ede0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.742+0000 7f104c349640 1 --2- 172.21.15.72:0/1655160228 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1044161ee0 0x7f1044070560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.742+0000 7f104c349640 1 --2- 172.21.15.72:0/1655160228 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f10440b6c20 0x7f10441f54a0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.742+0000 7f104c349640 1 --2- 172.21.15.72:0/1655160228 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f10440b67c0 0x7f104415ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.742+0000 7f104c349640 1 -- 172.21.15.72:0/1655160228 >> 172.21.15.72:0/1655160228 conn(0x7f10440b4010 msgr2=0x7f10440ce440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:53.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.742+0000 7f104c349640 1 -- 172.21.15.72:0/1655160228 shutdown_connections 2024-04-15T15:15:53.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.742+0000 7f104c349640 1 -- 172.21.15.72:0/1655160228 wait complete. 2024-04-15T15:15:53.755 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:15:53.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.859+0000 7f464193f640 1 Processor -- start 2024-04-15T15:15:53.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.859+0000 7f464193f640 1 -- start start 2024-04-15T15:15:53.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.860+0000 7f464193f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f463c15af20 0x7f463c06dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.860+0000 7f464193f640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f463c06d030 0x7f463c0d0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.860+0000 7f464193f640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f463c15c110 con 0x7f463c0ccb50 2024-04-15T15:15:53.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.860+0000 7f464193f640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f463c06ba90 con 0x7f463c15af20 2024-04-15T15:15:53.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.860+0000 7f464193f640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f463c0cfae0 con 0x7f463c06d030 2024-04-15T15:15:53.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.860+0000 7f463affd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f463c15af20 0x7f463c06dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:53.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.860+0000 7f463affd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f463c15af20 0x7f463c06dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48702/0 (socket says 172.21.15.72:48702) 2024-04-15T15:15:53.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.860+0000 7f463affd640 1 -- 172.21.15.72:0/3738221227 learned_addr learned my addr 172.21.15.72:0/3738221227 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:53.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.860+0000 7f463a7fc640 1 --2- 172.21.15.72:0/3738221227 >> v2:172.21.15.175:3300/0 conn(0x7f463c06d030 0x7f463c0d0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:53.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.860+0000 7f463affd640 1 -- 172.21.15.72:0/3738221227 >> v2:172.21.15.175:3300/0 conn(0x7f463c06d030 msgr2=0x7f463c0d0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:53.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.860+0000 7f463affd640 1 --2- 172.21.15.72:0/3738221227 >> v2:172.21.15.175:3300/0 conn(0x7f463c06d030 0x7f463c0d0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.860+0000 7f463affd640 1 -- 172.21.15.72:0/3738221227 >> v1:172.21.15.175:6789/0 conn(0x7f463c0ccb50 legacy=0x7f463c15e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:53.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.860+0000 7f463affd640 1 -- 172.21.15.72:0/3738221227 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f463c0cfdc0 con 0x7f463c15af20 2024-04-15T15:15:53.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.860+0000 7f463a7fc640 1 --2- 172.21.15.72:0/3738221227 >> v2:172.21.15.175:3300/0 conn(0x7f463c06d030 0x7f463c0d0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:53.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.861+0000 7f463affd640 1 --2- 172.21.15.72:0/3738221227 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f463c15af20 0x7f463c06dc10 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f463c07ac20 tx=0x7f462c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f092e57b35e80f75 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:53.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.861+0000 7f4639ffb640 1 -- 172.21.15.72:0/3738221227 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f462c02c9f0 con 0x7f463c15af20 2024-04-15T15:15:53.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.861+0000 7f4639ffb640 1 -- 172.21.15.72:0/3738221227 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f462c02cb90 con 0x7f463c15af20 2024-04-15T15:15:53.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.861+0000 7f4639ffb640 1 -- 172.21.15.72:0/3738221227 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f462c0343f0 con 0x7f463c15af20 2024-04-15T15:15:53.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.862+0000 7f464193f640 1 -- 172.21.15.72:0/3738221227 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f463c15af20 msgr2=0x7f463c06dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:53.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.862+0000 7f464193f640 1 --2- 172.21.15.72:0/3738221227 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f463c15af20 0x7f463c06dc10 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f463c07ac20 tx=0x7f462c02be50 comp rx=0 tx=0).stop 2024-04-15T15:15:53.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.862+0000 7f464193f640 1 -- 172.21.15.72:0/3738221227 shutdown_connections 2024-04-15T15:15:53.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.862+0000 7f464193f640 1 --2- 172.21.15.72:0/3738221227 >> v2:172.21.15.175:3300/0 conn(0x7f463c06d030 0x7f463c0d0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.862+0000 7f464193f640 1 --2- 172.21.15.72:0/3738221227 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f463c15af20 0x7f463c06dc10 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.862+0000 7f464193f640 1 -- 172.21.15.72:0/3738221227 >> 172.21.15.72:0/3738221227 conn(0x7f463c0b31d0 msgr2=0x7f463c0b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:53.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.862+0000 7f464193f640 1 -- 172.21.15.72:0/3738221227 shutdown_connections 2024-04-15T15:15:53.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.862+0000 7f464193f640 1 -- 172.21.15.72:0/3738221227 wait complete. 2024-04-15T15:15:53.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.863+0000 7f464193f640 1 Processor -- start 2024-04-15T15:15:53.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.863+0000 7f464193f640 1 -- start start 2024-04-15T15:15:53.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.864+0000 7f464193f640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f463c06d030 0x7f463c09a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.864+0000 7f464193f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f463c0ccb50 0x7f463c154b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.864+0000 7f464193f640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f463c15af20 0x7f463c09cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.864+0000 7f464193f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f463c0cc8f0 con 0x7f463c0ccb50 2024-04-15T15:15:53.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.864+0000 7f464193f640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f463c0cc770 con 0x7f463c15af20 2024-04-15T15:15:53.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.864+0000 7f464193f640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f463c0d06c0 con 0x7f463c06d030 2024-04-15T15:15:53.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.864+0000 7f463a7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f463c0ccb50 0x7f463c154b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:53.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.864+0000 7f463a7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f463c0ccb50 0x7f463c154b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48716/0 (socket says 172.21.15.72:48716) 2024-04-15T15:15:53.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.864+0000 7f463a7fc640 1 -- 172.21.15.72:0/3380058280 learned_addr learned my addr 172.21.15.72:0/3380058280 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:53.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.864+0000 7f463b7fe640 1 --2- 172.21.15.72:0/3380058280 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f463c15af20 0x7f463c09cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:53.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.864+0000 7f463affd640 1 --2- 172.21.15.72:0/3380058280 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f463c06d030 0x7f463c09a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:53.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.864+0000 7f463a7fc640 1 -- 172.21.15.72:0/3380058280 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f463c06d030 msgr2=0x7f463c09a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:53.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.864+0000 7f463a7fc640 1 --2- 172.21.15.72:0/3380058280 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f463c06d030 0x7f463c09a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.864+0000 7f463a7fc640 1 -- 172.21.15.72:0/3380058280 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f463c15af20 msgr2=0x7f463c09cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:53.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.864+0000 7f463a7fc640 1 --2- 172.21.15.72:0/3380058280 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f463c15af20 0x7f463c09cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.864+0000 7f463a7fc640 1 -- 172.21.15.72:0/3380058280 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f463c1550d0 con 0x7f463c0ccb50 2024-04-15T15:15:53.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.864+0000 7f463b7fe640 1 --2- 172.21.15.72:0/3380058280 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f463c15af20 0x7f463c09cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:53.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.864+0000 7f463a7fc640 1 --2- 172.21.15.72:0/3380058280 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f463c0ccb50 0x7f463c154b90 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f463000c7f0 tx=0x7f463000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:53.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.864+0000 7f4623fff640 1 -- 172.21.15.72:0/3380058280 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4630015070 con 0x7f463c0ccb50 2024-04-15T15:15:53.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.865+0000 7f4623fff640 1 -- 172.21.15.72:0/3380058280 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4630007a30 con 0x7f463c0ccb50 2024-04-15T15:15:53.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.865+0000 7f464193f640 1 -- 172.21.15.72:0/3380058280 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f463c09aad0 con 0x7f463c0ccb50 2024-04-15T15:15:53.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.865+0000 7f4623fff640 1 -- 172.21.15.72:0/3380058280 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4630007e00 con 0x7f463c0ccb50 2024-04-15T15:15:53.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.865+0000 7f464193f640 1 -- 172.21.15.72:0/3380058280 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f463c09b090 con 0x7f463c0ccb50 2024-04-15T15:15:53.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.865+0000 7f464193f640 1 -- 172.21.15.72:0/3380058280 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f463c075c50 con 0x7f463c0ccb50 2024-04-15T15:15:53.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.866+0000 7f4623fff640 1 -- 172.21.15.72:0/3380058280 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4630018070 con 0x7f463c0ccb50 2024-04-15T15:15:53.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.866+0000 7f4623fff640 1 --2- 172.21.15.72:0/3380058280 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f462806c920 0x7f462806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.866+0000 7f4623fff640 1 -- 172.21.15.72:0/3380058280 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f463008beb0 con 0x7f463c0ccb50 2024-04-15T15:15:53.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.866+0000 7f463affd640 1 --2- 172.21.15.72:0/3380058280 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f462806c920 0x7f462806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:53.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.867+0000 7f463affd640 1 --2- 172.21.15.72:0/3380058280 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f462806c920 0x7f462806ede0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f462c035000 tx=0x7f462c030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:53.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:53.871+0000 7f4623fff640 1 -- 172.21.15.72:0/3380058280 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4630059ef0 con 0x7f463c0ccb50 2024-04-15T15:15:54.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.043+0000 7f464193f640 1 -- 172.21.15.72:0/3380058280 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f463c073fd0 con 0x7f463c0ccb50 2024-04-15T15:15:54.045 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.044+0000 7f4623fff640 1 -- 172.21.15.72:0/3380058280 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f46300600c0 con 0x7f463c0ccb50 2024-04-15T15:15:54.045 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:54.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.046+0000 7f464193f640 1 -- 172.21.15.72:0/3380058280 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f462806c920 msgr2=0x7f462806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:54.047 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.046+0000 7f464193f640 1 --2- 172.21.15.72:0/3380058280 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f462806c920 0x7f462806ede0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f462c035000 tx=0x7f462c030000 comp rx=0 tx=0).stop 2024-04-15T15:15:54.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.046+0000 7f464193f640 1 -- 172.21.15.72:0/3380058280 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f463c0ccb50 msgr2=0x7f463c154b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:54.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.046+0000 7f464193f640 1 --2- 172.21.15.72:0/3380058280 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f463c0ccb50 0x7f463c154b90 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f463000c7f0 tx=0x7f463000cbf0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.046+0000 7f464193f640 1 -- 172.21.15.72:0/3380058280 shutdown_connections 2024-04-15T15:15:54.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.046+0000 7f464193f640 1 --2- 172.21.15.72:0/3380058280 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f462806c920 0x7f462806ede0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.046+0000 7f464193f640 1 --2- 172.21.15.72:0/3380058280 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f463c15af20 0x7f463c09cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.046+0000 7f464193f640 1 --2- 172.21.15.72:0/3380058280 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f463c0ccb50 0x7f463c154b90 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.046+0000 7f464193f640 1 --2- 172.21.15.72:0/3380058280 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f463c06d030 0x7f463c09a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.046+0000 7f464193f640 1 -- 172.21.15.72:0/3380058280 >> 172.21.15.72:0/3380058280 conn(0x7f463c0b31d0 msgr2=0x7f463c072aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:54.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.046+0000 7f464193f640 1 -- 172.21.15.72:0/3380058280 shutdown_connections 2024-04-15T15:15:54.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.046+0000 7f464193f640 1 -- 172.21.15.72:0/3380058280 wait complete. 2024-04-15T15:15:54.058 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:15:54.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.163+0000 7ff2d331c640 1 Processor -- start 2024-04-15T15:15:54.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.163+0000 7ff2d331c640 1 -- start start 2024-04-15T15:15:54.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.164+0000 7ff2d331c640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff2cc0d08f0 0x7ff2cc0cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:54.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.164+0000 7ff2d331c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff2cc0ce6d0 0x7ff2cc0cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:54.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.164+0000 7ff2d331c640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2cc06bbb0 con 0x7ff2cc0ce2f0 2024-04-15T15:15:54.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.164+0000 7ff2d331c640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7ff2cc06b960 con 0x7ff2cc0ce6d0 2024-04-15T15:15:54.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.164+0000 7ff2d331c640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7ff2cc06be90 con 0x7ff2cc0d08f0 2024-04-15T15:15:54.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.164+0000 7ff2d18b9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff2cc0ce6d0 0x7ff2cc0cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:54.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.164+0000 7ff2d18b9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff2cc0ce6d0 0x7ff2cc0cc5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48732/0 (socket says 172.21.15.72:48732) 2024-04-15T15:15:54.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.164+0000 7ff2d18b9640 1 -- 172.21.15.72:0/2400222761 learned_addr learned my addr 172.21.15.72:0/2400222761 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:54.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.164+0000 7ff2d08b7640 1 --2- 172.21.15.72:0/2400222761 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff2cc0d08f0 0x7ff2cc0cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:54.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.164+0000 7ff2d18b9640 1 -- 172.21.15.72:0/2400222761 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff2cc0d08f0 msgr2=0x7ff2cc0cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:54.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.164+0000 7ff2d18b9640 1 --2- 172.21.15.72:0/2400222761 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff2cc0d08f0 0x7ff2cc0cc0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.164+0000 7ff2d18b9640 1 -- 172.21.15.72:0/2400222761 >> v1:172.21.15.175:6789/0 conn(0x7ff2cc0ce2f0 legacy=0x7ff2cc15c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:54.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.164+0000 7ff2d18b9640 1 -- 172.21.15.72:0/2400222761 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2cc06c170 con 0x7ff2cc0ce6d0 2024-04-15T15:15:54.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.165+0000 7ff2d18b9640 1 --2- 172.21.15.72:0/2400222761 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2cc0ce6d0 0x7ff2cc0cc5f0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff2c4009e50 tx=0x7ff2c402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c61961884043aef server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:54.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.165+0000 7ff2c3fff640 1 -- 172.21.15.72:0/2400222761 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c402cb00 con 0x7ff2cc0ce6d0 2024-04-15T15:15:54.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.165+0000 7ff2c3fff640 1 -- 172.21.15.72:0/2400222761 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2c402cca0 con 0x7ff2cc0ce6d0 2024-04-15T15:15:54.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.165+0000 7ff2c3fff640 1 -- 172.21.15.72:0/2400222761 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c4034610 con 0x7ff2cc0ce6d0 2024-04-15T15:15:54.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.166+0000 7ff2d331c640 1 -- 172.21.15.72:0/2400222761 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2cc0ce6d0 msgr2=0x7ff2cc0cc5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:54.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.166+0000 7ff2d331c640 1 --2- 172.21.15.72:0/2400222761 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2cc0ce6d0 0x7ff2cc0cc5f0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff2c4009e50 tx=0x7ff2c402bf60 comp rx=0 tx=0).stop 2024-04-15T15:15:54.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.166+0000 7ff2d331c640 1 -- 172.21.15.72:0/2400222761 shutdown_connections 2024-04-15T15:15:54.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.166+0000 7ff2d331c640 1 --2- 172.21.15.72:0/2400222761 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2cc0ce6d0 0x7ff2cc0cc5f0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.166+0000 7ff2d331c640 1 --2- 172.21.15.72:0/2400222761 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff2cc0d08f0 0x7ff2cc0cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.166+0000 7ff2d331c640 1 -- 172.21.15.72:0/2400222761 >> 172.21.15.72:0/2400222761 conn(0x7ff2cc0b31d0 msgr2=0x7ff2cc0b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:54.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.166+0000 7ff2d331c640 1 -- 172.21.15.72:0/2400222761 shutdown_connections 2024-04-15T15:15:54.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.166+0000 7ff2d331c640 1 -- 172.21.15.72:0/2400222761 wait complete. 2024-04-15T15:15:54.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.167+0000 7ff2d331c640 1 Processor -- start 2024-04-15T15:15:54.169 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.167+0000 7ff2d331c640 1 -- start start 2024-04-15T15:15:54.171 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.167+0000 7ff2d331c640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff2cc0ce2f0 0x7ff2cc0cb910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:54.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.168+0000 7ff2d331c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2cc0ce6d0 0x7ff2cc15dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:54.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.168+0000 7ff2d331c640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff2cc0d08f0 0x7ff2cc0ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:54.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.168+0000 7ff2d331c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ff2cc155780 con 0x7ff2cc0ce6d0 2024-04-15T15:15:54.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.168+0000 7ff2d331c640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7ff2cc155600 con 0x7ff2cc0d08f0 2024-04-15T15:15:54.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.168+0000 7ff2d331c640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7ff2cc155900 con 0x7ff2cc0ce2f0 2024-04-15T15:15:54.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.168+0000 7ff2d08b7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2cc0ce6d0 0x7ff2cc15dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:54.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.168+0000 7ff2d08b7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2cc0ce6d0 0x7ff2cc15dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48742/0 (socket says 172.21.15.72:48742) 2024-04-15T15:15:54.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.168+0000 7ff2d08b7640 1 -- 172.21.15.72:0/759347415 learned_addr learned my addr 172.21.15.72:0/759347415 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:54.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.168+0000 7ff2d08b7640 1 -- 172.21.15.72:0/759347415 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff2cc0ce2f0 msgr2=0x7ff2cc0cb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:54.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.168+0000 7ff2d08b7640 1 --2- 172.21.15.72:0/759347415 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff2cc0ce2f0 0x7ff2cc0cb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.168+0000 7ff2d08b7640 1 -- 172.21.15.72:0/759347415 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff2cc0d08f0 msgr2=0x7ff2cc0ccc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:54.172 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.168+0000 7ff2d18b9640 1 --2- 172.21.15.72:0/759347415 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff2cc0d08f0 0x7ff2cc0ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:54.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.168+0000 7ff2d08b7640 1 --2- 172.21.15.72:0/759347415 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff2cc0d08f0 0x7ff2cc0ccc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.168+0000 7ff2d08b7640 1 -- 172.21.15.72:0/759347415 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2cc157e50 con 0x7ff2cc0ce6d0 2024-04-15T15:15:54.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.168+0000 7ff2d18b9640 1 --2- 172.21.15.72:0/759347415 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff2cc0d08f0 0x7ff2cc0ccc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:54.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.168+0000 7ff2d08b7640 1 --2- 172.21.15.72:0/759347415 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2cc0ce6d0 0x7ff2cc15dbc0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc00d620 tx=0x7ff2bc00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:54.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.168+0000 7ff2c1ffb640 1 -- 172.21.15.72:0/759347415 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2bc00b620 con 0x7ff2cc0ce6d0 2024-04-15T15:15:54.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.168+0000 7ff2c1ffb640 1 -- 172.21.15.72:0/759347415 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2bc019070 con 0x7ff2cc0ce6d0 2024-04-15T15:15:54.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.168+0000 7ff2c1ffb640 1 -- 172.21.15.72:0/759347415 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2bc00ba00 con 0x7ff2cc0ce6d0 2024-04-15T15:15:54.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.168+0000 7ff2d331c640 1 -- 172.21.15.72:0/759347415 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2cc158130 con 0x7ff2cc0ce6d0 2024-04-15T15:15:54.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.168+0000 7ff2d331c640 1 -- 172.21.15.72:0/759347415 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2cc1586f0 con 0x7ff2cc0ce6d0 2024-04-15T15:15:54.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.170+0000 7ff2d331c640 1 -- 172.21.15.72:0/759347415 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2cc075c50 con 0x7ff2cc0ce6d0 2024-04-15T15:15:54.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.170+0000 7ff2c1ffb640 1 -- 172.21.15.72:0/759347415 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff2bc00bba0 con 0x7ff2cc0ce6d0 2024-04-15T15:15:54.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.170+0000 7ff2c1ffb640 1 --2- 172.21.15.72:0/759347415 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff2b006c8f0 0x7ff2b006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:54.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.170+0000 7ff2c1ffb640 1 -- 172.21.15.72:0/759347415 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7ff2bc08c340 con 0x7ff2cc0ce6d0 2024-04-15T15:15:54.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.170+0000 7ff2d10b8640 1 --2- 172.21.15.72:0/759347415 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff2b006c8f0 0x7ff2b006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:54.174 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.171+0000 7ff2d10b8640 1 --2- 172.21.15.72:0/759347415 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff2b006c8f0 0x7ff2b006edb0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff2c8000f60 tx=0x7ff2c80079d0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:54.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.175+0000 7ff2c1ffb640 1 -- 172.21.15.72:0/759347415 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff2bc05a370 con 0x7ff2cc0ce6d0 2024-04-15T15:15:54.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.350+0000 7ff2d331c640 1 -- 172.21.15.72:0/759347415 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff2cc0cbe50 con 0x7ff2cc0ce6d0 2024-04-15T15:15:54.352 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.350+0000 7ff2c1ffb640 1 -- 172.21.15.72:0/759347415 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7ff2bc060030 con 0x7ff2cc0ce6d0 2024-04-15T15:15:54.352 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-04-15T15:15:54.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.352+0000 7ff2d331c640 1 -- 172.21.15.72:0/759347415 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff2b006c8f0 msgr2=0x7ff2b006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:54.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.352+0000 7ff2d331c640 1 --2- 172.21.15.72:0/759347415 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff2b006c8f0 0x7ff2b006edb0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff2c8000f60 tx=0x7ff2c80079d0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.352+0000 7ff2d331c640 1 -- 172.21.15.72:0/759347415 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2cc0ce6d0 msgr2=0x7ff2cc15dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:54.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.352+0000 7ff2d331c640 1 --2- 172.21.15.72:0/759347415 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2cc0ce6d0 0x7ff2cc15dbc0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7ff2bc00d620 tx=0x7ff2bc00dae0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.352+0000 7ff2d331c640 1 -- 172.21.15.72:0/759347415 shutdown_connections 2024-04-15T15:15:54.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.352+0000 7ff2d331c640 1 --2- 172.21.15.72:0/759347415 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff2b006c8f0 0x7ff2b006edb0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.352+0000 7ff2d331c640 1 --2- 172.21.15.72:0/759347415 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff2cc0d08f0 0x7ff2cc0ccc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.352+0000 7ff2d331c640 1 --2- 172.21.15.72:0/759347415 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff2cc0ce6d0 0x7ff2cc15dbc0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.352+0000 7ff2d331c640 1 --2- 172.21.15.72:0/759347415 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff2cc0ce2f0 0x7ff2cc0cb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.354 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.352+0000 7ff2d331c640 1 -- 172.21.15.72:0/759347415 >> 172.21.15.72:0/759347415 conn(0x7ff2cc0b31d0 msgr2=0x7ff2cc071bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:54.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.352+0000 7ff2d331c640 1 -- 172.21.15.72:0/759347415 shutdown_connections 2024-04-15T15:15:54.355 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.352+0000 7ff2d331c640 1 -- 172.21.15.72:0/759347415 wait complete. 2024-04-15T15:15:54.366 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:15:54.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.472+0000 7f4e5bba3640 1 Processor -- start 2024-04-15T15:15:54.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.472+0000 7f4e5bba3640 1 -- start start 2024-04-15T15:15:54.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.473+0000 7f4e5bba3640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4e5415a0c0 0x7f4e540ce030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:54.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.473+0000 7f4e5bba3640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4e5415d960 0x7f4e54073360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:54.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.473+0000 7f4e5bba3640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f4e540ce570 0x7f4e540738a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:54.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.473+0000 7f4e5bba3640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f4e5415cc00 con 0x7f4e5415d960 2024-04-15T15:15:54.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.473+0000 7f4e5bba3640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f4e540ceaf0 con 0x7f4e540ce570 2024-04-15T15:15:54.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.473+0000 7f4e5bba3640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f4e5415ce80 con 0x7f4e5415a0c0 2024-04-15T15:15:54.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.473+0000 7f4e5913e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4e5415d960 0x7f4e54073360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:54.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.473+0000 7f4e5913e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4e5415d960 0x7f4e54073360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48756/0 (socket says 172.21.15.72:48756) 2024-04-15T15:15:54.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.473+0000 7f4e5913e640 1 -- 172.21.15.72:0/34863778 learned_addr learned my addr 172.21.15.72:0/34863778 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:54.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.474+0000 7f4e5a140640 1 --2- 172.21.15.72:0/34863778 >> v2:172.21.15.175:3300/0 conn(0x7f4e540ce570 0x7f4e540738a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:54.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.474+0000 7f4e5993f640 1 --2- 172.21.15.72:0/34863778 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4e5415a0c0 0x7f4e540ce030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:54.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.474+0000 7f4e5913e640 1 -- 172.21.15.72:0/34863778 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4e5415a0c0 msgr2=0x7f4e540ce030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:54.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.474+0000 7f4e5913e640 1 --2- 172.21.15.72:0/34863778 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4e5415a0c0 0x7f4e540ce030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.474+0000 7f4e5913e640 1 -- 172.21.15.72:0/34863778 >> v2:172.21.15.175:3300/0 conn(0x7f4e540ce570 msgr2=0x7f4e540738a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:54.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.474+0000 7f4e5913e640 1 --2- 172.21.15.72:0/34863778 >> v2:172.21.15.175:3300/0 conn(0x7f4e540ce570 0x7f4e540738a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.474+0000 7f4e5913e640 1 -- 172.21.15.72:0/34863778 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e5415d160 con 0x7f4e5415d960 2024-04-15T15:15:54.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.474+0000 7f4e5a140640 1 --2- 172.21.15.72:0/34863778 >> v2:172.21.15.175:3300/0 conn(0x7f4e540ce570 0x7f4e540738a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:54.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.474+0000 7f4e5993f640 1 --2- 172.21.15.72:0/34863778 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4e5415a0c0 0x7f4e540ce030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:54.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.474+0000 7f4e5913e640 1 --2- 172.21.15.72:0/34863778 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e5415d960 0x7f4e54073360 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f4e54161310 tx=0x7f4e4402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b65a2134d52ed05 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:54.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.475+0000 7f4e5893d640 1 -- 172.21.15.72:0/34863778 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e4402e070 con 0x7f4e5415d960 2024-04-15T15:15:54.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.475+0000 7f4e5893d640 1 -- 172.21.15.72:0/34863778 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e44037070 con 0x7f4e5415d960 2024-04-15T15:15:54.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.475+0000 7f4e5893d640 1 -- 172.21.15.72:0/34863778 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e44031a00 con 0x7f4e5415d960 2024-04-15T15:15:54.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.475+0000 7f4e5bba3640 1 -- 172.21.15.72:0/34863778 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e5415d960 msgr2=0x7f4e54073360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:54.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.475+0000 7f4e5bba3640 1 --2- 172.21.15.72:0/34863778 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e5415d960 0x7f4e54073360 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f4e54161310 tx=0x7f4e4402b880 comp rx=0 tx=0).stop 2024-04-15T15:15:54.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.475+0000 7f4e5bba3640 1 -- 172.21.15.72:0/34863778 shutdown_connections 2024-04-15T15:15:54.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.475+0000 7f4e5bba3640 1 --2- 172.21.15.72:0/34863778 >> v2:172.21.15.175:3300/0 conn(0x7f4e540ce570 0x7f4e540738a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.475+0000 7f4e5bba3640 1 --2- 172.21.15.72:0/34863778 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e5415d960 0x7f4e54073360 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.475+0000 7f4e5bba3640 1 --2- 172.21.15.72:0/34863778 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4e5415a0c0 0x7f4e540ce030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.475+0000 7f4e5bba3640 1 -- 172.21.15.72:0/34863778 >> 172.21.15.72:0/34863778 conn(0x7f4e54071410 msgr2=0x7f4e54072c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:54.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.475+0000 7f4e5bba3640 1 -- 172.21.15.72:0/34863778 shutdown_connections 2024-04-15T15:15:54.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.475+0000 7f4e5bba3640 1 -- 172.21.15.72:0/34863778 wait complete. 2024-04-15T15:15:54.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.476+0000 7f4e5bba3640 1 Processor -- start 2024-04-15T15:15:54.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.477+0000 7f4e5bba3640 1 -- start start 2024-04-15T15:15:54.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.477+0000 7f4e5bba3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e540ce570 0x7f4e541ef7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:54.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.477+0000 7f4e5bba3640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4e5415a0c0 0x7f4e541f34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:54.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.477+0000 7f4e5bba3640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4e5415d960 0x7f4e54142ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:54.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.477+0000 7f4e5bba3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f4e5415a620 con 0x7f4e540ce570 2024-04-15T15:15:54.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.477+0000 7f4e5bba3640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f4e5415a4a0 con 0x7f4e5415a0c0 2024-04-15T15:15:54.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.477+0000 7f4e5bba3640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f4e54145510 con 0x7f4e5415d960 2024-04-15T15:15:54.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.477+0000 7f4e5993f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e540ce570 0x7f4e541ef7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:54.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.477+0000 7f4e5993f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e540ce570 0x7f4e541ef7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48758/0 (socket says 172.21.15.72:48758) 2024-04-15T15:15:54.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.477+0000 7f4e5993f640 1 -- 172.21.15.72:0/1401389622 learned_addr learned my addr 172.21.15.72:0/1401389622 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:54.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.478+0000 7f4e5993f640 1 -- 172.21.15.72:0/1401389622 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4e5415d960 msgr2=0x7f4e54142ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:54.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.478+0000 7f4e5993f640 1 --2- 172.21.15.72:0/1401389622 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4e5415d960 0x7f4e54142ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.478+0000 7f4e5993f640 1 -- 172.21.15.72:0/1401389622 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4e5415a0c0 msgr2=0x7f4e541f34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:54.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.478+0000 7f4e5993f640 1 --2- 172.21.15.72:0/1401389622 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4e5415a0c0 0x7f4e541f34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.478+0000 7f4e5993f640 1 -- 172.21.15.72:0/1401389622 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e54144940 con 0x7f4e540ce570 2024-04-15T15:15:54.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.478+0000 7f4e5993f640 1 --2- 172.21.15.72:0/1401389622 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e540ce570 0x7f4e541ef7e0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f4e50007fa0 tx=0x7f4e5000d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:54.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.478+0000 7f4e4a7fc640 1 -- 172.21.15.72:0/1401389622 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e5001a070 con 0x7f4e540ce570 2024-04-15T15:15:54.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.478+0000 7f4e4a7fc640 1 -- 172.21.15.72:0/1401389622 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e50003b90 con 0x7f4e540ce570 2024-04-15T15:15:54.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.478+0000 7f4e5bba3640 1 -- 172.21.15.72:0/1401389622 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e541efd20 con 0x7f4e540ce570 2024-04-15T15:15:54.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.478+0000 7f4e4a7fc640 1 -- 172.21.15.72:0/1401389622 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e50004030 con 0x7f4e540ce570 2024-04-15T15:15:54.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.478+0000 7f4e5bba3640 1 -- 172.21.15.72:0/1401389622 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e54143910 con 0x7f4e540ce570 2024-04-15T15:15:54.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.479+0000 7f4e5bba3640 1 -- 172.21.15.72:0/1401389622 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e54075c50 con 0x7f4e540ce570 2024-04-15T15:15:54.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.480+0000 7f4e4a7fc640 1 -- 172.21.15.72:0/1401389622 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4e5001d070 con 0x7f4e540ce570 2024-04-15T15:15:54.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.480+0000 7f4e4a7fc640 1 --2- 172.21.15.72:0/1401389622 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4e3806c920 0x7f4e3806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:54.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.480+0000 7f4e4a7fc640 1 -- 172.21.15.72:0/1401389622 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f4e5008c0b0 con 0x7f4e540ce570 2024-04-15T15:15:54.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.480+0000 7f4e5913e640 1 --2- 172.21.15.72:0/1401389622 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4e3806c920 0x7f4e3806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:54.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.480+0000 7f4e5913e640 1 --2- 172.21.15.72:0/1401389622 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4e3806c920 0x7f4e3806ede0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4e44036000 tx=0x7f4e44030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:54.485 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.484+0000 7f4e4a7fc640 1 -- 172.21.15.72:0/1401389622 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4e50010070 con 0x7f4e540ce570 2024-04-15T15:15:54.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.658+0000 7f4e5bba3640 1 -- 172.21.15.72:0/1401389622 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f4e540b6cf0 con 0x7f4e540ce570 2024-04-15T15:15:54.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.659+0000 7f4e4a7fc640 1 -- 172.21.15.72:0/1401389622 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f4e50059b20 con 0x7f4e540ce570 2024-04-15T15:15:54.661 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:15:54.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.661+0000 7f4e5bba3640 1 -- 172.21.15.72:0/1401389622 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4e3806c920 msgr2=0x7f4e3806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:54.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.661+0000 7f4e5bba3640 1 --2- 172.21.15.72:0/1401389622 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4e3806c920 0x7f4e3806ede0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4e44036000 tx=0x7f4e44030000 comp rx=0 tx=0).stop 2024-04-15T15:15:54.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.661+0000 7f4e5bba3640 1 -- 172.21.15.72:0/1401389622 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e540ce570 msgr2=0x7f4e541ef7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:54.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.661+0000 7f4e5bba3640 1 --2- 172.21.15.72:0/1401389622 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e540ce570 0x7f4e541ef7e0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f4e50007fa0 tx=0x7f4e5000d560 comp rx=0 tx=0).stop 2024-04-15T15:15:54.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.661+0000 7f4e5bba3640 1 -- 172.21.15.72:0/1401389622 shutdown_connections 2024-04-15T15:15:54.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.661+0000 7f4e5bba3640 1 --2- 172.21.15.72:0/1401389622 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4e3806c920 0x7f4e3806ede0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.661+0000 7f4e5bba3640 1 --2- 172.21.15.72:0/1401389622 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4e5415d960 0x7f4e54142ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.661+0000 7f4e5bba3640 1 --2- 172.21.15.72:0/1401389622 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4e5415a0c0 0x7f4e541f34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.661+0000 7f4e5bba3640 1 --2- 172.21.15.72:0/1401389622 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e540ce570 0x7f4e541ef7e0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.661+0000 7f4e5bba3640 1 -- 172.21.15.72:0/1401389622 >> 172.21.15.72:0/1401389622 conn(0x7f4e54071410 msgr2=0x7f4e54073250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:54.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.661+0000 7f4e5bba3640 1 -- 172.21.15.72:0/1401389622 shutdown_connections 2024-04-15T15:15:54.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.661+0000 7f4e5bba3640 1 -- 172.21.15.72:0/1401389622 wait complete. 2024-04-15T15:15:54.673 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:15:54.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.774+0000 7f893f331640 1 Processor -- start 2024-04-15T15:15:54.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.775+0000 7f893f331640 1 -- start start 2024-04-15T15:15:54.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.775+0000 7f893f331640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8938161e40 0x7f89380b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:54.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.775+0000 7f893f331640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f89380d45b0 0x7f89380d49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:54.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.775+0000 7f893f331640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8938162220 con 0x7f89380d12b0 2024-04-15T15:15:54.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.775+0000 7f893f331640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f89380d21f0 con 0x7f8938161e40 2024-04-15T15:15:54.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.775+0000 7f893f331640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f89380d4f20 con 0x7f89380d45b0 2024-04-15T15:15:54.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.776+0000 7f893d0cd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8938161e40 0x7f89380b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:54.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.776+0000 7f893c8cc640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f89380d12b0 0x7f89380d2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:47802/0 (socket says 172.21.15.72:47802) 2024-04-15T15:15:54.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.776+0000 7f893c8cc640 1 -- 172.21.15.72:0/526127689 learned_addr learned my addr 172.21.15.72:0/526127689 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:54.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.776+0000 7f893d0cd640 1 -- 172.21.15.72:0/526127689 >> v2:172.21.15.175:3300/0 conn(0x7f89380d45b0 msgr2=0x7f89380d49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:54.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.776+0000 7f893d0cd640 1 --2- 172.21.15.72:0/526127689 >> v2:172.21.15.175:3300/0 conn(0x7f89380d45b0 0x7f89380d49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.776+0000 7f893d0cd640 1 -- 172.21.15.72:0/526127689 >> v1:172.21.15.72:6789/0 conn(0x7f89380d12b0 legacy=0x7f89380d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:54.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.776+0000 7f892ffff640 1 -- 172.21.15.72:0/526127689 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1254796768 0 0) 0x7f8938162220 con 0x7f89380d12b0 2024-04-15T15:15:54.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.776+0000 7f893d0cd640 1 -- 172.21.15.72:0/526127689 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89380d3070 con 0x7f8938161e40 2024-04-15T15:15:54.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.777+0000 7f893d0cd640 1 --2- 172.21.15.72:0/526127689 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8938161e40 0x7f89380b37f0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f8934009d90 tx=0x7f893402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28d29daf9396e93 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:54.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.777+0000 7f892ffff640 1 -- 172.21.15.72:0/526127689 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8934039070 con 0x7f8938161e40 2024-04-15T15:15:54.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.777+0000 7f892ffff640 1 -- 172.21.15.72:0/526127689 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f893402cbd0 con 0x7f8938161e40 2024-04-15T15:15:54.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.777+0000 7f892ffff640 1 -- 172.21.15.72:0/526127689 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8934034440 con 0x7f8938161e40 2024-04-15T15:15:54.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.777+0000 7f893f331640 1 -- 172.21.15.72:0/526127689 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8938161e40 msgr2=0x7f89380b37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:54.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.777+0000 7f893f331640 1 --2- 172.21.15.72:0/526127689 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8938161e40 0x7f89380b37f0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f8934009d90 tx=0x7f893402be20 comp rx=0 tx=0).stop 2024-04-15T15:15:54.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.777+0000 7f893f331640 1 -- 172.21.15.72:0/526127689 shutdown_connections 2024-04-15T15:15:54.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.777+0000 7f893f331640 1 --2- 172.21.15.72:0/526127689 >> v2:172.21.15.175:3300/0 conn(0x7f89380d45b0 0x7f89380d49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.777+0000 7f893f331640 1 --2- 172.21.15.72:0/526127689 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8938161e40 0x7f89380b37f0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.777+0000 7f893f331640 1 -- 172.21.15.72:0/526127689 >> 172.21.15.72:0/526127689 conn(0x7f893809d970 msgr2=0x7f89380b95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:54.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.777+0000 7f893f331640 1 -- 172.21.15.72:0/526127689 shutdown_connections 2024-04-15T15:15:54.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.777+0000 7f893f331640 1 -- 172.21.15.72:0/526127689 wait complete. 2024-04-15T15:15:54.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.779+0000 7f893f331640 1 Processor -- start 2024-04-15T15:15:54.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.779+0000 7f893f331640 1 -- start start 2024-04-15T15:15:54.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.779+0000 7f893f331640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f89380d12b0 0x7f893809edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:54.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.779+0000 7f893f331640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89380d45b0 0x7f893815dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:54.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.779+0000 7f893f331640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8938161e40 0x7f89380b1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:54.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.779+0000 7f893f331640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f89380d1de0 con 0x7f89380d45b0 2024-04-15T15:15:54.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.779+0000 7f893f331640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f89380d1b00 con 0x7f8938161e40 2024-04-15T15:15:54.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.779+0000 7f893f331640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f89380d2440 con 0x7f89380d12b0 2024-04-15T15:15:54.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.779+0000 7f893c8cc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89380d45b0 0x7f893815dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:54.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.779+0000 7f893c8cc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89380d45b0 0x7f893815dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48782/0 (socket says 172.21.15.72:48782) 2024-04-15T15:15:54.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.779+0000 7f893c8cc640 1 -- 172.21.15.72:0/2495645165 learned_addr learned my addr 172.21.15.72:0/2495645165 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:54.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.780+0000 7f893c8cc640 1 -- 172.21.15.72:0/2495645165 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f89380d12b0 msgr2=0x7f893809edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:54.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.780+0000 7f893c8cc640 1 --2- 172.21.15.72:0/2495645165 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f89380d12b0 0x7f893809edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.780+0000 7f893c8cc640 1 -- 172.21.15.72:0/2495645165 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8938161e40 msgr2=0x7f89380b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:54.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.780+0000 7f893d8ce640 1 --2- 172.21.15.72:0/2495645165 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8938161e40 0x7f89380b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:54.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.780+0000 7f893c8cc640 1 --2- 172.21.15.72:0/2495645165 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8938161e40 0x7f89380b1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.780+0000 7f893c8cc640 1 -- 172.21.15.72:0/2495645165 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89380b1a40 con 0x7f89380d45b0 2024-04-15T15:15:54.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.780+0000 7f893c8cc640 1 --2- 172.21.15.72:0/2495645165 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89380d45b0 0x7f893815dc70 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f892800c4d0 tx=0x7f892800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:54.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.780+0000 7f892dffb640 1 -- 172.21.15.72:0/2495645165 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8928017070 con 0x7f89380d45b0 2024-04-15T15:15:54.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.780+0000 7f892dffb640 1 -- 172.21.15.72:0/2495645165 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8928005d00 con 0x7f89380d45b0 2024-04-15T15:15:54.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.780+0000 7f892dffb640 1 -- 172.21.15.72:0/2495645165 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89280072b0 con 0x7f89380d45b0 2024-04-15T15:15:54.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.780+0000 7f893f331640 1 -- 172.21.15.72:0/2495645165 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f893809f480 con 0x7f89380d45b0 2024-04-15T15:15:54.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.780+0000 7f893f331640 1 -- 172.21.15.72:0/2495645165 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f893809fa40 con 0x7f89380d45b0 2024-04-15T15:15:54.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.781+0000 7f893f331640 1 -- 172.21.15.72:0/2495645165 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89380689e0 con 0x7f89380d45b0 2024-04-15T15:15:54.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.782+0000 7f892dffb640 1 -- 172.21.15.72:0/2495645165 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f892801a070 con 0x7f89380d45b0 2024-04-15T15:15:54.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.782+0000 7f892dffb640 1 --2- 172.21.15.72:0/2495645165 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f891c06c920 0x7f891c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:54.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.782+0000 7f892dffb640 1 -- 172.21.15.72:0/2495645165 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f892808b220 con 0x7f89380d45b0 2024-04-15T15:15:54.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.782+0000 7f893d0cd640 1 --2- 172.21.15.72:0/2495645165 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f891c06c920 0x7f891c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:54.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.782+0000 7f893d0cd640 1 --2- 172.21.15.72:0/2495645165 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f891c06c920 0x7f891c06ede0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8934036000 tx=0x7f893402e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:54.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.786+0000 7f892dffb640 1 -- 172.21.15.72:0/2495645165 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f89280592e0 con 0x7f89380d45b0 2024-04-15T15:15:54.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.995+0000 7f893f331640 1 -- 172.21.15.72:0/2495645165 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f89380d51b0 con 0x7f89380d45b0 2024-04-15T15:15:54.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.996+0000 7f892dffb640 1 -- 172.21.15.72:0/2495645165 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5183 (secure 0 0 0) 0x7f8928058be0 con 0x7f89380d45b0 2024-04-15T15:15:54.997 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:54.998 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4267,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6833/1659343919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1659343919},{"type":"v1","addr":"172.21.15.175:6833","nonce":1659343919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6835/4113588141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6834","nonce":4113588141},{"type":"v1","addr":"172.21.15.175:6835","nonce":4113588141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4296,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/4151548261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4151548261},{"type":"v1","addr":"172.21.15.72:6836","nonce":4151548261}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:14:51.411091+0000","modified":"2024-04-15T15:15:07.485436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/3458067804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3458067804},{"type":"v1","addr":"172.21.15.72:6837","nonce":3458067804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4299,"qdb_cluster":[4299]},"id":1}]} 2024-04-15T15:15:54.998 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 9 2024-04-15T15:15:54.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.998+0000 7f893f331640 1 -- 172.21.15.72:0/2495645165 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f891c06c920 msgr2=0x7f891c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:54.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.998+0000 7f893f331640 1 --2- 172.21.15.72:0/2495645165 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f891c06c920 0x7f891c06ede0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8934036000 tx=0x7f893402e000 comp rx=0 tx=0).stop 2024-04-15T15:15:54.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.998+0000 7f893f331640 1 -- 172.21.15.72:0/2495645165 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89380d45b0 msgr2=0x7f893815dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:54.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.998+0000 7f893f331640 1 --2- 172.21.15.72:0/2495645165 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89380d45b0 0x7f893815dc70 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f892800c4d0 tx=0x7f892800c990 comp rx=0 tx=0).stop 2024-04-15T15:15:55.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.998+0000 7f893f331640 1 -- 172.21.15.72:0/2495645165 shutdown_connections 2024-04-15T15:15:55.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.998+0000 7f893f331640 1 --2- 172.21.15.72:0/2495645165 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f891c06c920 0x7f891c06ede0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.998+0000 7f893f331640 1 --2- 172.21.15.72:0/2495645165 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8938161e40 0x7f89380b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.998+0000 7f893f331640 1 --2- 172.21.15.72:0/2495645165 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f89380d45b0 0x7f893815dc70 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.998+0000 7f893f331640 1 --2- 172.21.15.72:0/2495645165 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f89380d12b0 0x7f893809edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.998+0000 7f893f331640 1 -- 172.21.15.72:0/2495645165 >> 172.21.15.72:0/2495645165 conn(0x7f893809d970 msgr2=0x7f893806d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:55.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.998+0000 7f893f331640 1 -- 172.21.15.72:0/2495645165 shutdown_connections 2024-04-15T15:15:55.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:54.998+0000 7f893f331640 1 -- 172.21.15.72:0/2495645165 wait complete. 2024-04-15T15:15:55.010 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:55.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.114+0000 7f8bafa74640 1 Processor -- start 2024-04-15T15:15:55.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.114+0000 7f8bafa74640 1 -- start start 2024-04-15T15:15:55.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.115+0000 7f8bafa74640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8ba815fe80 0x7f8ba8070310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.115+0000 7f8bafa74640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ba806f990 con 0x7f8ba81590d0 2024-04-15T15:15:55.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.115+0000 7f8bafa74640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ba815f3a0 con 0x7f8ba8070850 2024-04-15T15:15:55.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.115+0000 7f8bafa74640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f8ba8070cc0 con 0x7f8ba815fe80 2024-04-15T15:15:55.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.115+0000 7f8bad810640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f8ba81590d0 0x7f8ba806eeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:47812/0 (socket says 172.21.15.72:47812) 2024-04-15T15:15:55.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.115+0000 7f8bad810640 1 -- 172.21.15.72:0/184959945 learned_addr learned my addr 172.21.15.72:0/184959945 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:55.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.115+0000 7f8bad00f640 1 --2- 172.21.15.72:0/184959945 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8ba815fe80 0x7f8ba8070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.115+0000 7f8bac80e640 1 -- 172.21.15.72:0/184959945 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2224732613 0 0) 0x7f8ba806f990 con 0x7f8ba81590d0 2024-04-15T15:15:55.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.115+0000 7f8bac80e640 1 -- 172.21.15.72:0/184959945 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b94003520 con 0x7f8ba81590d0 2024-04-15T15:15:55.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.115+0000 7f8bac80e640 1 -- 172.21.15.72:0/184959945 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 886506643 0 0) 0x7f8ba815f3a0 con 0x7f8ba8070850 2024-04-15T15:15:55.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.115+0000 7f8bac80e640 1 -- 172.21.15.72:0/184959945 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ba806f990 con 0x7f8ba8070850 2024-04-15T15:15:55.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.115+0000 7f8bac80e640 1 -- 172.21.15.72:0/184959945 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2357542622 0 0) 0x7f8b94003520 con 0x7f8ba81590d0 2024-04-15T15:15:55.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.115+0000 7f8bac80e640 1 -- 172.21.15.72:0/184959945 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8ba815fe80 msgr2=0x7f8ba8070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:55.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.115+0000 7f8bac80e640 1 --2- 172.21.15.72:0/184959945 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8ba815fe80 0x7f8ba8070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.115+0000 7f8bac80e640 1 -- 172.21.15.72:0/184959945 >> v1:172.21.15.175:6789/0 conn(0x7f8ba8070850 legacy=0x7f8ba806d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.115+0000 7f8bac80e640 1 -- 172.21.15.72:0/184959945 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba815f680 con 0x7f8ba81590d0 2024-04-15T15:15:55.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.115+0000 7f8bac80e640 1 -- 172.21.15.72:0/184959945 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f8ba4001670 con 0x7f8ba81590d0 2024-04-15T15:15:55.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.115+0000 7f8bac80e640 1 -- 172.21.15.72:0/184959945 >> v1:172.21.15.72:6789/0 conn(0x7f8ba81590d0 legacy=0x7f8ba806eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.115+0000 7f8bac80e640 1 --2- 172.21.15.72:0/184959945 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8ba815fe80 0x7f8b94003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.115+0000 7f8bac80e640 1 --2- 172.21.15.72:0/184959945 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b940040d0 0x7f8b94004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.115+0000 7f8bac80e640 1 --2- 172.21.15.72:0/184959945 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b94004d40 0x7f8b940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.115+0000 7f8bac80e640 1 -- 172.21.15.72:0/184959945 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba8070cc0 con 0x7f8b94004d40 2024-04-15T15:15:55.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.115+0000 7f8bac80e640 1 -- 172.21.15.72:0/184959945 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9400baa0 con 0x7f8b940040d0 2024-04-15T15:15:55.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.115+0000 7f8bac80e640 1 -- 172.21.15.72:0/184959945 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f8b9400bca0 con 0x7f8ba815fe80 2024-04-15T15:15:55.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.115+0000 7f8bae011640 1 --2- 172.21.15.72:0/184959945 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b94004d40 0x7f8b940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.116+0000 7f8bad810640 1 --2- 172.21.15.72:0/184959945 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8ba815fe80 0x7f8b94003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.116+0000 7f8bad00f640 1 --2- 172.21.15.72:0/184959945 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b940040d0 0x7f8b94004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.116+0000 7f8bae011640 1 -- 172.21.15.72:0/184959945 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8ba815fe80 msgr2=0x7f8b94003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:55.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.116+0000 7f8bae011640 1 --2- 172.21.15.72:0/184959945 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8ba815fe80 0x7f8b94003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.116+0000 7f8bae011640 1 -- 172.21.15.72:0/184959945 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b940040d0 msgr2=0x7f8b94004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:55.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.116+0000 7f8bae011640 1 --2- 172.21.15.72:0/184959945 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b940040d0 0x7f8b94004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.116+0000 7f8bae011640 1 -- 172.21.15.72:0/184959945 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b94003520 con 0x7f8b94004d40 2024-04-15T15:15:55.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.116+0000 7f8bae011640 1 --2- 172.21.15.72:0/184959945 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b94004d40 0x7f8b940051c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f8ba0001fd0 tx=0x7f8b94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea51046630af39f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:55.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.117+0000 7f8bac80e640 1 -- 172.21.15.72:0/184959945 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba0001bf0 con 0x7f8b94004d40 2024-04-15T15:15:55.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.117+0000 7f8bac80e640 1 -- 172.21.15.72:0/184959945 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ba0003b70 con 0x7f8b94004d40 2024-04-15T15:15:55.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.117+0000 7f8bac80e640 1 -- 172.21.15.72:0/184959945 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba00050d0 con 0x7f8b94004d40 2024-04-15T15:15:55.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.117+0000 7f8bafa74640 1 -- 172.21.15.72:0/184959945 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b94004d40 msgr2=0x7f8b940051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.117+0000 7f8bafa74640 1 --2- 172.21.15.72:0/184959945 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b94004d40 0x7f8b940051c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f8ba0001fd0 tx=0x7f8b94005830 comp rx=0 tx=0).stop 2024-04-15T15:15:55.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.117+0000 7f8bae011640 1 -- 172.21.15.72:0/184959945 reap_dead start 2024-04-15T15:15:55.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.117+0000 7f8bafa74640 1 -- 172.21.15.72:0/184959945 shutdown_connections 2024-04-15T15:15:55.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.117+0000 7f8bafa74640 1 -- 172.21.15.72:0/184959945 >> 172.21.15.72:0/184959945 conn(0x7f8ba80cdd20 msgr2=0x7f8ba80ce100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:55.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.117+0000 7f8bafa74640 1 -- 172.21.15.72:0/184959945 shutdown_connections 2024-04-15T15:15:55.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.117+0000 7f8bafa74640 1 -- 172.21.15.72:0/184959945 wait complete. 2024-04-15T15:15:55.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.118+0000 7f8bafa74640 1 Processor -- start 2024-04-15T15:15:55.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.118+0000 7f8bafa74640 1 -- start start 2024-04-15T15:15:55.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.119+0000 7f8bafa74640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8b94004d40 0x7f8ba814cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.119+0000 7f8bafa74640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8ba8154240 0x7f8ba80cf740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.119+0000 7f8bafa74640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8ba80ca650 0x7f8ba80cfc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.119+0000 7f8bafa74640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba815c9c0 con 0x7f8ba80ca650 2024-04-15T15:15:55.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.119+0000 7f8bafa74640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba815f150 con 0x7f8ba8154240 2024-04-15T15:15:55.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.119+0000 7f8bafa74640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f8ba806f5c0 con 0x7f8b94004d40 2024-04-15T15:15:55.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.119+0000 7f8bae011640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8ba80ca650 0x7f8ba80cfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.119+0000 7f8bae011640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8ba80ca650 0x7f8ba80cfc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48804/0 (socket says 172.21.15.72:48804) 2024-04-15T15:15:55.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.119+0000 7f8bae011640 1 -- 172.21.15.72:0/598820251 learned_addr learned my addr 172.21.15.72:0/598820251 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:55.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.119+0000 7f8bad810640 1 --2- 172.21.15.72:0/598820251 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8b94004d40 0x7f8ba814cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.119+0000 7f8bad00f640 1 --2- 172.21.15.72:0/598820251 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8ba8154240 0x7f8ba80cf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.119+0000 7f8bae011640 1 -- 172.21.15.72:0/598820251 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8b94004d40 msgr2=0x7f8ba814cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.119+0000 7f8bae011640 1 --2- 172.21.15.72:0/598820251 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8b94004d40 0x7f8ba814cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.119+0000 7f8bae011640 1 -- 172.21.15.72:0/598820251 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8ba8154240 msgr2=0x7f8ba80cf740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.120+0000 7f8bae011640 1 --2- 172.21.15.72:0/598820251 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8ba8154240 0x7f8ba80cf740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.120+0000 7f8bae011640 1 -- 172.21.15.72:0/598820251 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba815dec0 con 0x7f8ba80ca650 2024-04-15T15:15:55.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.120+0000 7f8bad00f640 1 --2- 172.21.15.72:0/598820251 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8ba8154240 0x7f8ba80cf740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:55.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.120+0000 7f8bae011640 1 --2- 172.21.15.72:0/598820251 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8ba80ca650 0x7f8ba80cfc80 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f8ba0005040 tx=0x7f8ba0001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:55.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.120+0000 7f8b9e7fc640 1 -- 172.21.15.72:0/598820251 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba0042070 con 0x7f8ba80ca650 2024-04-15T15:15:55.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.120+0000 7f8b9e7fc640 1 -- 172.21.15.72:0/598820251 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ba0005400 con 0x7f8ba80ca650 2024-04-15T15:15:55.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.120+0000 7f8bafa74640 1 -- 172.21.15.72:0/598820251 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ba815cb40 con 0x7f8ba80ca650 2024-04-15T15:15:55.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.120+0000 7f8b9e7fc640 1 -- 172.21.15.72:0/598820251 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba0039040 con 0x7f8ba80ca650 2024-04-15T15:15:55.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.120+0000 7f8bafa74640 1 -- 172.21.15.72:0/598820251 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ba815d100 con 0x7f8ba80ca650 2024-04-15T15:15:55.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.121+0000 7f8bafa74640 1 -- 172.21.15.72:0/598820251 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ba80690f0 con 0x7f8ba80ca650 2024-04-15T15:15:55.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.121+0000 7f8b9e7fc640 1 -- 172.21.15.72:0/598820251 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8ba0045070 con 0x7f8ba80ca650 2024-04-15T15:15:55.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.122+0000 7f8b9e7fc640 1 --2- 172.21.15.72:0/598820251 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8b8c06c8f0 0x7f8b8c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.122+0000 7f8b9e7fc640 1 -- 172.21.15.72:0/598820251 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f8ba00ac970 con 0x7f8ba80ca650 2024-04-15T15:15:55.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.122+0000 7f8bad810640 1 --2- 172.21.15.72:0/598820251 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8b8c06c8f0 0x7f8b8c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.122+0000 7f8bad810640 1 --2- 172.21.15.72:0/598820251 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8b8c06c8f0 0x7f8b8c06edb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4001eb0 tx=0x7f8ba4001e20 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:55.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.126+0000 7f8b9e7fc640 1 -- 172.21.15.72:0/598820251 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8ba007a350 con 0x7f8ba80ca650 2024-04-15T15:15:55.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.301+0000 7f8bafa74640 1 -- 172.21.15.72:0/598820251 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8ba80d06b0 con 0x7f8ba80ca650 2024-04-15T15:15:55.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.302+0000 7f8b9e7fc640 1 -- 172.21.15.72:0/598820251 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18903 (secure 0 0 0) 0x7f8ba007e650 con 0x7f8ba80ca650 2024-04-15T15:15:55.304 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:55.304 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":26,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:15:32.421728+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:14:49.438366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:14:50.368379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:14:54.081157+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":1}]},{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:55.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.304+0000 7f8bafa74640 1 -- 172.21.15.72:0/598820251 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8b8c06c8f0 msgr2=0x7f8b8c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.304+0000 7f8bafa74640 1 --2- 172.21.15.72:0/598820251 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8b8c06c8f0 0x7f8b8c06edb0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4001eb0 tx=0x7f8ba4001e20 comp rx=0 tx=0).stop 2024-04-15T15:15:55.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.304+0000 7f8bafa74640 1 -- 172.21.15.72:0/598820251 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8ba80ca650 msgr2=0x7f8ba80cfc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.304+0000 7f8bafa74640 1 --2- 172.21.15.72:0/598820251 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8ba80ca650 0x7f8ba80cfc80 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f8ba0005040 tx=0x7f8ba0001650 comp rx=0 tx=0).stop 2024-04-15T15:15:55.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.304+0000 7f8bafa74640 1 -- 172.21.15.72:0/598820251 shutdown_connections 2024-04-15T15:15:55.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.304+0000 7f8bafa74640 1 --2- 172.21.15.72:0/598820251 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8b8c06c8f0 0x7f8b8c06edb0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.304+0000 7f8bafa74640 1 --2- 172.21.15.72:0/598820251 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8ba80ca650 0x7f8ba80cfc80 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.304+0000 7f8bafa74640 1 --2- 172.21.15.72:0/598820251 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8ba8154240 0x7f8ba80cf740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.304+0000 7f8bafa74640 1 --2- 172.21.15.72:0/598820251 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8b94004d40 0x7f8ba814cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.304+0000 7f8bafa74640 1 -- 172.21.15.72:0/598820251 >> 172.21.15.72:0/598820251 conn(0x7f8ba80cdd20 msgr2=0x7f8ba81596c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:55.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.305+0000 7f8bafa74640 1 -- 172.21.15.72:0/598820251 shutdown_connections 2024-04-15T15:15:55.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.305+0000 7f8bafa74640 1 -- 172.21.15.72:0/598820251 wait complete. 2024-04-15T15:15:55.317 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-15T15:15:55.318 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:15:55.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.419+0000 7fd34ce63640 1 Processor -- start 2024-04-15T15:15:55.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.419+0000 7fd34ce63640 1 -- start start 2024-04-15T15:15:55.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.420+0000 7fd34ce63640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd3480d12b0 0x7fd3480d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.420+0000 7fd34ce63640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd3480d2ea0 0x7fd3480d45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.420+0000 7fd34ce63640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd348162220 con 0x7fd348161e40 2024-04-15T15:15:55.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.420+0000 7fd34ce63640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fd3480b3f00 con 0x7fd3480d2ea0 2024-04-15T15:15:55.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.420+0000 7fd34ce63640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fd3480d21f0 con 0x7fd3480d12b0 2024-04-15T15:15:55.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.420+0000 7fd34659c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fd348161e40 0x7fd3480b37f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:47828/0 (socket says 172.21.15.72:47828) 2024-04-15T15:15:55.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.420+0000 7fd34659c640 1 -- 172.21.15.72:0/3815867803 learned_addr learned my addr 172.21.15.72:0/3815867803 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:55.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.421+0000 7fd345d9b640 1 --2- 172.21.15.72:0/3815867803 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd3480d12b0 0x7fd3480d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.421+0000 7fd346d9d640 1 --2- 172.21.15.72:0/3815867803 >> v2:172.21.15.175:3300/0 conn(0x7fd3480d2ea0 0x7fd3480d45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.421+0000 7fd34559a640 1 -- 172.21.15.72:0/3815867803 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1342380931 0 0) 0x7fd348162220 con 0x7fd348161e40 2024-04-15T15:15:55.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.421+0000 7fd34559a640 1 -- 172.21.15.72:0/3815867803 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd330003520 con 0x7fd348161e40 2024-04-15T15:15:55.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.421+0000 7fd34559a640 1 -- 172.21.15.72:0/3815867803 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1675751946 0 0) 0x7fd330003520 con 0x7fd348161e40 2024-04-15T15:15:55.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.421+0000 7fd34559a640 1 -- 172.21.15.72:0/3815867803 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd3480d12b0 msgr2=0x7fd3480d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:55.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.421+0000 7fd34559a640 1 --2- 172.21.15.72:0/3815867803 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd3480d12b0 0x7fd3480d2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.421+0000 7fd34559a640 1 -- 172.21.15.72:0/3815867803 >> v2:172.21.15.175:3300/0 conn(0x7fd3480d2ea0 msgr2=0x7fd3480d45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:55.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.421+0000 7fd34559a640 1 --2- 172.21.15.72:0/3815867803 >> v2:172.21.15.175:3300/0 conn(0x7fd3480d2ea0 0x7fd3480d45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.421+0000 7fd34559a640 1 -- 172.21.15.72:0/3815867803 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3480d1820 con 0x7fd348161e40 2024-04-15T15:15:55.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.421+0000 7fd34559a640 1 -- 172.21.15.72:0/3815867803 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7fd3380019f0 con 0x7fd348161e40 2024-04-15T15:15:55.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.421+0000 7fd34559a640 1 -- 172.21.15.72:0/3815867803 >> v1:172.21.15.72:6789/0 conn(0x7fd348161e40 legacy=0x7fd3480b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.421+0000 7fd34559a640 1 --2- 172.21.15.72:0/3815867803 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd3300039b0 0x7fd330003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.421+0000 7fd34559a640 1 --2- 172.21.15.72:0/3815867803 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd330004800 0x7fd330004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.421+0000 7fd34559a640 1 --2- 172.21.15.72:0/3815867803 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd3480d12b0 0x7fd3300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.421+0000 7fd34559a640 1 -- 172.21.15.72:0/3815867803 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd3480b3f00 con 0x7fd3300039b0 2024-04-15T15:15:55.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.421+0000 7fd34559a640 1 -- 172.21.15.72:0/3815867803 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fd3480d21f0 con 0x7fd330004800 2024-04-15T15:15:55.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.421+0000 7fd34559a640 1 -- 172.21.15.72:0/3815867803 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fd33000b9f0 con 0x7fd3480d12b0 2024-04-15T15:15:55.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.421+0000 7fd34659c640 1 --2- 172.21.15.72:0/3815867803 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd3300039b0 0x7fd330003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.421+0000 7fd345d9b640 1 --2- 172.21.15.72:0/3815867803 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd330004800 0x7fd330004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.421+0000 7fd346d9d640 1 --2- 172.21.15.72:0/3815867803 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd3480d12b0 0x7fd3300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.421+0000 7fd34659c640 1 -- 172.21.15.72:0/3815867803 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd3480d12b0 msgr2=0x7fd3300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:55.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.421+0000 7fd34659c640 1 --2- 172.21.15.72:0/3815867803 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd3480d12b0 0x7fd3300051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.421+0000 7fd34659c640 1 -- 172.21.15.72:0/3815867803 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd330004800 msgr2=0x7fd330004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:55.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.421+0000 7fd34659c640 1 --2- 172.21.15.72:0/3815867803 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd330004800 0x7fd330004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.421+0000 7fd34659c640 1 -- 172.21.15.72:0/3815867803 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd330003520 con 0x7fd3300039b0 2024-04-15T15:15:55.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.422+0000 7fd34659c640 1 --2- 172.21.15.72:0/3815867803 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd3300039b0 0x7fd330003dd0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fd3380013d0 tx=0x7fd330005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=102c2917c5cb7b2e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:55.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.422+0000 7fd34559a640 1 -- 172.21.15.72:0/3815867803 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd33800a4e0 con 0x7fd3300039b0 2024-04-15T15:15:55.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.422+0000 7fd34559a640 1 -- 172.21.15.72:0/3815867803 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd338034650 con 0x7fd3300039b0 2024-04-15T15:15:55.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.422+0000 7fd34559a640 1 -- 172.21.15.72:0/3815867803 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd338034a40 con 0x7fd3300039b0 2024-04-15T15:15:55.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.423+0000 7fd34ce63640 1 -- 172.21.15.72:0/3815867803 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd3300039b0 msgr2=0x7fd330003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.423+0000 7fd34ce63640 1 --2- 172.21.15.72:0/3815867803 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd3300039b0 0x7fd330003dd0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fd3380013d0 tx=0x7fd330005830 comp rx=0 tx=0).stop 2024-04-15T15:15:55.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.423+0000 7fd346d9d640 1 -- 172.21.15.72:0/3815867803 reap_dead start 2024-04-15T15:15:55.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.423+0000 7fd34ce63640 1 -- 172.21.15.72:0/3815867803 shutdown_connections 2024-04-15T15:15:55.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.423+0000 7fd34ce63640 1 -- 172.21.15.72:0/3815867803 >> 172.21.15.72:0/3815867803 conn(0x7fd34809d970 msgr2=0x7fd3480b95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:55.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.423+0000 7fd34ce63640 1 -- 172.21.15.72:0/3815867803 shutdown_connections 2024-04-15T15:15:55.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.423+0000 7fd34ce63640 1 -- 172.21.15.72:0/3815867803 wait complete. 2024-04-15T15:15:55.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.424+0000 7fd34ce63640 1 Processor -- start 2024-04-15T15:15:55.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.424+0000 7fd34ce63640 1 -- start start 2024-04-15T15:15:55.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.425+0000 7fd34ce63640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd3300039b0 0x7fd3480cd970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.425+0000 7fd34ce63640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd34815d960 0x7fd3480b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.425+0000 7fd34ce63640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd34814e2e0 0x7fd34814e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.425+0000 7fd34ce63640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd3480d2440 con 0x7fd3300039b0 2024-04-15T15:15:55.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.425+0000 7fd34ce63640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fd3480d1de0 con 0x7fd34814e2e0 2024-04-15T15:15:55.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.425+0000 7fd34ce63640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fd3480d2960 con 0x7fd34815d960 2024-04-15T15:15:55.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.425+0000 7fd34659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd3300039b0 0x7fd3480cd970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.425+0000 7fd34659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd3300039b0 0x7fd3480cd970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48812/0 (socket says 172.21.15.72:48812) 2024-04-15T15:15:55.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.425+0000 7fd34659c640 1 -- 172.21.15.72:0/732607006 learned_addr learned my addr 172.21.15.72:0/732607006 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:55.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.425+0000 7fd345d9b640 1 --2- 172.21.15.72:0/732607006 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd34815d960 0x7fd3480b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.425+0000 7fd346d9d640 1 --2- 172.21.15.72:0/732607006 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd34814e2e0 0x7fd34814e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.425+0000 7fd34659c640 1 -- 172.21.15.72:0/732607006 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd34815d960 msgr2=0x7fd3480b50d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.425+0000 7fd34659c640 1 --2- 172.21.15.72:0/732607006 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd34815d960 0x7fd3480b50d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.425+0000 7fd34659c640 1 -- 172.21.15.72:0/732607006 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd34814e2e0 msgr2=0x7fd34814e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.425+0000 7fd34659c640 1 --2- 172.21.15.72:0/732607006 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd34814e2e0 0x7fd34814e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.425+0000 7fd34659c640 1 -- 172.21.15.72:0/732607006 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3480b6000 con 0x7fd3300039b0 2024-04-15T15:15:55.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.426+0000 7fd34659c640 1 --2- 172.21.15.72:0/732607006 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd3300039b0 0x7fd3480cd970 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fd33800a0b0 tx=0x7fd338001360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:55.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.426+0000 7fd336ffd640 1 -- 172.21.15.72:0/732607006 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd338039040 con 0x7fd3300039b0 2024-04-15T15:15:55.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.426+0000 7fd336ffd640 1 -- 172.21.15.72:0/732607006 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd338043070 con 0x7fd3300039b0 2024-04-15T15:15:55.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.426+0000 7fd336ffd640 1 -- 172.21.15.72:0/732607006 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd338004d80 con 0x7fd3300039b0 2024-04-15T15:15:55.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.426+0000 7fd34ce63640 1 -- 172.21.15.72:0/732607006 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3480b5a40 con 0x7fd3300039b0 2024-04-15T15:15:55.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.426+0000 7fd34ce63640 1 -- 172.21.15.72:0/732607006 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd34814c5d0 con 0x7fd3300039b0 2024-04-15T15:15:55.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.427+0000 7fd34ce63640 1 -- 172.21.15.72:0/732607006 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3480689e0 con 0x7fd3300039b0 2024-04-15T15:15:55.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.427+0000 7fd336ffd640 1 -- 172.21.15.72:0/732607006 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd338034650 con 0x7fd3300039b0 2024-04-15T15:15:55.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.428+0000 7fd336ffd640 1 --2- 172.21.15.72:0/732607006 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd32806c920 0x7fd32806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.428+0000 7fd336ffd640 1 -- 172.21.15.72:0/732607006 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fd3380addb0 con 0x7fd3300039b0 2024-04-15T15:15:55.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.428+0000 7fd345d9b640 1 --2- 172.21.15.72:0/732607006 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd32806c920 0x7fd32806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.428+0000 7fd345d9b640 1 --2- 172.21.15.72:0/732607006 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd32806c920 0x7fd32806ede0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd34815dd80 tx=0x7fd33c009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:55.433 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.432+0000 7fd336ffd640 1 -- 172.21.15.72:0/732607006 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd338001a30 con 0x7fd3300039b0 2024-04-15T15:15:55.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.644+0000 7fd34ce63640 1 -- 172.21.15.72:0/732607006 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd3480d51b0 con 0x7fd3300039b0 2024-04-15T15:15:55.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.644+0000 7fd336ffd640 1 -- 172.21.15.72:0/732607006 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5183 (secure 0 0 0) 0x7fd33807bea0 con 0x7fd3300039b0 2024-04-15T15:15:55.646 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:55.646 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4267,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6833/1659343919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1659343919},{"type":"v1","addr":"172.21.15.175:6833","nonce":1659343919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6835/4113588141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6834","nonce":4113588141},{"type":"v1","addr":"172.21.15.175:6835","nonce":4113588141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4296,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/4151548261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4151548261},{"type":"v1","addr":"172.21.15.72:6836","nonce":4151548261}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:14:51.411091+0000","modified":"2024-04-15T15:15:07.485436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/3458067804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3458067804},{"type":"v1","addr":"172.21.15.72:6837","nonce":3458067804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4299,"qdb_cluster":[4299]},"id":1}]} 2024-04-15T15:15:55.646 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 9 2024-04-15T15:15:55.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.646+0000 7fd34ce63640 1 -- 172.21.15.72:0/732607006 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd32806c920 msgr2=0x7fd32806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.646+0000 7fd34ce63640 1 --2- 172.21.15.72:0/732607006 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd32806c920 0x7fd32806ede0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd34815dd80 tx=0x7fd33c009190 comp rx=0 tx=0).stop 2024-04-15T15:15:55.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.646+0000 7fd34ce63640 1 -- 172.21.15.72:0/732607006 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd3300039b0 msgr2=0x7fd3480cd970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.646+0000 7fd34ce63640 1 --2- 172.21.15.72:0/732607006 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd3300039b0 0x7fd3480cd970 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fd33800a0b0 tx=0x7fd338001360 comp rx=0 tx=0).stop 2024-04-15T15:15:55.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.646+0000 7fd34ce63640 1 -- 172.21.15.72:0/732607006 shutdown_connections 2024-04-15T15:15:55.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.646+0000 7fd34ce63640 1 --2- 172.21.15.72:0/732607006 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd32806c920 0x7fd32806ede0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.646+0000 7fd34ce63640 1 --2- 172.21.15.72:0/732607006 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd34814e2e0 0x7fd34814e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.646+0000 7fd34ce63640 1 --2- 172.21.15.72:0/732607006 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd34815d960 0x7fd3480b50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.646+0000 7fd34ce63640 1 --2- 172.21.15.72:0/732607006 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd3300039b0 0x7fd3480cd970 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.646+0000 7fd34ce63640 1 -- 172.21.15.72:0/732607006 >> 172.21.15.72:0/732607006 conn(0x7fd34809d970 msgr2=0x7fd3480b7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:55.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.646+0000 7fd34ce63640 1 -- 172.21.15.72:0/732607006 shutdown_connections 2024-04-15T15:15:55.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.646+0000 7fd34ce63640 1 -- 172.21.15.72:0/732607006 wait complete. 2024-04-15T15:15:55.659 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:15:55.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.760+0000 7f403f488640 1 Processor -- start 2024-04-15T15:15:55.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.760+0000 7f403f488640 1 -- start start 2024-04-15T15:15:55.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.761+0000 7f403f488640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f40380af400 0x7f40380ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.761+0000 7f403f488640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f403806f8f0 con 0x7f40380af860 2024-04-15T15:15:55.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.761+0000 7f403f488640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f403815ef80 con 0x7f403815eba0 2024-04-15T15:15:55.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.761+0000 7f403f488640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f40380ce2b0 con 0x7f40380af400 2024-04-15T15:15:55.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.761+0000 7f403da25640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f40380af860 0x7f40380ad0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:47840/0 (socket says 172.21.15.72:47840) 2024-04-15T15:15:55.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.761+0000 7f403da25640 1 -- 172.21.15.72:0/716293421 learned_addr learned my addr 172.21.15.72:0/716293421 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:55.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.761+0000 7f402ffff640 1 -- 172.21.15.72:0/716293421 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1634385692 0 0) 0x7f403806f8f0 con 0x7f40380af860 2024-04-15T15:15:55.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.761+0000 7f403ca23640 1 --2- 172.21.15.72:0/716293421 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f40380af400 0x7f40380ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.761+0000 7f402ffff640 1 -- 172.21.15.72:0/716293421 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4024003520 con 0x7f40380af860 2024-04-15T15:15:55.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.762+0000 7f402ffff640 1 -- 172.21.15.72:0/716293421 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3129679444 0 0) 0x7f4024003520 con 0x7f40380af860 2024-04-15T15:15:55.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.762+0000 7f402ffff640 1 -- 172.21.15.72:0/716293421 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f40380af400 msgr2=0x7f40380ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:55.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.762+0000 7f402ffff640 1 --2- 172.21.15.72:0/716293421 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f40380af400 0x7f40380ccb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.762+0000 7f402ffff640 1 -- 172.21.15.72:0/716293421 >> v1:172.21.15.175:6789/0 conn(0x7f403815eba0 legacy=0x7f403806f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.762+0000 7f402ffff640 1 -- 172.21.15.72:0/716293421 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40380ce590 con 0x7f40380af860 2024-04-15T15:15:55.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.762+0000 7f402ffff640 1 -- 172.21.15.72:0/716293421 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f40300019f0 con 0x7f40380af860 2024-04-15T15:15:55.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.762+0000 7f402ffff640 1 -- 172.21.15.72:0/716293421 >> v1:172.21.15.72:6789/0 conn(0x7f40380af860 legacy=0x7f40380ad0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.762+0000 7f402ffff640 1 --2- 172.21.15.72:0/716293421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f40240039e0 0x7f4024003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.762+0000 7f402ffff640 1 --2- 172.21.15.72:0/716293421 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f40380af400 0x7f4024004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.762+0000 7f402ffff640 1 --2- 172.21.15.72:0/716293421 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4024004d40 0x7f40240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.762+0000 7f402ffff640 1 -- 172.21.15.72:0/716293421 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f40380ce2b0 con 0x7f40240039e0 2024-04-15T15:15:55.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.762+0000 7f402ffff640 1 -- 172.21.15.72:0/716293421 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f402400baf0 con 0x7f4024004d40 2024-04-15T15:15:55.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.762+0000 7f402ffff640 1 -- 172.21.15.72:0/716293421 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f402400bcf0 con 0x7f40380af400 2024-04-15T15:15:55.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.762+0000 7f403d224640 1 --2- 172.21.15.72:0/716293421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f40240039e0 0x7f4024003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.762+0000 7f403ca23640 1 --2- 172.21.15.72:0/716293421 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f40380af400 0x7f4024004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.762+0000 7f403da25640 1 --2- 172.21.15.72:0/716293421 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4024004d40 0x7f40240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.762+0000 7f403d224640 1 -- 172.21.15.72:0/716293421 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f40380af400 msgr2=0x7f4024004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:55.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.762+0000 7f403d224640 1 --2- 172.21.15.72:0/716293421 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f40380af400 0x7f4024004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.762+0000 7f403d224640 1 -- 172.21.15.72:0/716293421 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4024004d40 msgr2=0x7f40240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:55.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.762+0000 7f403d224640 1 --2- 172.21.15.72:0/716293421 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4024004d40 0x7f40240051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.762+0000 7f403d224640 1 -- 172.21.15.72:0/716293421 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4024003520 con 0x7f40240039e0 2024-04-15T15:15:55.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.762+0000 7f403ca23640 1 --2- 172.21.15.72:0/716293421 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f40380af400 0x7f4024004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:55.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.763+0000 7f403d224640 1 --2- 172.21.15.72:0/716293421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f40240039e0 0x7f4024003e00 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f4034001750 tx=0x7f4024005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=632b0033ab8b1e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:55.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.763+0000 7f402ffff640 1 -- 172.21.15.72:0/716293421 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4034001370 con 0x7f40240039e0 2024-04-15T15:15:55.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.763+0000 7f402ffff640 1 -- 172.21.15.72:0/716293421 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40340052d0 con 0x7f40240039e0 2024-04-15T15:15:55.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.763+0000 7f402ffff640 1 -- 172.21.15.72:0/716293421 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4034003830 con 0x7f40240039e0 2024-04-15T15:15:55.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.763+0000 7f403f488640 1 -- 172.21.15.72:0/716293421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f40240039e0 msgr2=0x7f4024003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.763+0000 7f403f488640 1 --2- 172.21.15.72:0/716293421 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f40240039e0 0x7f4024003e00 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f4034001750 tx=0x7f4024005830 comp rx=0 tx=0).stop 2024-04-15T15:15:55.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.763+0000 7f403da25640 1 -- 172.21.15.72:0/716293421 reap_dead start 2024-04-15T15:15:55.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.764+0000 7f403f488640 1 -- 172.21.15.72:0/716293421 shutdown_connections 2024-04-15T15:15:55.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.764+0000 7f403f488640 1 -- 172.21.15.72:0/716293421 >> 172.21.15.72:0/716293421 conn(0x7f40380b1dd0 msgr2=0x7f40380b21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:55.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.764+0000 7f403f488640 1 -- 172.21.15.72:0/716293421 shutdown_connections 2024-04-15T15:15:55.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.764+0000 7f403f488640 1 -- 172.21.15.72:0/716293421 wait complete. 2024-04-15T15:15:55.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.765+0000 7f403f488640 1 Processor -- start 2024-04-15T15:15:55.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.765+0000 7f403f488640 1 -- start start 2024-04-15T15:15:55.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.766+0000 7f403f488640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f40240039e0 0x7f403814b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.766+0000 7f403f488640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4038152d90 0x7f403806b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.766+0000 7f403f488640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f403806bea0 0x7f403815ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.766+0000 7f403f488640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f40380b2b80 con 0x7f403806bea0 2024-04-15T15:15:55.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.766+0000 7f403f488640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f403806ecc0 con 0x7f4038152d90 2024-04-15T15:15:55.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.766+0000 7f403f488640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f40380ceb50 con 0x7f40240039e0 2024-04-15T15:15:55.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.766+0000 7f403da25640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f403806bea0 0x7f403815ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.766+0000 7f403da25640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f403806bea0 0x7f403815ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48832/0 (socket says 172.21.15.72:48832) 2024-04-15T15:15:55.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.766+0000 7f403da25640 1 -- 172.21.15.72:0/3494504898 learned_addr learned my addr 172.21.15.72:0/3494504898 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:55.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.766+0000 7f403ca23640 1 --2- 172.21.15.72:0/3494504898 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4038152d90 0x7f403806b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.766+0000 7f403da25640 1 -- 172.21.15.72:0/3494504898 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f40240039e0 msgr2=0x7f403814b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.766+0000 7f403d224640 1 --2- 172.21.15.72:0/3494504898 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f40240039e0 0x7f403814b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.766+0000 7f403da25640 1 --2- 172.21.15.72:0/3494504898 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f40240039e0 0x7f403814b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.766+0000 7f403da25640 1 -- 172.21.15.72:0/3494504898 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4038152d90 msgr2=0x7f403806b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.766+0000 7f403da25640 1 --2- 172.21.15.72:0/3494504898 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4038152d90 0x7f403806b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.766+0000 7f403da25640 1 -- 172.21.15.72:0/3494504898 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40380c94f0 con 0x7f403806bea0 2024-04-15T15:15:55.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.766+0000 7f403ca23640 1 --2- 172.21.15.72:0/3494504898 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4038152d90 0x7f403806b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:55.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.766+0000 7f403d224640 1 --2- 172.21.15.72:0/3494504898 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f40240039e0 0x7f403814b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:55.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.766+0000 7f403da25640 1 --2- 172.21.15.72:0/3494504898 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f403806bea0 0x7f403815ca40 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f40380ccb00 tx=0x7f403000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:55.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.766+0000 7f402dffb640 1 -- 172.21.15.72:0/3494504898 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f403000ebc0 con 0x7f403806bea0 2024-04-15T15:15:55.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.766+0000 7f402dffb640 1 -- 172.21.15.72:0/3494504898 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f403000ed60 con 0x7f403806bea0 2024-04-15T15:15:55.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.766+0000 7f403f488640 1 -- 172.21.15.72:0/3494504898 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40380c97d0 con 0x7f403806bea0 2024-04-15T15:15:55.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.766+0000 7f402dffb640 1 -- 172.21.15.72:0/3494504898 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4030014670 con 0x7f403806bea0 2024-04-15T15:15:55.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.767+0000 7f403f488640 1 -- 172.21.15.72:0/3494504898 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40380c9d90 con 0x7f403806bea0 2024-04-15T15:15:55.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.768+0000 7f403f488640 1 -- 172.21.15.72:0/3494504898 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40380689e0 con 0x7f403806bea0 2024-04-15T15:15:55.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.768+0000 7f402dffb640 1 -- 172.21.15.72:0/3494504898 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4030014810 con 0x7f403806bea0 2024-04-15T15:15:55.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.768+0000 7f402dffb640 1 --2- 172.21.15.72:0/3494504898 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f401c06c8f0 0x7f401c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.768+0000 7f402dffb640 1 -- 172.21.15.72:0/3494504898 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f403008bd40 con 0x7f403806bea0 2024-04-15T15:15:55.773 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.768+0000 7f403d224640 1 --2- 172.21.15.72:0/3494504898 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f401c06c8f0 0x7f401c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.769+0000 7f403d224640 1 --2- 172.21.15.72:0/3494504898 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f401c06c8f0 0x7f401c06edb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4034033000 tx=0x7f4034009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:55.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.773+0000 7f402dffb640 1 -- 172.21.15.72:0/3494504898 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4030059d70 con 0x7f403806bea0 2024-04-15T15:15:55.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.985+0000 7f403f488640 1 -- 172.21.15.72:0/3494504898 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f40380b04a0 con 0x7f403806bea0 2024-04-15T15:15:55.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.985+0000 7f402dffb640 1 -- 172.21.15.72:0/3494504898 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+5183 (secure 0 0 0) 0x7f4030059670 con 0x7f403806bea0 2024-04-15T15:15:55.987 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:55.987 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4267,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6833/1659343919","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":1659343919},{"type":"v1","addr":"172.21.15.175:6833","nonce":1659343919}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4270,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6835/4113588141","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6834","nonce":4113588141},{"type":"v1","addr":"172.21.15.175:6835","nonce":4113588141}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4296,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/4151548261","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4151548261},{"type":"v1","addr":"172.21.15.72:6836","nonce":4151548261}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:14:51.411091+0000","modified":"2024-04-15T15:15:07.485436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4299},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4299":{"gid":4299,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6837/3458067804","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":3458067804},{"type":"v1","addr":"172.21.15.72:6837","nonce":3458067804}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4299,"qdb_cluster":[4299]},"id":1}]} 2024-04-15T15:15:55.987 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 9 2024-04-15T15:15:55.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.987+0000 7f403f488640 1 -- 172.21.15.72:0/3494504898 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f401c06c8f0 msgr2=0x7f401c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.987+0000 7f403f488640 1 --2- 172.21.15.72:0/3494504898 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f401c06c8f0 0x7f401c06edb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4034033000 tx=0x7f4034009030 comp rx=0 tx=0).stop 2024-04-15T15:15:55.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.987+0000 7f403f488640 1 -- 172.21.15.72:0/3494504898 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f403806bea0 msgr2=0x7f403815ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.987+0000 7f403f488640 1 --2- 172.21.15.72:0/3494504898 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f403806bea0 0x7f403815ca40 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f40380ccb00 tx=0x7f403000e530 comp rx=0 tx=0).stop 2024-04-15T15:15:55.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.987+0000 7f403f488640 1 -- 172.21.15.72:0/3494504898 shutdown_connections 2024-04-15T15:15:55.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.987+0000 7f403f488640 1 --2- 172.21.15.72:0/3494504898 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f401c06c8f0 0x7f401c06edb0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.987+0000 7f403f488640 1 --2- 172.21.15.72:0/3494504898 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f403806bea0 0x7f403815ca40 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.987+0000 7f403f488640 1 --2- 172.21.15.72:0/3494504898 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4038152d90 0x7f403806b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.987+0000 7f403f488640 1 --2- 172.21.15.72:0/3494504898 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f40240039e0 0x7f403814b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.987+0000 7f403f488640 1 -- 172.21.15.72:0/3494504898 >> 172.21.15.72:0/3494504898 conn(0x7f40380b1dd0 msgr2=0x7f40380ccb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:55.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.988+0000 7f403f488640 1 -- 172.21.15.72:0/3494504898 shutdown_connections 2024-04-15T15:15:55.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:55.988+0000 7f403f488640 1 -- 172.21.15.72:0/3494504898 wait complete. 2024-04-15T15:15:56.000 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:56.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.104+0000 7f09c6841640 1 Processor -- start 2024-04-15T15:15:56.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.105+0000 7f09c6841640 1 -- start start 2024-04-15T15:15:56.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.105+0000 7f09c6841640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f09c015c480 0x7f09c006ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.105+0000 7f09c6841640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09c00d1d80 con 0x7f09c00ce670 2024-04-15T15:15:56.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.105+0000 7f09c6841640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09c00d2060 con 0x7f09c00cf4d0 2024-04-15T15:15:56.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.105+0000 7f09c6841640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f09c00d1b30 con 0x7f09c015c480 2024-04-15T15:15:56.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.105+0000 7f09bf7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f09c00ce670 0x7f09c00cedc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:47854/0 (socket says 172.21.15.72:47854) 2024-04-15T15:15:56.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.105+0000 7f09bf7fe640 1 -- 172.21.15.72:0/3059973644 learned_addr learned my addr 172.21.15.72:0/3059973644 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:56.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.106+0000 7f09bffff640 1 --2- 172.21.15.72:0/3059973644 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f09c015c480 0x7f09c006ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.106+0000 7f09beffd640 1 -- 172.21.15.72:0/3059973644 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 748590687 0 0) 0x7f09c00d1d80 con 0x7f09c00ce670 2024-04-15T15:15:56.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.106+0000 7f09beffd640 1 -- 172.21.15.72:0/3059973644 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f09a8003520 con 0x7f09c00ce670 2024-04-15T15:15:56.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.106+0000 7f09beffd640 1 -- 172.21.15.72:0/3059973644 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1422243781 0 0) 0x7f09a8003520 con 0x7f09c00ce670 2024-04-15T15:15:56.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.106+0000 7f09beffd640 1 -- 172.21.15.72:0/3059973644 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f09c015c480 msgr2=0x7f09c006ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:56.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.106+0000 7f09beffd640 1 --2- 172.21.15.72:0/3059973644 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f09c015c480 0x7f09c006ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.106+0000 7f09beffd640 1 -- 172.21.15.72:0/3059973644 >> v1:172.21.15.175:6789/0 conn(0x7f09c00cf4d0 legacy=0x7f09c00b52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:56.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.106+0000 7f09beffd640 1 -- 172.21.15.72:0/3059973644 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09c00ccc50 con 0x7f09c00ce670 2024-04-15T15:15:56.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.106+0000 7f09beffd640 1 -- 172.21.15.72:0/3059973644 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f09b4001670 con 0x7f09c00ce670 2024-04-15T15:15:56.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.106+0000 7f09beffd640 1 -- 172.21.15.72:0/3059973644 >> v1:172.21.15.72:6789/0 conn(0x7f09c00ce670 legacy=0x7f09c00cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:56.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.106+0000 7f09beffd640 1 --2- 172.21.15.72:0/3059973644 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f09c015c480 0x7f09a80039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.106+0000 7f09beffd640 1 --2- 172.21.15.72:0/3059973644 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09a8003f20 0x7f09a8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.106+0000 7f09beffd640 1 --2- 172.21.15.72:0/3059973644 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09a8004d40 0x7f09a80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.106+0000 7f09beffd640 1 -- 172.21.15.72:0/3059973644 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f09c00d1b30 con 0x7f09a8004d40 2024-04-15T15:15:56.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.106+0000 7f09beffd640 1 -- 172.21.15.72:0/3059973644 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f09a80079a0 con 0x7f09a8003f20 2024-04-15T15:15:56.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.106+0000 7f09beffd640 1 -- 172.21.15.72:0/3059973644 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f09a8007ba0 con 0x7f09c015c480 2024-04-15T15:15:56.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.106+0000 7f09c4dde640 1 --2- 172.21.15.72:0/3059973644 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09a8004d40 0x7f09a80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.106+0000 7f09bffff640 1 --2- 172.21.15.72:0/3059973644 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f09c015c480 0x7f09a80039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.106+0000 7f09bf7fe640 1 --2- 172.21.15.72:0/3059973644 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09a8003f20 0x7f09a8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.106+0000 7f09c4dde640 1 -- 172.21.15.72:0/3059973644 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f09c015c480 msgr2=0x7f09a80039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:56.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.106+0000 7f09c4dde640 1 --2- 172.21.15.72:0/3059973644 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f09c015c480 0x7f09a80039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.106+0000 7f09c4dde640 1 -- 172.21.15.72:0/3059973644 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09a8003f20 msgr2=0x7f09a8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:56.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.106+0000 7f09c4dde640 1 --2- 172.21.15.72:0/3059973644 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09a8003f20 0x7f09a8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.106+0000 7f09c4dde640 1 -- 172.21.15.72:0/3059973644 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09a8003520 con 0x7f09a8004d40 2024-04-15T15:15:56.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.106+0000 7f09bf7fe640 1 --2- 172.21.15.72:0/3059973644 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09a8003f20 0x7f09a8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:56.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.107+0000 7f09c4dde640 1 --2- 172.21.15.72:0/3059973644 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09a8004d40 0x7f09a80051c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f09b8001f30 tx=0x7f09a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=972810993215bf77 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:56.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.107+0000 7f09beffd640 1 -- 172.21.15.72:0/3059973644 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b8001b30 con 0x7f09a8004d40 2024-04-15T15:15:56.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.107+0000 7f09beffd640 1 -- 172.21.15.72:0/3059973644 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09b802d6e0 con 0x7f09a8004d40 2024-04-15T15:15:56.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.107+0000 7f09beffd640 1 -- 172.21.15.72:0/3059973644 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b802dad0 con 0x7f09a8004d40 2024-04-15T15:15:56.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.108+0000 7f09c6841640 1 -- 172.21.15.72:0/3059973644 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09a8004d40 msgr2=0x7f09a80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:56.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.108+0000 7f09c6841640 1 --2- 172.21.15.72:0/3059973644 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09a8004d40 0x7f09a80051c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f09b8001f30 tx=0x7f09a8005830 comp rx=0 tx=0).stop 2024-04-15T15:15:56.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.108+0000 7f09c4dde640 1 -- 172.21.15.72:0/3059973644 reap_dead start 2024-04-15T15:15:56.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.108+0000 7f09c6841640 1 -- 172.21.15.72:0/3059973644 shutdown_connections 2024-04-15T15:15:56.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.108+0000 7f09c6841640 1 -- 172.21.15.72:0/3059973644 >> 172.21.15.72:0/3059973644 conn(0x7f09c00b3fa0 msgr2=0x7f09c00b43b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:56.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.108+0000 7f09c6841640 1 -- 172.21.15.72:0/3059973644 shutdown_connections 2024-04-15T15:15:56.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.108+0000 7f09c6841640 1 -- 172.21.15.72:0/3059973644 wait complete. 2024-04-15T15:15:56.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.109+0000 7f09c6841640 1 Processor -- start 2024-04-15T15:15:56.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.109+0000 7f09c6841640 1 -- start start 2024-04-15T15:15:56.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.110+0000 7f09c6841640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f09a8004d40 0x7f09c01e48d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.110+0000 7f09c6841640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09c01e2380 0x7f09c015abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.110+0000 7f09c6841640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09c00cabf0 0x7f09c00cb070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.110+0000 7f09c6841640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f09c00cd170 con 0x7f09c01e2380 2024-04-15T15:15:56.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.110+0000 7f09c6841640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f09c00cca00 con 0x7f09c00cabf0 2024-04-15T15:15:56.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.110+0000 7f09c6841640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f09c015c860 con 0x7f09a8004d40 2024-04-15T15:15:56.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.110+0000 7f09bf7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09c01e2380 0x7f09c015abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.118 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.110+0000 7f09bf7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09c01e2380 0x7f09c015abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48850/0 (socket says 172.21.15.72:48850) 2024-04-15T15:15:56.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.110+0000 7f09bf7fe640 1 -- 172.21.15.72:0/1394356778 learned_addr learned my addr 172.21.15.72:0/1394356778 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:56.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.110+0000 7f09bffff640 1 --2- 172.21.15.72:0/1394356778 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f09a8004d40 0x7f09c01e48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.110+0000 7f09c4dde640 1 --2- 172.21.15.72:0/1394356778 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09c00cabf0 0x7f09c00cb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.110+0000 7f09bf7fe640 1 -- 172.21.15.72:0/1394356778 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f09a8004d40 msgr2=0x7f09c01e48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:56.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.110+0000 7f09bf7fe640 1 --2- 172.21.15.72:0/1394356778 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f09a8004d40 0x7f09c01e48d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.110+0000 7f09bf7fe640 1 -- 172.21.15.72:0/1394356778 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09c00cabf0 msgr2=0x7f09c00cb070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:56.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.110+0000 7f09bf7fe640 1 --2- 172.21.15.72:0/1394356778 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09c00cabf0 0x7f09c00cb070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.110+0000 7f09bf7fe640 1 -- 172.21.15.72:0/1394356778 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09c01e6230 con 0x7f09c01e2380 2024-04-15T15:15:56.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.110+0000 7f09bf7fe640 1 --2- 172.21.15.72:0/1394356778 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09c01e2380 0x7f09c015abe0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f09b400e7d0 tx=0x7f09b400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:56.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.111+0000 7f09bcff9640 1 -- 172.21.15.72:0/1394356778 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b400ee20 con 0x7f09c01e2380 2024-04-15T15:15:56.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.111+0000 7f09bcff9640 1 -- 172.21.15.72:0/1394356778 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09b4018070 con 0x7f09c01e2380 2024-04-15T15:15:56.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.111+0000 7f09c6841640 1 -- 172.21.15.72:0/1394356778 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09c01e5190 con 0x7f09c01e2380 2024-04-15T15:15:56.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.111+0000 7f09bcff9640 1 -- 172.21.15.72:0/1394356778 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b400cad0 con 0x7f09c01e2380 2024-04-15T15:15:56.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.111+0000 7f09c6841640 1 -- 172.21.15.72:0/1394356778 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09c01e5750 con 0x7f09c01e2380 2024-04-15T15:15:56.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.112+0000 7f09c6841640 1 -- 172.21.15.72:0/1394356778 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09c00690f0 con 0x7f09c01e2380 2024-04-15T15:15:56.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.112+0000 7f09bcff9640 1 -- 172.21.15.72:0/1394356778 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f09b400cc70 con 0x7f09c01e2380 2024-04-15T15:15:56.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.113+0000 7f09bcff9640 1 --2- 172.21.15.72:0/1394356778 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f09ac06c8f0 0x7f09ac06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.113+0000 7f09bffff640 1 --2- 172.21.15.72:0/1394356778 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f09ac06c8f0 0x7f09ac06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.113+0000 7f09bcff9640 1 -- 172.21.15.72:0/1394356778 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f09b408bd70 con 0x7f09c01e2380 2024-04-15T15:15:56.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.113+0000 7f09bffff640 1 --2- 172.21.15.72:0/1394356778 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f09ac06c8f0 0x7f09ac06edb0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f09b0009bc0 tx=0x7f09b0009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:56.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.117+0000 7f09bcff9640 1 -- 172.21.15.72:0/1394356778 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f09b4059da0 con 0x7f09c01e2380 2024-04-15T15:15:56.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.292+0000 7f09c6841640 1 -- 172.21.15.72:0/1394356778 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f09c0074e30 con 0x7f09c01e2380 2024-04-15T15:15:56.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.293+0000 7f09bcff9640 1 -- 172.21.15.72:0/1394356778 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18903 (secure 0 0 0) 0x7f09b40596a0 con 0x7f09c01e2380 2024-04-15T15:15:56.295 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:15:56.295 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":26,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:15:32.421728+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:14:49.438366+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:14:50.368379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:14:54.081157+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.2","mappings":[{"from":3,"to":1}]},{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.28","mappings":[{"from":5,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:56.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.295+0000 7f09c6841640 1 -- 172.21.15.72:0/1394356778 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f09ac06c8f0 msgr2=0x7f09ac06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:56.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.295+0000 7f09c6841640 1 --2- 172.21.15.72:0/1394356778 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f09ac06c8f0 0x7f09ac06edb0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f09b0009bc0 tx=0x7f09b0009190 comp rx=0 tx=0).stop 2024-04-15T15:15:56.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.295+0000 7f09c6841640 1 -- 172.21.15.72:0/1394356778 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09c01e2380 msgr2=0x7f09c015abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:56.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.295+0000 7f09c6841640 1 --2- 172.21.15.72:0/1394356778 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09c01e2380 0x7f09c015abe0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f09b400e7d0 tx=0x7f09b400ebd0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.295+0000 7f09c6841640 1 -- 172.21.15.72:0/1394356778 shutdown_connections 2024-04-15T15:15:56.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.295+0000 7f09c6841640 1 --2- 172.21.15.72:0/1394356778 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f09ac06c8f0 0x7f09ac06edb0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.295+0000 7f09c6841640 1 --2- 172.21.15.72:0/1394356778 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f09c00cabf0 0x7f09c00cb070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.295+0000 7f09c6841640 1 --2- 172.21.15.72:0/1394356778 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f09c01e2380 0x7f09c015abe0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.295+0000 7f09c6841640 1 --2- 172.21.15.72:0/1394356778 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f09a8004d40 0x7f09c01e48d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.295+0000 7f09c6841640 1 -- 172.21.15.72:0/1394356778 >> 172.21.15.72:0/1394356778 conn(0x7f09c00b3fa0 msgr2=0x7f09c006f310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:56.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.296+0000 7f09c6841640 1 -- 172.21.15.72:0/1394356778 shutdown_connections 2024-04-15T15:15:56.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.296+0000 7f09c6841640 1 -- 172.21.15.72:0/1394356778 wait complete. 2024-04-15T15:15:56.309 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-15T15:15:56.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.411+0000 7f5f862c8640 1 Processor -- start 2024-04-15T15:15:56.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.411+0000 7f5f862c8640 1 -- start start 2024-04-15T15:15:56.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.412+0000 7f5f862c8640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5f8015c630 0x7f5f8006cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.412+0000 7f5f862c8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5f800ccb90 0x7f5f800ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.412+0000 7f5f862c8640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f5f8006e620 0x7f5f8006eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.412+0000 7f5f862c8640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f5f8006d2f0 con 0x7f5f800ccb90 2024-04-15T15:15:56.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.412+0000 7f5f862c8640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f5f800cd530 con 0x7f5f8006e620 2024-04-15T15:15:56.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.412+0000 7f5f862c8640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f5f8006c920 con 0x7f5f8015c630 2024-04-15T15:15:56.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.412+0000 7f5f77fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5f800ccb90 0x7f5f800ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.412+0000 7f5f77fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f5f800ccb90 0x7f5f800ccff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48860/0 (socket says 172.21.15.72:48860) 2024-04-15T15:15:56.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.412+0000 7f5f77fff640 1 -- 172.21.15.72:0/2603265227 learned_addr learned my addr 172.21.15.72:0/2603265227 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:56.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.412+0000 7f5f84865640 1 --2- 172.21.15.72:0/2603265227 >> v2:172.21.15.175:3300/0 conn(0x7f5f8006e620 0x7f5f8006eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.412+0000 7f5f77fff640 1 -- 172.21.15.72:0/2603265227 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5f8015c630 msgr2=0x7f5f8006cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:56.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.412+0000 7f5f7ffff640 1 --2- 172.21.15.72:0/2603265227 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5f8015c630 0x7f5f8006cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.412+0000 7f5f77fff640 1 --2- 172.21.15.72:0/2603265227 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5f8015c630 0x7f5f8006cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.412+0000 7f5f77fff640 1 -- 172.21.15.72:0/2603265227 >> v2:172.21.15.175:3300/0 conn(0x7f5f8006e620 msgr2=0x7f5f8006eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:56.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.412+0000 7f5f77fff640 1 --2- 172.21.15.72:0/2603265227 >> v2:172.21.15.175:3300/0 conn(0x7f5f8006e620 0x7f5f8006eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.412+0000 7f5f77fff640 1 -- 172.21.15.72:0/2603265227 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f8006bd50 con 0x7f5f800ccb90 2024-04-15T15:15:56.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.412+0000 7f5f84865640 1 --2- 172.21.15.72:0/2603265227 >> v2:172.21.15.175:3300/0 conn(0x7f5f8006e620 0x7f5f8006eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:56.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.413+0000 7f5f77fff640 1 --2- 172.21.15.72:0/2603265227 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5f800ccb90 0x7f5f800ccff0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f5f800d11c0 tx=0x7f5f6802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67453936464addc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:56.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.413+0000 7f5f7f7fe640 1 -- 172.21.15.72:0/2603265227 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f68035020 con 0x7f5f800ccb90 2024-04-15T15:15:56.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.413+0000 7f5f7f7fe640 1 -- 172.21.15.72:0/2603265227 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f68031870 con 0x7f5f800ccb90 2024-04-15T15:15:56.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.413+0000 7f5f7f7fe640 1 -- 172.21.15.72:0/2603265227 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f68031c40 con 0x7f5f800ccb90 2024-04-15T15:15:56.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.414+0000 7f5f862c8640 1 -- 172.21.15.72:0/2603265227 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5f800ccb90 msgr2=0x7f5f800ccff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:56.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.414+0000 7f5f862c8640 1 --2- 172.21.15.72:0/2603265227 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5f800ccb90 0x7f5f800ccff0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f5f800d11c0 tx=0x7f5f6802b910 comp rx=0 tx=0).stop 2024-04-15T15:15:56.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.414+0000 7f5f862c8640 1 -- 172.21.15.72:0/2603265227 shutdown_connections 2024-04-15T15:15:56.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.414+0000 7f5f862c8640 1 --2- 172.21.15.72:0/2603265227 >> v2:172.21.15.175:3300/0 conn(0x7f5f8006e620 0x7f5f8006eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.414+0000 7f5f862c8640 1 --2- 172.21.15.72:0/2603265227 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5f800ccb90 0x7f5f800ccff0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.414+0000 7f5f862c8640 1 --2- 172.21.15.72:0/2603265227 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5f8015c630 0x7f5f8006cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.414+0000 7f5f862c8640 1 -- 172.21.15.72:0/2603265227 >> 172.21.15.72:0/2603265227 conn(0x7f5f800b31d0 msgr2=0x7f5f800b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:56.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.414+0000 7f5f862c8640 1 -- 172.21.15.72:0/2603265227 shutdown_connections 2024-04-15T15:15:56.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.414+0000 7f5f862c8640 1 -- 172.21.15.72:0/2603265227 wait complete. 2024-04-15T15:15:56.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.415+0000 7f5f862c8640 1 Processor -- start 2024-04-15T15:15:56.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.415+0000 7f5f862c8640 1 -- start start 2024-04-15T15:15:56.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.416+0000 7f5f862c8640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5f8006e620 0x7f5f80154be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.416+0000 7f5f862c8640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5f800ccb90 0x7f5f80159450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.416+0000 7f5f862c8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5f8015c630 0x7f5f8014b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.416+0000 7f5f862c8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f5f8006c5b0 con 0x7f5f8015c630 2024-04-15T15:15:56.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.416+0000 7f5f862c8640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f5f8006c2d0 con 0x7f5f8006e620 2024-04-15T15:15:56.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.416+0000 7f5f862c8640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f5f8015ca10 con 0x7f5f800ccb90 2024-04-15T15:15:56.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.416+0000 7f5f84865640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5f8015c630 0x7f5f8014b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.416+0000 7f5f84865640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5f8015c630 0x7f5f8014b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48866/0 (socket says 172.21.15.72:48866) 2024-04-15T15:15:56.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.416+0000 7f5f84865640 1 -- 172.21.15.72:0/2633812271 learned_addr learned my addr 172.21.15.72:0/2633812271 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:56.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.416+0000 7f5f7ffff640 1 --2- 172.21.15.72:0/2633812271 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5f8006e620 0x7f5f80154be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.416+0000 7f5f77fff640 1 --2- 172.21.15.72:0/2633812271 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5f800ccb90 0x7f5f80159450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.416+0000 7f5f84865640 1 -- 172.21.15.72:0/2633812271 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5f800ccb90 msgr2=0x7f5f80159450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:56.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.416+0000 7f5f84865640 1 --2- 172.21.15.72:0/2633812271 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5f800ccb90 0x7f5f80159450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.416+0000 7f5f84865640 1 -- 172.21.15.72:0/2633812271 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5f8006e620 msgr2=0x7f5f80154be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:56.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.416+0000 7f5f84865640 1 --2- 172.21.15.72:0/2633812271 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5f8006e620 0x7f5f80154be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.416+0000 7f5f84865640 1 -- 172.21.15.72:0/2633812271 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f80159bf0 con 0x7f5f8015c630 2024-04-15T15:15:56.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.416+0000 7f5f84865640 1 --2- 172.21.15.72:0/2633812271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5f8015c630 0x7f5f8014b750 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f5f7800cef0 tx=0x7f5f7800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:56.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.417+0000 7f5f7d7fa640 1 -- 172.21.15.72:0/2633812271 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f7801a070 con 0x7f5f8015c630 2024-04-15T15:15:56.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.417+0000 7f5f7d7fa640 1 -- 172.21.15.72:0/2633812271 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f7800d860 con 0x7f5f8015c630 2024-04-15T15:15:56.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.417+0000 7f5f862c8640 1 -- 172.21.15.72:0/2633812271 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f80159ed0 con 0x7f5f8015c630 2024-04-15T15:15:56.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.417+0000 7f5f7d7fa640 1 -- 172.21.15.72:0/2633812271 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f7800dc50 con 0x7f5f8015c630 2024-04-15T15:15:56.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.417+0000 7f5f862c8640 1 -- 172.21.15.72:0/2633812271 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f8015a490 con 0x7f5f8015c630 2024-04-15T15:15:56.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.418+0000 7f5f862c8640 1 -- 172.21.15.72:0/2633812271 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f80075c50 con 0x7f5f8015c630 2024-04-15T15:15:56.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.418+0000 7f5f7d7fa640 1 -- 172.21.15.72:0/2633812271 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5f7801d070 con 0x7f5f8015c630 2024-04-15T15:15:56.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.419+0000 7f5f7d7fa640 1 --2- 172.21.15.72:0/2633812271 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f5f6006c8f0 0x7f5f6006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.419+0000 7f5f7ffff640 1 --2- 172.21.15.72:0/2633812271 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f5f6006c8f0 0x7f5f6006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.419+0000 7f5f7d7fa640 1 -- 172.21.15.72:0/2633812271 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f5f7808c790 con 0x7f5f8015c630 2024-04-15T15:15:56.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.419+0000 7f5f7ffff640 1 --2- 172.21.15.72:0/2633812271 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f5f6006c8f0 0x7f5f6006edb0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5f8015aae0 tx=0x7f5f70008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:56.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.423+0000 7f5f7d7fa640 1 -- 172.21.15.72:0/2633812271 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5f7805a910 con 0x7f5f8015c630 2024-04-15T15:15:56.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.636+0000 7f5f862c8640 1 -- 172.21.15.72:0/2633812271 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f5f800d0f10 con 0x7f5f8015c630 2024-04-15T15:15:56.638 INFO:tasks.ceph.mon.a.smithi072.stderr:2024-04-15T15:15:56.637+0000 7f01d0611640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-15T15:15:56.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.638+0000 7f5f7d7fa640 1 -- 172.21.15.72:0/2633812271 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v10) ==== 161+0+0 (secure 0 0 0) 0x7f5f7805a210 con 0x7f5f8015c630 2024-04-15T15:15:56.639 INFO:teuthology.orchestra.run.smithi072.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-15T15:15:56.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.641+0000 7f5f862c8640 1 -- 172.21.15.72:0/2633812271 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f5f6006c8f0 msgr2=0x7f5f6006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:56.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.641+0000 7f5f862c8640 1 --2- 172.21.15.72:0/2633812271 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f5f6006c8f0 0x7f5f6006edb0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5f8015aae0 tx=0x7f5f70008040 comp rx=0 tx=0).stop 2024-04-15T15:15:56.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.641+0000 7f5f862c8640 1 -- 172.21.15.72:0/2633812271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5f8015c630 msgr2=0x7f5f8014b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:56.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.641+0000 7f5f862c8640 1 --2- 172.21.15.72:0/2633812271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5f8015c630 0x7f5f8014b750 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f5f7800cef0 tx=0x7f5f7800a710 comp rx=0 tx=0).stop 2024-04-15T15:15:56.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.641+0000 7f5f862c8640 1 -- 172.21.15.72:0/2633812271 shutdown_connections 2024-04-15T15:15:56.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.641+0000 7f5f862c8640 1 --2- 172.21.15.72:0/2633812271 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f5f6006c8f0 0x7f5f6006edb0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.641+0000 7f5f862c8640 1 --2- 172.21.15.72:0/2633812271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5f8015c630 0x7f5f8014b750 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.641+0000 7f5f862c8640 1 --2- 172.21.15.72:0/2633812271 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f5f800ccb90 0x7f5f80159450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.641+0000 7f5f862c8640 1 --2- 172.21.15.72:0/2633812271 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5f8006e620 0x7f5f80154be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.641+0000 7f5f862c8640 1 -- 172.21.15.72:0/2633812271 >> 172.21.15.72:0/2633812271 conn(0x7f5f800b31d0 msgr2=0x7f5f800cf860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:56.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.641+0000 7f5f862c8640 1 -- 172.21.15.72:0/2633812271 shutdown_connections 2024-04-15T15:15:56.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.641+0000 7f5f862c8640 1 -- 172.21.15.72:0/2633812271 wait complete. 2024-04-15T15:15:56.645 INFO:tasks.ceph.mds.c.smithi072.stderr: -4538> 2024-04-15T15:14:55.989+0000 7f7075518640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-15T15:15:56.657 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-15T15:15:56.678 INFO:tasks.ceph.mds.c.smithi072.stdout:starting mds.c at 2024-04-15T15:15:56.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.761+0000 7f69101a8640 1 Processor -- start 2024-04-15T15:15:56.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.761+0000 7f69101a8640 1 -- start start 2024-04-15T15:15:56.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.762+0000 7f69101a8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f69080b67c0 0x7f69080d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.762+0000 7f69101a8640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f69080b6c20 0x7f69080d1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.762+0000 7f69101a8640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69080ccfe0 con 0x7f6908161ee0 2024-04-15T15:15:56.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.762+0000 7f69101a8640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f69081622c0 con 0x7f69080b67c0 2024-04-15T15:15:56.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.762+0000 7f69101a8640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f69080cd2c0 con 0x7f69080b6c20 2024-04-15T15:15:56.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.762+0000 7f690d743640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f69080b67c0 0x7f69080d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.762+0000 7f690d743640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f69080b67c0 0x7f69080d1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48902/0 (socket says 172.21.15.72:48902) 2024-04-15T15:15:56.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.762+0000 7f690d743640 1 -- 172.21.15.72:0/329857999 learned_addr learned my addr 172.21.15.72:0/329857999 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:56.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.763+0000 7f690e745640 1 --2- 172.21.15.72:0/329857999 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f69080b6c20 0x7f69080d1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.763+0000 7f690d743640 1 -- 172.21.15.72:0/329857999 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f69080b6c20 msgr2=0x7f69080d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:56.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.763+0000 7f690d743640 1 --2- 172.21.15.72:0/329857999 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f69080b6c20 0x7f69080d1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.763+0000 7f690d743640 1 -- 172.21.15.72:0/329857999 >> v1:172.21.15.175:6789/0 conn(0x7f6908161ee0 legacy=0x7f69080c95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:56.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.763+0000 7f690d743640 1 -- 172.21.15.72:0/329857999 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69080cd5a0 con 0x7f69080b67c0 2024-04-15T15:15:56.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.763+0000 7f690d743640 1 --2- 172.21.15.72:0/329857999 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69080b67c0 0x7f69080d1100 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f68f8009870 tx=0x7f68f802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=822228351874cdc server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:56.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.764+0000 7f690cf42640 1 -- 172.21.15.72:0/329857999 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f8034070 con 0x7f69080b67c0 2024-04-15T15:15:56.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.764+0000 7f690cf42640 1 -- 172.21.15.72:0/329857999 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68f8030840 con 0x7f69080b67c0 2024-04-15T15:15:56.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.764+0000 7f690cf42640 1 -- 172.21.15.72:0/329857999 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f8030c10 con 0x7f69080b67c0 2024-04-15T15:15:56.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.764+0000 7f69101a8640 1 -- 172.21.15.72:0/329857999 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69080b67c0 msgr2=0x7f69080d1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:56.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.764+0000 7f69101a8640 1 --2- 172.21.15.72:0/329857999 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69080b67c0 0x7f69080d1100 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f68f8009870 tx=0x7f68f802b980 comp rx=0 tx=0).stop 2024-04-15T15:15:56.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.764+0000 7f69101a8640 1 -- 172.21.15.72:0/329857999 shutdown_connections 2024-04-15T15:15:56.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.764+0000 7f69101a8640 1 --2- 172.21.15.72:0/329857999 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f69080b6c20 0x7f69080d1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.764+0000 7f69101a8640 1 --2- 172.21.15.72:0/329857999 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69080b67c0 0x7f69080d1100 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.764+0000 7f69101a8640 1 -- 172.21.15.72:0/329857999 >> 172.21.15.72:0/329857999 conn(0x7f69080b4010 msgr2=0x7f69080b4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:56.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.764+0000 7f69101a8640 1 -- 172.21.15.72:0/329857999 shutdown_connections 2024-04-15T15:15:56.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.764+0000 7f69101a8640 1 -- 172.21.15.72:0/329857999 wait complete. 2024-04-15T15:15:56.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.765+0000 7f69101a8640 1 Processor -- start 2024-04-15T15:15:56.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.765+0000 7f69101a8640 1 -- start start 2024-04-15T15:15:56.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.766+0000 7f69101a8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69080b67c0 0x7f6908156f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.766+0000 7f69101a8640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69080b6c20 0x7f690815c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.766+0000 7f69101a8640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6908161ee0 0x7f690815c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.766+0000 7f69101a8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f69080cccf0 con 0x7f69080b67c0 2024-04-15T15:15:56.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.766+0000 7f69101a8640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f69080ccb70 con 0x7f69080b6c20 2024-04-15T15:15:56.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.766+0000 7f69101a8640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f69080ce080 con 0x7f6908161ee0 2024-04-15T15:15:56.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.766+0000 7f690df44640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69080b67c0 0x7f6908156f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.766+0000 7f690df44640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69080b67c0 0x7f6908156f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48918/0 (socket says 172.21.15.72:48918) 2024-04-15T15:15:56.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.766+0000 7f690df44640 1 -- 172.21.15.72:0/3940583831 learned_addr learned my addr 172.21.15.72:0/3940583831 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:56.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.767+0000 7f690d743640 1 --2- 172.21.15.72:0/3940583831 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69080b6c20 0x7f690815c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.767+0000 7f690e745640 1 --2- 172.21.15.72:0/3940583831 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6908161ee0 0x7f690815c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.767+0000 7f690df44640 1 -- 172.21.15.72:0/3940583831 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6908161ee0 msgr2=0x7f690815c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:56.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.767+0000 7f690df44640 1 --2- 172.21.15.72:0/3940583831 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6908161ee0 0x7f690815c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.770 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.767+0000 7f690df44640 1 -- 172.21.15.72:0/3940583831 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69080b6c20 msgr2=0x7f690815c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:56.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.767+0000 7f690df44640 1 --2- 172.21.15.72:0/3940583831 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69080b6c20 0x7f690815c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.767+0000 7f690df44640 1 -- 172.21.15.72:0/3940583831 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69081f5d30 con 0x7f69080b67c0 2024-04-15T15:15:56.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.767+0000 7f690d743640 1 --2- 172.21.15.72:0/3940583831 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69080b6c20 0x7f690815c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:56.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.767+0000 7f690df44640 1 --2- 172.21.15.72:0/3940583831 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69080b67c0 0x7f6908156f30 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f690400c760 tx=0x7f690400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:56.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.767+0000 7f68fe7fc640 1 -- 172.21.15.72:0/3940583831 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6904016020 con 0x7f69080b67c0 2024-04-15T15:15:56.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.767+0000 7f68fe7fc640 1 -- 172.21.15.72:0/3940583831 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6904006d50 con 0x7f69080b67c0 2024-04-15T15:15:56.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.767+0000 7f69101a8640 1 -- 172.21.15.72:0/3940583831 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69080c9010 con 0x7f69080b67c0 2024-04-15T15:15:56.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.767+0000 7f68fe7fc640 1 -- 172.21.15.72:0/3940583831 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6904003bb0 con 0x7f69080b67c0 2024-04-15T15:15:56.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.767+0000 7f69101a8640 1 -- 172.21.15.72:0/3940583831 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69081f5250 con 0x7f69080b67c0 2024-04-15T15:15:56.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.768+0000 7f69101a8640 1 -- 172.21.15.72:0/3940583831 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6908075c50 con 0x7f69080b67c0 2024-04-15T15:15:56.771 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.768+0000 7f68fe7fc640 1 -- 172.21.15.72:0/3940583831 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f690400a850 con 0x7f69080b67c0 2024-04-15T15:15:56.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.769+0000 7f68fe7fc640 1 --2- 172.21.15.72:0/3940583831 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f68ec06c8f0 0x7f68ec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.769+0000 7f68fe7fc640 1 -- 172.21.15.72:0/3940583831 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7f690408bbe0 con 0x7f69080b67c0 2024-04-15T15:15:56.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.769+0000 7f690d743640 1 --2- 172.21.15.72:0/3940583831 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f68ec06c8f0 0x7f68ec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.769+0000 7f690d743640 1 --2- 172.21.15.72:0/3940583831 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f68ec06c8f0 0x7f68ec06edb0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f68f8036000 tx=0x7f68f803b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:56.774 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.773+0000 7f68fe7fc640 1 -- 172.21.15.72:0/3940583831 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6904020020 con 0x7f69080b67c0 2024-04-15T15:15:56.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:56.985+0000 7f69101a8640 1 -- 172.21.15.72:0/3940583831 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f69080d1e50 con 0x7f69080b67c0 2024-04-15T15:15:57.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.646+0000 7f68fe7fc640 1 -- 172.21.15.72:0/3940583831 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v11) ==== 104+0+0 (secure 0 0 0) 0x7f6904019310 con 0x7f69080b67c0 2024-04-15T15:15:57.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.649+0000 7f69101a8640 1 -- 172.21.15.72:0/3940583831 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f68ec06c8f0 msgr2=0x7f68ec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:57.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.649+0000 7f69101a8640 1 --2- 172.21.15.72:0/3940583831 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f68ec06c8f0 0x7f68ec06edb0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f68f8036000 tx=0x7f68f803b040 comp rx=0 tx=0).stop 2024-04-15T15:15:57.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.649+0000 7f69101a8640 1 -- 172.21.15.72:0/3940583831 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69080b67c0 msgr2=0x7f6908156f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:57.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.649+0000 7f69101a8640 1 --2- 172.21.15.72:0/3940583831 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69080b67c0 0x7f6908156f30 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f690400c760 tx=0x7f690400cc20 comp rx=0 tx=0).stop 2024-04-15T15:15:57.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.649+0000 7f69101a8640 1 -- 172.21.15.72:0/3940583831 shutdown_connections 2024-04-15T15:15:57.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.649+0000 7f69101a8640 1 --2- 172.21.15.72:0/3940583831 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f68ec06c8f0 0x7f68ec06edb0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.649+0000 7f69101a8640 1 --2- 172.21.15.72:0/3940583831 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6908161ee0 0x7f690815c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.649+0000 7f69101a8640 1 --2- 172.21.15.72:0/3940583831 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69080b6c20 0x7f690815c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.649+0000 7f69101a8640 1 --2- 172.21.15.72:0/3940583831 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69080b67c0 0x7f6908156f30 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.649+0000 7f69101a8640 1 -- 172.21.15.72:0/3940583831 >> 172.21.15.72:0/3940583831 conn(0x7f69080b4010 msgr2=0x7f69080ca0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:57.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.650+0000 7f69101a8640 1 -- 172.21.15.72:0/3940583831 shutdown_connections 2024-04-15T15:15:57.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.650+0000 7f69101a8640 1 -- 172.21.15.72:0/3940583831 wait complete. 2024-04-15T15:15:57.667 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-15T15:15:57.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.819+0000 7f17a5841640 1 Processor -- start 2024-04-15T15:15:57.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.819+0000 7f17a5841640 1 -- start start 2024-04-15T15:15:57.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.820+0000 7f17a5841640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f17a015e9d0 0x7f17a00b3a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:57.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.820+0000 7f17a5841640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f17a00d4a40 0x7f17a00d2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:57.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.820+0000 7f17a5841640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17a00d50c0 con 0x7f17a00d1760 2024-04-15T15:15:57.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.820+0000 7f17a5841640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f17a00d22c0 con 0x7f17a00d4a40 2024-04-15T15:15:57.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.820+0000 7f17a5841640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f17a00d0e80 con 0x7f17a015e9d0 2024-04-15T15:15:57.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.820+0000 7f179f7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f17a00d4a40 0x7f17a00d2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:57.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.820+0000 7f179f7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f17a00d4a40 0x7f17a00d2750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48928/0 (socket says 172.21.15.72:48928) 2024-04-15T15:15:57.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.820+0000 7f179f7fe640 1 -- 172.21.15.72:0/1883958817 learned_addr learned my addr 172.21.15.72:0/1883958817 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:57.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.821+0000 7f179f7fe640 1 -- 172.21.15.72:0/1883958817 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f17a015e9d0 msgr2=0x7f17a00b3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:57.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.821+0000 7f179effd640 1 --2- 172.21.15.72:0/1883958817 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f17a015e9d0 0x7f17a00b3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:57.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.821+0000 7f179f7fe640 1 --2- 172.21.15.72:0/1883958817 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f17a015e9d0 0x7f17a00b3a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.821+0000 7f179f7fe640 1 -- 172.21.15.72:0/1883958817 >> v1:172.21.15.175:6789/0 conn(0x7f17a00d1760 legacy=0x7f17a00d4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:57.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.821+0000 7f179f7fe640 1 -- 172.21.15.72:0/1883958817 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17a00d05e0 con 0x7f17a00d4a40 2024-04-15T15:15:57.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.821+0000 7f179f7fe640 1 --2- 172.21.15.72:0/1883958817 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17a00d4a40 0x7f17a00d2750 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f1798009870 tx=0x7f179802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=895ddfd8beb0d687 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:57.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.822+0000 7f179dffb640 1 -- 172.21.15.72:0/1883958817 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1798035020 con 0x7f17a00d4a40 2024-04-15T15:15:57.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.822+0000 7f179dffb640 1 -- 172.21.15.72:0/1883958817 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1798031640 con 0x7f17a00d4a40 2024-04-15T15:15:57.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.822+0000 7f179dffb640 1 -- 172.21.15.72:0/1883958817 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f179802d040 con 0x7f17a00d4a40 2024-04-15T15:15:57.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.822+0000 7f17a5841640 1 -- 172.21.15.72:0/1883958817 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17a00d4a40 msgr2=0x7f17a00d2750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:57.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.822+0000 7f17a5841640 1 --2- 172.21.15.72:0/1883958817 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17a00d4a40 0x7f17a00d2750 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f1798009870 tx=0x7f179802b8f0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.822+0000 7f17a5841640 1 -- 172.21.15.72:0/1883958817 shutdown_connections 2024-04-15T15:15:57.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.822+0000 7f17a5841640 1 --2- 172.21.15.72:0/1883958817 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17a00d4a40 0x7f17a00d2750 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.822+0000 7f17a5841640 1 --2- 172.21.15.72:0/1883958817 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f17a015e9d0 0x7f17a00b3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.822+0000 7f17a5841640 1 -- 172.21.15.72:0/1883958817 >> 172.21.15.72:0/1883958817 conn(0x7f17a009d970 msgr2=0x7f17a00b9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:57.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.822+0000 7f17a5841640 1 -- 172.21.15.72:0/1883958817 shutdown_connections 2024-04-15T15:15:57.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.822+0000 7f17a5841640 1 -- 172.21.15.72:0/1883958817 wait complete. 2024-04-15T15:15:57.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.823+0000 7f17a5841640 1 Processor -- start 2024-04-15T15:15:57.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.824+0000 7f17a5841640 1 -- start start 2024-04-15T15:15:57.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.824+0000 7f17a5841640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f17a00d1760 0x7f17a00cc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:57.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.824+0000 7f17a5841640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17a00d4a40 0x7f17a015ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:57.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.824+0000 7f17a5841640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f17a015e9d0 0x7f17a01f47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:57.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.824+0000 7f17a5841640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f17a015edb0 con 0x7f17a00d4a40 2024-04-15T15:15:57.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.824+0000 7f17a5841640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f17a00d0ba0 con 0x7f17a00d1760 2024-04-15T15:15:57.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.824+0000 7f17a5841640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f17a00d2c90 con 0x7f17a015e9d0 2024-04-15T15:15:57.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.824+0000 7f179e7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17a00d4a40 0x7f17a015ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:57.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.824+0000 7f179e7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17a00d4a40 0x7f17a015ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48934/0 (socket says 172.21.15.72:48934) 2024-04-15T15:15:57.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.824+0000 7f179e7fc640 1 -- 172.21.15.72:0/881000868 learned_addr learned my addr 172.21.15.72:0/881000868 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:57.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.825+0000 7f179effd640 1 --2- 172.21.15.72:0/881000868 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f17a00d1760 0x7f17a00cc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:57.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.825+0000 7f179f7fe640 1 --2- 172.21.15.72:0/881000868 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f17a015e9d0 0x7f17a01f47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:57.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.825+0000 7f179e7fc640 1 -- 172.21.15.72:0/881000868 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f17a015e9d0 msgr2=0x7f17a01f47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:57.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.825+0000 7f179e7fc640 1 --2- 172.21.15.72:0/881000868 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f17a015e9d0 0x7f17a01f47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.825+0000 7f179e7fc640 1 -- 172.21.15.72:0/881000868 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f17a00d1760 msgr2=0x7f17a00cc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:57.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.825+0000 7f179e7fc640 1 --2- 172.21.15.72:0/881000868 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f17a00d1760 0x7f17a00cc0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.825+0000 7f179e7fc640 1 -- 172.21.15.72:0/881000868 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17a00cc680 con 0x7f17a00d4a40 2024-04-15T15:15:57.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.825+0000 7f179e7fc640 1 --2- 172.21.15.72:0/881000868 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17a00d4a40 0x7f17a015ae70 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f179400c430 tx=0x7f179400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:57.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.825+0000 7f1787fff640 1 -- 172.21.15.72:0/881000868 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1794011790 con 0x7f17a00d4a40 2024-04-15T15:15:57.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.825+0000 7f1787fff640 1 -- 172.21.15.72:0/881000868 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1794017070 con 0x7f17a00d4a40 2024-04-15T15:15:57.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.825+0000 7f17a5841640 1 -- 172.21.15.72:0/881000868 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17a00cc960 con 0x7f17a00d4a40 2024-04-15T15:15:57.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.825+0000 7f1787fff640 1 -- 172.21.15.72:0/881000868 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1794011b70 con 0x7f17a00d4a40 2024-04-15T15:15:57.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.825+0000 7f17a5841640 1 -- 172.21.15.72:0/881000868 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17a00ccf20 con 0x7f17a00d4a40 2024-04-15T15:15:57.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.826+0000 7f17a5841640 1 -- 172.21.15.72:0/881000868 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17a00689e0 con 0x7f17a00d4a40 2024-04-15T15:15:57.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.826+0000 7f1787fff640 1 -- 172.21.15.72:0/881000868 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1794011d10 con 0x7f17a00d4a40 2024-04-15T15:15:57.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.827+0000 7f1787fff640 1 --2- 172.21.15.72:0/881000868 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f178c06c8f0 0x7f178c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:57.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.827+0000 7f1787fff640 1 -- 172.21.15.72:0/881000868 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7f179408be50 con 0x7f17a00d4a40 2024-04-15T15:15:57.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.827+0000 7f179effd640 1 --2- 172.21.15.72:0/881000868 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f178c06c8f0 0x7f178c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:57.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.827+0000 7f179effd640 1 --2- 172.21.15.72:0/881000868 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f178c06c8f0 0x7f178c06edb0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f17900070b0 tx=0x7f1790007040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:57.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:57.831+0000 7f1787fff640 1 -- 172.21.15.72:0/881000868 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1794059eb0 con 0x7f17a00d4a40 2024-04-15T15:15:58.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.006+0000 7f17a5841640 1 -- 172.21.15.72:0/881000868 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f17a00d1f60 con 0x7f17a00d4a40 2024-04-15T15:15:58.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.648+0000 7f1787fff640 1 -- 172.21.15.72:0/881000868 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v28) ==== 181+0+0 (secure 0 0 0) 0x7f1794060080 con 0x7f17a00d4a40 2024-04-15T15:15:58.650 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_metadata' removed 2024-04-15T15:15:58.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.651+0000 7f17a5841640 1 -- 172.21.15.72:0/881000868 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f178c06c8f0 msgr2=0x7f178c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:58.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.651+0000 7f17a5841640 1 --2- 172.21.15.72:0/881000868 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f178c06c8f0 0x7f178c06edb0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f17900070b0 tx=0x7f1790007040 comp rx=0 tx=0).stop 2024-04-15T15:15:58.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.651+0000 7f17a5841640 1 -- 172.21.15.72:0/881000868 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17a00d4a40 msgr2=0x7f17a015ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:58.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.651+0000 7f17a5841640 1 --2- 172.21.15.72:0/881000868 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17a00d4a40 0x7f17a015ae70 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f179400c430 tx=0x7f179400c8f0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.651+0000 7f17a5841640 1 -- 172.21.15.72:0/881000868 shutdown_connections 2024-04-15T15:15:58.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.651+0000 7f17a5841640 1 --2- 172.21.15.72:0/881000868 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f178c06c8f0 0x7f178c06edb0 secure :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f17900070b0 tx=0x7f1790007040 comp rx=0 tx=0).stop 2024-04-15T15:15:58.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.651+0000 7f17a5841640 1 --2- 172.21.15.72:0/881000868 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f17a015e9d0 0x7f17a01f47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.651+0000 7f17a5841640 1 --2- 172.21.15.72:0/881000868 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17a00d4a40 0x7f17a015ae70 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.651+0000 7f17a5841640 1 --2- 172.21.15.72:0/881000868 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f17a00d1760 0x7f17a00cc0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.651+0000 7f17a5841640 1 -- 172.21.15.72:0/881000868 >> 172.21.15.72:0/881000868 conn(0x7f17a009d970 msgr2=0x7f17a00b32e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:58.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.651+0000 7f17a5841640 1 -- 172.21.15.72:0/881000868 shutdown_connections 2024-04-15T15:15:58.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.651+0000 7f17a5841640 1 -- 172.21.15.72:0/881000868 wait complete. 2024-04-15T15:15:58.667 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-15T15:15:58.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.774+0000 7f6cd9e67640 1 Processor -- start 2024-04-15T15:15:58.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.774+0000 7f6cd9e67640 1 -- start start 2024-04-15T15:15:58.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.775+0000 7f6cd9e67640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6cd415e9d0 0x7f6cd40b3a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:58.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.775+0000 7f6cd9e67640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6cd40d1760 0x7f6cd40d4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:58.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.775+0000 7f6cd9e67640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f6cd40d4870 0x7f6cd40b6040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:58.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.775+0000 7f6cd9e67640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f6cd40d22c0 con 0x7f6cd40d4870 2024-04-15T15:15:58.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.775+0000 7f6cd9e67640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f6cd40d0e80 con 0x7f6cd40d1760 2024-04-15T15:15:58.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.775+0000 7f6cd9e67640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f6cd40d4d00 con 0x7f6cd415e9d0 2024-04-15T15:15:58.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.775+0000 7f6cd3fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f6cd40d4870 0x7f6cd40b6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:58.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.775+0000 7f6cd3fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f6cd40d4870 0x7f6cd40b6040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48948/0 (socket says 172.21.15.72:48948) 2024-04-15T15:15:58.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.775+0000 7f6cd3fff640 1 -- 172.21.15.72:0/2895434710 learned_addr learned my addr 172.21.15.72:0/2895434710 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:58.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.775+0000 7f6ccbfff640 1 --2- 172.21.15.72:0/2895434710 >> v2:172.21.15.175:3300/0 conn(0x7f6cd40d1760 0x7f6cd40d4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:58.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.775+0000 7f6cd37fe640 1 --2- 172.21.15.72:0/2895434710 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6cd415e9d0 0x7f6cd40b3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:58.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.775+0000 7f6cd3fff640 1 -- 172.21.15.72:0/2895434710 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6cd415e9d0 msgr2=0x7f6cd40b3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:58.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.775+0000 7f6cd3fff640 1 --2- 172.21.15.72:0/2895434710 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6cd415e9d0 0x7f6cd40b3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.775+0000 7f6cd3fff640 1 -- 172.21.15.72:0/2895434710 >> v2:172.21.15.175:3300/0 conn(0x7f6cd40d1760 msgr2=0x7f6cd40d4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:58.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.775+0000 7f6cd3fff640 1 --2- 172.21.15.72:0/2895434710 >> v2:172.21.15.175:3300/0 conn(0x7f6cd40d1760 0x7f6cd40d4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.775+0000 7f6cd3fff640 1 -- 172.21.15.72:0/2895434710 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cd40d50c0 con 0x7f6cd40d4870 2024-04-15T15:15:58.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.775+0000 7f6cd37fe640 1 --2- 172.21.15.72:0/2895434710 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6cd415e9d0 0x7f6cd40b3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:58.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.775+0000 7f6ccbfff640 1 --2- 172.21.15.72:0/2895434710 >> v2:172.21.15.175:3300/0 conn(0x7f6cd40d1760 0x7f6cd40d4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:58.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.776+0000 7f6cd3fff640 1 --2- 172.21.15.72:0/2895434710 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6cd40d4870 0x7f6cd40b6040 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f6cd408dcd0 tx=0x7f6ccc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ac6a1ac4228199c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:58.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.776+0000 7f6cd2ffd640 1 -- 172.21.15.72:0/2895434710 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc02ca70 con 0x7f6cd40d4870 2024-04-15T15:15:58.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.776+0000 7f6cd2ffd640 1 -- 172.21.15.72:0/2895434710 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ccc02cc10 con 0x7f6cd40d4870 2024-04-15T15:15:58.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.776+0000 7f6cd2ffd640 1 -- 172.21.15.72:0/2895434710 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc0343f0 con 0x7f6cd40d4870 2024-04-15T15:15:58.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.777+0000 7f6cd9e67640 1 -- 172.21.15.72:0/2895434710 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6cd40d4870 msgr2=0x7f6cd40b6040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:58.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.777+0000 7f6cd9e67640 1 --2- 172.21.15.72:0/2895434710 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6cd40d4870 0x7f6cd40b6040 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f6cd408dcd0 tx=0x7f6ccc02bed0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.777+0000 7f6cd9e67640 1 -- 172.21.15.72:0/2895434710 shutdown_connections 2024-04-15T15:15:58.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.777+0000 7f6cd9e67640 1 --2- 172.21.15.72:0/2895434710 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6cd40d4870 0x7f6cd40b6040 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.777+0000 7f6cd9e67640 1 --2- 172.21.15.72:0/2895434710 >> v2:172.21.15.175:3300/0 conn(0x7f6cd40d1760 0x7f6cd40d4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.777+0000 7f6cd9e67640 1 --2- 172.21.15.72:0/2895434710 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6cd415e9d0 0x7f6cd40b3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.777+0000 7f6cd9e67640 1 -- 172.21.15.72:0/2895434710 >> 172.21.15.72:0/2895434710 conn(0x7f6cd409d970 msgr2=0x7f6cd40b9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:58.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.777+0000 7f6cd9e67640 1 -- 172.21.15.72:0/2895434710 shutdown_connections 2024-04-15T15:15:58.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.777+0000 7f6cd9e67640 1 -- 172.21.15.72:0/2895434710 wait complete. 2024-04-15T15:15:58.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.778+0000 7f6cd9e67640 1 Processor -- start 2024-04-15T15:15:58.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.778+0000 7f6cd9e67640 1 -- start start 2024-04-15T15:15:58.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.778+0000 7f6cd9e67640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6cd40d1760 0x7f6cd4160b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:58.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.778+0000 7f6cd9e67640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6cd40d4870 0x7f6cd40c80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:58.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.778+0000 7f6cd9e67640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6cd415e9d0 0x7f6cd40c9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:58.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.778+0000 7f6cd9e67640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6cd40d0ba0 con 0x7f6cd415e9d0 2024-04-15T15:15:58.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.778+0000 7f6cd9e67640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f6cd40d08c0 con 0x7f6cd40d1760 2024-04-15T15:15:58.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.778+0000 7f6cd9e67640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f6cd415edb0 con 0x7f6cd40d4870 2024-04-15T15:15:58.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.779+0000 7f6cd3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6cd415e9d0 0x7f6cd40c9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:58.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.779+0000 7f6cd3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6cd415e9d0 0x7f6cd40c9950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48954/0 (socket says 172.21.15.72:48954) 2024-04-15T15:15:58.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.779+0000 7f6cd3fff640 1 -- 172.21.15.72:0/1979574714 learned_addr learned my addr 172.21.15.72:0/1979574714 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:58.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.779+0000 7f6cd3fff640 1 -- 172.21.15.72:0/1979574714 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6cd40d4870 msgr2=0x7f6cd40c80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:58.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.779+0000 7f6cd3fff640 1 --2- 172.21.15.72:0/1979574714 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6cd40d4870 0x7f6cd40c80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.779+0000 7f6cd3fff640 1 -- 172.21.15.72:0/1979574714 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6cd40d1760 msgr2=0x7f6cd4160b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:58.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.779+0000 7f6cd37fe640 1 --2- 172.21.15.72:0/1979574714 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6cd40d1760 0x7f6cd4160b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:58.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.779+0000 7f6cd3fff640 1 --2- 172.21.15.72:0/1979574714 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6cd40d1760 0x7f6cd4160b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.779+0000 7f6cd3fff640 1 -- 172.21.15.72:0/1979574714 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cd40c8630 con 0x7f6cd415e9d0 2024-04-15T15:15:58.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.779+0000 7f6cd3fff640 1 --2- 172.21.15.72:0/1979574714 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6cd415e9d0 0x7f6cd40c9950 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc036040 tx=0x7f6ccc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:58.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.779+0000 7f6cd0ff9640 1 -- 172.21.15.72:0/1979574714 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc043020 con 0x7f6cd415e9d0 2024-04-15T15:15:58.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.779+0000 7f6cd0ff9640 1 -- 172.21.15.72:0/1979574714 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ccc03e450 con 0x7f6cd415e9d0 2024-04-15T15:15:58.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.779+0000 7f6cd9e67640 1 -- 172.21.15.72:0/1979574714 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cd40c88b0 con 0x7f6cd415e9d0 2024-04-15T15:15:58.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.779+0000 7f6cd0ff9640 1 -- 172.21.15.72:0/1979574714 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc03e820 con 0x7f6cd415e9d0 2024-04-15T15:15:58.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.779+0000 7f6cd9e67640 1 -- 172.21.15.72:0/1979574714 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cd40c8e70 con 0x7f6cd415e9d0 2024-04-15T15:15:58.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.780+0000 7f6cd9e67640 1 -- 172.21.15.72:0/1979574714 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cd40689e0 con 0x7f6cd415e9d0 2024-04-15T15:15:58.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.781+0000 7f6cd0ff9640 1 -- 172.21.15.72:0/1979574714 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6ccc046020 con 0x7f6cd415e9d0 2024-04-15T15:15:58.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.781+0000 7f6cd0ff9640 1 --2- 172.21.15.72:0/1979574714 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6cb406c8f0 0x7f6cb406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:58.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.781+0000 7f6cd0ff9640 1 -- 172.21.15.72:0/1979574714 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6383+0+0 (secure 0 0 0) 0x7f6ccc0ae780 con 0x7f6cd415e9d0 2024-04-15T15:15:58.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.781+0000 7f6cd37fe640 1 --2- 172.21.15.72:0/1979574714 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6cb406c8f0 0x7f6cb406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:58.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.782+0000 7f6cd37fe640 1 --2- 172.21.15.72:0/1979574714 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6cb406c8f0 0x7f6cb406edb0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6cc4008850 tx=0x7f6cc4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:58.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.785+0000 7f6cd0ff9640 1 -- 172.21.15.72:0/1979574714 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6ccc04d030 con 0x7f6cd415e9d0 2024-04-15T15:15:58.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:58.959+0000 7f6cd9e67640 1 -- 172.21.15.72:0/1979574714 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f6cd4162250 con 0x7f6cd415e9d0 2024-04-15T15:15:59.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.655+0000 7f6cd0ff9640 1 -- 172.21.15.72:0/1979574714 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v29) ==== 169+0+0 (secure 0 0 0) 0x7f6cd40c8630 con 0x7f6cd415e9d0 2024-04-15T15:15:59.657 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_data' removed 2024-04-15T15:15:59.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.658+0000 7f6cd9e67640 1 -- 172.21.15.72:0/1979574714 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6cb406c8f0 msgr2=0x7f6cb406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.658+0000 7f6cd9e67640 1 --2- 172.21.15.72:0/1979574714 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6cb406c8f0 0x7f6cb406edb0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f6cc4008850 tx=0x7f6cc4008040 comp rx=0 tx=0).stop 2024-04-15T15:15:59.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.658+0000 7f6cd9e67640 1 -- 172.21.15.72:0/1979574714 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6cd415e9d0 msgr2=0x7f6cd40c9950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.658+0000 7f6cd9e67640 1 --2- 172.21.15.72:0/1979574714 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6cd415e9d0 0x7f6cd40c9950 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc036040 tx=0x7f6ccc004a40 comp rx=0 tx=0).stop 2024-04-15T15:15:59.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.659+0000 7f6cd9e67640 1 -- 172.21.15.72:0/1979574714 shutdown_connections 2024-04-15T15:15:59.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.659+0000 7f6cd9e67640 1 --2- 172.21.15.72:0/1979574714 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6cb406c8f0 0x7f6cb406edb0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.659+0000 7f6cd9e67640 1 --2- 172.21.15.72:0/1979574714 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6cd415e9d0 0x7f6cd40c9950 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.659+0000 7f6cd9e67640 1 --2- 172.21.15.72:0/1979574714 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6cd40d4870 0x7f6cd40c80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.659+0000 7f6cd9e67640 1 --2- 172.21.15.72:0/1979574714 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6cd40d1760 0x7f6cd4160b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.659+0000 7f6cd9e67640 1 -- 172.21.15.72:0/1979574714 >> 172.21.15.72:0/1979574714 conn(0x7f6cd409d970 msgr2=0x7f6cd40b9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:59.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.659+0000 7f6cd9e67640 1 -- 172.21.15.72:0/1979574714 shutdown_connections 2024-04-15T15:15:59.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.659+0000 7f6cd9e67640 1 -- 172.21.15.72:0/1979574714 wait complete. 2024-04-15T15:15:59.674 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-15T15:15:59.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.782+0000 7f6ebd7bd640 1 Processor -- start 2024-04-15T15:15:59.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.782+0000 7f6ebd7bd640 1 -- start start 2024-04-15T15:15:59.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.783+0000 7f6ebd7bd640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6eb815c280 0x7f6eb8071740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.783+0000 7f6ebd7bd640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6eb80c8620 0x7f6eb8071c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.783+0000 7f6ebd7bd640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6eb80c7860 con 0x7f6eb806d4c0 2024-04-15T15:15:59.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.783+0000 7f6ebd7bd640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f6eb80d10c0 con 0x7f6eb815c280 2024-04-15T15:15:59.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.783+0000 7f6ebd7bd640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f6eb815fd70 con 0x7f6eb80c8620 2024-04-15T15:15:59.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.783+0000 7f6eb77fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f6eb806d4c0 0x7f6eb8162370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:47856/0 (socket says 172.21.15.72:47856) 2024-04-15T15:15:59.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.783+0000 7f6eb77fe640 1 -- 172.21.15.72:0/1366381838 learned_addr learned my addr 172.21.15.72:0/1366381838 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:15:59.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.783+0000 7f6eb5ffb640 1 -- 172.21.15.72:0/1366381838 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2162455406 0 0) 0x7f6eb80c7860 con 0x7f6eb806d4c0 2024-04-15T15:15:59.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.783+0000 7f6eb5ffb640 1 -- 172.21.15.72:0/1366381838 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ea4003520 con 0x7f6eb806d4c0 2024-04-15T15:15:59.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.783+0000 7f6eb6ffd640 1 --2- 172.21.15.72:0/1366381838 >> v2:172.21.15.175:3300/0 conn(0x7f6eb815c280 0x7f6eb8071740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.783+0000 7f6eb67fc640 1 --2- 172.21.15.72:0/1366381838 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6eb80c8620 0x7f6eb8071c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.783+0000 7f6eb5ffb640 1 -- 172.21.15.72:0/1366381838 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2041101726 0 0) 0x7f6ea4003520 con 0x7f6eb806d4c0 2024-04-15T15:15:59.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.784+0000 7f6eb5ffb640 1 -- 172.21.15.72:0/1366381838 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6eb80c8620 msgr2=0x7f6eb8071c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:59.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.784+0000 7f6eb5ffb640 1 --2- 172.21.15.72:0/1366381838 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6eb80c8620 0x7f6eb8071c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.784+0000 7f6eb5ffb640 1 -- 172.21.15.72:0/1366381838 >> v2:172.21.15.175:3300/0 conn(0x7f6eb815c280 msgr2=0x7f6eb8071740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:59.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.784+0000 7f6eb5ffb640 1 --2- 172.21.15.72:0/1366381838 >> v2:172.21.15.175:3300/0 conn(0x7f6eb815c280 0x7f6eb8071740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.784+0000 7f6eb5ffb640 1 -- 172.21.15.72:0/1366381838 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eb80c7b40 con 0x7f6eb806d4c0 2024-04-15T15:15:59.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.784+0000 7f6eb5ffb640 1 -- 172.21.15.72:0/1366381838 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f6eb0001670 con 0x7f6eb806d4c0 2024-04-15T15:15:59.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.784+0000 7f6eb5ffb640 1 -- 172.21.15.72:0/1366381838 >> v1:172.21.15.72:6789/0 conn(0x7f6eb806d4c0 legacy=0x7f6eb8162370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.784+0000 7f6eb5ffb640 1 --2- 172.21.15.72:0/1366381838 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6eb80c8620 0x7f6ea40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.784+0000 7f6eb5ffb640 1 --2- 172.21.15.72:0/1366381838 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ea4003f30 0x7f6ea4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.784+0000 7f6eb5ffb640 1 --2- 172.21.15.72:0/1366381838 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ea4004d40 0x7f6ea40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.784+0000 7f6eb5ffb640 1 -- 172.21.15.72:0/1366381838 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb80d10c0 con 0x7f6ea4003f30 2024-04-15T15:15:59.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.784+0000 7f6eb5ffb640 1 -- 172.21.15.72:0/1366381838 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb815fd70 con 0x7f6ea4004d40 2024-04-15T15:15:59.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.784+0000 7f6eb5ffb640 1 -- 172.21.15.72:0/1366381838 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f6ea400b9c0 con 0x7f6eb80c8620 2024-04-15T15:15:59.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.784+0000 7f6eb67fc640 1 --2- 172.21.15.72:0/1366381838 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ea4003f30 0x7f6ea4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.784+0000 7f6eb6ffd640 1 --2- 172.21.15.72:0/1366381838 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6eb80c8620 0x7f6ea40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.784+0000 7f6eb77fe640 1 --2- 172.21.15.72:0/1366381838 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ea4004d40 0x7f6ea40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.784+0000 7f6eb67fc640 1 -- 172.21.15.72:0/1366381838 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6eb80c8620 msgr2=0x7f6ea40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:59.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.784+0000 7f6eb67fc640 1 --2- 172.21.15.72:0/1366381838 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6eb80c8620 0x7f6ea40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.784+0000 7f6eb67fc640 1 -- 172.21.15.72:0/1366381838 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ea4004d40 msgr2=0x7f6ea40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:59.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.784+0000 7f6eb67fc640 1 --2- 172.21.15.72:0/1366381838 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ea4004d40 0x7f6ea40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.784+0000 7f6eb67fc640 1 -- 172.21.15.72:0/1366381838 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ea4003520 con 0x7f6ea4003f30 2024-04-15T15:15:59.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.784+0000 7f6eb77fe640 1 --2- 172.21.15.72:0/1366381838 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ea4004d40 0x7f6ea40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:59.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.784+0000 7f6eb6ffd640 1 --2- 172.21.15.72:0/1366381838 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6eb80c8620 0x7f6ea40039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:59.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.785+0000 7f6eb67fc640 1 --2- 172.21.15.72:0/1366381838 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ea4003f30 0x7f6ea4004800 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0009ac0 tx=0x7f6ea4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12a306b7cf906df5 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:59.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.785+0000 7f6eb5ffb640 1 -- 172.21.15.72:0/1366381838 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea0039070 con 0x7f6ea4003f30 2024-04-15T15:15:59.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.785+0000 7f6eb5ffb640 1 -- 172.21.15.72:0/1366381838 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ea002c6b0 con 0x7f6ea4003f30 2024-04-15T15:15:59.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.785+0000 7f6eb5ffb640 1 -- 172.21.15.72:0/1366381838 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea0009690 con 0x7f6ea4003f30 2024-04-15T15:15:59.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.785+0000 7f6ebd7bd640 1 -- 172.21.15.72:0/1366381838 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ea4003f30 msgr2=0x7f6ea4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.785+0000 7f6ebd7bd640 1 --2- 172.21.15.72:0/1366381838 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ea4003f30 0x7f6ea4004800 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0009ac0 tx=0x7f6ea4005830 comp rx=0 tx=0).stop 2024-04-15T15:15:59.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.785+0000 7f6eb77fe640 1 -- 172.21.15.72:0/1366381838 reap_dead start 2024-04-15T15:15:59.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.785+0000 7f6ebd7bd640 1 -- 172.21.15.72:0/1366381838 shutdown_connections 2024-04-15T15:15:59.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.785+0000 7f6ebd7bd640 1 -- 172.21.15.72:0/1366381838 >> 172.21.15.72:0/1366381838 conn(0x7f6eb80ceeb0 msgr2=0x7f6eb80cf2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:59.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.785+0000 7f6ebd7bd640 1 -- 172.21.15.72:0/1366381838 shutdown_connections 2024-04-15T15:15:59.791 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.786+0000 7f6ebd7bd640 1 -- 172.21.15.72:0/1366381838 wait complete. 2024-04-15T15:15:59.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.787+0000 7f6ebd7bd640 1 Processor -- start 2024-04-15T15:15:59.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.787+0000 7f6ebd7bd640 1 -- start start 2024-04-15T15:15:59.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.787+0000 7f6ebd7bd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ea4003f30 0x7f6eb8149fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.788+0000 7f6ebd7bd640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6eb80ca120 0x7f6eb80caf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.788+0000 7f6ebd7bd640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6eb815d9c0 0x7f6eb815de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.788+0000 7f6ebd7bd640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb806e0b0 con 0x7f6ea4003f30 2024-04-15T15:15:59.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.788+0000 7f6ebd7bd640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f6eb80c8190 con 0x7f6eb815d9c0 2024-04-15T15:15:59.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.788+0000 7f6ebd7bd640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f6eb815c660 con 0x7f6eb80ca120 2024-04-15T15:15:59.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.788+0000 7f6eb6ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ea4003f30 0x7f6eb8149fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.788+0000 7f6eb6ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ea4003f30 0x7f6eb8149fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48980/0 (socket says 172.21.15.72:48980) 2024-04-15T15:15:59.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.788+0000 7f6eb6ffd640 1 -- 172.21.15.72:0/3552355516 learned_addr learned my addr 172.21.15.72:0/3552355516 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:15:59.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.788+0000 7f6eb67fc640 1 --2- 172.21.15.72:0/3552355516 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6eb80ca120 0x7f6eb80caf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.788+0000 7f6eb77fe640 1 --2- 172.21.15.72:0/3552355516 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6eb815d9c0 0x7f6eb815de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.788+0000 7f6eb6ffd640 1 -- 172.21.15.72:0/3552355516 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6eb80ca120 msgr2=0x7f6eb80caf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.788+0000 7f6eb6ffd640 1 --2- 172.21.15.72:0/3552355516 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6eb80ca120 0x7f6eb80caf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.788+0000 7f6eb6ffd640 1 -- 172.21.15.72:0/3552355516 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6eb815d9c0 msgr2=0x7f6eb815de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.788+0000 7f6eb6ffd640 1 --2- 172.21.15.72:0/3552355516 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6eb815d9c0 0x7f6eb815de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.788+0000 7f6eb6ffd640 1 -- 172.21.15.72:0/3552355516 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6eb80cb810 con 0x7f6ea4003f30 2024-04-15T15:15:59.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.788+0000 7f6eb6ffd640 1 --2- 172.21.15.72:0/3552355516 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ea4003f30 0x7f6eb8149fd0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f6ea8010220 tx=0x7f6ea80106e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:59.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.788+0000 7f6eaf7fe640 1 -- 172.21.15.72:0/3552355516 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea801a030 con 0x7f6ea4003f30 2024-04-15T15:15:59.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.788+0000 7f6eaf7fe640 1 -- 172.21.15.72:0/3552355516 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ea8003990 con 0x7f6ea4003f30 2024-04-15T15:15:59.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.788+0000 7f6ebd7bd640 1 -- 172.21.15.72:0/3552355516 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6eb80cbaf0 con 0x7f6ea4003f30 2024-04-15T15:15:59.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.788+0000 7f6eaf7fe640 1 -- 172.21.15.72:0/3552355516 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea8003d60 con 0x7f6ea4003f30 2024-04-15T15:15:59.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.788+0000 7f6ebd7bd640 1 -- 172.21.15.72:0/3552355516 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6eb80cc0b0 con 0x7f6ea4003f30 2024-04-15T15:15:59.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.789+0000 7f6ebd7bd640 1 -- 172.21.15.72:0/3552355516 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6eb814a6f0 con 0x7f6ea4003f30 2024-04-15T15:15:59.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.790+0000 7f6eaf7fe640 1 -- 172.21.15.72:0/3552355516 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6ea801d070 con 0x7f6ea4003f30 2024-04-15T15:15:59.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.790+0000 7f6eaf7fe640 1 --2- 172.21.15.72:0/3552355516 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6e9806c8f0 0x7f6e9806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.790+0000 7f6eaf7fe640 1 -- 172.21.15.72:0/3552355516 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 5853+0+0 (secure 0 0 0) 0x7f6ea808cac0 con 0x7f6ea4003f30 2024-04-15T15:15:59.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.790+0000 7f6eb67fc640 1 --2- 172.21.15.72:0/3552355516 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6e9806c8f0 0x7f6e9806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.791+0000 7f6eb67fc640 1 --2- 172.21.15.72:0/3552355516 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6e9806c8f0 0x7f6e9806edb0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0036000 tx=0x7f6ea0030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:59.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.794+0000 7f6eaf7fe640 1 -- 172.21.15.72:0/3552355516 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6ea801de40 con 0x7f6ea4003f30 2024-04-15T15:15:59.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:15:59.969+0000 7f6ebd7bd640 1 -- 172.21.15.72:0/3552355516 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f6eb80689e0 con 0x7f6ea4003f30 2024-04-15T15:16:00.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.656+0000 7f6eaf7fe640 1 -- 172.21.15.72:0/3552355516 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v30) ==== 178+0+0 (secure 0 0 0) 0x7f6ea805aef0 con 0x7f6ea4003f30 2024-04-15T15:16:00.658 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_data_ec' removed 2024-04-15T15:16:00.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.660+0000 7f6ebd7bd640 1 -- 172.21.15.72:0/3552355516 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6e9806c8f0 msgr2=0x7f6e9806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:00.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.660+0000 7f6ebd7bd640 1 --2- 172.21.15.72:0/3552355516 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6e9806c8f0 0x7f6e9806edb0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0036000 tx=0x7f6ea0030000 comp rx=0 tx=0).stop 2024-04-15T15:16:00.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.660+0000 7f6ebd7bd640 1 -- 172.21.15.72:0/3552355516 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ea4003f30 msgr2=0x7f6eb8149fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:00.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.660+0000 7f6ebd7bd640 1 --2- 172.21.15.72:0/3552355516 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ea4003f30 0x7f6eb8149fd0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f6ea8010220 tx=0x7f6ea80106e0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.660+0000 7f6ebd7bd640 1 -- 172.21.15.72:0/3552355516 shutdown_connections 2024-04-15T15:16:00.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.660+0000 7f6ebd7bd640 1 --2- 172.21.15.72:0/3552355516 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6e9806c8f0 0x7f6e9806edb0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.660+0000 7f6ebd7bd640 1 --2- 172.21.15.72:0/3552355516 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6eb815d9c0 0x7f6eb815de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.660+0000 7f6ebd7bd640 1 --2- 172.21.15.72:0/3552355516 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6eb80ca120 0x7f6eb80caf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.660+0000 7f6ebd7bd640 1 --2- 172.21.15.72:0/3552355516 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ea4003f30 0x7f6eb8149fd0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.660+0000 7f6ebd7bd640 1 -- 172.21.15.72:0/3552355516 >> 172.21.15.72:0/3552355516 conn(0x7f6eb80ceeb0 msgr2=0x7f6eb8160130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:00.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.660+0000 7f6ebd7bd640 1 -- 172.21.15.72:0/3552355516 shutdown_connections 2024-04-15T15:16:00.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.660+0000 7f6ebd7bd640 1 -- 172.21.15.72:0/3552355516 wait complete. 2024-04-15T15:16:00.675 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-15T15:16:00.675 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-15T15:16:00.675 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-15T15:16:00.675 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:16:00.676 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-15T15:16:00.676 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-15T15:16:00.676 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-15T15:16:00.676 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:16:00.676 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-15T15:16:00.676 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-15T15:16:00.677 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-15T15:16:00.677 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:16:00.677 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-15T15:16:00.677 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-15T15:16:00.677 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-15T15:16:00.677 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:16:00.685 INFO:tasks.ceph.mds.a:Stopped 2024-04-15T15:16:00.685 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-15T15:16:00.686 INFO:tasks.ceph.mds.c:Stopped 2024-04-15T15:16:00.686 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-15T15:16:00.688 INFO:tasks.ceph.mds.d:Stopped 2024-04-15T15:16:00.688 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-15T15:16:00.689 INFO:tasks.ceph.mds.b:Stopped 2024-04-15T15:16:00.689 DEBUG:teuthology.orchestra.run.smithi175:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-15T15:16:00.728 INFO:tasks.ceph.mds.a:Started 2024-04-15T15:16:00.729 DEBUG:teuthology.parallel:result is None 2024-04-15T15:16:00.731 INFO:tasks.ceph.mds.c:Started 2024-04-15T15:16:00.731 DEBUG:teuthology.parallel:result is None 2024-04-15T15:16:00.732 INFO:tasks.ceph.mds.d:Started 2024-04-15T15:16:00.732 DEBUG:teuthology.parallel:result is None 2024-04-15T15:16:00.733 INFO:tasks.ceph.mds.b:Started 2024-04-15T15:16:00.733 DEBUG:teuthology.parallel:result is None 2024-04-15T15:16:00.734 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-15T15:16:00.809 INFO:tasks.ceph.mds.a.smithi072.stdout:starting mds.a at 2024-04-15T15:16:00.809 INFO:tasks.ceph.mds.c.smithi072.stdout:starting mds.c at 2024-04-15T15:16:00.815 INFO:tasks.ceph.mds.d.smithi175.stdout:starting mds.d at 2024-04-15T15:16:00.816 INFO:tasks.ceph.mds.b.smithi175.stdout:starting mds.b at 2024-04-15T15:16:00.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.840+0000 7f58c0ed9640 1 Processor -- start 2024-04-15T15:16:00.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.840+0000 7f58c0ed9640 1 -- start start 2024-04-15T15:16:00.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.841+0000 7f58c0ed9640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f58bc15e9f0 0x7f58bc0d3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:00.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.841+0000 7f58c0ed9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58bc0d2320 con 0x7f58bc06c400 2024-04-15T15:16:00.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.841+0000 7f58c0ed9640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58bc0d25c0 con 0x7f58bc15e610 2024-04-15T15:16:00.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.841+0000 7f58c0ed9640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f58bc0d31d0 con 0x7f58bc15e9f0 2024-04-15T15:16:00.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.841+0000 7f58bad9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f58bc06c400 0x7f58bc0b7940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:47900/0 (socket says 172.21.15.72:47900) 2024-04-15T15:16:00.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.841+0000 7f58bad9d640 1 -- 172.21.15.72:0/52722578 learned_addr learned my addr 172.21.15.72:0/52722578 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:00.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.842+0000 7f58b9d9b640 1 --2- 172.21.15.72:0/52722578 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f58bc15e9f0 0x7f58bc0d3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:00.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.842+0000 7f58b959a640 1 -- 172.21.15.72:0/52722578 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3094796755 0 0) 0x7f58bc0d2320 con 0x7f58bc06c400 2024-04-15T15:16:00.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.842+0000 7f58b959a640 1 -- 172.21.15.72:0/52722578 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58a4003520 con 0x7f58bc06c400 2024-04-15T15:16:00.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.842+0000 7f58b959a640 1 -- 172.21.15.72:0/52722578 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3402743710 0 0) 0x7f58bc0d25c0 con 0x7f58bc15e610 2024-04-15T15:16:00.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.842+0000 7f58b959a640 1 -- 172.21.15.72:0/52722578 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58bc0d2320 con 0x7f58bc15e610 2024-04-15T15:16:00.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.842+0000 7f58b9d9b640 1 -- 172.21.15.72:0/52722578 >> v1:172.21.15.175:6789/0 conn(0x7f58bc15e610 legacy=0x7f58bc0d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:00.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.842+0000 7f58b9d9b640 1 -- 172.21.15.72:0/52722578 >> v1:172.21.15.72:6789/0 conn(0x7f58bc06c400 legacy=0x7f58bc0b7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:00.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.842+0000 7f58b9d9b640 1 -- 172.21.15.72:0/52722578 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58bc0d28a0 con 0x7f58bc15e9f0 2024-04-15T15:16:00.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.843+0000 7f58b9d9b640 1 --2- 172.21.15.72:0/52722578 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f58bc15e9f0 0x7f58bc0d3660 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f58bc0d3350 tx=0x7f58b002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3e47e98ccd85d7a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:00.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.843+0000 7f58b959a640 1 -- 172.21.15.72:0/52722578 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b0035020 con 0x7f58bc15e9f0 2024-04-15T15:16:00.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.843+0000 7f58b959a640 1 -- 172.21.15.72:0/52722578 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58b0031890 con 0x7f58bc15e9f0 2024-04-15T15:16:00.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.843+0000 7f58b959a640 1 -- 172.21.15.72:0/52722578 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b0031c40 con 0x7f58bc15e9f0 2024-04-15T15:16:00.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.843+0000 7f58c0ed9640 1 -- 172.21.15.72:0/52722578 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f58bc15e9f0 msgr2=0x7f58bc0d3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:00.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.843+0000 7f58c0ed9640 1 --2- 172.21.15.72:0/52722578 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f58bc15e9f0 0x7f58bc0d3660 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f58bc0d3350 tx=0x7f58b002b930 comp rx=0 tx=0).stop 2024-04-15T15:16:00.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.844+0000 7f58c0ed9640 1 -- 172.21.15.72:0/52722578 shutdown_connections 2024-04-15T15:16:00.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.844+0000 7f58c0ed9640 1 --2- 172.21.15.72:0/52722578 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f58bc15e9f0 0x7f58bc0d3660 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.844+0000 7f58c0ed9640 1 -- 172.21.15.72:0/52722578 >> 172.21.15.72:0/52722578 conn(0x7f58bc0d4750 msgr2=0x7f58bc0d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:00.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.844+0000 7f58c0ed9640 1 -- 172.21.15.72:0/52722578 shutdown_connections 2024-04-15T15:16:00.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.844+0000 7f58c0ed9640 1 -- 172.21.15.72:0/52722578 wait complete. 2024-04-15T15:16:00.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.845+0000 7f58c0ed9640 1 Processor -- start 2024-04-15T15:16:00.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.845+0000 7f58c0ed9640 1 -- start start 2024-04-15T15:16:00.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.846+0000 7f58c0ed9640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f58bc06c400 0x7f58bc1f3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:00.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.846+0000 7f58c0ed9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f58bc15e610 0x7f58bc160b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:00.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.846+0000 7f58c0ed9640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f58bc15e9f0 0x7f58bc1f6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:00.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.846+0000 7f58c0ed9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f58bc06d130 con 0x7f58bc15e610 2024-04-15T15:16:00.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.846+0000 7f58c0ed9640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f58bc0d2e60 con 0x7f58bc15e9f0 2024-04-15T15:16:00.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.846+0000 7f58c0ed9640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f58bc0d3ba0 con 0x7f58bc06c400 2024-04-15T15:16:00.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.846+0000 7f58b9d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f58bc15e610 0x7f58bc160b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:00.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.846+0000 7f58b9d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f58bc15e610 0x7f58bc160b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49026/0 (socket says 172.21.15.72:49026) 2024-04-15T15:16:00.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.846+0000 7f58b9d9b640 1 -- 172.21.15.72:0/3728450100 learned_addr learned my addr 172.21.15.72:0/3728450100 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:00.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.846+0000 7f58bad9d640 1 --2- 172.21.15.72:0/3728450100 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f58bc15e9f0 0x7f58bc1f6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:00.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.846+0000 7f58ba59c640 1 --2- 172.21.15.72:0/3728450100 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f58bc06c400 0x7f58bc1f3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:00.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.846+0000 7f58b9d9b640 1 -- 172.21.15.72:0/3728450100 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f58bc06c400 msgr2=0x7f58bc1f3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:00.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.846+0000 7f58b9d9b640 1 --2- 172.21.15.72:0/3728450100 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f58bc06c400 0x7f58bc1f3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.846+0000 7f58b9d9b640 1 -- 172.21.15.72:0/3728450100 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f58bc15e9f0 msgr2=0x7f58bc1f6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:00.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.846+0000 7f58b9d9b640 1 --2- 172.21.15.72:0/3728450100 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f58bc15e9f0 0x7f58bc1f6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.846+0000 7f58b9d9b640 1 -- 172.21.15.72:0/3728450100 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58bc161080 con 0x7f58bc15e610 2024-04-15T15:16:00.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.846+0000 7f58bad9d640 1 --2- 172.21.15.72:0/3728450100 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f58bc15e9f0 0x7f58bc1f6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:00.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.846+0000 7f58b9d9b640 1 --2- 172.21.15.72:0/3728450100 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f58bc15e610 0x7f58bc160b40 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f58b002b970 tx=0x7f58b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:00.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.846+0000 7f58aaffd640 1 -- 172.21.15.72:0/3728450100 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b0035050 con 0x7f58bc15e610 2024-04-15T15:16:00.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.846+0000 7f58aaffd640 1 -- 172.21.15.72:0/3728450100 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58b0003820 con 0x7f58bc15e610 2024-04-15T15:16:00.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.846+0000 7f58c0ed9640 1 -- 172.21.15.72:0/3728450100 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58bc0cca10 con 0x7f58bc15e610 2024-04-15T15:16:00.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.846+0000 7f58aaffd640 1 -- 172.21.15.72:0/3728450100 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58b0003bf0 con 0x7f58bc15e610 2024-04-15T15:16:00.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.846+0000 7f58c0ed9640 1 -- 172.21.15.72:0/3728450100 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58bc0cccb0 con 0x7f58bc15e610 2024-04-15T15:16:00.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.848+0000 7f58c0ed9640 1 -- 172.21.15.72:0/3728450100 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58bc0cf740 con 0x7f58bc15e610 2024-04-15T15:16:00.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.848+0000 7f58aaffd640 1 -- 172.21.15.72:0/3728450100 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f58b0038020 con 0x7f58bc15e610 2024-04-15T15:16:00.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.848+0000 7f58aaffd640 1 --2- 172.21.15.72:0/3728450100 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f589c06c8f0 0x7f589c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:00.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.848+0000 7f58aaffd640 1 -- 172.21.15.72:0/3728450100 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5335+0+0 (secure 0 0 0) 0x7f58b00adbf0 con 0x7f58bc15e610 2024-04-15T15:16:00.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.849+0000 7f58ba59c640 1 --2- 172.21.15.72:0/3728450100 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f589c06c8f0 0x7f589c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:00.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.849+0000 7f58ba59c640 1 --2- 172.21.15.72:0/3728450100 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f589c06c8f0 0x7f589c06edb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f58bc0b57f0 tx=0x7f58ac009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:00.854 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:00.853+0000 7f58aaffd640 1 -- 172.21.15.72:0/3728450100 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f58bc161080 con 0x7f58bc15e610 2024-04-15T15:16:01.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.038+0000 7f58c0ed9640 1 -- 172.21.15.72:0/3728450100 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f58bc075c30 con 0x7f58bc15e610 2024-04-15T15:16:01.677 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.675+0000 7f58aaffd640 1 -- 172.21.15.72:0/3728450100 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v32) ==== 97+0+0 (secure 0 0 0) 0x7f58b002d040 con 0x7f58bc15e610 2024-04-15T15:16:01.677 INFO:teuthology.orchestra.run.smithi072.stderr: removed all blocklist entries 2024-04-15T15:16:01.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.677+0000 7f58c0ed9640 1 -- 172.21.15.72:0/3728450100 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f589c06c8f0 msgr2=0x7f589c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:01.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.677+0000 7f58c0ed9640 1 --2- 172.21.15.72:0/3728450100 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f589c06c8f0 0x7f589c06edb0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f58bc0b57f0 tx=0x7f58ac009040 comp rx=0 tx=0).stop 2024-04-15T15:16:01.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.677+0000 7f58c0ed9640 1 -- 172.21.15.72:0/3728450100 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f58bc15e610 msgr2=0x7f58bc160b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:01.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.677+0000 7f58c0ed9640 1 --2- 172.21.15.72:0/3728450100 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f58bc15e610 0x7f58bc160b40 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f58b002b970 tx=0x7f58b00046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.677+0000 7f58c0ed9640 1 -- 172.21.15.72:0/3728450100 shutdown_connections 2024-04-15T15:16:01.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.677+0000 7f58c0ed9640 1 --2- 172.21.15.72:0/3728450100 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f589c06c8f0 0x7f589c06edb0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.677+0000 7f58c0ed9640 1 --2- 172.21.15.72:0/3728450100 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f58bc15e9f0 0x7f58bc1f6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.677+0000 7f58c0ed9640 1 --2- 172.21.15.72:0/3728450100 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f58bc15e610 0x7f58bc160b40 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.677+0000 7f58c0ed9640 1 --2- 172.21.15.72:0/3728450100 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f58bc06c400 0x7f58bc1f3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.677+0000 7f58c0ed9640 1 -- 172.21.15.72:0/3728450100 >> 172.21.15.72:0/3728450100 conn(0x7f58bc0d4750 msgr2=0x7f58bc0cfcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:01.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.677+0000 7f58c0ed9640 1 -- 172.21.15.72:0/3728450100 shutdown_connections 2024-04-15T15:16:01.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.677+0000 7f58c0ed9640 1 -- 172.21.15.72:0/3728450100 wait complete. 2024-04-15T15:16:01.695 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-15T15:16:01.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.806+0000 7ff35cd7c640 1 Processor -- start 2024-04-15T15:16:01.808 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.806+0000 7ff35cd7c640 1 -- start start 2024-04-15T15:16:01.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.807+0000 7ff35cd7c640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff358156da0 0x7ff3580ccb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:01.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.807+0000 7ff35cd7c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3580cbad0 con 0x7ff3581569c0 2024-04-15T15:16:01.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.807+0000 7ff35cd7c640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3580cbdb0 con 0x7ff3580cd0b0 2024-04-15T15:16:01.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.807+0000 7ff35cd7c640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7ff3580cc090 con 0x7ff358156da0 2024-04-15T15:16:01.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.807+0000 7ff35659c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ff3581569c0 0x7ff35815d540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:58746/0 (socket says 172.21.15.72:58746) 2024-04-15T15:16:01.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.807+0000 7ff35659c640 1 -- 172.21.15.72:0/2855709324 learned_addr learned my addr 172.21.15.72:0/2855709324 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:01.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.807+0000 7ff355d9b640 1 -- 172.21.15.72:0/2855709324 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3023912898 0 0) 0x7ff3580cbad0 con 0x7ff3581569c0 2024-04-15T15:16:01.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.807+0000 7ff355d9b640 1 -- 172.21.15.72:0/2855709324 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff344003520 con 0x7ff3581569c0 2024-04-15T15:16:01.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.807+0000 7ff34ffff640 1 --2- 172.21.15.72:0/2855709324 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff358156da0 0x7ff3580ccb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:01.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.807+0000 7ff355d9b640 1 -- 172.21.15.72:0/2855709324 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 807494292 0 0) 0x7ff344003520 con 0x7ff3581569c0 2024-04-15T15:16:01.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.807+0000 7ff355d9b640 1 -- 172.21.15.72:0/2855709324 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff358156da0 msgr2=0x7ff3580ccb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:01.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.807+0000 7ff355d9b640 1 --2- 172.21.15.72:0/2855709324 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff358156da0 0x7ff3580ccb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.807+0000 7ff355d9b640 1 -- 172.21.15.72:0/2855709324 >> v1:172.21.15.175:6789/0 conn(0x7ff3580cd0b0 legacy=0x7ff35806c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:01.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.807+0000 7ff355d9b640 1 -- 172.21.15.72:0/2855709324 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3580cc370 con 0x7ff3581569c0 2024-04-15T15:16:01.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.807+0000 7ff355d9b640 1 -- 172.21.15.72:0/2855709324 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7ff348001670 con 0x7ff3581569c0 2024-04-15T15:16:01.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.807+0000 7ff355d9b640 1 -- 172.21.15.72:0/2855709324 >> v1:172.21.15.72:6789/0 conn(0x7ff3581569c0 legacy=0x7ff35815d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:01.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.807+0000 7ff355d9b640 1 --2- 172.21.15.72:0/2855709324 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3440039e0 0x7ff344003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:01.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.807+0000 7ff355d9b640 1 --2- 172.21.15.72:0/2855709324 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff344004800 0x7ff344004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:01.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.807+0000 7ff355d9b640 1 --2- 172.21.15.72:0/2855709324 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff358156da0 0x7ff344005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:01.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.808+0000 7ff355d9b640 1 -- 172.21.15.72:0/2855709324 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ff3580cc090 con 0x7ff3440039e0 2024-04-15T15:16:01.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.808+0000 7ff355d9b640 1 -- 172.21.15.72:0/2855709324 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7ff34400baf0 con 0x7ff344004800 2024-04-15T15:16:01.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.808+0000 7ff355d9b640 1 -- 172.21.15.72:0/2855709324 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7ff34400bcf0 con 0x7ff358156da0 2024-04-15T15:16:01.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.808+0000 7ff35659c640 1 --2- 172.21.15.72:0/2855709324 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3440039e0 0x7ff344003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:01.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.808+0000 7ff34ffff640 1 --2- 172.21.15.72:0/2855709324 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff344004800 0x7ff344004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:01.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.808+0000 7ff356d9d640 1 --2- 172.21.15.72:0/2855709324 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff358156da0 0x7ff344005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:01.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.808+0000 7ff35659c640 1 -- 172.21.15.72:0/2855709324 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff358156da0 msgr2=0x7ff344005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:01.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.808+0000 7ff35659c640 1 --2- 172.21.15.72:0/2855709324 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff358156da0 0x7ff344005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.808+0000 7ff35659c640 1 -- 172.21.15.72:0/2855709324 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff344004800 msgr2=0x7ff344004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:01.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.808+0000 7ff35659c640 1 --2- 172.21.15.72:0/2855709324 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff344004800 0x7ff344004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.808+0000 7ff35659c640 1 -- 172.21.15.72:0/2855709324 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff344003520 con 0x7ff3440039e0 2024-04-15T15:16:01.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.808+0000 7ff34ffff640 1 --2- 172.21.15.72:0/2855709324 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff344004800 0x7ff344004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:01.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.809+0000 7ff35659c640 1 --2- 172.21.15.72:0/2855709324 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3440039e0 0x7ff344003e00 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7ff34800c240 tx=0x7ff344005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7ac99e3caa69f71 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:01.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.809+0000 7ff355d9b640 1 -- 172.21.15.72:0/2855709324 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff348007cf0 con 0x7ff3440039e0 2024-04-15T15:16:01.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.809+0000 7ff355d9b640 1 -- 172.21.15.72:0/2855709324 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff348030070 con 0x7ff3440039e0 2024-04-15T15:16:01.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.809+0000 7ff355d9b640 1 -- 172.21.15.72:0/2855709324 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff348005d10 con 0x7ff3440039e0 2024-04-15T15:16:01.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.809+0000 7ff35cd7c640 1 -- 172.21.15.72:0/2855709324 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3440039e0 msgr2=0x7ff344003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:01.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.809+0000 7ff35cd7c640 1 --2- 172.21.15.72:0/2855709324 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff3440039e0 0x7ff344003e00 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7ff34800c240 tx=0x7ff344005830 comp rx=0 tx=0).stop 2024-04-15T15:16:01.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.809+0000 7ff356d9d640 1 -- 172.21.15.72:0/2855709324 reap_dead start 2024-04-15T15:16:01.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.809+0000 7ff35cd7c640 1 -- 172.21.15.72:0/2855709324 shutdown_connections 2024-04-15T15:16:01.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.809+0000 7ff35cd7c640 1 -- 172.21.15.72:0/2855709324 >> 172.21.15.72:0/2855709324 conn(0x7ff3580cd4a0 msgr2=0x7ff3580cd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:01.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.809+0000 7ff35cd7c640 1 -- 172.21.15.72:0/2855709324 shutdown_connections 2024-04-15T15:16:01.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.809+0000 7ff35cd7c640 1 -- 172.21.15.72:0/2855709324 wait complete. 2024-04-15T15:16:01.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.811+0000 7ff35cd7c640 1 Processor -- start 2024-04-15T15:16:01.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.811+0000 7ff35cd7c640 1 -- start start 2024-04-15T15:16:01.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.811+0000 7ff35cd7c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff35814f7a0 0x7ff3580b3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:01.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.811+0000 7ff35cd7c640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff35815b5c0 0x7ff35815ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:01.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.811+0000 7ff35cd7c640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff3580c6e80 0x7ff3580c7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:01.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.811+0000 7ff35cd7c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ff3580cb750 con 0x7ff35814f7a0 2024-04-15T15:16:01.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.811+0000 7ff35cd7c640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7ff3580b01a0 con 0x7ff3580c6e80 2024-04-15T15:16:01.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.811+0000 7ff35cd7c640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7ff35806ca80 con 0x7ff35815b5c0 2024-04-15T15:16:01.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.812+0000 7ff35659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff35814f7a0 0x7ff3580b3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:01.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.812+0000 7ff35659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff35814f7a0 0x7ff3580b3720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41936/0 (socket says 172.21.15.72:41936) 2024-04-15T15:16:01.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.812+0000 7ff35659c640 1 -- 172.21.15.72:0/3269115863 learned_addr learned my addr 172.21.15.72:0/3269115863 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:01.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.812+0000 7ff34ffff640 1 --2- 172.21.15.72:0/3269115863 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff35815b5c0 0x7ff35815ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:01.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.812+0000 7ff356d9d640 1 --2- 172.21.15.72:0/3269115863 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff3580c6e80 0x7ff3580c7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:01.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.812+0000 7ff35659c640 1 -- 172.21.15.72:0/3269115863 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff35815b5c0 msgr2=0x7ff35815ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:01.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.812+0000 7ff35659c640 1 --2- 172.21.15.72:0/3269115863 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff35815b5c0 0x7ff35815ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.812+0000 7ff35659c640 1 -- 172.21.15.72:0/3269115863 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff3580c6e80 msgr2=0x7ff3580c7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:01.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.812+0000 7ff35659c640 1 --2- 172.21.15.72:0/3269115863 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff3580c6e80 0x7ff3580c7300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.812+0000 7ff35659c640 1 -- 172.21.15.72:0/3269115863 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3580c60c0 con 0x7ff35814f7a0 2024-04-15T15:16:01.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.812+0000 7ff35659c640 1 --2- 172.21.15.72:0/3269115863 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff35814f7a0 0x7ff3580b3720 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7ff348001970 tx=0x7ff3580cbf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:01.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.812+0000 7ff34effd640 1 -- 172.21.15.72:0/3269115863 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff348004cb0 con 0x7ff35814f7a0 2024-04-15T15:16:01.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.812+0000 7ff34effd640 1 -- 172.21.15.72:0/3269115863 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff348004e50 con 0x7ff35814f7a0 2024-04-15T15:16:01.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.812+0000 7ff34effd640 1 -- 172.21.15.72:0/3269115863 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff348007270 con 0x7ff35814f7a0 2024-04-15T15:16:01.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.812+0000 7ff35cd7c640 1 -- 172.21.15.72:0/3269115863 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3580c63a0 con 0x7ff35814f7a0 2024-04-15T15:16:01.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.812+0000 7ff35cd7c640 1 -- 172.21.15.72:0/3269115863 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3580c7840 con 0x7ff35814f7a0 2024-04-15T15:16:01.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.814+0000 7ff35cd7c640 1 -- 172.21.15.72:0/3269115863 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3580689e0 con 0x7ff35814f7a0 2024-04-15T15:16:01.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.814+0000 7ff34effd640 1 -- 172.21.15.72:0/3269115863 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff348048020 con 0x7ff35814f7a0 2024-04-15T15:16:01.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.814+0000 7ff34effd640 1 --2- 172.21.15.72:0/3269115863 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff33806c920 0x7ff33806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:01.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.814+0000 7ff34effd640 1 -- 172.21.15.72:0/3269115863 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7ff3480acdb0 con 0x7ff35814f7a0 2024-04-15T15:16:01.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.814+0000 7ff34ffff640 1 --2- 172.21.15.72:0/3269115863 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff33806c920 0x7ff33806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:01.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.815+0000 7ff34ffff640 1 --2- 172.21.15.72:0/3269115863 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff33806c920 0x7ff33806ede0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff3580b6680 tx=0x7ff340008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:01.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:01.819+0000 7ff34effd640 1 -- 172.21.15.72:0/3269115863 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff348032320 con 0x7ff35814f7a0 2024-04-15T15:16:02.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.065+0000 7ff35cd7c640 1 -- 172.21.15.72:0/3269115863 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7ff3580cb530 con 0x7ff35814f7a0 2024-04-15T15:16:02.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.066+0000 7ff34effd640 1 -- 172.21.15.72:0/3269115863 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) ==== 80+0+6645 (secure 0 0 0) 0x7ff34807b430 con 0x7ff35814f7a0 2024-04-15T15:16:02.067 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:16:02.067 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-04-15T15:16:02.067 INFO:teuthology.orchestra.run.smithi072.stdout: "auth_dump": [ 2024-04-15T15:16:02.067 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:02.068 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.a", 2024-04-15T15:16:02.068 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQAxRB1mb8hFExAA+z3Hr0Di3fl2HSUuF4loaQ==", 2024-04-15T15:16:02.068 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-04-15T15:16:02.068 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-04-15T15:16:02.068 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-04-15T15:16:02.068 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-04-15T15:16:02.068 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-04-15T15:16:02.068 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:16:02.068 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:02.068 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:02.068 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.b", 2024-04-15T15:16:02.068 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQAxRB1mmEhOIBAA3r0+6X19RkbYSMGKLjo4wQ==", 2024-04-15T15:16:02.068 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-04-15T15:16:02.068 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-04-15T15:16:02.069 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-04-15T15:16:02.069 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-04-15T15:16:02.069 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-04-15T15:16:02.069 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:16:02.069 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:02.069 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:02.069 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.c", 2024-04-15T15:16:02.069 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQAxRB1mnrrtGhAAXFFmF2XPvAGcCa4ubdIbsA==", 2024-04-15T15:16:02.069 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-04-15T15:16:02.069 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-04-15T15:16:02.069 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-04-15T15:16:02.069 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-04-15T15:16:02.069 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-04-15T15:16:02.069 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:16:02.069 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:02.070 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:02.070 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mds.d", 2024-04-15T15:16:02.070 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQAxRB1mczkKKBAAmqVjJjk52ob0NzjfIVsMsg==", 2024-04-15T15:16:02.070 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-04-15T15:16:02.070 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-04-15T15:16:02.070 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-04-15T15:16:02.070 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-04-15T15:16:02.070 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-04-15T15:16:02.070 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:16:02.070 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:02.070 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:02.070 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.0", 2024-04-15T15:16:02.070 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQA1RB1mwxiNCBAAsgQtq6EfdWgfqR2lHVYO/A==", 2024-04-15T15:16:02.070 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-04-15T15:16:02.071 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-04-15T15:16:02.071 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-04-15T15:16:02.071 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-04-15T15:16:02.071 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:16:02.071 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:02.071 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:02.071 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.1", 2024-04-15T15:16:02.071 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQA4RB1mTmAgBxAAvuGNnF95Ma3ts7xme4REzg==", 2024-04-15T15:16:02.071 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-04-15T15:16:02.071 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-04-15T15:16:02.071 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-04-15T15:16:02.071 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-04-15T15:16:02.071 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:16:02.071 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:02.071 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:02.072 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.2", 2024-04-15T15:16:02.072 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQA7RB1mKV/EBRAATVdq1Hth//pt0NhGibGDyg==", 2024-04-15T15:16:02.072 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-04-15T15:16:02.072 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-04-15T15:16:02.072 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-04-15T15:16:02.072 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-04-15T15:16:02.072 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:16:02.072 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:02.072 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:02.072 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.3", 2024-04-15T15:16:02.072 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQA+RB1mpLCJBBAAmJmD6JCL4sA8dbrBkcWaoQ==", 2024-04-15T15:16:02.072 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-04-15T15:16:02.072 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-04-15T15:16:02.072 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-04-15T15:16:02.073 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-04-15T15:16:02.073 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:16:02.073 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:02.073 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:02.073 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.4", 2024-04-15T15:16:02.073 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBERB1mz26yDRAAiMp1uUjnORA7M3jIe7cL+A==", 2024-04-15T15:16:02.073 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-04-15T15:16:02.073 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-04-15T15:16:02.073 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-04-15T15:16:02.073 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-04-15T15:16:02.073 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:16:02.073 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:02.073 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:02.073 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.5", 2024-04-15T15:16:02.073 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBHRB1mdaVCDBAAG7XtCTHarMsw41QSGKWT7w==", 2024-04-15T15:16:02.074 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-04-15T15:16:02.074 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-04-15T15:16:02.074 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-04-15T15:16:02.074 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-04-15T15:16:02.074 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:16:02.074 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:02.074 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:02.074 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.6", 2024-04-15T15:16:02.074 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBKRB1mBadEChAAPPP/cq0rfWeb0vI0oEhy9Q==", 2024-04-15T15:16:02.074 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-04-15T15:16:02.074 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-04-15T15:16:02.074 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-04-15T15:16:02.074 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-04-15T15:16:02.074 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:16:02.075 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:02.075 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:02.075 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "osd.7", 2024-04-15T15:16:02.075 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBNRB1m1FK8CBAABbyg/11z7CwNO2k948Z4/Q==", 2024-04-15T15:16:02.075 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-04-15T15:16:02.075 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow profile osd", 2024-04-15T15:16:02.075 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile osd", 2024-04-15T15:16:02.075 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-04-15T15:16:02.075 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:16:02.075 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:02.075 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:02.075 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.0", 2024-04-15T15:16:02.075 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQAxRB1mycl8LBAA87k203Ws8lFHsc/7l5rqbg==", 2024-04-15T15:16:02.075 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-04-15T15:16:02.075 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-04-15T15:16:02.076 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow r", 2024-04-15T15:16:02.076 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow rw", 2024-04-15T15:16:02.076 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rwx" 2024-04-15T15:16:02.076 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:16:02.076 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:02.076 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:02.076 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.1", 2024-04-15T15:16:02.076 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQAxRB1mx0/iMxAAbLexbWtlBoCaESA8AH7sFg==", 2024-04-15T15:16:02.076 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-04-15T15:16:02.076 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-04-15T15:16:02.076 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow r", 2024-04-15T15:16:02.076 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow rw", 2024-04-15T15:16:02.076 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rwx" 2024-04-15T15:16:02.076 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:16:02.076 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:02.077 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:02.077 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.2", 2024-04-15T15:16:02.077 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQAxRB1m2XANMBAAc5PeZVI7uPVoptzUL2mlEQ==", 2024-04-15T15:16:02.077 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-04-15T15:16:02.077 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-04-15T15:16:02.077 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow r", 2024-04-15T15:16:02.077 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow rw", 2024-04-15T15:16:02.077 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rwx" 2024-04-15T15:16:02.077 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:16:02.077 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:02.077 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:02.077 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.3", 2024-04-15T15:16:02.077 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQAxRB1mF8myNxAADFNnmx+5pm9grpwJK3K3lw==", 2024-04-15T15:16:02.077 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-04-15T15:16:02.077 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow", 2024-04-15T15:16:02.077 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow r", 2024-04-15T15:16:02.078 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow rw", 2024-04-15T15:16:02.078 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow rwx" 2024-04-15T15:16:02.078 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:16:02.078 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:02.078 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:02.078 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.admin", 2024-04-15T15:16:02.078 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQAwRB1m279qMRAA0BqVgemyWquhOPxNdt73yg==", 2024-04-15T15:16:02.078 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-04-15T15:16:02.078 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow *", 2024-04-15T15:16:02.078 INFO:teuthology.orchestra.run.smithi072.stdout: "mgr": "allow *", 2024-04-15T15:16:02.078 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow *", 2024-04-15T15:16:02.078 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-04-15T15:16:02.078 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:16:02.078 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:02.079 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:02.079 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.bootstrap-mds", 2024-04-15T15:16:02.079 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTRB1ms+A1BhAAZyVoAnEkmd/o3nuViDw3eQ==", 2024-04-15T15:16:02.079 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-04-15T15:16:02.079 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile bootstrap-mds" 2024-04-15T15:16:02.079 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:16:02.079 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:02.079 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:02.079 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.bootstrap-mgr", 2024-04-15T15:16:02.079 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTRB1m9Og1BhAAORcrZZ6xeUGll30iGrcRYw==", 2024-04-15T15:16:02.079 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-04-15T15:16:02.079 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile bootstrap-mgr" 2024-04-15T15:16:02.079 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:16:02.079 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:02.079 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:02.080 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.bootstrap-osd", 2024-04-15T15:16:02.080 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTRB1mb+81BhAAetK5RjUh5DpOQqE7/1UifA==", 2024-04-15T15:16:02.080 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-04-15T15:16:02.080 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile bootstrap-osd" 2024-04-15T15:16:02.080 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:16:02.080 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:02.080 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:02.080 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.bootstrap-rbd", 2024-04-15T15:16:02.080 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTRB1mzfU1BhAAMEEQwGy6A/UgSXxWrJEqTQ==", 2024-04-15T15:16:02.080 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-04-15T15:16:02.080 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile bootstrap-rbd" 2024-04-15T15:16:02.080 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:16:02.080 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:02.080 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:02.080 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-04-15T15:16:02.081 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTRB1me/w1BhAARt7LgwDMEF1lzz3qGBjKGQ==", 2024-04-15T15:16:02.081 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-04-15T15:16:02.081 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-04-15T15:16:02.081 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:16:02.081 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:02.081 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:02.081 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "client.bootstrap-rgw", 2024-04-15T15:16:02.081 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQBTRB1mgAM2BhAARb+ElEA2iorXN+tmt1Vvyg==", 2024-04-15T15:16:02.081 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-04-15T15:16:02.081 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile bootstrap-rgw" 2024-04-15T15:16:02.081 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:16:02.081 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:02.081 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:02.081 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mgr.x", 2024-04-15T15:16:02.081 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQAxRB1mjTijDxAAFiyrGLVXMufbRf+Jjf3edg==", 2024-04-15T15:16:02.082 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-04-15T15:16:02.082 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow *", 2024-04-15T15:16:02.082 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile mgr", 2024-04-15T15:16:02.082 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-04-15T15:16:02.082 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:16:02.082 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:02.082 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:02.082 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": "mgr.y", 2024-04-15T15:16:02.082 INFO:teuthology.orchestra.run.smithi072.stdout: "key": "AQAxRB1mprGxCxAARc8uGtfv/vNbLoNQsdp9eA==", 2024-04-15T15:16:02.082 INFO:teuthology.orchestra.run.smithi072.stdout: "caps": { 2024-04-15T15:16:02.082 INFO:teuthology.orchestra.run.smithi072.stdout: "mds": "allow *", 2024-04-15T15:16:02.082 INFO:teuthology.orchestra.run.smithi072.stdout: "mon": "allow profile mgr", 2024-04-15T15:16:02.082 INFO:teuthology.orchestra.run.smithi072.stdout: "osd": "allow *" 2024-04-15T15:16:02.082 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:16:02.083 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:16:02.083 INFO:teuthology.orchestra.run.smithi072.stdout: ] 2024-04-15T15:16:02.083 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-04-15T15:16:02.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.067+0000 7ff35cd7c640 1 -- 172.21.15.72:0/3269115863 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff33806c920 msgr2=0x7ff33806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.067+0000 7ff35cd7c640 1 --2- 172.21.15.72:0/3269115863 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff33806c920 0x7ff33806ede0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff3580b6680 tx=0x7ff340008040 comp rx=0 tx=0).stop 2024-04-15T15:16:02.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.067+0000 7ff35cd7c640 1 -- 172.21.15.72:0/3269115863 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff35814f7a0 msgr2=0x7ff3580b3720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.067+0000 7ff35cd7c640 1 --2- 172.21.15.72:0/3269115863 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff35814f7a0 0x7ff3580b3720 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7ff348001970 tx=0x7ff3580cbf80 comp rx=0 tx=0).stop 2024-04-15T15:16:02.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.067+0000 7ff35cd7c640 1 -- 172.21.15.72:0/3269115863 shutdown_connections 2024-04-15T15:16:02.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.067+0000 7ff35cd7c640 1 --2- 172.21.15.72:0/3269115863 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff33806c920 0x7ff33806ede0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.067+0000 7ff35cd7c640 1 --2- 172.21.15.72:0/3269115863 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff3580c6e80 0x7ff3580c7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.067+0000 7ff35cd7c640 1 --2- 172.21.15.72:0/3269115863 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff35815b5c0 0x7ff35815ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.067+0000 7ff35cd7c640 1 --2- 172.21.15.72:0/3269115863 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff35814f7a0 0x7ff3580b3720 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.067+0000 7ff35cd7c640 1 -- 172.21.15.72:0/3269115863 >> 172.21.15.72:0/3269115863 conn(0x7ff3580cd4a0 msgr2=0x7ff3580b0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:02.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.068+0000 7ff35cd7c640 1 -- 172.21.15.72:0/3269115863 shutdown_connections 2024-04-15T15:16:02.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.068+0000 7ff35cd7c640 1 -- 172.21.15.72:0/3269115863 wait complete. 2024-04-15T15:16:02.086 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-04-15T15:16:02.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.187+0000 7f20d9759640 1 Processor -- start 2024-04-15T15:16:02.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.187+0000 7f20d9759640 1 -- start start 2024-04-15T15:16:02.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.188+0000 7f20d9759640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f20d415c250 0x7f20d406d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.188+0000 7f20d9759640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20d406df30 con 0x7f20d415f9f0 2024-04-15T15:16:02.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.188+0000 7f20d9759640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20d406e210 con 0x7f20d40d0010 2024-04-15T15:16:02.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.188+0000 7f20d9759640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f20d40cd1a0 con 0x7f20d415c250 2024-04-15T15:16:02.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.188+0000 7f20ca7fc640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f20d415f9f0 0x7f20d40cca90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:58752/0 (socket says 172.21.15.72:58752) 2024-04-15T15:16:02.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.188+0000 7f20ca7fc640 1 -- 172.21.15.72:0/3059006362 learned_addr learned my addr 172.21.15.72:0/3059006362 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:02.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.188+0000 7f20d27fc640 1 -- 172.21.15.72:0/3059006362 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2324532533 0 0) 0x7f20d406df30 con 0x7f20d415f9f0 2024-04-15T15:16:02.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.188+0000 7f20d27fc640 1 -- 172.21.15.72:0/3059006362 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20c0003520 con 0x7f20d415f9f0 2024-04-15T15:16:02.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.188+0000 7f20d2ffd640 1 --2- 172.21.15.72:0/3059006362 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f20d415c250 0x7f20d406d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.189+0000 7f20d27fc640 1 -- 172.21.15.72:0/3059006362 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2793288399 0 0) 0x7f20c0003520 con 0x7f20d415f9f0 2024-04-15T15:16:02.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.189+0000 7f20d27fc640 1 -- 172.21.15.72:0/3059006362 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f20d415c250 msgr2=0x7f20d406d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:02.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.189+0000 7f20d27fc640 1 --2- 172.21.15.72:0/3059006362 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f20d415c250 0x7f20d406d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.189+0000 7f20d27fc640 1 -- 172.21.15.72:0/3059006362 >> v1:172.21.15.175:6789/0 conn(0x7f20d40d0010 legacy=0x7f20d40b3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.189+0000 7f20d27fc640 1 -- 172.21.15.72:0/3059006362 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20d406e4f0 con 0x7f20d415f9f0 2024-04-15T15:16:02.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.189+0000 7f20d27fc640 1 -- 172.21.15.72:0/3059006362 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f20bc001670 con 0x7f20d415f9f0 2024-04-15T15:16:02.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.189+0000 7f20d27fc640 1 -- 172.21.15.72:0/3059006362 >> v1:172.21.15.72:6789/0 conn(0x7f20d415f9f0 legacy=0x7f20d40cca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.189+0000 7f20d27fc640 1 --2- 172.21.15.72:0/3059006362 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f20c00039e0 0x7f20c0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.189+0000 7f20d27fc640 1 --2- 172.21.15.72:0/3059006362 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f20d415c250 0x7f20c0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.189+0000 7f20d27fc640 1 --2- 172.21.15.72:0/3059006362 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20c0004d40 0x7f20c00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.189+0000 7f20d27fc640 1 -- 172.21.15.72:0/3059006362 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f20d40cd1a0 con 0x7f20c0004d40 2024-04-15T15:16:02.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.189+0000 7f20d27fc640 1 -- 172.21.15.72:0/3059006362 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f20c000bba0 con 0x7f20c00039e0 2024-04-15T15:16:02.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.189+0000 7f20d27fc640 1 -- 172.21.15.72:0/3059006362 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f20c000bda0 con 0x7f20d415c250 2024-04-15T15:16:02.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.189+0000 7f20d37fe640 1 --2- 172.21.15.72:0/3059006362 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20c0004d40 0x7f20c00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.189+0000 7f20d2ffd640 1 --2- 172.21.15.72:0/3059006362 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f20c00039e0 0x7f20c0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.189+0000 7f20ca7fc640 1 --2- 172.21.15.72:0/3059006362 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f20d415c250 0x7f20c0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.189+0000 7f20d37fe640 1 -- 172.21.15.72:0/3059006362 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f20d415c250 msgr2=0x7f20c0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:02.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.189+0000 7f20d37fe640 1 --2- 172.21.15.72:0/3059006362 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f20d415c250 0x7f20c0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.189+0000 7f20d37fe640 1 -- 172.21.15.72:0/3059006362 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f20c00039e0 msgr2=0x7f20c0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:02.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.189+0000 7f20d37fe640 1 --2- 172.21.15.72:0/3059006362 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f20c00039e0 0x7f20c0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.189+0000 7f20d37fe640 1 -- 172.21.15.72:0/3059006362 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20c0003520 con 0x7f20c0004d40 2024-04-15T15:16:02.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.190+0000 7f20d37fe640 1 --2- 172.21.15.72:0/3059006362 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20c0004d40 0x7f20c00051c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f20cc001e50 tx=0x7f20c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41840ef003d26b2d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:02.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.190+0000 7f20d27fc640 1 -- 172.21.15.72:0/3059006362 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20cc001a70 con 0x7f20c0004d40 2024-04-15T15:16:02.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.190+0000 7f20d27fc640 1 -- 172.21.15.72:0/3059006362 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20cc003b70 con 0x7f20c0004d40 2024-04-15T15:16:02.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.190+0000 7f20d27fc640 1 -- 172.21.15.72:0/3059006362 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20cc0050d0 con 0x7f20c0004d40 2024-04-15T15:16:02.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.190+0000 7f20d9759640 1 -- 172.21.15.72:0/3059006362 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20c0004d40 msgr2=0x7f20c00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.190+0000 7f20d9759640 1 --2- 172.21.15.72:0/3059006362 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20c0004d40 0x7f20c00051c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f20cc001e50 tx=0x7f20c0005830 comp rx=0 tx=0).stop 2024-04-15T15:16:02.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.191+0000 7f20d37fe640 1 -- 172.21.15.72:0/3059006362 reap_dead start 2024-04-15T15:16:02.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.191+0000 7f20d9759640 1 -- 172.21.15.72:0/3059006362 shutdown_connections 2024-04-15T15:16:02.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.191+0000 7f20d9759640 1 -- 172.21.15.72:0/3059006362 >> 172.21.15.72:0/3059006362 conn(0x7f20d40ceec0 msgr2=0x7f20d40cf2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:02.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.191+0000 7f20d9759640 1 -- 172.21.15.72:0/3059006362 shutdown_connections 2024-04-15T15:16:02.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.191+0000 7f20d9759640 1 -- 172.21.15.72:0/3059006362 wait complete. 2024-04-15T15:16:02.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.192+0000 7f20d9759640 1 Processor -- start 2024-04-15T15:16:02.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.192+0000 7f20d9759640 1 -- start start 2024-04-15T15:16:02.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.193+0000 7f20d9759640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f20c0004d40 0x7f20d41f5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.193+0000 7f20d9759640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f20d40cbff0 0x7f20d4157670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.193+0000 7f20d9759640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20d4158af0 0x7f20d415a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.193+0000 7f20d9759640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f20d415c630 con 0x7f20d4158af0 2024-04-15T15:16:02.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.193+0000 7f20d9759640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f20d40b38f0 con 0x7f20d40cbff0 2024-04-15T15:16:02.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.193+0000 7f20d9759640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f20d415c7b0 con 0x7f20c0004d40 2024-04-15T15:16:02.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.193+0000 7f20d37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20d4158af0 0x7f20d415a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.193+0000 7f20d37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20d4158af0 0x7f20d415a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41954/0 (socket says 172.21.15.72:41954) 2024-04-15T15:16:02.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.193+0000 7f20d37fe640 1 -- 172.21.15.72:0/2385185083 learned_addr learned my addr 172.21.15.72:0/2385185083 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:02.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.193+0000 7f20ca7fc640 1 --2- 172.21.15.72:0/2385185083 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f20d40cbff0 0x7f20d4157670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.193+0000 7f20d2ffd640 1 --2- 172.21.15.72:0/2385185083 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f20c0004d40 0x7f20d41f5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.193+0000 7f20d37fe640 1 -- 172.21.15.72:0/2385185083 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f20c0004d40 msgr2=0x7f20d41f5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.193+0000 7f20d37fe640 1 --2- 172.21.15.72:0/2385185083 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f20c0004d40 0x7f20d41f5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.193+0000 7f20d37fe640 1 -- 172.21.15.72:0/2385185083 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f20d40cbff0 msgr2=0x7f20d4157670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.193+0000 7f20d37fe640 1 --2- 172.21.15.72:0/2385185083 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f20d40cbff0 0x7f20d4157670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.193+0000 7f20d37fe640 1 -- 172.21.15.72:0/2385185083 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20d41596d0 con 0x7f20d4158af0 2024-04-15T15:16:02.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.193+0000 7f20d2ffd640 1 --2- 172.21.15.72:0/2385185083 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f20c0004d40 0x7f20d41f5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:02.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.193+0000 7f20ca7fc640 1 --2- 172.21.15.72:0/2385185083 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f20d40cbff0 0x7f20d4157670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:02.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.193+0000 7f20d37fe640 1 --2- 172.21.15.72:0/2385185083 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20d4158af0 0x7f20d415a2b0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f20cc001f70 tx=0x7f20cc0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:02.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.193+0000 7f20cbfff640 1 -- 172.21.15.72:0/2385185083 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20cc047070 con 0x7f20d4158af0 2024-04-15T15:16:02.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.193+0000 7f20cbfff640 1 -- 172.21.15.72:0/2385185083 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20cc006b50 con 0x7f20d4158af0 2024-04-15T15:16:02.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.193+0000 7f20d9759640 1 -- 172.21.15.72:0/2385185083 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20d40c9640 con 0x7f20d4158af0 2024-04-15T15:16:02.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.193+0000 7f20cbfff640 1 -- 172.21.15.72:0/2385185083 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20cc007db0 con 0x7f20d4158af0 2024-04-15T15:16:02.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.194+0000 7f20d9759640 1 -- 172.21.15.72:0/2385185083 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20d40c98e0 con 0x7f20d4158af0 2024-04-15T15:16:02.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.194+0000 7f20d9759640 1 -- 172.21.15.72:0/2385185083 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20d4075c50 con 0x7f20d4158af0 2024-04-15T15:16:02.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.195+0000 7f20cbfff640 1 -- 172.21.15.72:0/2385185083 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f20cc04a070 con 0x7f20d4158af0 2024-04-15T15:16:02.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.195+0000 7f20cbfff640 1 --2- 172.21.15.72:0/2385185083 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f20b406c8f0 0x7f20b406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.195+0000 7f20cbfff640 1 -- 172.21.15.72:0/2385185083 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f20cc0ad030 con 0x7f20d4158af0 2024-04-15T15:16:02.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.195+0000 7f20d2ffd640 1 --2- 172.21.15.72:0/2385185083 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f20b406c8f0 0x7f20b406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.196+0000 7f20d2ffd640 1 --2- 172.21.15.72:0/2385185083 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f20b406c8f0 0x7f20b406edb0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f20c4006fd0 tx=0x7f20c4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:02.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.200+0000 7f20cbfff640 1 -- 172.21.15.72:0/2385185083 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f20cc033020 con 0x7f20d4158af0 2024-04-15T15:16:02.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.446+0000 7f20d9759640 1 -- 172.21.15.72:0/2385185083 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7f20d40b83c0 con 0x7f20d4158af0 2024-04-15T15:16:02.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.448+0000 7f20cbfff640 1 -- 172.21.15.72:0/2385185083 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) ==== 90+0+0 (secure 0 0 0) 0x7f20cc07b6b0 con 0x7f20d4158af0 2024-04-15T15:16:02.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.450+0000 7f20d9759640 1 -- 172.21.15.72:0/2385185083 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f20b406c8f0 msgr2=0x7f20b406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.450+0000 7f20d9759640 1 --2- 172.21.15.72:0/2385185083 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f20b406c8f0 0x7f20b406edb0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f20c4006fd0 tx=0x7f20c4008040 comp rx=0 tx=0).stop 2024-04-15T15:16:02.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.450+0000 7f20d9759640 1 -- 172.21.15.72:0/2385185083 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20d4158af0 msgr2=0x7f20d415a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.450+0000 7f20d9759640 1 --2- 172.21.15.72:0/2385185083 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20d4158af0 0x7f20d415a2b0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f20cc001f70 tx=0x7f20cc0014a0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.450+0000 7f20d9759640 1 -- 172.21.15.72:0/2385185083 shutdown_connections 2024-04-15T15:16:02.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.450+0000 7f20d9759640 1 --2- 172.21.15.72:0/2385185083 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f20b406c8f0 0x7f20b406edb0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.450+0000 7f20d9759640 1 --2- 172.21.15.72:0/2385185083 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f20d4158af0 0x7f20d415a2b0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.450+0000 7f20d9759640 1 --2- 172.21.15.72:0/2385185083 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f20d40cbff0 0x7f20d4157670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.450+0000 7f20d9759640 1 --2- 172.21.15.72:0/2385185083 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f20c0004d40 0x7f20d41f5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.450+0000 7f20d9759640 1 -- 172.21.15.72:0/2385185083 >> 172.21.15.72:0/2385185083 conn(0x7f20d40ceec0 msgr2=0x7f20d4160c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:02.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.450+0000 7f20d9759640 1 -- 172.21.15.72:0/2385185083 shutdown_connections 2024-04-15T15:16:02.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.450+0000 7f20d9759640 1 -- 172.21.15.72:0/2385185083 wait complete. 2024-04-15T15:16:02.462 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-04-15T15:16:02.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.565+0000 7fbf8c550640 1 Processor -- start 2024-04-15T15:16:02.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.566+0000 7fbf8c550640 1 -- start start 2024-04-15T15:16:02.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.566+0000 7fbf8c550640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbf8415e610 0x7fbf840d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.567+0000 7fbf8c550640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fbf8406c400 0x7fbf8406c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.567+0000 7fbf8c550640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf840d2320 con 0x7fbf8415e9f0 2024-04-15T15:16:02.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.567+0000 7fbf8c550640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fbf840d31d0 con 0x7fbf8406c400 2024-04-15T15:16:02.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.567+0000 7fbf8c550640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fbf840d25c0 con 0x7fbf8415e610 2024-04-15T15:16:02.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.567+0000 7fbf8aaed640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fbf8406c400 0x7fbf8406c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.567+0000 7fbf8aaed640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fbf8406c400 0x7fbf8406c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41966/0 (socket says 172.21.15.72:41966) 2024-04-15T15:16:02.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.567+0000 7fbf8aaed640 1 -- 172.21.15.72:0/421446191 learned_addr learned my addr 172.21.15.72:0/421446191 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:02.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.567+0000 7fbf8aaed640 1 -- 172.21.15.72:0/421446191 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbf8415e610 msgr2=0x7fbf840d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:02.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.567+0000 7fbf8a2ec640 1 --2- 172.21.15.72:0/421446191 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbf8415e610 0x7fbf840d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.567+0000 7fbf8aaed640 1 --2- 172.21.15.72:0/421446191 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbf8415e610 0x7fbf840d0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.567+0000 7fbf8aaed640 1 -- 172.21.15.72:0/421446191 >> v1:172.21.15.175:6789/0 conn(0x7fbf8415e9f0 legacy=0x7fbf840d3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:02.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.567+0000 7fbf8aaed640 1 -- 172.21.15.72:0/421446191 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf840d28a0 con 0x7fbf8406c400 2024-04-15T15:16:02.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.568+0000 7fbf8aaed640 1 --2- 172.21.15.72:0/421446191 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf8406c400 0x7fbf8406c890 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fbf74009e50 tx=0x7fbf7402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6498ceafc5f13ca9 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:02.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.568+0000 7fbf892ea640 1 -- 172.21.15.72:0/421446191 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf7402cb00 con 0x7fbf8406c400 2024-04-15T15:16:02.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.568+0000 7fbf892ea640 1 -- 172.21.15.72:0/421446191 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf7402cca0 con 0x7fbf8406c400 2024-04-15T15:16:02.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.568+0000 7fbf892ea640 1 -- 172.21.15.72:0/421446191 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf74034610 con 0x7fbf8406c400 2024-04-15T15:16:02.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.568+0000 7fbf8c550640 1 -- 172.21.15.72:0/421446191 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf8406c400 msgr2=0x7fbf8406c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.568+0000 7fbf8c550640 1 --2- 172.21.15.72:0/421446191 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf8406c400 0x7fbf8406c890 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fbf74009e50 tx=0x7fbf7402bf60 comp rx=0 tx=0).stop 2024-04-15T15:16:02.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.569+0000 7fbf8c550640 1 -- 172.21.15.72:0/421446191 shutdown_connections 2024-04-15T15:16:02.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.569+0000 7fbf8c550640 1 --2- 172.21.15.72:0/421446191 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf8406c400 0x7fbf8406c890 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.569+0000 7fbf8c550640 1 --2- 172.21.15.72:0/421446191 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbf8415e610 0x7fbf840d0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.569+0000 7fbf8c550640 1 -- 172.21.15.72:0/421446191 >> 172.21.15.72:0/421446191 conn(0x7fbf840d4750 msgr2=0x7fbf840d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:02.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.569+0000 7fbf8c550640 1 -- 172.21.15.72:0/421446191 shutdown_connections 2024-04-15T15:16:02.571 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.569+0000 7fbf8c550640 1 -- 172.21.15.72:0/421446191 wait complete. 2024-04-15T15:16:02.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.570+0000 7fbf8c550640 1 Processor -- start 2024-04-15T15:16:02.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.570+0000 7fbf8c550640 1 -- start start 2024-04-15T15:16:02.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.570+0000 7fbf8c550640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbf8406c400 0x7fbf840ca140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.570+0000 7fbf8c550640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf8415e610 0x7fbf841e8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.571+0000 7fbf8c550640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbf8415e9f0 0x7fbf841e8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.571+0000 7fbf8c550640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8406d130 con 0x7fbf8415e610 2024-04-15T15:16:02.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.571+0000 7fbf8c550640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fbf840d2e60 con 0x7fbf8415e9f0 2024-04-15T15:16:02.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.571+0000 7fbf8c550640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fbf840d0bd0 con 0x7fbf8406c400 2024-04-15T15:16:02.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.571+0000 7fbf89aeb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf8415e610 0x7fbf841e8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.571+0000 7fbf89aeb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf8415e610 0x7fbf841e8250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41970/0 (socket says 172.21.15.72:41970) 2024-04-15T15:16:02.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.571+0000 7fbf89aeb640 1 -- 172.21.15.72:0/772986757 learned_addr learned my addr 172.21.15.72:0/772986757 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:02.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.571+0000 7fbf8a2ec640 1 --2- 172.21.15.72:0/772986757 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbf8406c400 0x7fbf840ca140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.571+0000 7fbf8aaed640 1 --2- 172.21.15.72:0/772986757 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbf8415e9f0 0x7fbf841e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.571+0000 7fbf89aeb640 1 -- 172.21.15.72:0/772986757 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbf8406c400 msgr2=0x7fbf840ca140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.571+0000 7fbf89aeb640 1 --2- 172.21.15.72:0/772986757 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbf8406c400 0x7fbf840ca140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.571+0000 7fbf89aeb640 1 -- 172.21.15.72:0/772986757 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbf8415e9f0 msgr2=0x7fbf841e8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.571+0000 7fbf89aeb640 1 --2- 172.21.15.72:0/772986757 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbf8415e9f0 0x7fbf841e8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.571+0000 7fbf89aeb640 1 -- 172.21.15.72:0/772986757 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf840cac00 con 0x7fbf8415e610 2024-04-15T15:16:02.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.571+0000 7fbf8aaed640 1 --2- 172.21.15.72:0/772986757 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbf8415e9f0 0x7fbf841e8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:02.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.571+0000 7fbf89aeb640 1 --2- 172.21.15.72:0/772986757 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf8415e610 0x7fbf841e8250 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fbf7800b3f0 tx=0x7fbf7800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:02.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.571+0000 7fbf82ffd640 1 -- 172.21.15.72:0/772986757 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf78015020 con 0x7fbf8415e610 2024-04-15T15:16:02.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.571+0000 7fbf82ffd640 1 -- 172.21.15.72:0/772986757 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf78011830 con 0x7fbf8415e610 2024-04-15T15:16:02.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.571+0000 7fbf8c550640 1 -- 172.21.15.72:0/772986757 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf840caee0 con 0x7fbf8415e610 2024-04-15T15:16:02.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.571+0000 7fbf82ffd640 1 -- 172.21.15.72:0/772986757 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf78011c20 con 0x7fbf8415e610 2024-04-15T15:16:02.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.571+0000 7fbf8c550640 1 -- 172.21.15.72:0/772986757 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf841562b0 con 0x7fbf8415e610 2024-04-15T15:16:02.575 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.572+0000 7fbf8c550640 1 -- 172.21.15.72:0/772986757 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf84075c50 con 0x7fbf8415e610 2024-04-15T15:16:02.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.573+0000 7fbf82ffd640 1 -- 172.21.15.72:0/772986757 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbf78018020 con 0x7fbf8415e610 2024-04-15T15:16:02.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.573+0000 7fbf82ffd640 1 --2- 172.21.15.72:0/772986757 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fbf7006c8f0 0x7fbf7006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.573+0000 7fbf8a2ec640 1 --2- 172.21.15.72:0/772986757 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fbf7006c8f0 0x7fbf7006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.573+0000 7fbf82ffd640 1 -- 172.21.15.72:0/772986757 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7fbf7808b550 con 0x7fbf8415e610 2024-04-15T15:16:02.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.574+0000 7fbf8a2ec640 1 --2- 172.21.15.72:0/772986757 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fbf7006c8f0 0x7fbf7006edb0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c00aac0 tx=0x7fbf7c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:02.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.578+0000 7fbf82ffd640 1 -- 172.21.15.72:0/772986757 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbf7801f030 con 0x7fbf8415e610 2024-04-15T15:16:02.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.821+0000 7fbf8c550640 1 -- 172.21.15.72:0/772986757 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7fbf840cf740 con 0x7fbf8415e610 2024-04-15T15:16:02.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.822+0000 7fbf82ffd640 1 -- 172.21.15.72:0/772986757 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) ==== 90+0+0 (secure 0 0 0) 0x7fbf78059bd0 con 0x7fbf8415e610 2024-04-15T15:16:02.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.824+0000 7fbf8c550640 1 -- 172.21.15.72:0/772986757 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fbf7006c8f0 msgr2=0x7fbf7006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.824+0000 7fbf8c550640 1 --2- 172.21.15.72:0/772986757 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fbf7006c8f0 0x7fbf7006edb0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c00aac0 tx=0x7fbf7c008040 comp rx=0 tx=0).stop 2024-04-15T15:16:02.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.824+0000 7fbf8c550640 1 -- 172.21.15.72:0/772986757 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf8415e610 msgr2=0x7fbf841e8250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.824+0000 7fbf8c550640 1 --2- 172.21.15.72:0/772986757 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf8415e610 0x7fbf841e8250 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fbf7800b3f0 tx=0x7fbf7800b8b0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.824+0000 7fbf8c550640 1 -- 172.21.15.72:0/772986757 shutdown_connections 2024-04-15T15:16:02.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.824+0000 7fbf8c550640 1 --2- 172.21.15.72:0/772986757 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fbf7006c8f0 0x7fbf7006edb0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.824+0000 7fbf8c550640 1 --2- 172.21.15.72:0/772986757 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbf8415e9f0 0x7fbf841e8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.824+0000 7fbf8c550640 1 --2- 172.21.15.72:0/772986757 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf8415e610 0x7fbf841e8250 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.825+0000 7fbf8c550640 1 --2- 172.21.15.72:0/772986757 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbf8406c400 0x7fbf840ca140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.825+0000 7fbf8c550640 1 -- 172.21.15.72:0/772986757 >> 172.21.15.72:0/772986757 conn(0x7fbf840d4750 msgr2=0x7fbf840b5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:02.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.825+0000 7fbf8c550640 1 -- 172.21.15.72:0/772986757 shutdown_connections 2024-04-15T15:16:02.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.825+0000 7fbf8c550640 1 -- 172.21.15.72:0/772986757 wait complete. 2024-04-15T15:16:02.837 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-04-15T15:16:02.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.939+0000 7f026feba640 1 Processor -- start 2024-04-15T15:16:02.941 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.940+0000 7f026feba640 1 -- start start 2024-04-15T15:16:02.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.940+0000 7f026feba640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f026815b520 0x7f026815efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.940+0000 7f026feba640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f026806f550 0x7f02680b4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.940+0000 7f026feba640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f026815fa20 con 0x7f026815f500 2024-04-15T15:16:02.942 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.940+0000 7f026feba640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f02680b43b0 con 0x7f026806f550 2024-04-15T15:16:02.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.940+0000 7f026feba640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f026815fd00 con 0x7f026815b520 2024-04-15T15:16:02.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.940+0000 7f026e457640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f026806f550 0x7f02680b4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.941+0000 7f026e457640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f026806f550 0x7f02680b4e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41980/0 (socket says 172.21.15.72:41980) 2024-04-15T15:16:02.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.941+0000 7f026e457640 1 -- 172.21.15.72:0/1119138517 learned_addr learned my addr 172.21.15.72:0/1119138517 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:02.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.941+0000 7f026e457640 1 -- 172.21.15.72:0/1119138517 >> v2:172.21.15.175:3300/0 conn(0x7f026815b520 msgr2=0x7f026815efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:02.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.941+0000 7f026e457640 1 --2- 172.21.15.72:0/1119138517 >> v2:172.21.15.175:3300/0 conn(0x7f026815b520 0x7f026815efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.941+0000 7f026e457640 1 -- 172.21.15.72:0/1119138517 >> v1:172.21.15.175:6789/0 conn(0x7f026815f500 legacy=0x7f026806ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:02.943 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.941+0000 7f026e457640 1 -- 172.21.15.72:0/1119138517 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02680b4600 con 0x7f026806f550 2024-04-15T15:16:02.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.941+0000 7f026e457640 1 --2- 172.21.15.72:0/1119138517 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f026806f550 0x7f02680b4e00 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f0268072560 tx=0x7f026002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fdf937fff1d821f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:02.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.942+0000 7f026cc54640 1 -- 172.21.15.72:0/1119138517 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f026002ca80 con 0x7f026806f550 2024-04-15T15:16:02.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.942+0000 7f026cc54640 1 -- 172.21.15.72:0/1119138517 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f026002cc20 con 0x7f026806f550 2024-04-15T15:16:02.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.942+0000 7f026cc54640 1 -- 172.21.15.72:0/1119138517 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02600343f0 con 0x7f026806f550 2024-04-15T15:16:02.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.942+0000 7f026feba640 1 -- 172.21.15.72:0/1119138517 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f026806f550 msgr2=0x7f02680b4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.942+0000 7f026feba640 1 --2- 172.21.15.72:0/1119138517 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f026806f550 0x7f02680b4e00 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f0268072560 tx=0x7f026002bee0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.942+0000 7f026feba640 1 -- 172.21.15.72:0/1119138517 shutdown_connections 2024-04-15T15:16:02.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.942+0000 7f026feba640 1 --2- 172.21.15.72:0/1119138517 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f026806f550 0x7f02680b4e00 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.942+0000 7f026feba640 1 --2- 172.21.15.72:0/1119138517 >> v2:172.21.15.175:3300/0 conn(0x7f026815b520 0x7f026815efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.942+0000 7f026feba640 1 -- 172.21.15.72:0/1119138517 >> 172.21.15.72:0/1119138517 conn(0x7f02680703f0 msgr2=0x7f0268070800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:02.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.942+0000 7f026feba640 1 -- 172.21.15.72:0/1119138517 shutdown_connections 2024-04-15T15:16:02.945 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.942+0000 7f026feba640 1 -- 172.21.15.72:0/1119138517 wait complete. 2024-04-15T15:16:02.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.943+0000 7f026feba640 1 Processor -- start 2024-04-15T15:16:02.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.943+0000 7f026feba640 1 -- start start 2024-04-15T15:16:02.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.944+0000 7f026feba640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f026806f550 0x7f0268152a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.944+0000 7f026feba640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f026815b520 0x7f0268157df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.944+0000 7f026feba640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f026815f500 0x7f0268153a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.944+0000 7f026feba640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f02680b54c0 con 0x7f026815f500 2024-04-15T15:16:02.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.944+0000 7f026feba640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f02680b5340 con 0x7f026806f550 2024-04-15T15:16:02.947 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.944+0000 7f026feba640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f02680b5640 con 0x7f026815b520 2024-04-15T15:16:02.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.944+0000 7f026e457640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f026815f500 0x7f0268153a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.944+0000 7f026e457640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f026815f500 0x7f0268153a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:41996/0 (socket says 172.21.15.72:41996) 2024-04-15T15:16:02.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.944+0000 7f026e457640 1 -- 172.21.15.72:0/2068137033 learned_addr learned my addr 172.21.15.72:0/2068137033 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:02.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.944+0000 7f026e457640 1 -- 172.21.15.72:0/2068137033 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f026815b520 msgr2=0x7f0268157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.944+0000 7f026e457640 1 --2- 172.21.15.72:0/2068137033 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f026815b520 0x7f0268157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.945+0000 7f026e457640 1 -- 172.21.15.72:0/2068137033 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f026806f550 msgr2=0x7f0268152a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.945+0000 7f026e457640 1 --2- 172.21.15.72:0/2068137033 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f026806f550 0x7f0268152a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.945+0000 7f026e457640 1 -- 172.21.15.72:0/2068137033 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02681f1670 con 0x7f026815f500 2024-04-15T15:16:02.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.945+0000 7f026e457640 1 --2- 172.21.15.72:0/2068137033 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f026815f500 0x7f0268153a50 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f0260036040 tx=0x7f0260004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:02.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.945+0000 7f02667fc640 1 -- 172.21.15.72:0/2068137033 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0260043020 con 0x7f026815f500 2024-04-15T15:16:02.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.945+0000 7f02667fc640 1 -- 172.21.15.72:0/2068137033 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f026003e450 con 0x7f026815f500 2024-04-15T15:16:02.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.945+0000 7f026feba640 1 -- 172.21.15.72:0/2068137033 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0268152fd0 con 0x7f026815f500 2024-04-15T15:16:02.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.945+0000 7f02667fc640 1 -- 172.21.15.72:0/2068137033 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f026003e820 con 0x7f026815f500 2024-04-15T15:16:02.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.945+0000 7f026feba640 1 -- 172.21.15.72:0/2068137033 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02681f1910 con 0x7f026815f500 2024-04-15T15:16:02.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.946+0000 7f026feba640 1 -- 172.21.15.72:0/2068137033 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0268075c50 con 0x7f026815f500 2024-04-15T15:16:02.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.946+0000 7f02667fc640 1 -- 172.21.15.72:0/2068137033 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0260046020 con 0x7f026815f500 2024-04-15T15:16:02.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.947+0000 7f02667fc640 1 --2- 172.21.15.72:0/2068137033 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f025406c8f0 0x7f025406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.947+0000 7f026dc56640 1 --2- 172.21.15.72:0/2068137033 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f025406c8f0 0x7f025406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.947+0000 7f02667fc640 1 -- 172.21.15.72:0/2068137033 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f02600ae330 con 0x7f026815f500 2024-04-15T15:16:02.950 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.947+0000 7f026dc56640 1 --2- 172.21.15.72:0/2068137033 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f025406c8f0 0x7f025406edb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0258009710 tx=0x7f0258006c90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:02.953 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:02.952+0000 7f02667fc640 1 -- 172.21.15.72:0/2068137033 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f026004d030 con 0x7f026815f500 2024-04-15T15:16:03.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.193+0000 7f026feba640 1 -- 172.21.15.72:0/2068137033 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7f026806ec20 con 0x7f026815f500 2024-04-15T15:16:03.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.195+0000 7f02667fc640 1 -- 172.21.15.72:0/2068137033 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) ==== 90+0+0 (secure 0 0 0) 0x7f026007c920 con 0x7f026815f500 2024-04-15T15:16:03.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.197+0000 7f026feba640 1 -- 172.21.15.72:0/2068137033 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f025406c8f0 msgr2=0x7f025406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.197+0000 7f026feba640 1 --2- 172.21.15.72:0/2068137033 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f025406c8f0 0x7f025406edb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0258009710 tx=0x7f0258006c90 comp rx=0 tx=0).stop 2024-04-15T15:16:03.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.197+0000 7f026feba640 1 -- 172.21.15.72:0/2068137033 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f026815f500 msgr2=0x7f0268153a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.197+0000 7f026feba640 1 --2- 172.21.15.72:0/2068137033 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f026815f500 0x7f0268153a50 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f0260036040 tx=0x7f0260004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:03.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.197+0000 7f026feba640 1 -- 172.21.15.72:0/2068137033 shutdown_connections 2024-04-15T15:16:03.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.197+0000 7f026feba640 1 --2- 172.21.15.72:0/2068137033 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f025406c8f0 0x7f025406edb0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.197+0000 7f026feba640 1 --2- 172.21.15.72:0/2068137033 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f026815f500 0x7f0268153a50 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.197+0000 7f026feba640 1 --2- 172.21.15.72:0/2068137033 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f026815b520 0x7f0268157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.197+0000 7f026feba640 1 --2- 172.21.15.72:0/2068137033 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f026806f550 0x7f0268152a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.197+0000 7f026feba640 1 -- 172.21.15.72:0/2068137033 >> 172.21.15.72:0/2068137033 conn(0x7f02680703f0 msgr2=0x7f026815ed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:03.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.197+0000 7f026feba640 1 -- 172.21.15.72:0/2068137033 shutdown_connections 2024-04-15T15:16:03.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.197+0000 7f026feba640 1 -- 172.21.15.72:0/2068137033 wait complete. 2024-04-15T15:16:03.209 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-04-15T15:16:03.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.312+0000 7fc1c9f59640 1 Processor -- start 2024-04-15T15:16:03.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.312+0000 7fc1c9f59640 1 -- start start 2024-04-15T15:16:03.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.312+0000 7fc1c9f59640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc1c406c400 0x7fc1c406c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.312+0000 7fc1c9f59640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1c40d2320 con 0x7fc1c415e610 2024-04-15T15:16:03.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.312+0000 7fc1c9f59640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1c40d25c0 con 0x7fc1c415e9f0 2024-04-15T15:16:03.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.312+0000 7fc1c9f59640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fc1c40d31d0 con 0x7fc1c406c400 2024-04-15T15:16:03.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.313+0000 7fc1c3fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc1c406c400 0x7fc1c406c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.313+0000 7fc1c3fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc1c406c400 0x7fc1c406c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42000/0 (socket says 172.21.15.72:42000) 2024-04-15T15:16:03.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.313+0000 7fc1c37fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fc1c415e610 0x7fc1c40d0690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:58768/0 (socket says 172.21.15.72:58768) 2024-04-15T15:16:03.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.313+0000 7fc1c3fff640 1 -- 172.21.15.72:0/1732877803 learned_addr learned my addr 172.21.15.72:0/1732877803 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:03.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.313+0000 7fc1c3fff640 1 -- 172.21.15.72:0/1732877803 >> v1:172.21.15.175:6789/0 conn(0x7fc1c415e9f0 legacy=0x7fc1c40d3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:03.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.313+0000 7fc1c3fff640 1 -- 172.21.15.72:0/1732877803 >> v1:172.21.15.72:6789/0 conn(0x7fc1c415e610 legacy=0x7fc1c40d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.313+0000 7fc1c3fff640 1 -- 172.21.15.72:0/1732877803 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1c40d28a0 con 0x7fc1c406c400 2024-04-15T15:16:03.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.313+0000 7fc1c27fc640 1 -- 172.21.15.72:0/1732877803 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 365497472 0 0) 0x7fc1c40d2320 con 0x7fc1c415e610 2024-04-15T15:16:03.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.314+0000 7fc1c3fff640 1 --2- 172.21.15.72:0/1732877803 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1c406c400 0x7fc1c406c890 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc009e50 tx=0x7fc1bc02c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d461adcd60547c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:03.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.314+0000 7fc1c27fc640 1 -- 172.21.15.72:0/1732877803 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc02cbb0 con 0x7fc1c406c400 2024-04-15T15:16:03.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.314+0000 7fc1c27fc640 1 -- 172.21.15.72:0/1732877803 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1bc02cd50 con 0x7fc1c406c400 2024-04-15T15:16:03.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.314+0000 7fc1c27fc640 1 -- 172.21.15.72:0/1732877803 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1bc034610 con 0x7fc1c406c400 2024-04-15T15:16:03.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.314+0000 7fc1c9f59640 1 -- 172.21.15.72:0/1732877803 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1c406c400 msgr2=0x7fc1c406c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.314+0000 7fc1c9f59640 1 --2- 172.21.15.72:0/1732877803 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1c406c400 0x7fc1c406c890 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc009e50 tx=0x7fc1bc02c010 comp rx=0 tx=0).stop 2024-04-15T15:16:03.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.314+0000 7fc1c9f59640 1 -- 172.21.15.72:0/1732877803 shutdown_connections 2024-04-15T15:16:03.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.314+0000 7fc1c9f59640 1 --2- 172.21.15.72:0/1732877803 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1c406c400 0x7fc1c406c890 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.314+0000 7fc1c9f59640 1 -- 172.21.15.72:0/1732877803 >> 172.21.15.72:0/1732877803 conn(0x7fc1c40d4750 msgr2=0x7fc1c40d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:03.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.314+0000 7fc1c9f59640 1 -- 172.21.15.72:0/1732877803 shutdown_connections 2024-04-15T15:16:03.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.314+0000 7fc1c9f59640 1 -- 172.21.15.72:0/1732877803 wait complete. 2024-04-15T15:16:03.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.315+0000 7fc1c9f59640 1 Processor -- start 2024-04-15T15:16:03.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.316+0000 7fc1c9f59640 1 -- start start 2024-04-15T15:16:03.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.316+0000 7fc1c9f59640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1c406c400 0x7fc1c414cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.316+0000 7fc1c9f59640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc1c415e610 0x7fc1c40cd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.316+0000 7fc1c9f59640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1c415e9f0 0x7fc1c40cb940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.316+0000 7fc1c9f59640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c406d130 con 0x7fc1c406c400 2024-04-15T15:16:03.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.316+0000 7fc1c9f59640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c40d2e60 con 0x7fc1c415e9f0 2024-04-15T15:16:03.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.316+0000 7fc1c9f59640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fc1c40b7940 con 0x7fc1c415e610 2024-04-15T15:16:03.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.316+0000 7fc1c37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1c406c400 0x7fc1c414cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.316+0000 7fc1c37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1c406c400 0x7fc1c414cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42012/0 (socket says 172.21.15.72:42012) 2024-04-15T15:16:03.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.316+0000 7fc1c37fe640 1 -- 172.21.15.72:0/2371644645 learned_addr learned my addr 172.21.15.72:0/2371644645 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:03.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.317+0000 7fc1c2ffd640 1 --2- 172.21.15.72:0/2371644645 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc1c415e610 0x7fc1c40cd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.317+0000 7fc1c3fff640 1 --2- 172.21.15.72:0/2371644645 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1c415e9f0 0x7fc1c40cb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.317+0000 7fc1c37fe640 1 -- 172.21.15.72:0/2371644645 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc1c415e610 msgr2=0x7fc1c40cd4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.317+0000 7fc1c37fe640 1 --2- 172.21.15.72:0/2371644645 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc1c415e610 0x7fc1c40cd4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.317+0000 7fc1c37fe640 1 -- 172.21.15.72:0/2371644645 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1c415e9f0 msgr2=0x7fc1c40cb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.317+0000 7fc1c37fe640 1 --2- 172.21.15.72:0/2371644645 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1c415e9f0 0x7fc1c40cb940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.317+0000 7fc1c37fe640 1 -- 172.21.15.72:0/2371644645 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1c415a180 con 0x7fc1c406c400 2024-04-15T15:16:03.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.317+0000 7fc1c2ffd640 1 --2- 172.21.15.72:0/2371644645 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc1c415e610 0x7fc1c40cd4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:03.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.317+0000 7fc1c3fff640 1 --2- 172.21.15.72:0/2371644645 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1c415e9f0 0x7fc1c40cb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:03.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.317+0000 7fc1c37fe640 1 --2- 172.21.15.72:0/2371644645 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1c406c400 0x7fc1c414cc70 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fc1b400c4e0 tx=0x7fc1b400c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:03.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.317+0000 7fc1abfff640 1 -- 172.21.15.72:0/2371644645 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1b4018070 con 0x7fc1c406c400 2024-04-15T15:16:03.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.317+0000 7fc1abfff640 1 -- 172.21.15.72:0/2371644645 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1b40074c0 con 0x7fc1c406c400 2024-04-15T15:16:03.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.317+0000 7fc1c9f59640 1 -- 172.21.15.72:0/2371644645 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1c415a460 con 0x7fc1c406c400 2024-04-15T15:16:03.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.317+0000 7fc1abfff640 1 -- 172.21.15.72:0/2371644645 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1b4004a20 con 0x7fc1c406c400 2024-04-15T15:16:03.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.317+0000 7fc1c9f59640 1 -- 172.21.15.72:0/2371644645 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1c40ccfa0 con 0x7fc1c406c400 2024-04-15T15:16:03.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.318+0000 7fc1c9f59640 1 -- 172.21.15.72:0/2371644645 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1c4075c50 con 0x7fc1c406c400 2024-04-15T15:16:03.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.319+0000 7fc1abfff640 1 -- 172.21.15.72:0/2371644645 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc1b401b070 con 0x7fc1c406c400 2024-04-15T15:16:03.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.319+0000 7fc1abfff640 1 --2- 172.21.15.72:0/2371644645 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc1b006c8f0 0x7fc1b006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.319+0000 7fc1abfff640 1 -- 172.21.15.72:0/2371644645 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7fc1b408ada0 con 0x7fc1c406c400 2024-04-15T15:16:03.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.319+0000 7fc1c2ffd640 1 --2- 172.21.15.72:0/2371644645 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc1b006c8f0 0x7fc1b006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.320+0000 7fc1c2ffd640 1 --2- 172.21.15.72:0/2371644645 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc1b006c8f0 0x7fc1b006edb0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc1c40cdd50 tx=0x7fc1b8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:03.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.323+0000 7fc1abfff640 1 -- 172.21.15.72:0/2371644645 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc1b4059450 con 0x7fc1c406c400 2024-04-15T15:16:03.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.567+0000 7fc1c9f59640 1 -- 172.21.15.72:0/2371644645 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7fc1c40b7080 con 0x7fc1c406c400 2024-04-15T15:16:03.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.569+0000 7fc1abfff640 1 -- 172.21.15.72:0/2371644645 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) ==== 90+0+0 (secure 0 0 0) 0x7fc1b4058d50 con 0x7fc1c406c400 2024-04-15T15:16:03.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.571+0000 7fc1c9f59640 1 -- 172.21.15.72:0/2371644645 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc1b006c8f0 msgr2=0x7fc1b006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.571+0000 7fc1c9f59640 1 --2- 172.21.15.72:0/2371644645 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc1b006c8f0 0x7fc1b006edb0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc1c40cdd50 tx=0x7fc1b8008040 comp rx=0 tx=0).stop 2024-04-15T15:16:03.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.571+0000 7fc1c9f59640 1 -- 172.21.15.72:0/2371644645 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1c406c400 msgr2=0x7fc1c414cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.571+0000 7fc1c9f59640 1 --2- 172.21.15.72:0/2371644645 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1c406c400 0x7fc1c414cc70 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fc1b400c4e0 tx=0x7fc1b400c9a0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.571+0000 7fc1c9f59640 1 -- 172.21.15.72:0/2371644645 shutdown_connections 2024-04-15T15:16:03.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.571+0000 7fc1c9f59640 1 --2- 172.21.15.72:0/2371644645 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc1b006c8f0 0x7fc1b006edb0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.571+0000 7fc1c9f59640 1 --2- 172.21.15.72:0/2371644645 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1c415e9f0 0x7fc1c40cb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.573 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.571+0000 7fc1c9f59640 1 --2- 172.21.15.72:0/2371644645 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc1c415e610 0x7fc1c40cd4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.571+0000 7fc1c9f59640 1 --2- 172.21.15.72:0/2371644645 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1c406c400 0x7fc1c414cc70 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.571+0000 7fc1c9f59640 1 -- 172.21.15.72:0/2371644645 >> 172.21.15.72:0/2371644645 conn(0x7fc1c40d4750 msgr2=0x7fc1c40b5890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:03.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.571+0000 7fc1c9f59640 1 -- 172.21.15.72:0/2371644645 shutdown_connections 2024-04-15T15:16:03.574 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.571+0000 7fc1c9f59640 1 -- 172.21.15.72:0/2371644645 wait complete. 2024-04-15T15:16:03.584 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-04-15T15:16:03.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.687+0000 7ff8aa28f640 1 Processor -- start 2024-04-15T15:16:03.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.687+0000 7ff8aa28f640 1 -- start start 2024-04-15T15:16:03.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.688+0000 7ff8aa28f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff8a40cfe30 0x7ff8a40cf040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.688+0000 7ff8aa28f640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8a40ae160 con 0x7ff8a415e770 2024-04-15T15:16:03.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.688+0000 7ff8aa28f640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8a40b5310 con 0x7ff8a40b4dc0 2024-04-15T15:16:03.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.688+0000 7ff8aa28f640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7ff8a40cf580 con 0x7ff8a40cfe30 2024-04-15T15:16:03.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.688+0000 7ff8a37fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff8a40cfe30 0x7ff8a40cf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.688+0000 7ff8a3fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ff8a415e770 0x7ff8a40cf720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:58782/0 (socket says 172.21.15.72:58782) 2024-04-15T15:16:03.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.688+0000 7ff8a3fff640 1 -- 172.21.15.72:0/2356466623 learned_addr learned my addr 172.21.15.72:0/2356466623 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:03.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.689+0000 7ff8a37fe640 1 -- 172.21.15.72:0/2356466623 >> v1:172.21.15.175:6789/0 conn(0x7ff8a40b4dc0 legacy=0x7ff8a40ae600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:03.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.689+0000 7ff8a37fe640 1 -- 172.21.15.72:0/2356466623 >> v1:172.21.15.72:6789/0 conn(0x7ff8a415e770 legacy=0x7ff8a40cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.689+0000 7ff8a37fe640 1 -- 172.21.15.72:0/2356466623 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8a40b55f0 con 0x7ff8a40cfe30 2024-04-15T15:16:03.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.689+0000 7ff8a2ffd640 1 -- 172.21.15.72:0/2356466623 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1607959088 0 0) 0x7ff8a40ae160 con 0x7ff8a415e770 2024-04-15T15:16:03.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.689+0000 7ff8a37fe640 1 --2- 172.21.15.72:0/2356466623 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8a40cfe30 0x7ff8a40cf040 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff8980097a0 tx=0x7ff89802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45b321f9502b8b11 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:03.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.690+0000 7ff8a2ffd640 1 -- 172.21.15.72:0/2356466623 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff898035020 con 0x7ff8a40cfe30 2024-04-15T15:16:03.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.690+0000 7ff8a2ffd640 1 -- 172.21.15.72:0/2356466623 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff898031830 con 0x7ff8a40cfe30 2024-04-15T15:16:03.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.690+0000 7ff8a2ffd640 1 -- 172.21.15.72:0/2356466623 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff898031c00 con 0x7ff8a40cfe30 2024-04-15T15:16:03.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.690+0000 7ff8aa28f640 1 -- 172.21.15.72:0/2356466623 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8a40cfe30 msgr2=0x7ff8a40cf040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.690+0000 7ff8aa28f640 1 --2- 172.21.15.72:0/2356466623 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8a40cfe30 0x7ff8a40cf040 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff8980097a0 tx=0x7ff89802b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.692 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.690+0000 7ff8aa28f640 1 -- 172.21.15.72:0/2356466623 shutdown_connections 2024-04-15T15:16:03.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.690+0000 7ff8aa28f640 1 --2- 172.21.15.72:0/2356466623 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8a40cfe30 0x7ff8a40cf040 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.690+0000 7ff8aa28f640 1 -- 172.21.15.72:0/2356466623 >> 172.21.15.72:0/2356466623 conn(0x7ff8a40d13a0 msgr2=0x7ff8a40d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:03.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.690+0000 7ff8aa28f640 1 -- 172.21.15.72:0/2356466623 shutdown_connections 2024-04-15T15:16:03.693 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.690+0000 7ff8aa28f640 1 -- 172.21.15.72:0/2356466623 wait complete. 2024-04-15T15:16:03.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.691+0000 7ff8aa28f640 1 Processor -- start 2024-04-15T15:16:03.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.692+0000 7ff8aa28f640 1 -- start start 2024-04-15T15:16:03.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.692+0000 7ff8aa28f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8a40b4dc0 0x7ff8a414bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.692+0000 7ff8aa28f640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff8a40cfe30 0x7ff8a41e70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.692+0000 7ff8aa28f640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff8a415e770 0x7ff8a406ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.692+0000 7ff8aa28f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a415eb50 con 0x7ff8a40b4dc0 2024-04-15T15:16:03.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.692+0000 7ff8aa28f640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a40b5bb0 con 0x7ff8a415e770 2024-04-15T15:16:03.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.692+0000 7ff8aa28f640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7ff8a415ecd0 con 0x7ff8a40cfe30 2024-04-15T15:16:03.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.692+0000 7ff8a3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8a40b4dc0 0x7ff8a414bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.692+0000 7ff8a3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8a40b4dc0 0x7ff8a414bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42022/0 (socket says 172.21.15.72:42022) 2024-04-15T15:16:03.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.692+0000 7ff8a3fff640 1 -- 172.21.15.72:0/3440480027 learned_addr learned my addr 172.21.15.72:0/3440480027 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:03.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.692+0000 7ff8a37fe640 1 --2- 172.21.15.72:0/3440480027 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff8a40cfe30 0x7ff8a41e70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.692+0000 7ff8a882c640 1 --2- 172.21.15.72:0/3440480027 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff8a415e770 0x7ff8a406ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.693+0000 7ff8a3fff640 1 -- 172.21.15.72:0/3440480027 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff8a40cfe30 msgr2=0x7ff8a41e70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.693+0000 7ff8a3fff640 1 --2- 172.21.15.72:0/3440480027 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff8a40cfe30 0x7ff8a41e70e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.693+0000 7ff8a3fff640 1 -- 172.21.15.72:0/3440480027 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff8a415e770 msgr2=0x7ff8a406ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.693+0000 7ff8a3fff640 1 --2- 172.21.15.72:0/3440480027 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff8a415e770 0x7ff8a406ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.693+0000 7ff8a3fff640 1 -- 172.21.15.72:0/3440480027 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8a406df30 con 0x7ff8a40b4dc0 2024-04-15T15:16:03.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.693+0000 7ff8a3fff640 1 --2- 172.21.15.72:0/3440480027 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8a40b4dc0 0x7ff8a414bf00 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7ff89400c400 tx=0x7ff89400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:03.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.693+0000 7ff8a0ff9640 1 -- 172.21.15.72:0/3440480027 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff894017070 con 0x7ff8a40b4dc0 2024-04-15T15:16:03.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.693+0000 7ff8a0ff9640 1 -- 172.21.15.72:0/3440480027 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff894005ce0 con 0x7ff8a40b4dc0 2024-04-15T15:16:03.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.693+0000 7ff8aa28f640 1 -- 172.21.15.72:0/3440480027 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8a406e210 con 0x7ff8a40b4dc0 2024-04-15T15:16:03.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.693+0000 7ff8a0ff9640 1 -- 172.21.15.72:0/3440480027 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff894007310 con 0x7ff8a40b4dc0 2024-04-15T15:16:03.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.693+0000 7ff8aa28f640 1 -- 172.21.15.72:0/3440480027 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8a4159970 con 0x7ff8a40b4dc0 2024-04-15T15:16:03.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.694+0000 7ff8aa28f640 1 -- 172.21.15.72:0/3440480027 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8a4075c50 con 0x7ff8a40b4dc0 2024-04-15T15:16:03.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.694+0000 7ff8a0ff9640 1 -- 172.21.15.72:0/3440480027 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff89401a070 con 0x7ff8a40b4dc0 2024-04-15T15:16:03.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.695+0000 7ff8a0ff9640 1 --2- 172.21.15.72:0/3440480027 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff89006c8f0 0x7ff89006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.695+0000 7ff8a0ff9640 1 -- 172.21.15.72:0/3440480027 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7ff89408adb0 con 0x7ff8a40b4dc0 2024-04-15T15:16:03.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.695+0000 7ff8a37fe640 1 --2- 172.21.15.72:0/3440480027 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff89006c8f0 0x7ff89006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.695+0000 7ff8a37fe640 1 --2- 172.21.15.72:0/3440480027 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff89006c8f0 0x7ff89006edb0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff898036000 tx=0x7ff898008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:03.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.699+0000 7ff8a0ff9640 1 -- 172.21.15.72:0/3440480027 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff8940593a0 con 0x7ff8a40b4dc0 2024-04-15T15:16:03.944 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.942+0000 7ff8aa28f640 1 -- 172.21.15.72:0/3440480027 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7ff8a4073b60 con 0x7ff8a40b4dc0 2024-04-15T15:16:03.946 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.944+0000 7ff8a0ff9640 1 -- 172.21.15.72:0/3440480027 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) ==== 97+0+0 (secure 0 0 0) 0x7ff894058ca0 con 0x7ff8a40b4dc0 2024-04-15T15:16:03.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.946+0000 7ff8aa28f640 1 -- 172.21.15.72:0/3440480027 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff89006c8f0 msgr2=0x7ff89006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.946+0000 7ff8aa28f640 1 --2- 172.21.15.72:0/3440480027 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff89006c8f0 0x7ff89006edb0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff898036000 tx=0x7ff898008030 comp rx=0 tx=0).stop 2024-04-15T15:16:03.948 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.946+0000 7ff8aa28f640 1 -- 172.21.15.72:0/3440480027 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8a40b4dc0 msgr2=0x7ff8a414bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.946+0000 7ff8aa28f640 1 --2- 172.21.15.72:0/3440480027 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8a40b4dc0 0x7ff8a414bf00 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7ff89400c400 tx=0x7ff89400c8c0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.946+0000 7ff8aa28f640 1 -- 172.21.15.72:0/3440480027 shutdown_connections 2024-04-15T15:16:03.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.946+0000 7ff8aa28f640 1 --2- 172.21.15.72:0/3440480027 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff89006c8f0 0x7ff89006edb0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.946+0000 7ff8aa28f640 1 --2- 172.21.15.72:0/3440480027 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff8a415e770 0x7ff8a406ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.946+0000 7ff8aa28f640 1 --2- 172.21.15.72:0/3440480027 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff8a40cfe30 0x7ff8a41e70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.946+0000 7ff8aa28f640 1 --2- 172.21.15.72:0/3440480027 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8a40b4dc0 0x7ff8a414bf00 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.947+0000 7ff8aa28f640 1 -- 172.21.15.72:0/3440480027 >> 172.21.15.72:0/3440480027 conn(0x7ff8a40d13a0 msgr2=0x7ff8a415da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:03.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.947+0000 7ff8aa28f640 1 -- 172.21.15.72:0/3440480027 shutdown_connections 2024-04-15T15:16:03.949 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:03.947+0000 7ff8aa28f640 1 -- 172.21.15.72:0/3440480027 wait complete. 2024-04-15T15:16:03.959 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-04-15T15:16:04.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.062+0000 7f6faad15640 1 Processor -- start 2024-04-15T15:16:04.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.062+0000 7f6faad15640 1 -- start start 2024-04-15T15:16:04.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.062+0000 7f6faad15640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6fa415c630 0x7f6fa406cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.062+0000 7f6faad15640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f6fa40ccb90 0x7f6fa40ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.062+0000 7f6faad15640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fa406bd50 con 0x7f6fa406e620 2024-04-15T15:16:04.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.062+0000 7f6faad15640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f6fa406d2f0 con 0x7f6fa40ccb90 2024-04-15T15:16:04.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.062+0000 7f6faad15640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f6fa40cd530 con 0x7f6fa415c630 2024-04-15T15:16:04.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.063+0000 7f6f9bfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f6fa40ccb90 0x7f6fa40ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.063+0000 7f6fa92b2640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f6fa406e620 0x7f6fa406eab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:58796/0 (socket says 172.21.15.72:58796) 2024-04-15T15:16:04.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.063+0000 7f6f9bfff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f6fa40ccb90 0x7f6fa40ccff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42032/0 (socket says 172.21.15.72:42032) 2024-04-15T15:16:04.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.063+0000 7f6f9bfff640 1 -- 172.21.15.72:0/2281431055 learned_addr learned my addr 172.21.15.72:0/2281431055 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:04.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.063+0000 7f6f9bfff640 1 -- 172.21.15.72:0/2281431055 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6fa415c630 msgr2=0x7f6fa406cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:04.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.063+0000 7f6f9bfff640 1 --2- 172.21.15.72:0/2281431055 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6fa415c630 0x7f6fa406cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.063+0000 7f6f9bfff640 1 -- 172.21.15.72:0/2281431055 >> v1:172.21.15.72:6789/0 conn(0x7f6fa406e620 legacy=0x7f6fa406eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.063+0000 7f6f9b7fe640 1 -- 172.21.15.72:0/2281431055 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3785779334 0 0) 0x7f6fa406bd50 con 0x7f6fa406e620 2024-04-15T15:16:04.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.063+0000 7f6f9bfff640 1 -- 172.21.15.72:0/2281431055 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fa406bff0 con 0x7f6fa40ccb90 2024-04-15T15:16:04.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.064+0000 7f6f9bfff640 1 --2- 172.21.15.72:0/2281431055 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fa40ccb90 0x7f6fa40ccff0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f6f94009870 tx=0x7f6f9402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5b1f419f55d9b43 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.064+0000 7f6f9b7fe640 1 -- 172.21.15.72:0/2281431055 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f94034070 con 0x7f6fa40ccb90 2024-04-15T15:16:04.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.064+0000 7f6f9b7fe640 1 -- 172.21.15.72:0/2281431055 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f94030840 con 0x7f6fa40ccb90 2024-04-15T15:16:04.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.064+0000 7f6f9b7fe640 1 -- 172.21.15.72:0/2281431055 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f94030c10 con 0x7f6fa40ccb90 2024-04-15T15:16:04.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.064+0000 7f6faad15640 1 -- 172.21.15.72:0/2281431055 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fa40ccb90 msgr2=0x7f6fa40ccff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.064+0000 7f6faad15640 1 --2- 172.21.15.72:0/2281431055 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fa40ccb90 0x7f6fa40ccff0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f6f94009870 tx=0x7f6f9402b980 comp rx=0 tx=0).stop 2024-04-15T15:16:04.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.064+0000 7f6faad15640 1 -- 172.21.15.72:0/2281431055 shutdown_connections 2024-04-15T15:16:04.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.064+0000 7f6faad15640 1 --2- 172.21.15.72:0/2281431055 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fa40ccb90 0x7f6fa40ccff0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.064+0000 7f6faad15640 1 --2- 172.21.15.72:0/2281431055 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6fa415c630 0x7f6fa406cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.064+0000 7f6faad15640 1 -- 172.21.15.72:0/2281431055 >> 172.21.15.72:0/2281431055 conn(0x7f6fa40b31d0 msgr2=0x7f6fa40b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.064+0000 7f6faad15640 1 -- 172.21.15.72:0/2281431055 shutdown_connections 2024-04-15T15:16:04.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.064+0000 7f6faad15640 1 -- 172.21.15.72:0/2281431055 wait complete. 2024-04-15T15:16:04.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.066+0000 7f6faad15640 1 Processor -- start 2024-04-15T15:16:04.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.066+0000 7f6faad15640 1 -- start start 2024-04-15T15:16:04.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.066+0000 7f6faad15640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6fa406e620 0x7f6fa40725c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.066+0000 7f6faad15640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6fa40ccb90 0x7f6fa4157150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.066+0000 7f6faad15640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fa415c630 0x7f6fa4152770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.066+0000 7f6faad15640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa406c5b0 con 0x7f6fa415c630 2024-04-15T15:16:04.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.066+0000 7f6faad15640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa406c920 con 0x7f6fa406e620 2024-04-15T15:16:04.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.066+0000 7f6faad15640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f6fa415ca10 con 0x7f6fa40ccb90 2024-04-15T15:16:04.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.067+0000 7f6fa92b2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fa415c630 0x7f6fa4152770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.067+0000 7f6fa92b2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fa415c630 0x7f6fa4152770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42040/0 (socket says 172.21.15.72:42040) 2024-04-15T15:16:04.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.067+0000 7f6fa92b2640 1 -- 172.21.15.72:0/3914811910 learned_addr learned my addr 172.21.15.72:0/3914811910 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:04.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.067+0000 7f6f9bfff640 1 --2- 172.21.15.72:0/3914811910 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6fa40ccb90 0x7f6fa4157150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.067+0000 7f6fa8ab1640 1 --2- 172.21.15.72:0/3914811910 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6fa406e620 0x7f6fa40725c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.067+0000 7f6fa92b2640 1 -- 172.21.15.72:0/3914811910 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6fa40ccb90 msgr2=0x7f6fa4157150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.067+0000 7f6fa92b2640 1 --2- 172.21.15.72:0/3914811910 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6fa40ccb90 0x7f6fa4157150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.067+0000 7f6fa92b2640 1 -- 172.21.15.72:0/3914811910 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6fa406e620 msgr2=0x7f6fa40725c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.067+0000 7f6fa92b2640 1 --2- 172.21.15.72:0/3914811910 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6fa406e620 0x7f6fa40725c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.067+0000 7f6fa92b2640 1 -- 172.21.15.72:0/3914811910 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fa40caea0 con 0x7f6fa415c630 2024-04-15T15:16:04.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.067+0000 7f6fa8ab1640 1 --2- 172.21.15.72:0/3914811910 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6fa406e620 0x7f6fa40725c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:04.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.067+0000 7f6fa92b2640 1 --2- 172.21.15.72:0/3914811910 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fa415c630 0x7f6fa4152770 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c00c700 tx=0x7f6f9c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.067+0000 7f6f997fa640 1 -- 172.21.15.72:0/3914811910 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c018070 con 0x7f6fa415c630 2024-04-15T15:16:04.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.067+0000 7f6f997fa640 1 -- 172.21.15.72:0/3914811910 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f9c00ce10 con 0x7f6fa415c630 2024-04-15T15:16:04.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.067+0000 7f6faad15640 1 -- 172.21.15.72:0/3914811910 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fa40cb180 con 0x7f6fa415c630 2024-04-15T15:16:04.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.067+0000 7f6f997fa640 1 -- 172.21.15.72:0/3914811910 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f9c00a860 con 0x7f6fa415c630 2024-04-15T15:16:04.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.067+0000 7f6faad15640 1 -- 172.21.15.72:0/3914811910 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fa40cb740 con 0x7f6fa415c630 2024-04-15T15:16:04.075 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.068+0000 7f6faad15640 1 -- 172.21.15.72:0/3914811910 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fa4075c50 con 0x7f6fa415c630 2024-04-15T15:16:04.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.069+0000 7f6f997fa640 1 -- 172.21.15.72:0/3914811910 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6f9c00aa00 con 0x7f6fa415c630 2024-04-15T15:16:04.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.069+0000 7f6f997fa640 1 --2- 172.21.15.72:0/3914811910 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6f9006c8f0 0x7f6f9006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.069+0000 7f6f997fa640 1 -- 172.21.15.72:0/3914811910 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f6f9c08af80 con 0x7f6fa415c630 2024-04-15T15:16:04.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.069+0000 7f6fa8ab1640 1 --2- 172.21.15.72:0/3914811910 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6f9006c8f0 0x7f6f9006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.070+0000 7f6fa8ab1640 1 --2- 172.21.15.72:0/3914811910 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6f9006c8f0 0x7f6f9006edb0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0006fd0 tx=0x7f6fa0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.073+0000 7f6f997fa640 1 -- 172.21.15.72:0/3914811910 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6f9c059580 con 0x7f6fa415c630 2024-04-15T15:16:04.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.319+0000 7f6faad15640 1 -- 172.21.15.72:0/3914811910 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7f6fa40b05c0 con 0x7f6fa415c630 2024-04-15T15:16:04.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.321+0000 7f6f997fa640 1 -- 172.21.15.72:0/3914811910 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) ==== 90+0+0 (secure 0 0 0) 0x7f6f9c058e80 con 0x7f6fa415c630 2024-04-15T15:16:04.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.323+0000 7f6faad15640 1 -- 172.21.15.72:0/3914811910 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6f9006c8f0 msgr2=0x7f6f9006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.323+0000 7f6faad15640 1 --2- 172.21.15.72:0/3914811910 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6f9006c8f0 0x7f6f9006edb0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0006fd0 tx=0x7f6fa0008040 comp rx=0 tx=0).stop 2024-04-15T15:16:04.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.323+0000 7f6faad15640 1 -- 172.21.15.72:0/3914811910 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fa415c630 msgr2=0x7f6fa4152770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.323+0000 7f6faad15640 1 --2- 172.21.15.72:0/3914811910 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fa415c630 0x7f6fa4152770 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c00c700 tx=0x7f6f9c00cbc0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.323+0000 7f6faad15640 1 -- 172.21.15.72:0/3914811910 shutdown_connections 2024-04-15T15:16:04.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.323+0000 7f6faad15640 1 --2- 172.21.15.72:0/3914811910 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6f9006c8f0 0x7f6f9006edb0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.323+0000 7f6faad15640 1 --2- 172.21.15.72:0/3914811910 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6fa415c630 0x7f6fa4152770 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.323+0000 7f6faad15640 1 --2- 172.21.15.72:0/3914811910 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6fa40ccb90 0x7f6fa4157150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.323+0000 7f6faad15640 1 --2- 172.21.15.72:0/3914811910 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6fa406e620 0x7f6fa40725c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.323+0000 7f6faad15640 1 -- 172.21.15.72:0/3914811910 >> 172.21.15.72:0/3914811910 conn(0x7f6fa40b31d0 msgr2=0x7f6fa40afe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.323+0000 7f6faad15640 1 -- 172.21.15.72:0/3914811910 shutdown_connections 2024-04-15T15:16:04.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.323+0000 7f6faad15640 1 -- 172.21.15.72:0/3914811910 wait complete. 2024-04-15T15:16:04.336 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:04.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.437+0000 7f83ea575640 1 Processor -- start 2024-04-15T15:16:04.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.437+0000 7f83ea575640 1 -- start start 2024-04-15T15:16:04.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.438+0000 7f83ea575640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f83e415e7e0 0x7f83e40b67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.438+0000 7f83ea575640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83e4162800 con 0x7f83e40ae360 2024-04-15T15:16:04.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.438+0000 7f83ea575640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83e4162ae0 con 0x7f83e4073290 2024-04-15T15:16:04.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.438+0000 7f83ea575640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f83e4162dc0 con 0x7f83e415e7e0 2024-04-15T15:16:04.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.438+0000 7f83e37fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f83e40ae360 0x7f83e4072b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:58802/0 (socket says 172.21.15.72:58802) 2024-04-15T15:16:04.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.438+0000 7f83e37fe640 1 -- 172.21.15.72:0/2704351596 learned_addr learned my addr 172.21.15.72:0/2704351596 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:04.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.438+0000 7f83e3fff640 1 --2- 172.21.15.72:0/2704351596 >> v2:172.21.15.175:3300/0 conn(0x7f83e415e7e0 0x7f83e40b67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.438+0000 7f83e2ffd640 1 -- 172.21.15.72:0/2704351596 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2165003718 0 0) 0x7f83e4162800 con 0x7f83e40ae360 2024-04-15T15:16:04.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.439+0000 7f83e2ffd640 1 -- 172.21.15.72:0/2704351596 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83cc003520 con 0x7f83e40ae360 2024-04-15T15:16:04.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.439+0000 7f83e2ffd640 1 -- 172.21.15.72:0/2704351596 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1113468597 0 0) 0x7f83e4162ae0 con 0x7f83e4073290 2024-04-15T15:16:04.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.439+0000 7f83e2ffd640 1 -- 172.21.15.72:0/2704351596 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83e4162800 con 0x7f83e4073290 2024-04-15T15:16:04.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.439+0000 7f83e2ffd640 1 -- 172.21.15.72:0/2704351596 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1094722612 0 0) 0x7f83cc003520 con 0x7f83e40ae360 2024-04-15T15:16:04.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.439+0000 7f83e3fff640 1 -- 172.21.15.72:0/2704351596 >> v1:172.21.15.175:6789/0 conn(0x7f83e4073290 legacy=0x7f83e40b46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.439+0000 7f83e3fff640 1 -- 172.21.15.72:0/2704351596 >> v1:172.21.15.72:6789/0 conn(0x7f83e40ae360 legacy=0x7f83e4072b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.439+0000 7f83e3fff640 1 -- 172.21.15.72:0/2704351596 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83e41630a0 con 0x7f83e415e7e0 2024-04-15T15:16:04.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.440+0000 7f83e3fff640 1 --2- 172.21.15.72:0/2704351596 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f83e415e7e0 0x7f83e40b67b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f83d40097a0 tx=0x7f83d402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d44fb277a760274f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.440+0000 7f83e2ffd640 1 -- 172.21.15.72:0/2704351596 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d4038070 con 0x7f83e415e7e0 2024-04-15T15:16:04.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.440+0000 7f83e2ffd640 1 -- 172.21.15.72:0/2704351596 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83d4031500 con 0x7f83e415e7e0 2024-04-15T15:16:04.442 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.440+0000 7f83e2ffd640 1 -- 172.21.15.72:0/2704351596 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83d8001670 con 0x7f83e415e7e0 2024-04-15T15:16:04.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.440+0000 7f83ea575640 1 -- 172.21.15.72:0/2704351596 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f83e415e7e0 msgr2=0x7f83e40b67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.440+0000 7f83ea575640 1 --2- 172.21.15.72:0/2704351596 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f83e415e7e0 0x7f83e40b67b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f83d40097a0 tx=0x7f83d402b820 comp rx=0 tx=0).stop 2024-04-15T15:16:04.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.440+0000 7f83ea575640 1 -- 172.21.15.72:0/2704351596 shutdown_connections 2024-04-15T15:16:04.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.440+0000 7f83ea575640 1 --2- 172.21.15.72:0/2704351596 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f83e415e7e0 0x7f83e40b67b0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.440+0000 7f83ea575640 1 -- 172.21.15.72:0/2704351596 >> 172.21.15.72:0/2704351596 conn(0x7f83e40b31b0 msgr2=0x7f83e40b3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.440+0000 7f83ea575640 1 -- 172.21.15.72:0/2704351596 shutdown_connections 2024-04-15T15:16:04.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.440+0000 7f83ea575640 1 -- 172.21.15.72:0/2704351596 wait complete. 2024-04-15T15:16:04.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.441+0000 7f83ea575640 1 Processor -- start 2024-04-15T15:16:04.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.442+0000 7f83ea575640 1 -- start start 2024-04-15T15:16:04.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.442+0000 7f83ea575640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f83e4073290 0x7f83e4148a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.442+0000 7f83ea575640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f83e40ae360 0x7f83e40b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.442+0000 7f83ea575640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83e415e7e0 0x7f83e40b5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.442+0000 7f83ea575640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f83e4162550 con 0x7f83e415e7e0 2024-04-15T15:16:04.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.442+0000 7f83ea575640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f83e41623d0 con 0x7f83e40ae360 2024-04-15T15:16:04.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.442+0000 7f83ea575640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f83e406fd30 con 0x7f83e4073290 2024-04-15T15:16:04.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.443+0000 7f83e8b12640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83e415e7e0 0x7f83e40b5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.443+0000 7f83e8b12640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83e415e7e0 0x7f83e40b5240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42042/0 (socket says 172.21.15.72:42042) 2024-04-15T15:16:04.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.443+0000 7f83e8b12640 1 -- 172.21.15.72:0/3249189337 learned_addr learned my addr 172.21.15.72:0/3249189337 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:04.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.443+0000 7f83e3fff640 1 --2- 172.21.15.72:0/3249189337 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f83e4073290 0x7f83e4148a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.443+0000 7f83e37fe640 1 --2- 172.21.15.72:0/3249189337 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f83e40ae360 0x7f83e40b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.443+0000 7f83e8b12640 1 -- 172.21.15.72:0/3249189337 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f83e4073290 msgr2=0x7f83e4148a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.443+0000 7f83e8b12640 1 --2- 172.21.15.72:0/3249189337 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f83e4073290 0x7f83e4148a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.443+0000 7f83e8b12640 1 -- 172.21.15.72:0/3249189337 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f83e40ae360 msgr2=0x7f83e40b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.443+0000 7f83e8b12640 1 --2- 172.21.15.72:0/3249189337 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f83e40ae360 0x7f83e40b4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.443+0000 7f83e8b12640 1 -- 172.21.15.72:0/3249189337 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83e414b140 con 0x7f83e415e7e0 2024-04-15T15:16:04.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.443+0000 7f83e8b12640 1 --2- 172.21.15.72:0/3249189337 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83e415e7e0 0x7f83e40b5240 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f83dc00c730 tx=0x7f83dc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.443+0000 7f83e0ff9640 1 -- 172.21.15.72:0/3249189337 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83dc003b70 con 0x7f83e415e7e0 2024-04-15T15:16:04.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.443+0000 7f83e0ff9640 1 -- 172.21.15.72:0/3249189337 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83dc003d10 con 0x7f83e415e7e0 2024-04-15T15:16:04.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.443+0000 7f83ea575640 1 -- 172.21.15.72:0/3249189337 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83e414a0a0 con 0x7f83e415e7e0 2024-04-15T15:16:04.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.443+0000 7f83e0ff9640 1 -- 172.21.15.72:0/3249189337 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83dc00aa90 con 0x7f83e415e7e0 2024-04-15T15:16:04.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.443+0000 7f83ea575640 1 -- 172.21.15.72:0/3249189337 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83e414a660 con 0x7f83e415e7e0 2024-04-15T15:16:04.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.444+0000 7f83ea575640 1 -- 172.21.15.72:0/3249189337 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83e40689e0 con 0x7f83e415e7e0 2024-04-15T15:16:04.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.445+0000 7f83e0ff9640 1 -- 172.21.15.72:0/3249189337 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f83dc00ac30 con 0x7f83e415e7e0 2024-04-15T15:16:04.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.445+0000 7f83e0ff9640 1 --2- 172.21.15.72:0/3249189337 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f83d006c8f0 0x7f83d006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.445+0000 7f83e0ff9640 1 -- 172.21.15.72:0/3249189337 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f83dc08b850 con 0x7f83e415e7e0 2024-04-15T15:16:04.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.445+0000 7f83e3fff640 1 --2- 172.21.15.72:0/3249189337 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f83d006c8f0 0x7f83d006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.445+0000 7f83e3fff640 1 --2- 172.21.15.72:0/3249189337 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f83d006c8f0 0x7f83d006edb0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f83d4035000 tx=0x7f83d4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.452 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.449+0000 7f83e0ff9640 1 -- 172.21.15.72:0/3249189337 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f83dc059ed0 con 0x7f83e415e7e0 2024-04-15T15:16:04.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.626+0000 7f83ea575640 1 -- 172.21.15.72:0/3249189337 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f83e40b47e0 con 0x7f83e415e7e0 2024-04-15T15:16:04.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.627+0000 7f83e0ff9640 1 -- 172.21.15.72:0/3249189337 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) ==== 74+0+13164 (secure 0 0 0) 0x7f83dc0597d0 con 0x7f83e415e7e0 2024-04-15T15:16:04.632 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:16:04.632 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":34,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:16:01.682001+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:04.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.629+0000 7f83ea575640 1 -- 172.21.15.72:0/3249189337 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f83d006c8f0 msgr2=0x7f83d006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.629+0000 7f83ea575640 1 --2- 172.21.15.72:0/3249189337 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f83d006c8f0 0x7f83d006edb0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f83d4035000 tx=0x7f83d4008030 comp rx=0 tx=0).stop 2024-04-15T15:16:04.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.629+0000 7f83ea575640 1 -- 172.21.15.72:0/3249189337 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83e415e7e0 msgr2=0x7f83e40b5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.629+0000 7f83ea575640 1 --2- 172.21.15.72:0/3249189337 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83e415e7e0 0x7f83e40b5240 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f83dc00c730 tx=0x7f83dc00cbf0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.629+0000 7f83ea575640 1 -- 172.21.15.72:0/3249189337 shutdown_connections 2024-04-15T15:16:04.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.629+0000 7f83ea575640 1 --2- 172.21.15.72:0/3249189337 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f83d006c8f0 0x7f83d006edb0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.629+0000 7f83ea575640 1 --2- 172.21.15.72:0/3249189337 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83e415e7e0 0x7f83e40b5240 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.629+0000 7f83ea575640 1 --2- 172.21.15.72:0/3249189337 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f83e40ae360 0x7f83e40b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.629+0000 7f83ea575640 1 --2- 172.21.15.72:0/3249189337 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f83e4073290 0x7f83e4148a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.629+0000 7f83ea575640 1 -- 172.21.15.72:0/3249189337 >> 172.21.15.72:0/3249189337 conn(0x7f83e40b31b0 msgr2=0x7f83e406d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.629+0000 7f83ea575640 1 -- 172.21.15.72:0/3249189337 shutdown_connections 2024-04-15T15:16:04.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.629+0000 7f83ea575640 1 -- 172.21.15.72:0/3249189337 wait complete. 2024-04-15T15:16:04.642 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:39.359102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:16:04.642 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:04.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.747+0000 7fe4fdf9f640 1 Processor -- start 2024-04-15T15:16:04.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.747+0000 7fe4fdf9f640 1 -- start start 2024-04-15T15:16:04.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.748+0000 7fe4fdf9f640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe4f80d0520 0x7fe4f815ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.748+0000 7fe4fdf9f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe4f8162480 0x7fe4f80ce8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.748+0000 7fe4fdf9f640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4f81629f0 con 0x7fe4f815f780 2024-04-15T15:16:04.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.748+0000 7fe4fdf9f640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fe4f8162cd0 con 0x7fe4f8162480 2024-04-15T15:16:04.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.748+0000 7fe4fdf9f640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fe4f8162fb0 con 0x7fe4f80d0520 2024-04-15T15:16:04.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.748+0000 7fe4fd79e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe4f8162480 0x7fe4f80ce8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.748+0000 7fe4fd79e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe4f8162480 0x7fe4f80ce8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42044/0 (socket says 172.21.15.72:42044) 2024-04-15T15:16:04.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.748+0000 7fe4fd79e640 1 -- 172.21.15.72:0/2805378301 learned_addr learned my addr 172.21.15.72:0/2805378301 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:04.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.748+0000 7fe4f7fff640 1 --2- 172.21.15.72:0/2805378301 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe4f80d0520 0x7fe4f815ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.748+0000 7fe4fd79e640 1 -- 172.21.15.72:0/2805378301 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe4f80d0520 msgr2=0x7fe4f815ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:04.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.748+0000 7fe4fd79e640 1 --2- 172.21.15.72:0/2805378301 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe4f80d0520 0x7fe4f815ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.748+0000 7fe4fd79e640 1 -- 172.21.15.72:0/2805378301 >> v1:172.21.15.175:6789/0 conn(0x7fe4f815f780 legacy=0x7fe4f80cfe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:04.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.748+0000 7fe4fd79e640 1 -- 172.21.15.72:0/2805378301 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4f8163290 con 0x7fe4f8162480 2024-04-15T15:16:04.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.748+0000 7fe4f7fff640 1 --2- 172.21.15.72:0/2805378301 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe4f80d0520 0x7fe4f815ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:04.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.749+0000 7fe4fd79e640 1 --2- 172.21.15.72:0/2805378301 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4f8162480 0x7fe4f80ce8b0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fe4f0009e50 tx=0x7fe4f002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=214bc4a7e589345e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.749+0000 7fe4f77fe640 1 -- 172.21.15.72:0/2805378301 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f002ca70 con 0x7fe4f8162480 2024-04-15T15:16:04.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.749+0000 7fe4f77fe640 1 -- 172.21.15.72:0/2805378301 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4f002cc10 con 0x7fe4f8162480 2024-04-15T15:16:04.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.749+0000 7fe4f77fe640 1 -- 172.21.15.72:0/2805378301 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4f00343f0 con 0x7fe4f8162480 2024-04-15T15:16:04.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.749+0000 7fe4fdf9f640 1 -- 172.21.15.72:0/2805378301 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4f8162480 msgr2=0x7fe4f80ce8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.749+0000 7fe4fdf9f640 1 --2- 172.21.15.72:0/2805378301 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4f8162480 0x7fe4f80ce8b0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fe4f0009e50 tx=0x7fe4f002bed0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.749+0000 7fe4fdf9f640 1 -- 172.21.15.72:0/2805378301 shutdown_connections 2024-04-15T15:16:04.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.749+0000 7fe4fdf9f640 1 --2- 172.21.15.72:0/2805378301 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4f8162480 0x7fe4f80ce8b0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.749+0000 7fe4fdf9f640 1 --2- 172.21.15.72:0/2805378301 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe4f80d0520 0x7fe4f815ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.749+0000 7fe4fdf9f640 1 -- 172.21.15.72:0/2805378301 >> 172.21.15.72:0/2805378301 conn(0x7fe4f80b3d70 msgr2=0x7fe4f80b4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.750+0000 7fe4fdf9f640 1 -- 172.21.15.72:0/2805378301 shutdown_connections 2024-04-15T15:16:04.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.750+0000 7fe4fdf9f640 1 -- 172.21.15.72:0/2805378301 wait complete. 2024-04-15T15:16:04.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.751+0000 7fe4fdf9f640 1 Processor -- start 2024-04-15T15:16:04.757 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.751+0000 7fe4fdf9f640 1 -- start start 2024-04-15T15:16:04.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.752+0000 7fe4fdf9f640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe4f80d0520 0x7fe4f80b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.752+0000 7fe4fdf9f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4f815f780 0x7fe4f8154890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.752+0000 7fe4fdf9f640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe4f8162480 0x7fe4f80b6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.760 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.752+0000 7fe4fdf9f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe4f80b37e0 con 0x7fe4f815f780 2024-04-15T15:16:04.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.752+0000 7fe4fdf9f640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fe4f80b3660 con 0x7fe4f8162480 2024-04-15T15:16:04.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.752+0000 7fe4fdf9f640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fe4f815fb60 con 0x7fe4f80d0520 2024-04-15T15:16:04.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.752+0000 7fe4f7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4f815f780 0x7fe4f8154890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.752+0000 7fe4f7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4f815f780 0x7fe4f8154890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42058/0 (socket says 172.21.15.72:42058) 2024-04-15T15:16:04.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.752+0000 7fe4f7fff640 1 -- 172.21.15.72:0/918232656 learned_addr learned my addr 172.21.15.72:0/918232656 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:04.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.752+0000 7fe4f7fff640 1 -- 172.21.15.72:0/918232656 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe4f80d0520 msgr2=0x7fe4f80b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.752+0000 7fe4f7fff640 1 --2- 172.21.15.72:0/918232656 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe4f80d0520 0x7fe4f80b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.752+0000 7fe4f7fff640 1 -- 172.21.15.72:0/918232656 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe4f8162480 msgr2=0x7fe4f80b6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.752+0000 7fe4f7fff640 1 --2- 172.21.15.72:0/918232656 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe4f8162480 0x7fe4f80b6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.752+0000 7fe4f7fff640 1 -- 172.21.15.72:0/918232656 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4f80c6cd0 con 0x7fe4f815f780 2024-04-15T15:16:04.762 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.752+0000 7fe4f7fff640 1 --2- 172.21.15.72:0/918232656 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4f815f780 0x7fe4f8154890 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fe4e400c7f0 tx=0x7fe4e400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.752+0000 7fe4f57fa640 1 -- 172.21.15.72:0/918232656 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4e4015070 con 0x7fe4f815f780 2024-04-15T15:16:04.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.752+0000 7fe4f57fa640 1 -- 172.21.15.72:0/918232656 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4e4007a30 con 0x7fe4f815f780 2024-04-15T15:16:04.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.752+0000 7fe4fdf9f640 1 -- 172.21.15.72:0/918232656 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4f80c6fb0 con 0x7fe4f815f780 2024-04-15T15:16:04.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.752+0000 7fe4f57fa640 1 -- 172.21.15.72:0/918232656 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4e4007e00 con 0x7fe4f815f780 2024-04-15T15:16:04.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.752+0000 7fe4fdf9f640 1 -- 172.21.15.72:0/918232656 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4f80c7570 con 0x7fe4f815f780 2024-04-15T15:16:04.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.754+0000 7fe4fdf9f640 1 -- 172.21.15.72:0/918232656 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4f8075aa0 con 0x7fe4f815f780 2024-04-15T15:16:04.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.754+0000 7fe4f57fa640 1 -- 172.21.15.72:0/918232656 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe4e4018070 con 0x7fe4f815f780 2024-04-15T15:16:04.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.754+0000 7fe4f57fa640 1 --2- 172.21.15.72:0/918232656 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe4dc06c8f0 0x7fe4dc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.755+0000 7fe4f57fa640 1 -- 172.21.15.72:0/918232656 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7fe4e408b970 con 0x7fe4f815f780 2024-04-15T15:16:04.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.755+0000 7fe4fcf9d640 1 --2- 172.21.15.72:0/918232656 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe4dc06c8f0 0x7fe4dc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.755+0000 7fe4fcf9d640 1 --2- 172.21.15.72:0/918232656 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe4dc06c8f0 0x7fe4dc06edb0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec003cc0 tx=0x7fe4ec0079b0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.763 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.759+0000 7fe4f57fa640 1 -- 172.21.15.72:0/918232656 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe4e4059f60 con 0x7fe4f815f780 2024-04-15T15:16:04.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.932+0000 7fe4fdf9f640 1 -- 172.21.15.72:0/918232656 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe4f80b4db0 con 0x7fe4f815f780 2024-04-15T15:16:04.934 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.933+0000 7fe4f57fa640 1 -- 172.21.15.72:0/918232656 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) ==== 93+0+10 (secure 0 0 0) 0x7fe4e4059860 con 0x7fe4f815f780 2024-04-15T15:16:04.935 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-15T15:16:04.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.935+0000 7fe4fdf9f640 1 -- 172.21.15.72:0/918232656 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe4dc06c8f0 msgr2=0x7fe4dc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.935+0000 7fe4fdf9f640 1 --2- 172.21.15.72:0/918232656 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe4dc06c8f0 0x7fe4dc06edb0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec003cc0 tx=0x7fe4ec0079b0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.935+0000 7fe4fdf9f640 1 -- 172.21.15.72:0/918232656 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4f815f780 msgr2=0x7fe4f8154890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.935+0000 7fe4fdf9f640 1 --2- 172.21.15.72:0/918232656 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4f815f780 0x7fe4f8154890 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fe4e400c7f0 tx=0x7fe4e400cbf0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.935+0000 7fe4fdf9f640 1 -- 172.21.15.72:0/918232656 shutdown_connections 2024-04-15T15:16:04.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.935+0000 7fe4fdf9f640 1 --2- 172.21.15.72:0/918232656 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe4dc06c8f0 0x7fe4dc06edb0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.935+0000 7fe4fdf9f640 1 --2- 172.21.15.72:0/918232656 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe4f8162480 0x7fe4f80b6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.935+0000 7fe4fdf9f640 1 --2- 172.21.15.72:0/918232656 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe4f815f780 0x7fe4f8154890 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.935+0000 7fe4fdf9f640 1 --2- 172.21.15.72:0/918232656 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe4f80d0520 0x7fe4f80b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.935+0000 7fe4fdf9f640 1 -- 172.21.15.72:0/918232656 >> 172.21.15.72:0/918232656 conn(0x7fe4f80b3d70 msgr2=0x7fe4f806f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.935+0000 7fe4fdf9f640 1 -- 172.21.15.72:0/918232656 shutdown_connections 2024-04-15T15:16:04.938 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:04.935+0000 7fe4fdf9f640 1 -- 172.21.15.72:0/918232656 wait complete. 2024-04-15T15:16:04.948 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-15T15:16:05.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.050+0000 7f1716ad9640 1 Processor -- start 2024-04-15T15:16:05.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.050+0000 7f1716ad9640 1 -- start start 2024-04-15T15:16:05.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.051+0000 7f1716ad9640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f171015eba0 0x7f171006f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.051+0000 7f1716ad9640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f171006f720 0x7f17100af400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.051+0000 7f1716ad9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f17100af940 0x7f17100ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.051+0000 7f1716ad9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f171015ef80 con 0x7f17100af940 2024-04-15T15:16:05.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.051+0000 7f1716ad9640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f17100ce2b0 con 0x7f171015eba0 2024-04-15T15:16:05.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.051+0000 7f1716ad9640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f17100ce590 con 0x7f171006f720 2024-04-15T15:16:05.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.051+0000 7f1715076640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f17100af940 0x7f17100ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.051+0000 7f1715076640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f17100af940 0x7f17100ccb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42074/0 (socket says 172.21.15.72:42074) 2024-04-15T15:16:05.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.051+0000 7f1715076640 1 -- 172.21.15.72:0/3647765610 learned_addr learned my addr 172.21.15.72:0/3647765610 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:05.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.051+0000 7f170ffff640 1 --2- 172.21.15.72:0/3647765610 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f171006f720 0x7f17100af400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.054 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.051+0000 7f1714875640 1 --2- 172.21.15.72:0/3647765610 >> v2:172.21.15.175:3300/0 conn(0x7f171015eba0 0x7f171006f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.051+0000 7f1715076640 1 -- 172.21.15.72:0/3647765610 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f171006f720 msgr2=0x7f17100af400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:05.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.051+0000 7f1715076640 1 --2- 172.21.15.72:0/3647765610 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f171006f720 0x7f17100af400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.051+0000 7f1715076640 1 -- 172.21.15.72:0/3647765610 >> v2:172.21.15.175:3300/0 conn(0x7f171015eba0 msgr2=0x7f171006f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:05.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.051+0000 7f1715076640 1 --2- 172.21.15.72:0/3647765610 >> v2:172.21.15.175:3300/0 conn(0x7f171015eba0 0x7f171006f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.051+0000 7f1715076640 1 -- 172.21.15.72:0/3647765610 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17100ce870 con 0x7f17100af940 2024-04-15T15:16:05.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.052+0000 7f1715076640 1 --2- 172.21.15.72:0/3647765610 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17100af940 0x7f17100ccb90 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f17100b2e90 tx=0x7f170802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b32b7adf8a74a814 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:05.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.052+0000 7f170f7fe640 1 -- 172.21.15.72:0/3647765610 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f170802c9e0 con 0x7f17100af940 2024-04-15T15:16:05.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.052+0000 7f170f7fe640 1 -- 172.21.15.72:0/3647765610 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f170802cb80 con 0x7f17100af940 2024-04-15T15:16:05.055 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.052+0000 7f170f7fe640 1 -- 172.21.15.72:0/3647765610 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17080343f0 con 0x7f17100af940 2024-04-15T15:16:05.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.053+0000 7f1716ad9640 1 -- 172.21.15.72:0/3647765610 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17100af940 msgr2=0x7f17100ccb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.053+0000 7f1716ad9640 1 --2- 172.21.15.72:0/3647765610 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17100af940 0x7f17100ccb90 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f17100b2e90 tx=0x7f170802be40 comp rx=0 tx=0).stop 2024-04-15T15:16:05.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.053+0000 7f1716ad9640 1 -- 172.21.15.72:0/3647765610 shutdown_connections 2024-04-15T15:16:05.058 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.053+0000 7f1716ad9640 1 --2- 172.21.15.72:0/3647765610 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17100af940 0x7f17100ccb90 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.053+0000 7f1716ad9640 1 --2- 172.21.15.72:0/3647765610 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f171006f720 0x7f17100af400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.053+0000 7f1716ad9640 1 --2- 172.21.15.72:0/3647765610 >> v2:172.21.15.175:3300/0 conn(0x7f171015eba0 0x7f171006f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.053+0000 7f1716ad9640 1 -- 172.21.15.72:0/3647765610 >> 172.21.15.72:0/3647765610 conn(0x7f17100b1dd0 msgr2=0x7f17100b21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:05.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.053+0000 7f1716ad9640 1 -- 172.21.15.72:0/3647765610 shutdown_connections 2024-04-15T15:16:05.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.053+0000 7f1716ad9640 1 -- 172.21.15.72:0/3647765610 wait complete. 2024-04-15T15:16:05.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.054+0000 7f1716ad9640 1 Processor -- start 2024-04-15T15:16:05.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.054+0000 7f1716ad9640 1 -- start start 2024-04-15T15:16:05.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.055+0000 7f1716ad9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17100af940 0x7f171014d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.055+0000 7f1716ad9640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f171015eba0 0x7f17100ca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.055+0000 7f1716ad9640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f17100caa30 0x7f17101f5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.055+0000 7f1716ad9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f17100ceb50 con 0x7f17100af940 2024-04-15T15:16:05.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.055+0000 7f1716ad9640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f17100b2b80 con 0x7f17100caa30 2024-04-15T15:16:05.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.055+0000 7f1716ad9640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f17100cd0d0 con 0x7f171015eba0 2024-04-15T15:16:05.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.055+0000 7f1714875640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17100af940 0x7f171014d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.055+0000 7f1714875640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17100af940 0x7f171014d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42084/0 (socket says 172.21.15.72:42084) 2024-04-15T15:16:05.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.055+0000 7f1714875640 1 -- 172.21.15.72:0/2128519963 learned_addr learned my addr 172.21.15.72:0/2128519963 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:05.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.055+0000 7f1715076640 1 --2- 172.21.15.72:0/2128519963 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f17100caa30 0x7f17101f5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.055+0000 7f170ffff640 1 --2- 172.21.15.72:0/2128519963 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f171015eba0 0x7f17100ca4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.055+0000 7f1714875640 1 -- 172.21.15.72:0/2128519963 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f171015eba0 msgr2=0x7f17100ca4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.055+0000 7f1714875640 1 --2- 172.21.15.72:0/2128519963 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f171015eba0 0x7f17100ca4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.055+0000 7f1714875640 1 -- 172.21.15.72:0/2128519963 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f17100caa30 msgr2=0x7f17101f5790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.055+0000 7f1714875640 1 --2- 172.21.15.72:0/2128519963 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f17100caa30 0x7f17101f5790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.055+0000 7f1714875640 1 -- 172.21.15.72:0/2128519963 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f171015a190 con 0x7f17100af940 2024-04-15T15:16:05.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.055+0000 7f1715076640 1 --2- 172.21.15.72:0/2128519963 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f17100caa30 0x7f17101f5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:05.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.055+0000 7f170ffff640 1 --2- 172.21.15.72:0/2128519963 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f171015eba0 0x7f17100ca4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:05.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.056+0000 7f1714875640 1 --2- 172.21.15.72:0/2128519963 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17100af940 0x7f171014d9b0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f170000c820 tx=0x7f170000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:05.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.056+0000 7f170d7fa640 1 -- 172.21.15.72:0/2128519963 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1700015070 con 0x7f17100af940 2024-04-15T15:16:05.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.056+0000 7f170d7fa640 1 -- 172.21.15.72:0/2128519963 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1700004020 con 0x7f17100af940 2024-04-15T15:16:05.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.056+0000 7f170d7fa640 1 -- 172.21.15.72:0/2128519963 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17000043f0 con 0x7f17100af940 2024-04-15T15:16:05.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.056+0000 7f1716ad9640 1 -- 172.21.15.72:0/2128519963 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f171015a470 con 0x7f17100af940 2024-04-15T15:16:05.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.056+0000 7f1716ad9640 1 -- 172.21.15.72:0/2128519963 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f171015aa30 con 0x7f17100af940 2024-04-15T15:16:05.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.057+0000 7f1716ad9640 1 -- 172.21.15.72:0/2128519963 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17100d12f0 con 0x7f17100af940 2024-04-15T15:16:05.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.057+0000 7f170d7fa640 1 -- 172.21.15.72:0/2128519963 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1700007830 con 0x7f17100af940 2024-04-15T15:16:05.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.058+0000 7f170d7fa640 1 --2- 172.21.15.72:0/2128519963 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f16fc06c8f0 0x7f16fc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.058+0000 7f170ffff640 1 --2- 172.21.15.72:0/2128519963 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f16fc06c8f0 0x7f16fc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.058+0000 7f170d7fa640 1 -- 172.21.15.72:0/2128519963 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f170008b750 con 0x7f17100af940 2024-04-15T15:16:05.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.058+0000 7f170ffff640 1 --2- 172.21.15.72:0/2128519963 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f16fc06c8f0 0x7f16fc06edb0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1704006fd0 tx=0x7f1704008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:05.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.062+0000 7f170d7fa640 1 -- 172.21.15.72:0/2128519963 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f17100d12f0 con 0x7f17100af940 2024-04-15T15:16:05.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.240+0000 7f1716ad9640 1 -- 172.21.15.72:0/2128519963 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f17100689e0 con 0x7f17100af940 2024-04-15T15:16:05.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.241+0000 7f170d7fa640 1 -- 172.21.15.72:0/2128519963 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v34) ==== 84+0+65 (secure 0 0 0) 0x7f1700059d40 con 0x7f17100af940 2024-04-15T15:16:05.242 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:16:05.242 INFO:teuthology.orchestra.run.smithi072.stdout:[ 2024-04-15T15:16:05.242 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:05.243 INFO:teuthology.orchestra.run.smithi072.stdout: "poolnum": 1, 2024-04-15T15:16:05.243 INFO:teuthology.orchestra.run.smithi072.stdout: "poolname": ".mgr" 2024-04-15T15:16:05.243 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:16:05.243 INFO:teuthology.orchestra.run.smithi072.stdout:] 2024-04-15T15:16:05.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.242+0000 7f1716ad9640 1 -- 172.21.15.72:0/2128519963 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f16fc06c8f0 msgr2=0x7f16fc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.242+0000 7f1716ad9640 1 --2- 172.21.15.72:0/2128519963 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f16fc06c8f0 0x7f16fc06edb0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f1704006fd0 tx=0x7f1704008040 comp rx=0 tx=0).stop 2024-04-15T15:16:05.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.242+0000 7f1716ad9640 1 -- 172.21.15.72:0/2128519963 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17100af940 msgr2=0x7f171014d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.242+0000 7f1716ad9640 1 --2- 172.21.15.72:0/2128519963 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17100af940 0x7f171014d9b0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f170000c820 tx=0x7f170000cce0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.243+0000 7f1716ad9640 1 -- 172.21.15.72:0/2128519963 shutdown_connections 2024-04-15T15:16:05.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.243+0000 7f1716ad9640 1 --2- 172.21.15.72:0/2128519963 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f16fc06c8f0 0x7f16fc06edb0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.243+0000 7f1716ad9640 1 --2- 172.21.15.72:0/2128519963 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f17100caa30 0x7f17101f5790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.243+0000 7f1716ad9640 1 --2- 172.21.15.72:0/2128519963 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f171015eba0 0x7f17100ca4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.243+0000 7f1716ad9640 1 --2- 172.21.15.72:0/2128519963 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17100af940 0x7f171014d9b0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.243+0000 7f1716ad9640 1 -- 172.21.15.72:0/2128519963 >> 172.21.15.72:0/2128519963 conn(0x7f17100b1dd0 msgr2=0x7f17101529b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:05.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.243+0000 7f1716ad9640 1 -- 172.21.15.72:0/2128519963 shutdown_connections 2024-04-15T15:16:05.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.243+0000 7f1716ad9640 1 -- 172.21.15.72:0/2128519963 wait complete. 2024-04-15T15:16:05.255 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-15T15:16:05.256 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-15T15:16:05.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.361+0000 7fc3b9235640 1 Processor -- start 2024-04-15T15:16:05.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.361+0000 7fc3b9235640 1 -- start start 2024-04-15T15:16:05.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.361+0000 7fc3b9235640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc3b415e9d0 0x7fc3b40b3a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.361+0000 7fc3b9235640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc3b40d1760 0x7fc3b40d4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.361+0000 7fc3b9235640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3b40d50c0 con 0x7fc3b40d4870 2024-04-15T15:16:05.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.361+0000 7fc3b9235640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fc3b40d22c0 con 0x7fc3b415e9d0 2024-04-15T15:16:05.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.361+0000 7fc3b9235640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fc3b40d0e80 con 0x7fc3b40d1760 2024-04-15T15:16:05.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.362+0000 7fc3b2d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc3b415e9d0 0x7fc3b40b3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.362+0000 7fc3b2d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc3b415e9d0 0x7fc3b40b3a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42090/0 (socket says 172.21.15.72:42090) 2024-04-15T15:16:05.364 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.362+0000 7fc3b2d9d640 1 -- 172.21.15.72:0/896624517 learned_addr learned my addr 172.21.15.72:0/896624517 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:05.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.362+0000 7fc3b2d9d640 1 -- 172.21.15.72:0/896624517 >> v2:172.21.15.175:3300/0 conn(0x7fc3b40d1760 msgr2=0x7fc3b40d4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:05.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.362+0000 7fc3b2d9d640 1 --2- 172.21.15.72:0/896624517 >> v2:172.21.15.175:3300/0 conn(0x7fc3b40d1760 0x7fc3b40d4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.362+0000 7fc3b2d9d640 1 -- 172.21.15.72:0/896624517 >> v1:172.21.15.175:6789/0 conn(0x7fc3b40d4870 legacy=0x7fc3b40b6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:05.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.362+0000 7fc3b2d9d640 1 -- 172.21.15.72:0/896624517 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3b40d05e0 con 0x7fc3b415e9d0 2024-04-15T15:16:05.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.362+0000 7fc3b2d9d640 1 --2- 172.21.15.72:0/896624517 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3b415e9d0 0x7fc3b40b3a60 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fc3b406ede0 tx=0x7fc3a402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a307039eb6e3dbbb server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:05.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.363+0000 7fc3b259c640 1 -- 172.21.15.72:0/896624517 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a4035020 con 0x7fc3b415e9d0 2024-04-15T15:16:05.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.363+0000 7fc3b259c640 1 -- 172.21.15.72:0/896624517 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3a4031870 con 0x7fc3b415e9d0 2024-04-15T15:16:05.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.363+0000 7fc3b259c640 1 -- 172.21.15.72:0/896624517 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a4031c40 con 0x7fc3b415e9d0 2024-04-15T15:16:05.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.363+0000 7fc3b9235640 1 -- 172.21.15.72:0/896624517 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3b415e9d0 msgr2=0x7fc3b40b3a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.363+0000 7fc3b9235640 1 --2- 172.21.15.72:0/896624517 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3b415e9d0 0x7fc3b40b3a60 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fc3b406ede0 tx=0x7fc3a402b910 comp rx=0 tx=0).stop 2024-04-15T15:16:05.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.363+0000 7fc3b9235640 1 -- 172.21.15.72:0/896624517 shutdown_connections 2024-04-15T15:16:05.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.363+0000 7fc3b9235640 1 --2- 172.21.15.72:0/896624517 >> v2:172.21.15.175:3300/0 conn(0x7fc3b40d1760 0x7fc3b40d4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.363+0000 7fc3b9235640 1 --2- 172.21.15.72:0/896624517 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3b415e9d0 0x7fc3b40b3a60 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.363+0000 7fc3b9235640 1 -- 172.21.15.72:0/896624517 >> 172.21.15.72:0/896624517 conn(0x7fc3b409d970 msgr2=0x7fc3b40b9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:05.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.363+0000 7fc3b9235640 1 -- 172.21.15.72:0/896624517 shutdown_connections 2024-04-15T15:16:05.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.363+0000 7fc3b9235640 1 -- 172.21.15.72:0/896624517 wait complete. 2024-04-15T15:16:05.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.365+0000 7fc3b9235640 1 Processor -- start 2024-04-15T15:16:05.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.365+0000 7fc3b9235640 1 -- start start 2024-04-15T15:16:05.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.365+0000 7fc3b9235640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3b40d1760 0x7fc3b40ca190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.365+0000 7fc3b9235640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc3b40d4870 0x7fc3b40b3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.365+0000 7fc3b9235640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc3b415e9d0 0x7fc3b40b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.365+0000 7fc3b9235640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fc3b40d4cc0 con 0x7fc3b40d1760 2024-04-15T15:16:05.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.365+0000 7fc3b9235640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fc3b40d0ba0 con 0x7fc3b40d4870 2024-04-15T15:16:05.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.365+0000 7fc3b9235640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fc3b415edb0 con 0x7fc3b415e9d0 2024-04-15T15:16:05.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.365+0000 7fc3b2d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3b40d1760 0x7fc3b40ca190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.365+0000 7fc3b2d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3b40d1760 0x7fc3b40ca190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42100/0 (socket says 172.21.15.72:42100) 2024-04-15T15:16:05.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.365+0000 7fc3b2d9d640 1 -- 172.21.15.72:0/4278080497 learned_addr learned my addr 172.21.15.72:0/4278080497 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:05.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.366+0000 7fc3b2d9d640 1 -- 172.21.15.72:0/4278080497 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc3b415e9d0 msgr2=0x7fc3b40b6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.366+0000 7fc3b2d9d640 1 --2- 172.21.15.72:0/4278080497 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc3b415e9d0 0x7fc3b40b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.366+0000 7fc3b2d9d640 1 -- 172.21.15.72:0/4278080497 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc3b40d4870 msgr2=0x7fc3b40b3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.366+0000 7fc3b2d9d640 1 --2- 172.21.15.72:0/4278080497 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc3b40d4870 0x7fc3b40b3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.366+0000 7fc3b2d9d640 1 -- 172.21.15.72:0/4278080497 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3b40b4850 con 0x7fc3b40d1760 2024-04-15T15:16:05.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.366+0000 7fc3b2d9d640 1 --2- 172.21.15.72:0/4278080497 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3b40d1760 0x7fc3b40ca190 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fc3a402b950 tx=0x7fc3a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:05.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.366+0000 7fc3ab7fe640 1 -- 172.21.15.72:0/4278080497 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a403ad60 con 0x7fc3b40d1760 2024-04-15T15:16:05.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.366+0000 7fc3ab7fe640 1 -- 172.21.15.72:0/4278080497 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3a40443f0 con 0x7fc3b40d1760 2024-04-15T15:16:05.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.366+0000 7fc3ab7fe640 1 -- 172.21.15.72:0/4278080497 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a40447e0 con 0x7fc3b40d1760 2024-04-15T15:16:05.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.366+0000 7fc3b9235640 1 -- 172.21.15.72:0/4278080497 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3b40b4290 con 0x7fc3b40d1760 2024-04-15T15:16:05.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.366+0000 7fc3b9235640 1 -- 172.21.15.72:0/4278080497 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3b40b5a40 con 0x7fc3b40d1760 2024-04-15T15:16:05.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.367+0000 7fc3b9235640 1 -- 172.21.15.72:0/4278080497 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3b40689e0 con 0x7fc3b40d1760 2024-04-15T15:16:05.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.368+0000 7fc3ab7fe640 1 -- 172.21.15.72:0/4278080497 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc3a4038020 con 0x7fc3b40d1760 2024-04-15T15:16:05.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.368+0000 7fc3ab7fe640 1 --2- 172.21.15.72:0/4278080497 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc39406c8f0 0x7fc39406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.368+0000 7fc3ab7fe640 1 -- 172.21.15.72:0/4278080497 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7fc3a40adbd0 con 0x7fc3b40d1760 2024-04-15T15:16:05.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.368+0000 7fc3abfff640 1 --2- 172.21.15.72:0/4278080497 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc39406c8f0 0x7fc39406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.368+0000 7fc3abfff640 1 --2- 172.21.15.72:0/4278080497 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc39406c8f0 0x7fc39406edb0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc3b40d3a60 tx=0x7fc39c006bf0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:05.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.372+0000 7fc3ab7fe640 1 -- 172.21.15.72:0/4278080497 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc3a403f070 con 0x7fc3b40d1760 2024-04-15T15:16:05.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.547+0000 7fc3b9235640 1 -- 172.21.15.72:0/4278080497 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fc3b40d1f60 con 0x7fc3b40d1760 2024-04-15T15:16:05.683 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.681+0000 7fc3ab7fe640 1 -- 172.21.15.72:0/4278080497 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v35) ==== 138+0+0 (secure 0 0 0) 0x7fc3a407c1c0 con 0x7fc3b40d1760 2024-04-15T15:16:05.683 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_metadata' created 2024-04-15T15:16:05.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.684+0000 7fc3b9235640 1 -- 172.21.15.72:0/4278080497 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc39406c8f0 msgr2=0x7fc39406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.684+0000 7fc3b9235640 1 --2- 172.21.15.72:0/4278080497 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc39406c8f0 0x7fc39406edb0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc3b40d3a60 tx=0x7fc39c006bf0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.684+0000 7fc3b9235640 1 -- 172.21.15.72:0/4278080497 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3b40d1760 msgr2=0x7fc3b40ca190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.684+0000 7fc3b9235640 1 --2- 172.21.15.72:0/4278080497 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3b40d1760 0x7fc3b40ca190 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fc3a402b950 tx=0x7fc3a40046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.684+0000 7fc3b9235640 1 -- 172.21.15.72:0/4278080497 shutdown_connections 2024-04-15T15:16:05.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.684+0000 7fc3b9235640 1 --2- 172.21.15.72:0/4278080497 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc39406c8f0 0x7fc39406edb0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.684+0000 7fc3b9235640 1 --2- 172.21.15.72:0/4278080497 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc3b415e9d0 0x7fc3b40b6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.684+0000 7fc3b9235640 1 --2- 172.21.15.72:0/4278080497 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc3b40d4870 0x7fc3b40b3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.684+0000 7fc3b9235640 1 --2- 172.21.15.72:0/4278080497 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3b40d1760 0x7fc3b40ca190 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.684+0000 7fc3b9235640 1 -- 172.21.15.72:0/4278080497 >> 172.21.15.72:0/4278080497 conn(0x7fc3b409d970 msgr2=0x7fc3b40b32e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:05.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.684+0000 7fc3b9235640 1 -- 172.21.15.72:0/4278080497 shutdown_connections 2024-04-15T15:16:05.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.684+0000 7fc3b9235640 1 -- 172.21.15.72:0/4278080497 wait complete. 2024-04-15T15:16:05.702 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-15T15:16:05.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.812+0000 7f87d9414640 1 Processor -- start 2024-04-15T15:16:05.814 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.812+0000 7f87d9414640 1 -- start start 2024-04-15T15:16:05.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.813+0000 7f87d9414640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f87d40d08f0 0x7f87d40cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.813+0000 7f87d9414640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87d406bbb0 con 0x7f87d40ce6d0 2024-04-15T15:16:05.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.813+0000 7f87d9414640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87d406be90 con 0x7f87d40ce2f0 2024-04-15T15:16:05.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.813+0000 7f87d9414640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f87d406b960 con 0x7f87d40d08f0 2024-04-15T15:16:05.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.813+0000 7f87d27fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f87d40d08f0 0x7f87d40cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.813+0000 7f87d27fc640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f87d40d08f0 0x7f87d40cc0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42106/0 (socket says 172.21.15.72:42106) 2024-04-15T15:16:05.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.813+0000 7f87d27fc640 1 -- 172.21.15.72:0/4024234405 learned_addr learned my addr 172.21.15.72:0/4024234405 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:05.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.813+0000 7f87d37fe640 1 --1- 172.21.15.72:0/4024234405 >> v1:172.21.15.72:6789/0 conn(0x7f87d40ce6d0 0x7f87d4155600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:58816/0 (socket says 172.21.15.72:58816) 2024-04-15T15:16:05.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.813+0000 7f87d27fc640 1 -- 172.21.15.72:0/4024234405 >> v1:172.21.15.175:6789/0 conn(0x7f87d40ce2f0 legacy=0x7f87d415c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:05.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.813+0000 7f87d27fc640 1 -- 172.21.15.72:0/4024234405 >> v1:172.21.15.72:6789/0 conn(0x7f87d40ce6d0 legacy=0x7f87d4155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.813+0000 7f87d27fc640 1 -- 172.21.15.72:0/4024234405 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87d406c170 con 0x7f87d40d08f0 2024-04-15T15:16:05.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.814+0000 7f87d27fc640 1 --2- 172.21.15.72:0/4024234405 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87d40d08f0 0x7f87d40cc0b0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f87d406bae0 tx=0x7f87c802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2e635b16407cc72 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:05.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.814+0000 7f87d1ffb640 1 -- 172.21.15.72:0/4024234405 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c8034070 con 0x7f87d40d08f0 2024-04-15T15:16:05.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.814+0000 7f87d1ffb640 1 -- 172.21.15.72:0/4024234405 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87c8030840 con 0x7f87d40d08f0 2024-04-15T15:16:05.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.814+0000 7f87d1ffb640 1 -- 172.21.15.72:0/4024234405 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c8030c10 con 0x7f87d40d08f0 2024-04-15T15:16:05.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.815+0000 7f87d9414640 1 -- 172.21.15.72:0/4024234405 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87d40d08f0 msgr2=0x7f87d40cc0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.815+0000 7f87d9414640 1 --2- 172.21.15.72:0/4024234405 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87d40d08f0 0x7f87d40cc0b0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f87d406bae0 tx=0x7f87c802b980 comp rx=0 tx=0).stop 2024-04-15T15:16:05.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.815+0000 7f87d9414640 1 -- 172.21.15.72:0/4024234405 shutdown_connections 2024-04-15T15:16:05.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.815+0000 7f87d9414640 1 --2- 172.21.15.72:0/4024234405 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87d40d08f0 0x7f87d40cc0b0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.815+0000 7f87d9414640 1 -- 172.21.15.72:0/4024234405 >> 172.21.15.72:0/4024234405 conn(0x7f87d40b31d0 msgr2=0x7f87d40b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:05.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.815+0000 7f87d9414640 1 -- 172.21.15.72:0/4024234405 shutdown_connections 2024-04-15T15:16:05.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.815+0000 7f87d9414640 1 -- 172.21.15.72:0/4024234405 wait complete. 2024-04-15T15:16:05.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.816+0000 7f87d9414640 1 Processor -- start 2024-04-15T15:16:05.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.816+0000 7f87d9414640 1 -- start start 2024-04-15T15:16:05.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.817+0000 7f87d9414640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f87d40ce2f0 0x7f87d41560f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.817+0000 7f87d9414640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87d40ce6d0 0x7f87d4154cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.817+0000 7f87d9414640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87d40d08f0 0x7f87d4145980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.817+0000 7f87d9414640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f87d406ef50 con 0x7f87d40ce6d0 2024-04-15T15:16:05.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.817+0000 7f87d9414640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f87d406edd0 con 0x7f87d40d08f0 2024-04-15T15:16:05.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.817+0000 7f87d9414640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f87d406f0d0 con 0x7f87d40ce2f0 2024-04-15T15:16:05.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.817+0000 7f87d27fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87d40ce6d0 0x7f87d4154cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.817+0000 7f87d27fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87d40ce6d0 0x7f87d4154cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42118/0 (socket says 172.21.15.72:42118) 2024-04-15T15:16:05.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.817+0000 7f87d27fc640 1 -- 172.21.15.72:0/3960567587 learned_addr learned my addr 172.21.15.72:0/3960567587 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:05.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.817+0000 7f87d37fe640 1 --2- 172.21.15.72:0/3960567587 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87d40d08f0 0x7f87d4145980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.817+0000 7f87d2ffd640 1 --2- 172.21.15.72:0/3960567587 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f87d40ce2f0 0x7f87d41560f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.817+0000 7f87d27fc640 1 -- 172.21.15.72:0/3960567587 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f87d40ce2f0 msgr2=0x7f87d41560f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.817+0000 7f87d27fc640 1 --2- 172.21.15.72:0/3960567587 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f87d40ce2f0 0x7f87d41560f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.817+0000 7f87d27fc640 1 -- 172.21.15.72:0/3960567587 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87d40d08f0 msgr2=0x7f87d4145980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.817+0000 7f87d27fc640 1 --2- 172.21.15.72:0/3960567587 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87d40d08f0 0x7f87d4145980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.817+0000 7f87d27fc640 1 -- 172.21.15.72:0/3960567587 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87d4155770 con 0x7f87d40ce6d0 2024-04-15T15:16:05.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.817+0000 7f87d2ffd640 1 --2- 172.21.15.72:0/3960567587 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f87d40ce2f0 0x7f87d41560f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:05.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.817+0000 7f87d27fc640 1 --2- 172.21.15.72:0/3960567587 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87d40ce6d0 0x7f87d4154cb0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f87c802b9c0 tx=0x7f87c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:05.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.818+0000 7f87bb7fe640 1 -- 172.21.15.72:0/3960567587 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c8034040 con 0x7f87d40ce6d0 2024-04-15T15:16:05.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.818+0000 7f87bb7fe640 1 -- 172.21.15.72:0/3960567587 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87c8039d90 con 0x7f87d40ce6d0 2024-04-15T15:16:05.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.818+0000 7f87bb7fe640 1 -- 172.21.15.72:0/3960567587 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87c8043630 con 0x7f87d40ce6d0 2024-04-15T15:16:05.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.818+0000 7f87d9414640 1 -- 172.21.15.72:0/3960567587 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87d41551f0 con 0x7f87d40ce6d0 2024-04-15T15:16:05.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.818+0000 7f87d9414640 1 -- 172.21.15.72:0/3960567587 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87d4155490 con 0x7f87d40ce6d0 2024-04-15T15:16:05.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.819+0000 7f87d9414640 1 -- 172.21.15.72:0/3960567587 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87d4075c50 con 0x7f87d40ce6d0 2024-04-15T15:16:05.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.819+0000 7f87bb7fe640 1 -- 172.21.15.72:0/3960567587 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f87c8037070 con 0x7f87d40ce6d0 2024-04-15T15:16:05.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.820+0000 7f87bb7fe640 1 --2- 172.21.15.72:0/3960567587 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f87c006c8f0 0x7f87c006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.820+0000 7f87bb7fe640 1 -- 172.21.15.72:0/3960567587 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5619+0+0 (secure 0 0 0) 0x7f87c80adcf0 con 0x7f87d40ce6d0 2024-04-15T15:16:05.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.820+0000 7f87d2ffd640 1 --2- 172.21.15.72:0/3960567587 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f87c006c8f0 0x7f87c006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.820+0000 7f87d2ffd640 1 --2- 172.21.15.72:0/3960567587 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f87c006c8f0 0x7f87c006edb0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f87c4007fd0 tx=0x7f87c4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:05.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.824+0000 7f87bb7fe640 1 -- 172.21.15.72:0/3960567587 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f87c80b2070 con 0x7f87d40ce6d0 2024-04-15T15:16:06.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:05.998+0000 7f87d9414640 1 -- 172.21.15.72:0/3960567587 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f87d406cab0 con 0x7f87d40ce6d0 2024-04-15T15:16:06.685 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.682+0000 7f87bb7fe640 1 -- 172.21.15.72:0/3960567587 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v36) ==== 170+0+0 (secure 0 0 0) 0x7f87c803e070 con 0x7f87d40ce6d0 2024-04-15T15:16:06.685 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_data' created 2024-04-15T15:16:06.688 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.686+0000 7f87d9414640 1 -- 172.21.15.72:0/3960567587 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f87c006c8f0 msgr2=0x7f87c006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.686+0000 7f87d9414640 1 --2- 172.21.15.72:0/3960567587 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f87c006c8f0 0x7f87c006edb0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f87c4007fd0 tx=0x7f87c4009040 comp rx=0 tx=0).stop 2024-04-15T15:16:06.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.686+0000 7f87d9414640 1 -- 172.21.15.72:0/3960567587 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87d40ce6d0 msgr2=0x7f87d4154cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.686+0000 7f87d9414640 1 --2- 172.21.15.72:0/3960567587 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87d40ce6d0 0x7f87d4154cb0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f87c802b9c0 tx=0x7f87c80046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.686+0000 7f87d9414640 1 -- 172.21.15.72:0/3960567587 shutdown_connections 2024-04-15T15:16:06.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.686+0000 7f87d9414640 1 --2- 172.21.15.72:0/3960567587 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f87c006c8f0 0x7f87c006edb0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.686+0000 7f87d9414640 1 --2- 172.21.15.72:0/3960567587 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87d40d08f0 0x7f87d4145980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.686+0000 7f87d9414640 1 --2- 172.21.15.72:0/3960567587 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87d40ce6d0 0x7f87d4154cb0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.686+0000 7f87d9414640 1 --2- 172.21.15.72:0/3960567587 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f87d40ce2f0 0x7f87d41560f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.686+0000 7f87d9414640 1 -- 172.21.15.72:0/3960567587 >> 172.21.15.72:0/3960567587 conn(0x7f87d40b31d0 msgr2=0x7f87d40ccb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:06.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.686+0000 7f87d9414640 1 -- 172.21.15.72:0/3960567587 shutdown_connections 2024-04-15T15:16:06.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.686+0000 7f87d9414640 1 -- 172.21.15.72:0/3960567587 wait complete. 2024-04-15T15:16:06.712 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-15T15:16:06.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.819+0000 7f6f4b982640 1 Processor -- start 2024-04-15T15:16:06.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.820+0000 7f6f4b982640 1 -- start start 2024-04-15T15:16:06.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.820+0000 7f6f4b982640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6f440ce2f0 0x7f6f4415c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.820+0000 7f6f4b982640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f6f440d08f0 0x7f6f440cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.820+0000 7f6f4b982640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f4406bbb0 con 0x7f6f440ce6d0 2024-04-15T15:16:06.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.821+0000 7f6f4b982640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f6f4406b960 con 0x7f6f440d08f0 2024-04-15T15:16:06.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.821+0000 7f6f4b982640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f6f4406be90 con 0x7f6f440ce2f0 2024-04-15T15:16:06.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.821+0000 7f6f48f1d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f6f440d08f0 0x7f6f440cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.821+0000 7f6f48f1d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f6f440d08f0 0x7f6f440cc0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42130/0 (socket says 172.21.15.72:42130) 2024-04-15T15:16:06.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.821+0000 7f6f48f1d640 1 -- 172.21.15.72:0/2762621927 learned_addr learned my addr 172.21.15.72:0/2762621927 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:06.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.821+0000 7f6f4971e640 1 --2- 172.21.15.72:0/2762621927 >> v2:172.21.15.175:3300/0 conn(0x7f6f440ce2f0 0x7f6f4415c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.821+0000 7f6f48f1d640 1 -- 172.21.15.72:0/2762621927 >> v2:172.21.15.175:3300/0 conn(0x7f6f440ce2f0 msgr2=0x7f6f4415c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:06.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.821+0000 7f6f48f1d640 1 --2- 172.21.15.72:0/2762621927 >> v2:172.21.15.175:3300/0 conn(0x7f6f440ce2f0 0x7f6f4415c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.821+0000 7f6f48f1d640 1 -- 172.21.15.72:0/2762621927 >> v1:172.21.15.175:6789/0 conn(0x7f6f440ce6d0 legacy=0x7f6f44155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.824 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.821+0000 7f6f48f1d640 1 -- 172.21.15.72:0/2762621927 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f4406c170 con 0x7f6f440d08f0 2024-04-15T15:16:06.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.822+0000 7f6f48f1d640 1 --2- 172.21.15.72:0/2762621927 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f440d08f0 0x7f6f440cc0b0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f6f440cdf10 tx=0x7f6f3802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24b8184dd1a757dd server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:06.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.822+0000 7f6f43fff640 1 -- 172.21.15.72:0/2762621927 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f3802cb20 con 0x7f6f440d08f0 2024-04-15T15:16:06.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.822+0000 7f6f43fff640 1 -- 172.21.15.72:0/2762621927 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f3802ccc0 con 0x7f6f440d08f0 2024-04-15T15:16:06.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.822+0000 7f6f43fff640 1 -- 172.21.15.72:0/2762621927 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f38034610 con 0x7f6f440d08f0 2024-04-15T15:16:06.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.822+0000 7f6f4b982640 1 -- 172.21.15.72:0/2762621927 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f440d08f0 msgr2=0x7f6f440cc0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.822+0000 7f6f4b982640 1 --2- 172.21.15.72:0/2762621927 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f440d08f0 0x7f6f440cc0b0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f6f440cdf10 tx=0x7f6f3802bf80 comp rx=0 tx=0).stop 2024-04-15T15:16:06.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.822+0000 7f6f4b982640 1 -- 172.21.15.72:0/2762621927 shutdown_connections 2024-04-15T15:16:06.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.822+0000 7f6f4b982640 1 --2- 172.21.15.72:0/2762621927 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f440d08f0 0x7f6f440cc0b0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.822+0000 7f6f4b982640 1 --2- 172.21.15.72:0/2762621927 >> v2:172.21.15.175:3300/0 conn(0x7f6f440ce2f0 0x7f6f4415c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.822+0000 7f6f4b982640 1 -- 172.21.15.72:0/2762621927 >> 172.21.15.72:0/2762621927 conn(0x7f6f440b31d0 msgr2=0x7f6f440b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:06.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.823+0000 7f6f4b982640 1 -- 172.21.15.72:0/2762621927 shutdown_connections 2024-04-15T15:16:06.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.823+0000 7f6f4b982640 1 -- 172.21.15.72:0/2762621927 wait complete. 2024-04-15T15:16:06.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.824+0000 7f6f4b982640 1 Processor -- start 2024-04-15T15:16:06.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.824+0000 7f6f4b982640 1 -- start start 2024-04-15T15:16:06.830 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.824+0000 7f6f4b982640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f440ce2f0 0x7f6f44156100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.824+0000 7f6f4b982640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6f440ce6d0 0x7f6f44157bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.824+0000 7f6f4b982640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f440d08f0 0x7f6f44149600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.825+0000 7f6f4b982640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6f44155d10 con 0x7f6f440d08f0 2024-04-15T15:16:06.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.825+0000 7f6f4b982640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f6f4415ca70 con 0x7f6f440ce2f0 2024-04-15T15:16:06.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.825+0000 7f6f4b982640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f6f44155e90 con 0x7f6f440ce6d0 2024-04-15T15:16:06.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.825+0000 7f6f49f1f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f440d08f0 0x7f6f44149600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.825+0000 7f6f49f1f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f440d08f0 0x7f6f44149600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42144/0 (socket says 172.21.15.72:42144) 2024-04-15T15:16:06.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.825+0000 7f6f49f1f640 1 -- 172.21.15.72:0/4006066329 learned_addr learned my addr 172.21.15.72:0/4006066329 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:06.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.825+0000 7f6f48f1d640 1 --2- 172.21.15.72:0/4006066329 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6f440ce6d0 0x7f6f44157bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.825+0000 7f6f49f1f640 1 -- 172.21.15.72:0/4006066329 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6f440ce6d0 msgr2=0x7f6f44157bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.825+0000 7f6f49f1f640 1 --2- 172.21.15.72:0/4006066329 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6f440ce6d0 0x7f6f44157bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.825+0000 7f6f49f1f640 1 -- 172.21.15.72:0/4006066329 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f440ce2f0 msgr2=0x7f6f44156100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.825+0000 7f6f4971e640 1 --2- 172.21.15.72:0/4006066329 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f440ce2f0 0x7f6f44156100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.825+0000 7f6f49f1f640 1 --2- 172.21.15.72:0/4006066329 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f440ce2f0 0x7f6f44156100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.825+0000 7f6f49f1f640 1 -- 172.21.15.72:0/4006066329 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f441e6ff0 con 0x7f6f440d08f0 2024-04-15T15:16:06.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.825+0000 7f6f4971e640 1 --2- 172.21.15.72:0/4006066329 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f440ce2f0 0x7f6f44156100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:06.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.825+0000 7f6f49f1f640 1 --2- 172.21.15.72:0/4006066329 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f440d08f0 0x7f6f44149600 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f6f3400c400 tx=0x7f6f3400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:06.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.825+0000 7f6f41ffb640 1 -- 172.21.15.72:0/4006066329 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f34017070 con 0x7f6f440d08f0 2024-04-15T15:16:06.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.825+0000 7f6f41ffb640 1 -- 172.21.15.72:0/4006066329 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f340052d0 con 0x7f6f440d08f0 2024-04-15T15:16:06.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.825+0000 7f6f4b982640 1 -- 172.21.15.72:0/4006066329 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f441e72d0 con 0x7f6f440d08f0 2024-04-15T15:16:06.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.825+0000 7f6f41ffb640 1 -- 172.21.15.72:0/4006066329 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f34003810 con 0x7f6f440d08f0 2024-04-15T15:16:06.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.826+0000 7f6f4b982640 1 -- 172.21.15.72:0/4006066329 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f441e7b70 con 0x7f6f440d08f0 2024-04-15T15:16:06.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.826+0000 7f6f4b982640 1 -- 172.21.15.72:0/4006066329 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f44075c50 con 0x7f6f440d08f0 2024-04-15T15:16:06.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.827+0000 7f6f41ffb640 1 -- 172.21.15.72:0/4006066329 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6f3401a070 con 0x7f6f440d08f0 2024-04-15T15:16:06.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.827+0000 7f6f41ffb640 1 --2- 172.21.15.72:0/4006066329 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6f3006c8f0 0x7f6f3006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.828+0000 7f6f41ffb640 1 -- 172.21.15.72:0/4006066329 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6001+0+0 (secure 0 0 0) 0x7f6f3408b050 con 0x7f6f440d08f0 2024-04-15T15:16:06.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.828+0000 7f6f4971e640 1 --2- 172.21.15.72:0/4006066329 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6f3006c8f0 0x7f6f3006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.828+0000 7f6f4971e640 1 --2- 172.21.15.72:0/4006066329 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6f3006c8f0 0x7f6f3006edb0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c006fd0 tx=0x7f6f3c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:06.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:06.832+0000 7f6f41ffb640 1 -- 172.21.15.72:0/4006066329 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6f34059350 con 0x7f6f440d08f0 2024-04-15T15:16:07.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.048+0000 7f6f4b982640 1 -- 172.21.15.72:0/4006066329 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f6f4406cab0 con 0x7f6f440d08f0 2024-04-15T15:16:07.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.701+0000 7f6f41ffb640 1 -- 172.21.15.72:0/4006066329 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v14) ==== 172+0+0 (secure 0 0 0) 0x7f6f34058c50 con 0x7f6f440d08f0 2024-04-15T15:16:07.704 INFO:teuthology.orchestra.run.smithi072.stderr:new fs with metadata pool 5 and data pool 6 2024-04-15T15:16:07.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.713+0000 7f6f4b982640 1 -- 172.21.15.72:0/4006066329 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6f3006c8f0 msgr2=0x7f6f3006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.713+0000 7f6f4b982640 1 --2- 172.21.15.72:0/4006066329 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6f3006c8f0 0x7f6f3006edb0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c006fd0 tx=0x7f6f3c008040 comp rx=0 tx=0).stop 2024-04-15T15:16:07.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.713+0000 7f6f4b982640 1 -- 172.21.15.72:0/4006066329 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f440d08f0 msgr2=0x7f6f44149600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.713+0000 7f6f4b982640 1 --2- 172.21.15.72:0/4006066329 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f440d08f0 0x7f6f44149600 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f6f3400c400 tx=0x7f6f3400c8c0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.714+0000 7f6f4b982640 1 -- 172.21.15.72:0/4006066329 shutdown_connections 2024-04-15T15:16:07.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.714+0000 7f6f4b982640 1 --2- 172.21.15.72:0/4006066329 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6f3006c8f0 0x7f6f3006edb0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.714+0000 7f6f4b982640 1 --2- 172.21.15.72:0/4006066329 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6f440d08f0 0x7f6f44149600 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.714+0000 7f6f4b982640 1 --2- 172.21.15.72:0/4006066329 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6f440ce6d0 0x7f6f44157bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.714+0000 7f6f4b982640 1 --2- 172.21.15.72:0/4006066329 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6f440ce2f0 0x7f6f44156100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.714+0000 7f6f4b982640 1 -- 172.21.15.72:0/4006066329 >> 172.21.15.72:0/4006066329 conn(0x7f6f440b31d0 msgr2=0x7f6f440ad350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:07.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.715+0000 7f6f4b982640 1 -- 172.21.15.72:0/4006066329 shutdown_connections 2024-04-15T15:16:07.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.716+0000 7f6f4b982640 1 -- 172.21.15.72:0/4006066329 wait complete. 2024-04-15T15:16:07.735 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:07.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.836+0000 7fc1fde4e640 1 Processor -- start 2024-04-15T15:16:07.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.836+0000 7fc1fde4e640 1 -- start start 2024-04-15T15:16:07.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.837+0000 7fc1fde4e640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc1f806fc20 0x7fc1f80b4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.837+0000 7fc1fde4e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1f8141e70 con 0x7fc1f813f610 2024-04-15T15:16:07.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.837+0000 7fc1fde4e640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1f8142150 con 0x7fc1f8160240 2024-04-15T15:16:07.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.837+0000 7fc1fde4e640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fc1f8142430 con 0x7fc1f806fc20 2024-04-15T15:16:07.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.837+0000 7fc1f77fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fc1f813f610 0x7fc1f806f510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:58822/0 (socket says 172.21.15.72:58822) 2024-04-15T15:16:07.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.837+0000 7fc1f77fe640 1 -- 172.21.15.72:0/3730399943 learned_addr learned my addr 172.21.15.72:0/3730399943 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:07.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.837+0000 7fc1f6ffd640 1 --2- 172.21.15.72:0/3730399943 >> v2:172.21.15.175:3300/0 conn(0x7fc1f806fc20 0x7fc1f80b4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.837+0000 7fc1f67fc640 1 -- 172.21.15.72:0/3730399943 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 859363699 0 0) 0x7fc1f8141e70 con 0x7fc1f813f610 2024-04-15T15:16:07.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.837+0000 7fc1f67fc640 1 -- 172.21.15.72:0/3730399943 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1e0003520 con 0x7fc1f813f610 2024-04-15T15:16:07.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.837+0000 7fc1f67fc640 1 -- 172.21.15.72:0/3730399943 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 449429399 0 0) 0x7fc1e0003520 con 0x7fc1f813f610 2024-04-15T15:16:07.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.837+0000 7fc1f67fc640 1 -- 172.21.15.72:0/3730399943 >> v2:172.21.15.175:3300/0 conn(0x7fc1f806fc20 msgr2=0x7fc1f80b4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:07.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.837+0000 7fc1f67fc640 1 --2- 172.21.15.72:0/3730399943 >> v2:172.21.15.175:3300/0 conn(0x7fc1f806fc20 0x7fc1f80b4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.837+0000 7fc1f67fc640 1 -- 172.21.15.72:0/3730399943 >> v1:172.21.15.175:6789/0 conn(0x7fc1f8160240 legacy=0x7fc1f815cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.837+0000 7fc1f67fc640 1 -- 172.21.15.72:0/3730399943 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1f8142710 con 0x7fc1f813f610 2024-04-15T15:16:07.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.837+0000 7fc1f67fc640 1 -- 172.21.15.72:0/3730399943 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7fc1e8001a10 con 0x7fc1f813f610 2024-04-15T15:16:07.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.837+0000 7fc1f67fc640 1 -- 172.21.15.72:0/3730399943 >> v1:172.21.15.72:6789/0 conn(0x7fc1f813f610 legacy=0x7fc1f806f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.837+0000 7fc1f67fc640 1 --2- 172.21.15.72:0/3730399943 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1e00039b0 0x7fc1e0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.837+0000 7fc1f67fc640 1 --2- 172.21.15.72:0/3730399943 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc1e0004800 0x7fc1e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.838+0000 7fc1f67fc640 1 --2- 172.21.15.72:0/3730399943 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1e00051c0 0x7fc1e000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.838+0000 7fc1f67fc640 1 -- 172.21.15.72:0/3730399943 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fc1f8142430 con 0x7fc1e00039b0 2024-04-15T15:16:07.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.838+0000 7fc1f67fc640 1 -- 172.21.15.72:0/3730399943 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fc1e000c060 con 0x7fc1e00051c0 2024-04-15T15:16:07.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.838+0000 7fc1f67fc640 1 -- 172.21.15.72:0/3730399943 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fc1e000c260 con 0x7fc1e0004800 2024-04-15T15:16:07.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.838+0000 7fc1f77fe640 1 --2- 172.21.15.72:0/3730399943 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1e00039b0 0x7fc1e0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.838+0000 7fc1f6ffd640 1 --2- 172.21.15.72:0/3730399943 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc1e0004800 0x7fc1e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.838+0000 7fc1f7fff640 1 --2- 172.21.15.72:0/3730399943 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1e00051c0 0x7fc1e000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.838+0000 7fc1f77fe640 1 -- 172.21.15.72:0/3730399943 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc1e0004800 msgr2=0x7fc1e0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:07.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.838+0000 7fc1f77fe640 1 --2- 172.21.15.72:0/3730399943 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc1e0004800 0x7fc1e0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.838+0000 7fc1f77fe640 1 -- 172.21.15.72:0/3730399943 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1e00051c0 msgr2=0x7fc1e000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:07.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.838+0000 7fc1f77fe640 1 --2- 172.21.15.72:0/3730399943 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1e00051c0 0x7fc1e000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.838+0000 7fc1f7fff640 1 -- 172.21.15.72:0/3730399943 reap_dead start 2024-04-15T15:16:07.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.838+0000 7fc1f77fe640 1 -- 172.21.15.72:0/3730399943 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1e0003520 con 0x7fc1e00039b0 2024-04-15T15:16:07.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.839+0000 7fc1f77fe640 1 --2- 172.21.15.72:0/3730399943 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1e00039b0 0x7fc1e0003dd0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fc1e800a850 tx=0x7fc1e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1b01dba5c227f66 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:07.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.839+0000 7fc1f67fc640 1 -- 172.21.15.72:0/3730399943 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e800a4a0 con 0x7fc1e00039b0 2024-04-15T15:16:07.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.839+0000 7fc1f67fc640 1 -- 172.21.15.72:0/3730399943 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1e8034650 con 0x7fc1e00039b0 2024-04-15T15:16:07.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.839+0000 7fc1f67fc640 1 -- 172.21.15.72:0/3730399943 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e8034a40 con 0x7fc1e00039b0 2024-04-15T15:16:07.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.839+0000 7fc1fde4e640 1 -- 172.21.15.72:0/3730399943 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1e00039b0 msgr2=0x7fc1e0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.839+0000 7fc1fde4e640 1 --2- 172.21.15.72:0/3730399943 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1e00039b0 0x7fc1e0003dd0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fc1e800a850 tx=0x7fc1e0005830 comp rx=0 tx=0).stop 2024-04-15T15:16:07.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.839+0000 7fc1fde4e640 1 -- 172.21.15.72:0/3730399943 shutdown_connections 2024-04-15T15:16:07.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.839+0000 7fc1fde4e640 1 --2- 172.21.15.72:0/3730399943 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1e00039b0 0x7fc1e0003dd0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.839+0000 7fc1fde4e640 1 -- 172.21.15.72:0/3730399943 >> 172.21.15.72:0/3730399943 conn(0x7fc1f80b31b0 msgr2=0x7fc1f80b3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:07.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.839+0000 7fc1fde4e640 1 -- 172.21.15.72:0/3730399943 shutdown_connections 2024-04-15T15:16:07.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.839+0000 7fc1fde4e640 1 -- 172.21.15.72:0/3730399943 wait complete. 2024-04-15T15:16:07.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.840+0000 7fc1fde4e640 1 Processor -- start 2024-04-15T15:16:07.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.841+0000 7fc1fde4e640 1 -- start start 2024-04-15T15:16:07.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.841+0000 7fc1fde4e640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc1e00039b0 0x7fc1f8155da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.841+0000 7fc1fde4e640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1f8157310 0x7fc1f81304d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.841+0000 7fc1fde4e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1f8157790 0x7fc1f8130a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.841+0000 7fc1fde4e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fc1f806f510 con 0x7fc1f8157790 2024-04-15T15:16:07.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.841+0000 7fc1fde4e640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fc1f813f9f0 con 0x7fc1f8157310 2024-04-15T15:16:07.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.841+0000 7fc1fde4e640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fc1f806f690 con 0x7fc1e00039b0 2024-04-15T15:16:07.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.841+0000 7fc1f7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1f8157790 0x7fc1f8130a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.841+0000 7fc1f7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1f8157790 0x7fc1f8130a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42174/0 (socket says 172.21.15.72:42174) 2024-04-15T15:16:07.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.841+0000 7fc1f7fff640 1 -- 172.21.15.72:0/720776996 learned_addr learned my addr 172.21.15.72:0/720776996 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:07.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.842+0000 7fc1f6ffd640 1 --2- 172.21.15.72:0/720776996 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1f8157310 0x7fc1f81304d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.842+0000 7fc1f77fe640 1 --2- 172.21.15.72:0/720776996 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc1e00039b0 0x7fc1f8155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.842+0000 7fc1f7fff640 1 -- 172.21.15.72:0/720776996 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc1e00039b0 msgr2=0x7fc1f8155da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.842+0000 7fc1f7fff640 1 --2- 172.21.15.72:0/720776996 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc1e00039b0 0x7fc1f8155da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.842+0000 7fc1f7fff640 1 -- 172.21.15.72:0/720776996 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1f8157310 msgr2=0x7fc1f81304d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.842+0000 7fc1f7fff640 1 --2- 172.21.15.72:0/720776996 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1f8157310 0x7fc1f81304d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.842+0000 7fc1f7fff640 1 -- 172.21.15.72:0/720776996 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1f8158ff0 con 0x7fc1f8157790 2024-04-15T15:16:07.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.842+0000 7fc1f7fff640 1 --2- 172.21.15.72:0/720776996 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1f8157790 0x7fc1f8130a10 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fc1f8146390 tx=0x7fc1f000e5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:07.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.842+0000 7fc1dffff640 1 -- 172.21.15.72:0/720776996 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f0019070 con 0x7fc1f8157790 2024-04-15T15:16:07.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.842+0000 7fc1dffff640 1 -- 172.21.15.72:0/720776996 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1f0005110 con 0x7fc1f8157790 2024-04-15T15:16:07.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.842+0000 7fc1fde4e640 1 -- 172.21.15.72:0/720776996 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1f8143d70 con 0x7fc1f8157790 2024-04-15T15:16:07.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.842+0000 7fc1dffff640 1 -- 172.21.15.72:0/720776996 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f0007030 con 0x7fc1f8157790 2024-04-15T15:16:07.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.842+0000 7fc1fde4e640 1 -- 172.21.15.72:0/720776996 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1f8144330 con 0x7fc1f8157790 2024-04-15T15:16:07.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.843+0000 7fc1fde4e640 1 -- 172.21.15.72:0/720776996 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1f80689e0 con 0x7fc1f8157790 2024-04-15T15:16:07.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.843+0000 7fc1dffff640 1 -- 172.21.15.72:0/720776996 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc1f001c070 con 0x7fc1f8157790 2024-04-15T15:16:07.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.844+0000 7fc1dffff640 1 --2- 172.21.15.72:0/720776996 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc1e406c920 0x7fc1e406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.844+0000 7fc1dffff640 1 -- 172.21.15.72:0/720776996 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6101+0+0 (secure 0 0 0) 0x7fc1f008aca0 con 0x7fc1f8157790 2024-04-15T15:16:07.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.844+0000 7fc1f77fe640 1 --2- 172.21.15.72:0/720776996 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc1e406c920 0x7fc1e406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.844+0000 7fc1f77fe640 1 --2- 172.21.15.72:0/720776996 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc1e406c920 0x7fc1e406ede0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc1e8034000 tx=0x7fc1e802f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:07.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:07.849+0000 7fc1dffff640 1 -- 172.21.15.72:0/720776996 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc1f8158ff0 con 0x7fc1f8157790 2024-04-15T15:16:08.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.029+0000 7fc1fde4e640 1 -- 172.21.15.72:0/720776996 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc1f8072f00 con 0x7fc1f8157790 2024-04-15T15:16:08.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.030+0000 7fc1dffff640 1 -- 172.21.15.72:0/720776996 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) ==== 74+0+16955 (secure 0 0 0) 0x7fc1f0058980 con 0x7fc1f8157790 2024-04-15T15:16:08.033 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:16:08.033 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":37,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:16:07.050116+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:05.551499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:06.003152+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:08.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.032+0000 7fc1fde4e640 1 -- 172.21.15.72:0/720776996 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc1e406c920 msgr2=0x7fc1e406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.032+0000 7fc1fde4e640 1 --2- 172.21.15.72:0/720776996 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc1e406c920 0x7fc1e406ede0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc1e8034000 tx=0x7fc1e802f000 comp rx=0 tx=0).stop 2024-04-15T15:16:08.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.032+0000 7fc1fde4e640 1 -- 172.21.15.72:0/720776996 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1f8157790 msgr2=0x7fc1f8130a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.032+0000 7fc1fde4e640 1 --2- 172.21.15.72:0/720776996 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1f8157790 0x7fc1f8130a10 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fc1f8146390 tx=0x7fc1f000e5f0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.032+0000 7fc1fde4e640 1 -- 172.21.15.72:0/720776996 shutdown_connections 2024-04-15T15:16:08.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.032+0000 7fc1fde4e640 1 --2- 172.21.15.72:0/720776996 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc1e406c920 0x7fc1e406ede0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.032+0000 7fc1fde4e640 1 --2- 172.21.15.72:0/720776996 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc1f8157790 0x7fc1f8130a10 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.032+0000 7fc1fde4e640 1 --2- 172.21.15.72:0/720776996 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc1f8157310 0x7fc1f81304d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.032+0000 7fc1fde4e640 1 --2- 172.21.15.72:0/720776996 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc1e00039b0 0x7fc1f8155da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.032+0000 7fc1fde4e640 1 -- 172.21.15.72:0/720776996 >> 172.21.15.72:0/720776996 conn(0x7fc1f80b31b0 msgr2=0x7fc1f80b3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:08.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.033+0000 7fc1fde4e640 1 -- 172.21.15.72:0/720776996 shutdown_connections 2024-04-15T15:16:08.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.033+0000 7fc1fde4e640 1 -- 172.21.15.72:0/720776996 wait complete. 2024-04-15T15:16:08.047 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:08.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.149+0000 7f96bf62a640 1 Processor -- start 2024-04-15T15:16:08.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.149+0000 7f96bf62a640 1 -- start start 2024-04-15T15:16:08.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.150+0000 7f96bf62a640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f96b806f510 0x7f96b806f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.150+0000 7f96bf62a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96b80b4ed0 con 0x7f96b80b4a50 2024-04-15T15:16:08.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.150+0000 7f96bf62a640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96b806fe30 con 0x7f96b80b6350 2024-04-15T15:16:08.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.150+0000 7f96bf62a640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f96b80b7420 con 0x7f96b806f510 2024-04-15T15:16:08.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.150+0000 7f96bdbc7640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f96b80b4a50 0x7f96b8160110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:58826/0 (socket says 172.21.15.72:58826) 2024-04-15T15:16:08.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.150+0000 7f96bdbc7640 1 -- 172.21.15.72:0/1218134308 learned_addr learned my addr 172.21.15.72:0/1218134308 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:08.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.151+0000 7f96bd3c6640 1 --2- 172.21.15.72:0/1218134308 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f96b806f510 0x7f96b806f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.151+0000 7f96b7fff640 1 -- 172.21.15.72:0/1218134308 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4011931001 0 0) 0x7f96b80b4ed0 con 0x7f96b80b4a50 2024-04-15T15:16:08.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.151+0000 7f96b7fff640 1 -- 172.21.15.72:0/1218134308 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96a0003520 con 0x7f96b80b4a50 2024-04-15T15:16:08.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.151+0000 7f96b7fff640 1 -- 172.21.15.72:0/1218134308 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 416460928 0 0) 0x7f96b806fe30 con 0x7f96b80b6350 2024-04-15T15:16:08.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.151+0000 7f96b7fff640 1 -- 172.21.15.72:0/1218134308 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96b80b4ed0 con 0x7f96b80b6350 2024-04-15T15:16:08.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.151+0000 7f96b7fff640 1 -- 172.21.15.72:0/1218134308 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2279597287 0 0) 0x7f96a0003520 con 0x7f96b80b4a50 2024-04-15T15:16:08.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.151+0000 7f96b7fff640 1 -- 172.21.15.72:0/1218134308 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f96b806f510 msgr2=0x7f96b806f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:08.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.151+0000 7f96b7fff640 1 --2- 172.21.15.72:0/1218134308 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f96b806f510 0x7f96b806f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.151+0000 7f96b7fff640 1 -- 172.21.15.72:0/1218134308 >> v1:172.21.15.175:6789/0 conn(0x7f96b80b6350 legacy=0x7f96b80b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.151+0000 7f96b7fff640 1 -- 172.21.15.72:0/1218134308 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96b815cc30 con 0x7f96b80b4a50 2024-04-15T15:16:08.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.151+0000 7f96b7fff640 1 -- 172.21.15.72:0/1218134308 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f96b00019f0 con 0x7f96b80b4a50 2024-04-15T15:16:08.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.151+0000 7f96b7fff640 1 -- 172.21.15.72:0/1218134308 >> v1:172.21.15.72:6789/0 conn(0x7f96b80b4a50 legacy=0x7f96b8160110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.151+0000 7f96b7fff640 1 --2- 172.21.15.72:0/1218134308 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f96a0003b70 0x7f96a0003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.151+0000 7f96b7fff640 1 --2- 172.21.15.72:0/1218134308 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f96a0004800 0x7f96a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.151+0000 7f96b7fff640 1 --2- 172.21.15.72:0/1218134308 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f96a00051c0 0x7f96a000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.151+0000 7f96b7fff640 1 -- 172.21.15.72:0/1218134308 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f96b80b7420 con 0x7f96a00051c0 2024-04-15T15:16:08.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.151+0000 7f96b7fff640 1 -- 172.21.15.72:0/1218134308 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f96a000bfb0 con 0x7f96a0004800 2024-04-15T15:16:08.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.151+0000 7f96b7fff640 1 -- 172.21.15.72:0/1218134308 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f96a000c1b0 con 0x7f96a0003b70 2024-04-15T15:16:08.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.151+0000 7f96bd3c6640 1 --2- 172.21.15.72:0/1218134308 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f96b806f510 0x7f96b806f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:08.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.151+0000 7f96bdbc7640 1 --2- 172.21.15.72:0/1218134308 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f96a00051c0 0x7f96a000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.151+0000 7f96bcbc5640 1 --2- 172.21.15.72:0/1218134308 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f96a0004800 0x7f96a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.151+0000 7f96bd3c6640 1 --2- 172.21.15.72:0/1218134308 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f96a0003b70 0x7f96a0003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.151+0000 7f96bdbc7640 1 -- 172.21.15.72:0/1218134308 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f96a0003b70 msgr2=0x7f96a0003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:08.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.151+0000 7f96bdbc7640 1 --2- 172.21.15.72:0/1218134308 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f96a0003b70 0x7f96a0003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.151+0000 7f96bdbc7640 1 -- 172.21.15.72:0/1218134308 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f96a0004800 msgr2=0x7f96a0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:08.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.151+0000 7f96bdbc7640 1 --2- 172.21.15.72:0/1218134308 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f96a0004800 0x7f96a0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.151+0000 7f96bdbc7640 1 -- 172.21.15.72:0/1218134308 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96a0003520 con 0x7f96a00051c0 2024-04-15T15:16:08.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.151+0000 7f96bd3c6640 1 --2- 172.21.15.72:0/1218134308 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f96a0003b70 0x7f96a0003f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:08.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.152+0000 7f96bdbc7640 1 --2- 172.21.15.72:0/1218134308 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f96a00051c0 0x7f96a000b9c0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f96b000a820 tx=0x7f96a0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4f3acee9ff162f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:08.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.152+0000 7f96b7fff640 1 -- 172.21.15.72:0/1218134308 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b000a450 con 0x7f96a00051c0 2024-04-15T15:16:08.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.152+0000 7f96b7fff640 1 -- 172.21.15.72:0/1218134308 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96b00072d0 con 0x7f96a00051c0 2024-04-15T15:16:08.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.152+0000 7f96b7fff640 1 -- 172.21.15.72:0/1218134308 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b0007740 con 0x7f96a00051c0 2024-04-15T15:16:08.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.153+0000 7f96bf62a640 1 -- 172.21.15.72:0/1218134308 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f96a00051c0 msgr2=0x7f96a000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.153+0000 7f96bf62a640 1 --2- 172.21.15.72:0/1218134308 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f96a00051c0 0x7f96a000b9c0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f96b000a820 tx=0x7f96a0005830 comp rx=0 tx=0).stop 2024-04-15T15:16:08.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.153+0000 7f96bdbc7640 1 -- 172.21.15.72:0/1218134308 reap_dead start 2024-04-15T15:16:08.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.153+0000 7f96bf62a640 1 -- 172.21.15.72:0/1218134308 shutdown_connections 2024-04-15T15:16:08.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.153+0000 7f96bf62a640 1 -- 172.21.15.72:0/1218134308 >> 172.21.15.72:0/1218134308 conn(0x7f96b80cdd20 msgr2=0x7f96b80ce100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:08.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.153+0000 7f96bf62a640 1 -- 172.21.15.72:0/1218134308 shutdown_connections 2024-04-15T15:16:08.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.153+0000 7f96bf62a640 1 -- 172.21.15.72:0/1218134308 wait complete. 2024-04-15T15:16:08.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.154+0000 7f96bf62a640 1 Processor -- start 2024-04-15T15:16:08.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.154+0000 7f96bf62a640 1 -- start start 2024-04-15T15:16:08.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.155+0000 7f96bf62a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f96a00051c0 0x7f96b815ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.155+0000 7f96bf62a640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f96b8156560 0x7f96b815f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.155+0000 7f96bf62a640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f96b8144720 0x7f96b8144ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.155+0000 7f96bf62a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f96b815d1d0 con 0x7f96a00051c0 2024-04-15T15:16:08.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.155+0000 7f96bf62a640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f96b815d820 con 0x7f96b8156560 2024-04-15T15:16:08.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.155+0000 7f96bf62a640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f96b815d4b0 con 0x7f96b8144720 2024-04-15T15:16:08.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.155+0000 7f96bd3c6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f96a00051c0 0x7f96b815ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.155+0000 7f96bd3c6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f96a00051c0 0x7f96b815ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42180/0 (socket says 172.21.15.72:42180) 2024-04-15T15:16:08.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.155+0000 7f96bd3c6640 1 -- 172.21.15.72:0/3114484062 learned_addr learned my addr 172.21.15.72:0/3114484062 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:08.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.155+0000 7f96bcbc5640 1 --2- 172.21.15.72:0/3114484062 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f96b8156560 0x7f96b815f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.155+0000 7f96bdbc7640 1 --2- 172.21.15.72:0/3114484062 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f96b8144720 0x7f96b8144ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.155+0000 7f96bd3c6640 1 -- 172.21.15.72:0/3114484062 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f96b8144720 msgr2=0x7f96b8144ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.155+0000 7f96bd3c6640 1 --2- 172.21.15.72:0/3114484062 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f96b8144720 0x7f96b8144ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.155+0000 7f96bd3c6640 1 -- 172.21.15.72:0/3114484062 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f96b8156560 msgr2=0x7f96b815f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.155+0000 7f96bd3c6640 1 --2- 172.21.15.72:0/3114484062 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f96b8156560 0x7f96b815f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.155+0000 7f96bd3c6640 1 -- 172.21.15.72:0/3114484062 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96b8142f30 con 0x7f96a00051c0 2024-04-15T15:16:08.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.155+0000 7f96bd3c6640 1 --2- 172.21.15.72:0/3114484062 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f96a00051c0 0x7f96b815ef90 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f96a8011910 tx=0x7f96a8011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:08.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.155+0000 7f96b5ffb640 1 -- 172.21.15.72:0/3114484062 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a800fbd0 con 0x7f96a00051c0 2024-04-15T15:16:08.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.156+0000 7f96b5ffb640 1 -- 172.21.15.72:0/3114484062 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96a800fd70 con 0x7f96a00051c0 2024-04-15T15:16:08.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.156+0000 7f96bf62a640 1 -- 172.21.15.72:0/3114484062 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96b8143210 con 0x7f96a00051c0 2024-04-15T15:16:08.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.156+0000 7f96b5ffb640 1 -- 172.21.15.72:0/3114484062 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a8015660 con 0x7f96a00051c0 2024-04-15T15:16:08.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.156+0000 7f96bf62a640 1 -- 172.21.15.72:0/3114484062 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96b81437d0 con 0x7f96a00051c0 2024-04-15T15:16:08.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.157+0000 7f96bf62a640 1 -- 172.21.15.72:0/3114484062 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96b80690f0 con 0x7f96a00051c0 2024-04-15T15:16:08.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.157+0000 7f96b5ffb640 1 -- 172.21.15.72:0/3114484062 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f96a8015800 con 0x7f96a00051c0 2024-04-15T15:16:08.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.158+0000 7f96b5ffb640 1 --2- 172.21.15.72:0/3114484062 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f96a406c8f0 0x7f96a406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.158+0000 7f96b5ffb640 1 -- 172.21.15.72:0/3114484062 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6101+0+0 (secure 0 0 0) 0x7f96a808c770 con 0x7f96a00051c0 2024-04-15T15:16:08.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.158+0000 7f96bcbc5640 1 --2- 172.21.15.72:0/3114484062 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f96a406c8f0 0x7f96a406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.158+0000 7f96bcbc5640 1 --2- 172.21.15.72:0/3114484062 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f96a406c8f0 0x7f96a406edb0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f96b8142d10 tx=0x7f96ac009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:08.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.162+0000 7f96b5ffb640 1 -- 172.21.15.72:0/3114484062 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f96a8019030 con 0x7f96a00051c0 2024-04-15T15:16:08.342 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.340+0000 7f96bf62a640 1 -- 172.21.15.72:0/3114484062 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f96b813f8f0 con 0x7f96a00051c0 2024-04-15T15:16:08.343 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.341+0000 7f96b5ffb640 1 -- 172.21.15.72:0/3114484062 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) ==== 74+0+16955 (secure 0 0 0) 0x7f96a805aa10 con 0x7f96a00051c0 2024-04-15T15:16:08.343 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:16:08.344 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":37,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:16:07.050116+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:05.551499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:06.003152+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:08.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.343+0000 7f96bf62a640 1 -- 172.21.15.72:0/3114484062 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f96a406c8f0 msgr2=0x7f96a406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.343+0000 7f96bf62a640 1 --2- 172.21.15.72:0/3114484062 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f96a406c8f0 0x7f96a406edb0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f96b8142d10 tx=0x7f96ac009040 comp rx=0 tx=0).stop 2024-04-15T15:16:08.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.343+0000 7f96bf62a640 1 -- 172.21.15.72:0/3114484062 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f96a00051c0 msgr2=0x7f96b815ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.343+0000 7f96bf62a640 1 --2- 172.21.15.72:0/3114484062 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f96a00051c0 0x7f96b815ef90 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f96a8011910 tx=0x7f96a8011dd0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.343+0000 7f96bf62a640 1 -- 172.21.15.72:0/3114484062 shutdown_connections 2024-04-15T15:16:08.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.343+0000 7f96bf62a640 1 --2- 172.21.15.72:0/3114484062 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f96a406c8f0 0x7f96a406edb0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.343+0000 7f96bf62a640 1 --2- 172.21.15.72:0/3114484062 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f96b8144720 0x7f96b8144ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.343+0000 7f96bf62a640 1 --2- 172.21.15.72:0/3114484062 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f96b8156560 0x7f96b815f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.343+0000 7f96bf62a640 1 --2- 172.21.15.72:0/3114484062 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f96a00051c0 0x7f96b815ef90 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.343+0000 7f96bf62a640 1 -- 172.21.15.72:0/3114484062 >> 172.21.15.72:0/3114484062 conn(0x7f96b80cdd20 msgr2=0x7f96b80cd540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:08.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.343+0000 7f96bf62a640 1 -- 172.21.15.72:0/3114484062 shutdown_connections 2024-04-15T15:16:08.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.343+0000 7f96bf62a640 1 -- 172.21.15.72:0/3114484062 wait complete. 2024-04-15T15:16:08.357 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-15T15:16:08.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.461+0000 7ff8e4075640 1 Processor -- start 2024-04-15T15:16:08.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.461+0000 7ff8e4075640 1 -- start start 2024-04-15T15:16:08.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.462+0000 7ff8e4075640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff8dc15e960 0x7ff8dc0d1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.462+0000 7ff8e4075640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff8dc0d5010 0x7ff8dc0cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.462+0000 7ff8e4075640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8dc0b65b0 con 0x7ff8dc0b11d0 2024-04-15T15:16:08.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.462+0000 7ff8e4075640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7ff8dc0b6890 con 0x7ff8dc15e960 2024-04-15T15:16:08.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.462+0000 7ff8e4075640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7ff8dc0b6b70 con 0x7ff8dc0d5010 2024-04-15T15:16:08.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.462+0000 7ff8e1e11640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff8dc15e960 0x7ff8dc0d1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.462+0000 7ff8e2612640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff8dc0d5010 0x7ff8dc0cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.462+0000 7ff8e1610640 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7ff8dc0b11d0 0x7ff8dc0d4900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.72:40118/0 (socket says 172.21.15.72:40118) 2024-04-15T15:16:08.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.462+0000 7ff8e1610640 1 -- 172.21.15.72:0/773064862 learned_addr learned my addr 172.21.15.72:0/773064862 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:08.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.463+0000 7ff8e1e11640 1 -- 172.21.15.72:0/773064862 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff8dc0d5010 msgr2=0x7ff8dc0cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:08.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.463+0000 7ff8e1e11640 1 --2- 172.21.15.72:0/773064862 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff8dc0d5010 0x7ff8dc0cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.463+0000 7ff8e1e11640 1 -- 172.21.15.72:0/773064862 >> v1:172.21.15.175:6789/0 conn(0x7ff8dc0b11d0 legacy=0x7ff8dc0d4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.463+0000 7ff8e0e0f640 1 -- 172.21.15.72:0/773064862 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3555731840 0 0) 0x7ff8dc0b65b0 con 0x7ff8dc0b11d0 2024-04-15T15:16:08.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.463+0000 7ff8e1e11640 1 -- 172.21.15.72:0/773064862 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8dc0b6e50 con 0x7ff8dc15e960 2024-04-15T15:16:08.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.463+0000 7ff8e1e11640 1 --2- 172.21.15.72:0/773064862 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff8dc15e960 0x7ff8dc0d1920 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff8d80097a0 tx=0x7ff8d802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3835e1b8521565ca server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:08.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.464+0000 7ff8e0e0f640 1 -- 172.21.15.72:0/773064862 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d8035020 con 0x7ff8dc15e960 2024-04-15T15:16:08.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.464+0000 7ff8e0e0f640 1 -- 172.21.15.72:0/773064862 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8d8031830 con 0x7ff8dc15e960 2024-04-15T15:16:08.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.464+0000 7ff8e0e0f640 1 -- 172.21.15.72:0/773064862 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d8031c00 con 0x7ff8dc15e960 2024-04-15T15:16:08.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.464+0000 7ff8e4075640 1 -- 172.21.15.72:0/773064862 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff8dc15e960 msgr2=0x7ff8dc0d1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.464+0000 7ff8e4075640 1 --2- 172.21.15.72:0/773064862 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff8dc15e960 0x7ff8dc0d1920 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff8d80097a0 tx=0x7ff8d802b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.464+0000 7ff8e4075640 1 -- 172.21.15.72:0/773064862 shutdown_connections 2024-04-15T15:16:08.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.464+0000 7ff8e4075640 1 --2- 172.21.15.72:0/773064862 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff8dc0d5010 0x7ff8dc0cffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.464+0000 7ff8e4075640 1 --2- 172.21.15.72:0/773064862 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff8dc15e960 0x7ff8dc0d1920 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.464+0000 7ff8e4075640 1 -- 172.21.15.72:0/773064862 >> 172.21.15.72:0/773064862 conn(0x7ff8dc0b9580 msgr2=0x7ff8dc0b9990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:08.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.464+0000 7ff8e4075640 1 -- 172.21.15.72:0/773064862 shutdown_connections 2024-04-15T15:16:08.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.464+0000 7ff8e4075640 1 -- 172.21.15.72:0/773064862 wait complete. 2024-04-15T15:16:08.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.465+0000 7ff8e4075640 1 Processor -- start 2024-04-15T15:16:08.469 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.465+0000 7ff8e4075640 1 -- start start 2024-04-15T15:16:08.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.466+0000 7ff8e4075640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8dc0b11d0 0x7ff8dc157a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.466+0000 7ff8e4075640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff8dc0d5010 0x7ff8dc15fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.466+0000 7ff8e4075640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff8dc15e960 0x7ff8dc1602d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.466+0000 7ff8e4075640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ff8dc0cfb40 con 0x7ff8dc0b11d0 2024-04-15T15:16:08.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.466+0000 7ff8e4075640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7ff8dc0cf9c0 con 0x7ff8dc0d5010 2024-04-15T15:16:08.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.466+0000 7ff8e4075640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7ff8dc0b6040 con 0x7ff8dc15e960 2024-04-15T15:16:08.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.466+0000 7ff8e1e11640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8dc0b11d0 0x7ff8dc157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.466+0000 7ff8e1e11640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8dc0b11d0 0x7ff8dc157a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42188/0 (socket says 172.21.15.72:42188) 2024-04-15T15:16:08.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.466+0000 7ff8e1e11640 1 -- 172.21.15.72:0/2853406619 learned_addr learned my addr 172.21.15.72:0/2853406619 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:08.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.467+0000 7ff8e2612640 1 --2- 172.21.15.72:0/2853406619 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff8dc15e960 0x7ff8dc1602d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.467+0000 7ff8e1610640 1 --2- 172.21.15.72:0/2853406619 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff8dc0d5010 0x7ff8dc15fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.467+0000 7ff8e1e11640 1 -- 172.21.15.72:0/2853406619 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff8dc15e960 msgr2=0x7ff8dc1602d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.467+0000 7ff8e1e11640 1 --2- 172.21.15.72:0/2853406619 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff8dc15e960 0x7ff8dc1602d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.467+0000 7ff8e1e11640 1 -- 172.21.15.72:0/2853406619 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff8dc0d5010 msgr2=0x7ff8dc15fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.467+0000 7ff8e1e11640 1 --2- 172.21.15.72:0/2853406619 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff8dc0d5010 0x7ff8dc15fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.467+0000 7ff8e1e11640 1 -- 172.21.15.72:0/2853406619 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8dc157fd0 con 0x7ff8dc0b11d0 2024-04-15T15:16:08.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.467+0000 7ff8e1e11640 1 --2- 172.21.15.72:0/2853406619 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8dc0b11d0 0x7ff8dc157a90 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8000c00 tx=0x7ff8d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:08.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.467+0000 7ff8d27fc640 1 -- 172.21.15.72:0/2853406619 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d8035040 con 0x7ff8dc0b11d0 2024-04-15T15:16:08.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.467+0000 7ff8d27fc640 1 -- 172.21.15.72:0/2853406619 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8d8004250 con 0x7ff8dc0b11d0 2024-04-15T15:16:08.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.467+0000 7ff8e4075640 1 -- 172.21.15.72:0/2853406619 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8dc160900 con 0x7ff8dc0b11d0 2024-04-15T15:16:08.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.467+0000 7ff8d27fc640 1 -- 172.21.15.72:0/2853406619 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d803a640 con 0x7ff8dc0b11d0 2024-04-15T15:16:08.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.467+0000 7ff8e4075640 1 -- 172.21.15.72:0/2853406619 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8dc160ec0 con 0x7ff8dc0b11d0 2024-04-15T15:16:08.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.468+0000 7ff8e4075640 1 -- 172.21.15.72:0/2853406619 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8dc075c50 con 0x7ff8dc0b11d0 2024-04-15T15:16:08.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.468+0000 7ff8d27fc640 1 -- 172.21.15.72:0/2853406619 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff8d8038020 con 0x7ff8dc0b11d0 2024-04-15T15:16:08.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.469+0000 7ff8d27fc640 1 --2- 172.21.15.72:0/2853406619 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff8c006c8f0 0x7ff8c006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.469+0000 7ff8e1610640 1 --2- 172.21.15.72:0/2853406619 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff8c006c8f0 0x7ff8c006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.469+0000 7ff8d27fc640 1 -- 172.21.15.72:0/2853406619 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6101+0+0 (secure 0 0 0) 0x7ff8d80ada00 con 0x7ff8dc0b11d0 2024-04-15T15:16:08.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.469+0000 7ff8e1610640 1 --2- 172.21.15.72:0/2853406619 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff8c006c8f0 0x7ff8c006edb0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc007fd0 tx=0x7ff8cc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:08.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.473+0000 7ff8d27fc640 1 -- 172.21.15.72:0/2853406619 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff8d807bcb0 con 0x7ff8dc0b11d0 2024-04-15T15:16:08.689 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.686+0000 7ff8e4075640 1 -- 172.21.15.72:0/2853406619 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7ff8dc0b4930 con 0x7ff8dc0b11d0 2024-04-15T15:16:08.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.701+0000 7ff8d27fc640 1 -- 172.21.15.72:0/2853406619 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) ==== 118+0+0 (secure 0 0 0) 0x7ff8d807b5b0 con 0x7ff8dc0b11d0 2024-04-15T15:16:08.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.703+0000 7ff8e4075640 1 -- 172.21.15.72:0/2853406619 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff8c006c8f0 msgr2=0x7ff8c006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.703+0000 7ff8e4075640 1 --2- 172.21.15.72:0/2853406619 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff8c006c8f0 0x7ff8c006edb0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc007fd0 tx=0x7ff8cc009040 comp rx=0 tx=0).stop 2024-04-15T15:16:08.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.703+0000 7ff8e4075640 1 -- 172.21.15.72:0/2853406619 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8dc0b11d0 msgr2=0x7ff8dc157a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.703+0000 7ff8e4075640 1 --2- 172.21.15.72:0/2853406619 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8dc0b11d0 0x7ff8dc157a90 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7ff8d8000c00 tx=0x7ff8d80046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.704+0000 7ff8e4075640 1 -- 172.21.15.72:0/2853406619 shutdown_connections 2024-04-15T15:16:08.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.704+0000 7ff8e4075640 1 --2- 172.21.15.72:0/2853406619 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff8c006c8f0 0x7ff8c006edb0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.704+0000 7ff8e4075640 1 --2- 172.21.15.72:0/2853406619 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff8dc15e960 0x7ff8dc1602d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.704+0000 7ff8e4075640 1 --2- 172.21.15.72:0/2853406619 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff8dc0d5010 0x7ff8dc15fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.704+0000 7ff8e4075640 1 --2- 172.21.15.72:0/2853406619 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff8dc0b11d0 0x7ff8dc157a90 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.704+0000 7ff8e4075640 1 -- 172.21.15.72:0/2853406619 >> 172.21.15.72:0/2853406619 conn(0x7ff8dc0b9580 msgr2=0x7ff8dc0b9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:08.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.704+0000 7ff8e4075640 1 -- 172.21.15.72:0/2853406619 shutdown_connections 2024-04-15T15:16:08.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.704+0000 7ff8e4075640 1 -- 172.21.15.72:0/2853406619 wait complete. 2024-04-15T15:16:08.719 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-15T15:16:08.719 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-15T15:16:08.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.823+0000 7fda41e7e640 1 Processor -- start 2024-04-15T15:16:08.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.824+0000 7fda41e7e640 1 -- start start 2024-04-15T15:16:08.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.824+0000 7fda41e7e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fda3c06c730 0x7fda3c06cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.824+0000 7fda41e7e640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fda3c1667c0 0x7fda3c168bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.824+0000 7fda41e7e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda3c0cf8f0 con 0x7fda3c06c350 2024-04-15T15:16:08.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.824+0000 7fda41e7e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fda3c0d27d0 con 0x7fda3c06c730 2024-04-15T15:16:08.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.824+0000 7fda41e7e640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fda3c0b3ae0 con 0x7fda3c1667c0 2024-04-15T15:16:08.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.824+0000 7fda32ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fda3c06c730 0x7fda3c06cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.824+0000 7fda3b7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fda3c06c350 0x7fda3c0d0710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:58842/0 (socket says 172.21.15.72:58842) 2024-04-15T15:16:08.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.824+0000 7fda32ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fda3c06c730 0x7fda3c06cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42194/0 (socket says 172.21.15.72:42194) 2024-04-15T15:16:08.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.824+0000 7fda3b7fe640 1 -- 172.21.15.72:0/2725334876 learned_addr learned my addr 172.21.15.72:0/2725334876 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:08.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.825+0000 7fda32ffd640 1 -- 172.21.15.72:0/2725334876 >> v2:172.21.15.175:3300/0 conn(0x7fda3c1667c0 msgr2=0x7fda3c168bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:08.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.825+0000 7fda32ffd640 1 --2- 172.21.15.72:0/2725334876 >> v2:172.21.15.175:3300/0 conn(0x7fda3c1667c0 0x7fda3c168bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.825+0000 7fda32ffd640 1 -- 172.21.15.72:0/2725334876 >> v1:172.21.15.72:6789/0 conn(0x7fda3c06c350 legacy=0x7fda3c0d0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.825+0000 7fda3affd640 1 -- 172.21.15.72:0/2725334876 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 439889257 0 0) 0x7fda3c0cf8f0 con 0x7fda3c06c350 2024-04-15T15:16:08.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.825+0000 7fda32ffd640 1 -- 172.21.15.72:0/2725334876 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda3c0d0e20 con 0x7fda3c06c730 2024-04-15T15:16:08.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.825+0000 7fda32ffd640 1 --2- 172.21.15.72:0/2725334876 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda3c06c730 0x7fda3c06cb90 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fda240097a0 tx=0x7fda2402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e65d55065980642a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:08.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.826+0000 7fda3affd640 1 -- 172.21.15.72:0/2725334876 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda240315a0 con 0x7fda3c06c730 2024-04-15T15:16:08.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.826+0000 7fda3affd640 1 -- 172.21.15.72:0/2725334876 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda24037070 con 0x7fda3c06c730 2024-04-15T15:16:08.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.826+0000 7fda3affd640 1 -- 172.21.15.72:0/2725334876 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda24031960 con 0x7fda3c06c730 2024-04-15T15:16:08.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.826+0000 7fda41e7e640 1 -- 172.21.15.72:0/2725334876 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda3c06c730 msgr2=0x7fda3c06cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.826+0000 7fda41e7e640 1 --2- 172.21.15.72:0/2725334876 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda3c06c730 0x7fda3c06cb90 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fda240097a0 tx=0x7fda2402b840 comp rx=0 tx=0).stop 2024-04-15T15:16:08.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.826+0000 7fda41e7e640 1 -- 172.21.15.72:0/2725334876 shutdown_connections 2024-04-15T15:16:08.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.826+0000 7fda41e7e640 1 --2- 172.21.15.72:0/2725334876 >> v2:172.21.15.175:3300/0 conn(0x7fda3c1667c0 0x7fda3c168bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.826+0000 7fda41e7e640 1 --2- 172.21.15.72:0/2725334876 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda3c06c730 0x7fda3c06cb90 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.826+0000 7fda41e7e640 1 -- 172.21.15.72:0/2725334876 >> 172.21.15.72:0/2725334876 conn(0x7fda3c0d48c0 msgr2=0x7fda3c0d4cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:08.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.826+0000 7fda41e7e640 1 -- 172.21.15.72:0/2725334876 shutdown_connections 2024-04-15T15:16:08.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.826+0000 7fda41e7e640 1 -- 172.21.15.72:0/2725334876 wait complete. 2024-04-15T15:16:08.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.827+0000 7fda41e7e640 1 Processor -- start 2024-04-15T15:16:08.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.828+0000 7fda41e7e640 1 -- start start 2024-04-15T15:16:08.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.828+0000 7fda41e7e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda3c06c350 0x7fda3c14c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.828+0000 7fda41e7e640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fda3c06c730 0x7fda3c14f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.828+0000 7fda41e7e640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda3c1667c0 0x7fda3c14f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.828+0000 7fda41e7e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fda3c0b3240 con 0x7fda3c06c350 2024-04-15T15:16:08.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.828+0000 7fda41e7e640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fda3c0d3c70 con 0x7fda3c1667c0 2024-04-15T15:16:08.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.828+0000 7fda41e7e640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fda3c0b3520 con 0x7fda3c06c730 2024-04-15T15:16:08.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.828+0000 7fda3b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda3c06c350 0x7fda3c14c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.828+0000 7fda3b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda3c06c350 0x7fda3c14c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42196/0 (socket says 172.21.15.72:42196) 2024-04-15T15:16:08.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.828+0000 7fda3b7fe640 1 -- 172.21.15.72:0/1054127607 learned_addr learned my addr 172.21.15.72:0/1054127607 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:08.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.828+0000 7fda3bfff640 1 --2- 172.21.15.72:0/1054127607 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda3c1667c0 0x7fda3c14f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.828+0000 7fda32ffd640 1 --2- 172.21.15.72:0/1054127607 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fda3c06c730 0x7fda3c14f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.829+0000 7fda3b7fe640 1 -- 172.21.15.72:0/1054127607 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fda3c06c730 msgr2=0x7fda3c14f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.829+0000 7fda3b7fe640 1 --2- 172.21.15.72:0/1054127607 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fda3c06c730 0x7fda3c14f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.829+0000 7fda3b7fe640 1 -- 172.21.15.72:0/1054127607 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda3c1667c0 msgr2=0x7fda3c14f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.829+0000 7fda3b7fe640 1 --2- 172.21.15.72:0/1054127607 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda3c1667c0 0x7fda3c14f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.829+0000 7fda3b7fe640 1 -- 172.21.15.72:0/1054127607 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda3c14dc60 con 0x7fda3c06c350 2024-04-15T15:16:08.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.829+0000 7fda3b7fe640 1 --2- 172.21.15.72:0/1054127607 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda3c06c350 0x7fda3c14c460 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fda2c00c4d0 tx=0x7fda2c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:08.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.829+0000 7fda38ff9640 1 -- 172.21.15.72:0/1054127607 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda2c018070 con 0x7fda3c06c350 2024-04-15T15:16:08.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.829+0000 7fda38ff9640 1 -- 172.21.15.72:0/1054127607 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda2c007500 con 0x7fda3c06c350 2024-04-15T15:16:08.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.829+0000 7fda38ff9640 1 -- 172.21.15.72:0/1054127607 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda2c004a20 con 0x7fda3c06c350 2024-04-15T15:16:08.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.829+0000 7fda41e7e640 1 -- 172.21.15.72:0/1054127607 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda3c14df40 con 0x7fda3c06c350 2024-04-15T15:16:08.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.829+0000 7fda41e7e640 1 -- 172.21.15.72:0/1054127607 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda3c14e500 con 0x7fda3c06c350 2024-04-15T15:16:08.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.830+0000 7fda41e7e640 1 -- 172.21.15.72:0/1054127607 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda3c0cb040 con 0x7fda3c06c350 2024-04-15T15:16:08.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.831+0000 7fda38ff9640 1 -- 172.21.15.72:0/1054127607 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fda2c01b070 con 0x7fda3c06c350 2024-04-15T15:16:08.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.831+0000 7fda38ff9640 1 --2- 172.21.15.72:0/1054127607 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fda1c06c920 0x7fda1c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.831+0000 7fda38ff9640 1 -- 172.21.15.72:0/1054127607 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fda2c08b040 con 0x7fda3c06c350 2024-04-15T15:16:08.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.831+0000 7fda32ffd640 1 --2- 172.21.15.72:0/1054127607 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fda1c06c920 0x7fda1c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.831+0000 7fda32ffd640 1 --2- 172.21.15.72:0/1054127607 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fda1c06c920 0x7fda1c06ede0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fda24036000 tx=0x7fda2403d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:08.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:08.835+0000 7fda38ff9640 1 -- 172.21.15.72:0/1054127607 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fda2c0592e0 con 0x7fda3c06c350 2024-04-15T15:16:09.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.050+0000 7fda41e7e640 1 -- 172.21.15.72:0/1054127607 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fda3c075c30 con 0x7fda3c06c350 2024-04-15T15:16:09.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.710+0000 7fda38ff9640 1 -- 172.21.15.72:0/1054127607 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v16) ==== 122+0+0 (secure 0 0 0) 0x7fda2c058be0 con 0x7fda3c06c350 2024-04-15T15:16:09.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.712+0000 7fda41e7e640 1 -- 172.21.15.72:0/1054127607 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fda1c06c920 msgr2=0x7fda1c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.712+0000 7fda41e7e640 1 --2- 172.21.15.72:0/1054127607 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fda1c06c920 0x7fda1c06ede0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fda24036000 tx=0x7fda2403d040 comp rx=0 tx=0).stop 2024-04-15T15:16:09.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.712+0000 7fda41e7e640 1 -- 172.21.15.72:0/1054127607 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda3c06c350 msgr2=0x7fda3c14c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.712+0000 7fda41e7e640 1 --2- 172.21.15.72:0/1054127607 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda3c06c350 0x7fda3c14c460 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fda2c00c4d0 tx=0x7fda2c00c990 comp rx=0 tx=0).stop 2024-04-15T15:16:09.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.712+0000 7fda41e7e640 1 -- 172.21.15.72:0/1054127607 shutdown_connections 2024-04-15T15:16:09.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.712+0000 7fda41e7e640 1 --2- 172.21.15.72:0/1054127607 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fda1c06c920 0x7fda1c06ede0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.712+0000 7fda41e7e640 1 --2- 172.21.15.72:0/1054127607 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fda3c1667c0 0x7fda3c14f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.712+0000 7fda41e7e640 1 --2- 172.21.15.72:0/1054127607 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fda3c06c730 0x7fda3c14f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.712+0000 7fda41e7e640 1 --2- 172.21.15.72:0/1054127607 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fda3c06c350 0x7fda3c14c460 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.712+0000 7fda41e7e640 1 -- 172.21.15.72:0/1054127607 >> 172.21.15.72:0/1054127607 conn(0x7fda3c0d48c0 msgr2=0x7fda3c06cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:09.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.713+0000 7fda41e7e640 1 -- 172.21.15.72:0/1054127607 shutdown_connections 2024-04-15T15:16:09.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.713+0000 7fda41e7e640 1 -- 172.21.15.72:0/1054127607 wait complete. 2024-04-15T15:16:09.728 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-15T15:16:09.728 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:16:09.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.832+0000 7f7d31b74640 1 Processor -- start 2024-04-15T15:16:09.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.832+0000 7f7d31b74640 1 -- start start 2024-04-15T15:16:09.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.832+0000 7f7d31b74640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7d2c15c210 0x7f7d2c0b52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.832+0000 7f7d31b74640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f7d2c0d0c70 0x7f7d2c15f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.832+0000 7f7d31b74640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d2c0b5960 con 0x7f7d2c0cca00 2024-04-15T15:16:09.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.832+0000 7f7d31b74640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f7d2c0d0190 con 0x7f7d2c0d0c70 2024-04-15T15:16:09.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.832+0000 7f7d31b74640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f7d2c0d0470 con 0x7f7d2c15c210 2024-04-15T15:16:09.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.833+0000 7f7d2b7fe640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7d2c15c210 0x7f7d2c0b52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.833+0000 7f7d2affd640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f7d2c0d0c70 0x7f7d2c15f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.833+0000 7f7d2bfff640 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f7d2c0cca00 0x7f7d2c162270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.72:40124/0 (socket says 172.21.15.72:40124) 2024-04-15T15:16:09.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.833+0000 7f7d2bfff640 1 -- 172.21.15.72:0/2825433438 learned_addr learned my addr 172.21.15.72:0/2825433438 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:09.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.833+0000 7f7d2affd640 1 -- 172.21.15.72:0/2825433438 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7d2c15c210 msgr2=0x7f7d2c0b52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:09.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.833+0000 7f7d2affd640 1 --2- 172.21.15.72:0/2825433438 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7d2c15c210 0x7f7d2c0b52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.833+0000 7f7d2affd640 1 -- 172.21.15.72:0/2825433438 >> v1:172.21.15.175:6789/0 conn(0x7f7d2c0cca00 legacy=0x7f7d2c162270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.833+0000 7f7d2a7fc640 1 -- 172.21.15.72:0/2825433438 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2458398312 0 0) 0x7f7d2c0b5960 con 0x7f7d2c0cca00 2024-04-15T15:16:09.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.833+0000 7f7d2affd640 1 -- 172.21.15.72:0/2825433438 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d2c0d0750 con 0x7f7d2c0d0c70 2024-04-15T15:16:09.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.833+0000 7f7d2b7fe640 1 --2- 172.21.15.72:0/2825433438 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7d2c15c210 0x7f7d2c0b52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:09.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.834+0000 7f7d2affd640 1 --2- 172.21.15.72:0/2825433438 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7d2c0d0c70 0x7f7d2c15f770 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7d20009d90 tx=0x7f7d2002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7992fe875cfa2d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:09.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.834+0000 7f7d2a7fc640 1 -- 172.21.15.72:0/2825433438 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d20039070 con 0x7f7d2c0d0c70 2024-04-15T15:16:09.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.834+0000 7f7d2a7fc640 1 -- 172.21.15.72:0/2825433438 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d2002cc50 con 0x7f7d2c0d0c70 2024-04-15T15:16:09.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.834+0000 7f7d2a7fc640 1 -- 172.21.15.72:0/2825433438 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d20034440 con 0x7f7d2c0d0c70 2024-04-15T15:16:09.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.835+0000 7f7d31b74640 1 -- 172.21.15.72:0/2825433438 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7d2c0d0c70 msgr2=0x7f7d2c15f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.835+0000 7f7d31b74640 1 --2- 172.21.15.72:0/2825433438 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7d2c0d0c70 0x7f7d2c15f770 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7d20009d90 tx=0x7f7d2002bea0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.835+0000 7f7d31b74640 1 -- 172.21.15.72:0/2825433438 shutdown_connections 2024-04-15T15:16:09.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.835+0000 7f7d31b74640 1 --2- 172.21.15.72:0/2825433438 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7d2c0d0c70 0x7f7d2c15f770 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.835+0000 7f7d31b74640 1 --2- 172.21.15.72:0/2825433438 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7d2c15c210 0x7f7d2c0b52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.835+0000 7f7d31b74640 1 -- 172.21.15.72:0/2825433438 >> 172.21.15.72:0/2825433438 conn(0x7f7d2c0b3fa0 msgr2=0x7f7d2c0b43b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:09.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.835+0000 7f7d31b74640 1 -- 172.21.15.72:0/2825433438 shutdown_connections 2024-04-15T15:16:09.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.835+0000 7f7d31b74640 1 -- 172.21.15.72:0/2825433438 wait complete. 2024-04-15T15:16:09.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.836+0000 7f7d31b74640 1 Processor -- start 2024-04-15T15:16:09.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.836+0000 7f7d31b74640 1 -- start start 2024-04-15T15:16:09.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.837+0000 7f7d31b74640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7d2c0cca00 0x7f7d2c156e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.837+0000 7f7d31b74640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d2c0d0c70 0x7f7d2c1f4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.837+0000 7f7d31b74640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7d2c15c210 0x7f7d2c157ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.837+0000 7f7d31b74640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f7d2c0cd010 con 0x7f7d2c0d0c70 2024-04-15T15:16:09.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.837+0000 7f7d31b74640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f7d2c0cce90 con 0x7f7d2c0cca00 2024-04-15T15:16:09.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.837+0000 7f7d31b74640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f7d2c0cd190 con 0x7f7d2c15c210 2024-04-15T15:16:09.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.837+0000 7f7d2affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d2c0d0c70 0x7f7d2c1f4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.837+0000 7f7d2affd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d2c0d0c70 0x7f7d2c1f4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42198/0 (socket says 172.21.15.72:42198) 2024-04-15T15:16:09.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.837+0000 7f7d2affd640 1 -- 172.21.15.72:0/847800549 learned_addr learned my addr 172.21.15.72:0/847800549 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:09.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.837+0000 7f7d2b7fe640 1 --2- 172.21.15.72:0/847800549 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7d2c0cca00 0x7f7d2c156e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.837+0000 7f7d2bfff640 1 --2- 172.21.15.72:0/847800549 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7d2c15c210 0x7f7d2c157ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.837+0000 7f7d2affd640 1 -- 172.21.15.72:0/847800549 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7d2c15c210 msgr2=0x7f7d2c157ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.837+0000 7f7d2affd640 1 --2- 172.21.15.72:0/847800549 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7d2c15c210 0x7f7d2c157ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.837+0000 7f7d2affd640 1 -- 172.21.15.72:0/847800549 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7d2c0cca00 msgr2=0x7f7d2c156e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.837+0000 7f7d2affd640 1 --2- 172.21.15.72:0/847800549 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7d2c0cca00 0x7f7d2c156e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.837+0000 7f7d2affd640 1 -- 172.21.15.72:0/847800549 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d2c0c95b0 con 0x7f7d2c0d0c70 2024-04-15T15:16:09.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.838+0000 7f7d2affd640 1 --2- 172.21.15.72:0/847800549 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d2c0d0c70 0x7f7d2c1f4450 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f7d20009d90 tx=0x7f7d20004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:09.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.838+0000 7f7d13fff640 1 -- 172.21.15.72:0/847800549 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d20039050 con 0x7f7d2c0d0c70 2024-04-15T15:16:09.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.838+0000 7f7d13fff640 1 -- 172.21.15.72:0/847800549 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d20045070 con 0x7f7d2c0d0c70 2024-04-15T15:16:09.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.838+0000 7f7d13fff640 1 -- 172.21.15.72:0/847800549 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2003e610 con 0x7f7d2c0d0c70 2024-04-15T15:16:09.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.838+0000 7f7d31b74640 1 -- 172.21.15.72:0/847800549 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d2c0c9890 con 0x7f7d2c0d0c70 2024-04-15T15:16:09.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.838+0000 7f7d31b74640 1 -- 172.21.15.72:0/847800549 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d2c158400 con 0x7f7d2c0d0c70 2024-04-15T15:16:09.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.839+0000 7f7d31b74640 1 -- 172.21.15.72:0/847800549 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d2c0690f0 con 0x7f7d2c0d0c70 2024-04-15T15:16:09.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.839+0000 7f7d13fff640 1 -- 172.21.15.72:0/847800549 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7d2003e7b0 con 0x7f7d2c0d0c70 2024-04-15T15:16:09.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.840+0000 7f7d13fff640 1 --2- 172.21.15.72:0/847800549 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7d1806c920 0x7f7d1806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.840+0000 7f7d13fff640 1 -- 172.21.15.72:0/847800549 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f7d200ae760 con 0x7f7d2c0d0c70 2024-04-15T15:16:09.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.840+0000 7f7d2b7fe640 1 --2- 172.21.15.72:0/847800549 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7d1806c920 0x7f7d1806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.840+0000 7f7d2b7fe640 1 --2- 172.21.15.72:0/847800549 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7d1806c920 0x7f7d1806ede0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c009560 tx=0x7f7d1c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:09.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:09.844+0000 7f7d13fff640 1 -- 172.21.15.72:0/847800549 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7d2007ca90 con 0x7f7d2c0d0c70 2024-04-15T15:16:10.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.056+0000 7f7d31b74640 1 -- 172.21.15.72:0/847800549 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7d2c0d1aa0 con 0x7f7d2c0d0c70 2024-04-15T15:16:10.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.057+0000 7f7d13fff640 1 -- 172.21.15.72:0/847800549 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5184 (secure 0 0 0) 0x7f7d2007c390 con 0x7f7d2c0d0c70 2024-04-15T15:16:10.060 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:16:10.060 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4883,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3363272836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3363272836},{"type":"v1","addr":"172.21.15.175:6834","nonce":3363272836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5134,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6835/1059187505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":1059187505},{"type":"v1","addr":"172.21.15.175:6835","nonce":1059187505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5556,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/690132275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":690132275},{"type":"v1","addr":"172.21.15.72:6837","nonce":690132275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:16:07.050078+0000","modified":"2024-04-15T15:16:09.052120+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5559},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/4270793613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4270793613},{"type":"v1","addr":"172.21.15.72:6836","nonce":4270793613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5559,"qdb_cluster":[5559]},"id":2}]} 2024-04-15T15:16:10.061 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 16 2024-04-15T15:16:10.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.059+0000 7f7d31b74640 1 -- 172.21.15.72:0/847800549 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7d1806c920 msgr2=0x7f7d1806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.059+0000 7f7d31b74640 1 --2- 172.21.15.72:0/847800549 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7d1806c920 0x7f7d1806ede0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7d1c009560 tx=0x7f7d1c008040 comp rx=0 tx=0).stop 2024-04-15T15:16:10.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.059+0000 7f7d31b74640 1 -- 172.21.15.72:0/847800549 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d2c0d0c70 msgr2=0x7f7d2c1f4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.059+0000 7f7d31b74640 1 --2- 172.21.15.72:0/847800549 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d2c0d0c70 0x7f7d2c1f4450 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f7d20009d90 tx=0x7f7d20004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:10.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.059+0000 7f7d31b74640 1 -- 172.21.15.72:0/847800549 shutdown_connections 2024-04-15T15:16:10.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.059+0000 7f7d31b74640 1 --2- 172.21.15.72:0/847800549 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7d1806c920 0x7f7d1806ede0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.059+0000 7f7d31b74640 1 --2- 172.21.15.72:0/847800549 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7d2c15c210 0x7f7d2c157ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.059+0000 7f7d31b74640 1 --2- 172.21.15.72:0/847800549 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7d2c0d0c70 0x7f7d2c1f4450 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.059+0000 7f7d31b74640 1 --2- 172.21.15.72:0/847800549 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7d2c0cca00 0x7f7d2c156e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.059+0000 7f7d31b74640 1 -- 172.21.15.72:0/847800549 >> 172.21.15.72:0/847800549 conn(0x7f7d2c0b3fa0 msgr2=0x7f7d2c0b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:10.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.059+0000 7f7d31b74640 1 -- 172.21.15.72:0/847800549 shutdown_connections 2024-04-15T15:16:10.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.059+0000 7f7d31b74640 1 -- 172.21.15.72:0/847800549 wait complete. 2024-04-15T15:16:10.073 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:10.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.173+0000 7fb408ff0640 1 Processor -- start 2024-04-15T15:16:10.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.173+0000 7fb408ff0640 1 -- start start 2024-04-15T15:16:10.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.174+0000 7fb408ff0640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb404070310 0x7fb404070770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.174+0000 7fb408ff0640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb404162140 con 0x7fb4040ae340 2024-04-15T15:16:10.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.174+0000 7fb408ff0640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4040b42d0 con 0x7fb404161d60 2024-04-15T15:16:10.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.174+0000 7fb408ff0640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fb4040b5650 con 0x7fb404070310 2024-04-15T15:16:10.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.174+0000 7fb402d9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fb4040ae340 0x7fb4040ae790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:58850/0 (socket says 172.21.15.72:58850) 2024-04-15T15:16:10.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.175+0000 7fb402d9d640 1 -- 172.21.15.72:0/1778299537 learned_addr learned my addr 172.21.15.72:0/1778299537 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:10.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.175+0000 7fb401d9b640 1 --2- 172.21.15.72:0/1778299537 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb404070310 0x7fb404070770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.175+0000 7fb40159a640 1 -- 172.21.15.72:0/1778299537 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1195594622 0 0) 0x7fb404162140 con 0x7fb4040ae340 2024-04-15T15:16:10.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.175+0000 7fb40159a640 1 -- 172.21.15.72:0/1778299537 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb3ec003520 con 0x7fb4040ae340 2024-04-15T15:16:10.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.175+0000 7fb40159a640 1 -- 172.21.15.72:0/1778299537 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2465735904 0 0) 0x7fb3ec003520 con 0x7fb4040ae340 2024-04-15T15:16:10.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.175+0000 7fb40159a640 1 -- 172.21.15.72:0/1778299537 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb404070310 msgr2=0x7fb404070770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:10.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.175+0000 7fb40159a640 1 --2- 172.21.15.72:0/1778299537 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb404070310 0x7fb404070770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.175+0000 7fb40159a640 1 -- 172.21.15.72:0/1778299537 >> v1:172.21.15.175:6789/0 conn(0x7fb404161d60 legacy=0x7fb40406f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.175+0000 7fb40159a640 1 -- 172.21.15.72:0/1778299537 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4040b45b0 con 0x7fb4040ae340 2024-04-15T15:16:10.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.175+0000 7fb40159a640 1 -- 172.21.15.72:0/1778299537 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7fb3fc001670 con 0x7fb4040ae340 2024-04-15T15:16:10.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.175+0000 7fb40159a640 1 -- 172.21.15.72:0/1778299537 >> v1:172.21.15.72:6789/0 conn(0x7fb4040ae340 legacy=0x7fb4040ae790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.175+0000 7fb40159a640 1 --2- 172.21.15.72:0/1778299537 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb3ec0039e0 0x7fb3ec003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.175+0000 7fb40159a640 1 --2- 172.21.15.72:0/1778299537 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb404070310 0x7fb3ec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.175+0000 7fb40159a640 1 --2- 172.21.15.72:0/1778299537 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3ec004d40 0x7fb3ec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.175+0000 7fb40159a640 1 -- 172.21.15.72:0/1778299537 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ec00b9c0 con 0x7fb3ec004d40 2024-04-15T15:16:10.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.175+0000 7fb40159a640 1 -- 172.21.15.72:0/1778299537 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ec00bb40 con 0x7fb3ec0039e0 2024-04-15T15:16:10.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.175+0000 7fb40159a640 1 -- 172.21.15.72:0/1778299537 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fb3ec00bd40 con 0x7fb404070310 2024-04-15T15:16:10.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.175+0000 7fb402d9d640 1 --2- 172.21.15.72:0/1778299537 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3ec004d40 0x7fb3ec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.175+0000 7fb401d9b640 1 --2- 172.21.15.72:0/1778299537 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb404070310 0x7fb3ec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.175+0000 7fb40259c640 1 --2- 172.21.15.72:0/1778299537 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb3ec0039e0 0x7fb3ec003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.175+0000 7fb402d9d640 1 -- 172.21.15.72:0/1778299537 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb404070310 msgr2=0x7fb3ec004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:10.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.175+0000 7fb402d9d640 1 --2- 172.21.15.72:0/1778299537 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb404070310 0x7fb3ec004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.175+0000 7fb402d9d640 1 -- 172.21.15.72:0/1778299537 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb3ec0039e0 msgr2=0x7fb3ec003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:10.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.175+0000 7fb402d9d640 1 --2- 172.21.15.72:0/1778299537 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb3ec0039e0 0x7fb3ec003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.176+0000 7fb402d9d640 1 -- 172.21.15.72:0/1778299537 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3ec003520 con 0x7fb3ec004d40 2024-04-15T15:16:10.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.176+0000 7fb402d9d640 1 --2- 172.21.15.72:0/1778299537 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3ec004d40 0x7fb3ec0051c0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00c210 tx=0x7fb3ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a27624cb6b23aed server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:10.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.176+0000 7fb40159a640 1 -- 172.21.15.72:0/1778299537 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc007ce0 con 0x7fb3ec004d40 2024-04-15T15:16:10.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.176+0000 7fb40159a640 1 -- 172.21.15.72:0/1778299537 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3fc030070 con 0x7fb3ec004d40 2024-04-15T15:16:10.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.176+0000 7fb40159a640 1 -- 172.21.15.72:0/1778299537 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc005d30 con 0x7fb3ec004d40 2024-04-15T15:16:10.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.177+0000 7fb408ff0640 1 -- 172.21.15.72:0/1778299537 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3ec004d40 msgr2=0x7fb3ec0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.177+0000 7fb408ff0640 1 --2- 172.21.15.72:0/1778299537 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3ec004d40 0x7fb3ec0051c0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00c210 tx=0x7fb3ec005830 comp rx=0 tx=0).stop 2024-04-15T15:16:10.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.177+0000 7fb402d9d640 1 -- 172.21.15.72:0/1778299537 reap_dead start 2024-04-15T15:16:10.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.177+0000 7fb408ff0640 1 -- 172.21.15.72:0/1778299537 shutdown_connections 2024-04-15T15:16:10.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.177+0000 7fb408ff0640 1 -- 172.21.15.72:0/1778299537 >> 172.21.15.72:0/1778299537 conn(0x7fb4040cdd20 msgr2=0x7fb4040ce100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:10.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.177+0000 7fb408ff0640 1 -- 172.21.15.72:0/1778299537 shutdown_connections 2024-04-15T15:16:10.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.177+0000 7fb408ff0640 1 -- 172.21.15.72:0/1778299537 wait complete. 2024-04-15T15:16:10.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.178+0000 7fb408ff0640 1 Processor -- start 2024-04-15T15:16:10.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.178+0000 7fb408ff0640 1 -- start start 2024-04-15T15:16:10.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.179+0000 7fb408ff0640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb3ec004d40 0x7fb40406ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.179+0000 7fb408ff0640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb40415a850 0x7fb4041f7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.179+0000 7fb408ff0640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb40415b820 0x7fb40406d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.179+0000 7fb408ff0640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb4040b4e50 con 0x7fb40415b820 2024-04-15T15:16:10.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.179+0000 7fb408ff0640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fb4040b4b70 con 0x7fb40415a850 2024-04-15T15:16:10.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.179+0000 7fb408ff0640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fb4040b5130 con 0x7fb3ec004d40 2024-04-15T15:16:10.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.179+0000 7fb402d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb40415b820 0x7fb40406d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.179+0000 7fb402d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb40415b820 0x7fb40406d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42222/0 (socket says 172.21.15.72:42222) 2024-04-15T15:16:10.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.179+0000 7fb402d9d640 1 -- 172.21.15.72:0/221089390 learned_addr learned my addr 172.21.15.72:0/221089390 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:10.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.179+0000 7fb40259c640 1 --2- 172.21.15.72:0/221089390 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb3ec004d40 0x7fb40406ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.179+0000 7fb401d9b640 1 --2- 172.21.15.72:0/221089390 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb40415a850 0x7fb4041f7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.179+0000 7fb402d9d640 1 -- 172.21.15.72:0/221089390 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb3ec004d40 msgr2=0x7fb40406ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.179+0000 7fb402d9d640 1 --2- 172.21.15.72:0/221089390 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb3ec004d40 0x7fb40406ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.179+0000 7fb402d9d640 1 -- 172.21.15.72:0/221089390 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb40415a850 msgr2=0x7fb4041f7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.179+0000 7fb402d9d640 1 --2- 172.21.15.72:0/221089390 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb40415a850 0x7fb4041f7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.179+0000 7fb402d9d640 1 -- 172.21.15.72:0/221089390 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4041f7cc0 con 0x7fb40415b820 2024-04-15T15:16:10.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.180+0000 7fb40259c640 1 --2- 172.21.15.72:0/221089390 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb3ec004d40 0x7fb40406ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:10.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.180+0000 7fb402d9d640 1 --2- 172.21.15.72:0/221089390 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb40415b820 0x7fb40406d7c0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00c330 tx=0x7fb3fc02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:10.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.180+0000 7fb3f2ffd640 1 -- 172.21.15.72:0/221089390 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc007ce0 con 0x7fb40415b820 2024-04-15T15:16:10.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.180+0000 7fb3f2ffd640 1 -- 172.21.15.72:0/221089390 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3fc0049e0 con 0x7fb40415b820 2024-04-15T15:16:10.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.180+0000 7fb408ff0640 1 -- 172.21.15.72:0/221089390 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4041f7fa0 con 0x7fb40415b820 2024-04-15T15:16:10.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.180+0000 7fb3f2ffd640 1 -- 172.21.15.72:0/221089390 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc004d90 con 0x7fb40415b820 2024-04-15T15:16:10.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.180+0000 7fb408ff0640 1 -- 172.21.15.72:0/221089390 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4041f7890 con 0x7fb40415b820 2024-04-15T15:16:10.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.181+0000 7fb408ff0640 1 -- 172.21.15.72:0/221089390 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4040690f0 con 0x7fb40415b820 2024-04-15T15:16:10.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.181+0000 7fb3f2ffd640 1 -- 172.21.15.72:0/221089390 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb3fc007070 con 0x7fb40415b820 2024-04-15T15:16:10.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.182+0000 7fb3f2ffd640 1 --2- 172.21.15.72:0/221089390 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb3e406c8f0 0x7fb3e406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.182+0000 7fb40259c640 1 --2- 172.21.15.72:0/221089390 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb3e406c8f0 0x7fb3e406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.182+0000 7fb3f2ffd640 1 -- 172.21.15.72:0/221089390 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fb3fc0acd00 con 0x7fb40415b820 2024-04-15T15:16:10.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.182+0000 7fb40259c640 1 --2- 172.21.15.72:0/221089390 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb3e406c8f0 0x7fb3e406edb0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4000f80 tx=0x7fb3f4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:10.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.186+0000 7fb3f2ffd640 1 -- 172.21.15.72:0/221089390 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb3fc07b030 con 0x7fb40415b820 2024-04-15T15:16:10.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.362+0000 7fb408ff0640 1 -- 172.21.15.72:0/221089390 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb4041f8170 con 0x7fb40415b820 2024-04-15T15:16:10.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.363+0000 7fb3f2ffd640 1 -- 172.21.15.72:0/221089390 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+16942 (secure 0 0 0) 0x7fb3fc07a930 con 0x7fb40415b820 2024-04-15T15:16:10.366 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:16:10.366 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":38,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:16:08.702307+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:05.551499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:06.003152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:10.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.365+0000 7fb408ff0640 1 -- 172.21.15.72:0/221089390 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb3e406c8f0 msgr2=0x7fb3e406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.365+0000 7fb408ff0640 1 --2- 172.21.15.72:0/221089390 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb3e406c8f0 0x7fb3e406edb0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4000f80 tx=0x7fb3f4009040 comp rx=0 tx=0).stop 2024-04-15T15:16:10.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.365+0000 7fb408ff0640 1 -- 172.21.15.72:0/221089390 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb40415b820 msgr2=0x7fb40406d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.365+0000 7fb408ff0640 1 --2- 172.21.15.72:0/221089390 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb40415b820 0x7fb40406d7c0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00c330 tx=0x7fb3fc02e070 comp rx=0 tx=0).stop 2024-04-15T15:16:10.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.365+0000 7fb408ff0640 1 -- 172.21.15.72:0/221089390 shutdown_connections 2024-04-15T15:16:10.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.365+0000 7fb408ff0640 1 --2- 172.21.15.72:0/221089390 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb3e406c8f0 0x7fb3e406edb0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.365+0000 7fb408ff0640 1 --2- 172.21.15.72:0/221089390 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb40415b820 0x7fb40406d7c0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.365+0000 7fb408ff0640 1 --2- 172.21.15.72:0/221089390 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb40415a850 0x7fb4041f7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.365+0000 7fb408ff0640 1 --2- 172.21.15.72:0/221089390 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb3ec004d40 0x7fb40406ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.365+0000 7fb408ff0640 1 -- 172.21.15.72:0/221089390 >> 172.21.15.72:0/221089390 conn(0x7fb4040cdd20 msgr2=0x7fb4040b2400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:10.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.365+0000 7fb408ff0640 1 -- 172.21.15.72:0/221089390 shutdown_connections 2024-04-15T15:16:10.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.365+0000 7fb408ff0640 1 -- 172.21.15.72:0/221089390 wait complete. 2024-04-15T15:16:10.379 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-15T15:16:10.379 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:16:10.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.479+0000 7fd987e93640 1 Processor -- start 2024-04-15T15:16:10.482 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.480+0000 7fd987e93640 1 -- start start 2024-04-15T15:16:10.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.480+0000 7fd987e93640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd980073290 0x7fd9800b67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.480+0000 7fd987e93640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd980162800 con 0x7fd9800ae360 2024-04-15T15:16:10.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.481+0000 7fd987e93640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd980162ae0 con 0x7fd98015e7e0 2024-04-15T15:16:10.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.481+0000 7fd987e93640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fd980162dc0 con 0x7fd980073290 2024-04-15T15:16:10.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.481+0000 7fd986430640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fd9800ae360 0x7fd9800b46b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:58862/0 (socket says 172.21.15.72:58862) 2024-04-15T15:16:10.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.481+0000 7fd986430640 1 -- 172.21.15.72:0/3363047802 learned_addr learned my addr 172.21.15.72:0/3363047802 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:10.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.481+0000 7fd98542e640 1 --2- 172.21.15.72:0/3363047802 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd980073290 0x7fd9800b67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.481+0000 7fd984c2d640 1 -- 172.21.15.72:0/3363047802 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1878278478 0 0) 0x7fd980162800 con 0x7fd9800ae360 2024-04-15T15:16:10.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.481+0000 7fd984c2d640 1 -- 172.21.15.72:0/3363047802 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd96c003520 con 0x7fd9800ae360 2024-04-15T15:16:10.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.481+0000 7fd984c2d640 1 -- 172.21.15.72:0/3363047802 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 720563676 0 0) 0x7fd96c003520 con 0x7fd9800ae360 2024-04-15T15:16:10.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.481+0000 7fd984c2d640 1 -- 172.21.15.72:0/3363047802 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd980073290 msgr2=0x7fd9800b67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:10.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.481+0000 7fd984c2d640 1 --2- 172.21.15.72:0/3363047802 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd980073290 0x7fd9800b67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.481+0000 7fd984c2d640 1 -- 172.21.15.72:0/3363047802 >> v1:172.21.15.175:6789/0 conn(0x7fd98015e7e0 legacy=0x7fd980072b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.481+0000 7fd984c2d640 1 -- 172.21.15.72:0/3363047802 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9801630a0 con 0x7fd9800ae360 2024-04-15T15:16:10.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.481+0000 7fd984c2d640 1 -- 172.21.15.72:0/3363047802 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7fd9780019f0 con 0x7fd9800ae360 2024-04-15T15:16:10.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.481+0000 7fd984c2d640 1 -- 172.21.15.72:0/3363047802 >> v1:172.21.15.72:6789/0 conn(0x7fd9800ae360 legacy=0x7fd9800b46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.481+0000 7fd984c2d640 1 --2- 172.21.15.72:0/3363047802 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd96c0039e0 0x7fd96c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.481+0000 7fd984c2d640 1 --2- 172.21.15.72:0/3363047802 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd96c004800 0x7fd96c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.481+0000 7fd984c2d640 1 --2- 172.21.15.72:0/3363047802 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd980073290 0x7fd96c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.481+0000 7fd984c2d640 1 -- 172.21.15.72:0/3363047802 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd980162dc0 con 0x7fd96c0039e0 2024-04-15T15:16:10.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.481+0000 7fd984c2d640 1 -- 172.21.15.72:0/3363047802 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fd96c00ba70 con 0x7fd96c004800 2024-04-15T15:16:10.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.481+0000 7fd984c2d640 1 -- 172.21.15.72:0/3363047802 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fd96c00bc70 con 0x7fd980073290 2024-04-15T15:16:10.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.481+0000 7fd985c2f640 1 --2- 172.21.15.72:0/3363047802 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd96c0039e0 0x7fd96c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.481+0000 7fd98542e640 1 --2- 172.21.15.72:0/3363047802 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd96c004800 0x7fd96c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.482+0000 7fd986430640 1 --2- 172.21.15.72:0/3363047802 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd980073290 0x7fd96c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.482+0000 7fd985c2f640 1 -- 172.21.15.72:0/3363047802 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd980073290 msgr2=0x7fd96c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:10.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.482+0000 7fd985c2f640 1 --2- 172.21.15.72:0/3363047802 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd980073290 0x7fd96c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.482+0000 7fd985c2f640 1 -- 172.21.15.72:0/3363047802 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd96c004800 msgr2=0x7fd96c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:10.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.482+0000 7fd985c2f640 1 --2- 172.21.15.72:0/3363047802 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd96c004800 0x7fd96c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.482+0000 7fd985c2f640 1 -- 172.21.15.72:0/3363047802 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd96c003520 con 0x7fd96c0039e0 2024-04-15T15:16:10.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.482+0000 7fd98542e640 1 --2- 172.21.15.72:0/3363047802 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd96c004800 0x7fd96c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:10.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.482+0000 7fd985c2f640 1 --2- 172.21.15.72:0/3363047802 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd96c0039e0 0x7fd96c003e00 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fd974001750 tx=0x7fd96c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b98e5d797d21187 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:10.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.483+0000 7fd984c2d640 1 -- 172.21.15.72:0/3363047802 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd974001370 con 0x7fd96c0039e0 2024-04-15T15:16:10.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.483+0000 7fd984c2d640 1 -- 172.21.15.72:0/3363047802 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9740052d0 con 0x7fd96c0039e0 2024-04-15T15:16:10.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.483+0000 7fd984c2d640 1 -- 172.21.15.72:0/3363047802 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd974003830 con 0x7fd96c0039e0 2024-04-15T15:16:10.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.483+0000 7fd987e93640 1 -- 172.21.15.72:0/3363047802 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd96c0039e0 msgr2=0x7fd96c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.483+0000 7fd987e93640 1 --2- 172.21.15.72:0/3363047802 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd96c0039e0 0x7fd96c003e00 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fd974001750 tx=0x7fd96c005830 comp rx=0 tx=0).stop 2024-04-15T15:16:10.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.483+0000 7fd986430640 1 -- 172.21.15.72:0/3363047802 reap_dead start 2024-04-15T15:16:10.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.483+0000 7fd987e93640 1 -- 172.21.15.72:0/3363047802 shutdown_connections 2024-04-15T15:16:10.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.483+0000 7fd987e93640 1 -- 172.21.15.72:0/3363047802 >> 172.21.15.72:0/3363047802 conn(0x7fd9800b31b0 msgr2=0x7fd9800b3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:10.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.483+0000 7fd987e93640 1 -- 172.21.15.72:0/3363047802 shutdown_connections 2024-04-15T15:16:10.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.483+0000 7fd987e93640 1 -- 172.21.15.72:0/3363047802 wait complete. 2024-04-15T15:16:10.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.484+0000 7fd987e93640 1 Processor -- start 2024-04-15T15:16:10.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.484+0000 7fd987e93640 1 -- start start 2024-04-15T15:16:10.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.485+0000 7fd987e93640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd96c0039e0 0x7fd9801577a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.485+0000 7fd987e93640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd98006edf0 0x7fd9801ea540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.485+0000 7fd987e93640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd980153f60 0x7fd9801611b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.485+0000 7fd987e93640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd980162550 con 0x7fd98006edf0 2024-04-15T15:16:10.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.485+0000 7fd987e93640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fd9801623d0 con 0x7fd980153f60 2024-04-15T15:16:10.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.485+0000 7fd987e93640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fd98006fd30 con 0x7fd96c0039e0 2024-04-15T15:16:10.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.485+0000 7fd98542e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd98006edf0 0x7fd9801ea540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.485+0000 7fd98542e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd98006edf0 0x7fd9801ea540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42244/0 (socket says 172.21.15.72:42244) 2024-04-15T15:16:10.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.485+0000 7fd98542e640 1 -- 172.21.15.72:0/3695158684 learned_addr learned my addr 172.21.15.72:0/3695158684 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:10.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.485+0000 7fd986430640 1 --2- 172.21.15.72:0/3695158684 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd980153f60 0x7fd9801611b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.485+0000 7fd985c2f640 1 --2- 172.21.15.72:0/3695158684 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd96c0039e0 0x7fd9801577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.486+0000 7fd98542e640 1 -- 172.21.15.72:0/3695158684 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd96c0039e0 msgr2=0x7fd9801577a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.486+0000 7fd98542e640 1 --2- 172.21.15.72:0/3695158684 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd96c0039e0 0x7fd9801577a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.486+0000 7fd98542e640 1 -- 172.21.15.72:0/3695158684 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd980153f60 msgr2=0x7fd9801611b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.486+0000 7fd98542e640 1 --2- 172.21.15.72:0/3695158684 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd980153f60 0x7fd9801611b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.486+0000 7fd98542e640 1 -- 172.21.15.72:0/3695158684 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd98015f7f0 con 0x7fd98006edf0 2024-04-15T15:16:10.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.486+0000 7fd985c2f640 1 --2- 172.21.15.72:0/3695158684 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd96c0039e0 0x7fd9801577a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:10.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.486+0000 7fd986430640 1 --2- 172.21.15.72:0/3695158684 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd980153f60 0x7fd9801611b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:10.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.486+0000 7fd98542e640 1 --2- 172.21.15.72:0/3695158684 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd98006edf0 0x7fd9801ea540 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fd97c011600 tx=0x7fd97c011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:10.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.486+0000 7fd9727fc640 1 -- 172.21.15.72:0/3695158684 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd97c00f920 con 0x7fd98006edf0 2024-04-15T15:16:10.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.486+0000 7fd9727fc640 1 -- 172.21.15.72:0/3695158684 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd97c00fac0 con 0x7fd98006edf0 2024-04-15T15:16:10.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.486+0000 7fd987e93640 1 -- 172.21.15.72:0/3695158684 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd98015fad0 con 0x7fd98006edf0 2024-04-15T15:16:10.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.486+0000 7fd9727fc640 1 -- 172.21.15.72:0/3695158684 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd97c00fe90 con 0x7fd98006edf0 2024-04-15T15:16:10.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.486+0000 7fd987e93640 1 -- 172.21.15.72:0/3695158684 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9801543e0 con 0x7fd98006edf0 2024-04-15T15:16:10.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.487+0000 7fd987e93640 1 -- 172.21.15.72:0/3695158684 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9801eaa80 con 0x7fd98006edf0 2024-04-15T15:16:10.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.487+0000 7fd9727fc640 1 -- 172.21.15.72:0/3695158684 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd97c015460 con 0x7fd98006edf0 2024-04-15T15:16:10.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.488+0000 7fd9727fc640 1 --2- 172.21.15.72:0/3695158684 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd96406ca90 0x7fd96406ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.488+0000 7fd9727fc640 1 -- 172.21.15.72:0/3695158684 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fd97c08c4f0 con 0x7fd98006edf0 2024-04-15T15:16:10.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.488+0000 7fd985c2f640 1 --2- 172.21.15.72:0/3695158684 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd96406ca90 0x7fd96406ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.494 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.488+0000 7fd985c2f640 1 --2- 172.21.15.72:0/3695158684 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd96406ca90 0x7fd96406ef50 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd974033000 tx=0x7fd974009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:10.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.492+0000 7fd9727fc640 1 -- 172.21.15.72:0/3695158684 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd97c019030 con 0x7fd98006edf0 2024-04-15T15:16:10.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.663+0000 7fd987e93640 1 -- 172.21.15.72:0/3695158684 --> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd98006f250 con 0x7fd96406ca90 2024-04-15T15:16:10.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.669+0000 7fd9727fc640 1 -- 172.21.15.72:0/3695158684 <== mgr.4101 v2:172.21.15.72:6832/3503807805 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+237200 (secure 0 0 0) 0x7fd974005250 con 0x7fd96406ca90 2024-04-15T15:16:10.675 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:16:10.676 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-04-15T15:16:10.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.672+0000 7fd987e93640 1 -- 172.21.15.72:0/3695158684 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd96406ca90 msgr2=0x7fd96406ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.672+0000 7fd987e93640 1 --2- 172.21.15.72:0/3695158684 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd96406ca90 0x7fd96406ef50 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd974033000 tx=0x7fd974009030 comp rx=0 tx=0).stop 2024-04-15T15:16:10.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.672+0000 7fd987e93640 1 -- 172.21.15.72:0/3695158684 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd98006edf0 msgr2=0x7fd9801ea540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.678 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.672+0000 7fd987e93640 1 --2- 172.21.15.72:0/3695158684 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd98006edf0 0x7fd9801ea540 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fd97c011600 tx=0x7fd97c011ac0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.672+0000 7fd987e93640 1 -- 172.21.15.72:0/3695158684 shutdown_connections 2024-04-15T15:16:10.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.672+0000 7fd987e93640 1 --2- 172.21.15.72:0/3695158684 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd96406ca90 0x7fd96406ef50 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.672+0000 7fd987e93640 1 --2- 172.21.15.72:0/3695158684 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd980153f60 0x7fd9801611b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.672+0000 7fd987e93640 1 --2- 172.21.15.72:0/3695158684 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd98006edf0 0x7fd9801ea540 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.672+0000 7fd987e93640 1 --2- 172.21.15.72:0/3695158684 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd96c0039e0 0x7fd9801577a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.672+0000 7fd987e93640 1 -- 172.21.15.72:0/3695158684 >> 172.21.15.72:0/3695158684 conn(0x7fd9800b31b0 msgr2=0x7fd9800b45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:10.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.672+0000 7fd987e93640 1 -- 172.21.15.72:0/3695158684 shutdown_connections 2024-04-15T15:16:10.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.672+0000 7fd987e93640 1 -- 172.21.15.72:0/3695158684 wait complete. 2024-04-15T15:16:10.686 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-04-15T15:16:09.359072+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":66,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":220420,"kb_used_data":4200,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754300,"statfs":{"total":773094113280,"available":772868403200,"internally_reserved":0,"allocated":4300800,"data_stored":2487144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000702"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":194,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-15T15:16:01.690588+0000","last_change":"2024-04-15T15:14:40.494279+0000","last_active":"2024-04-15T15:16:01.690588+0000","last_peered":"2024-04-15T15:16:01.690588+0000","last_clean":"2024-04-15T15:16:01.690588+0000","last_became_active":"2024-04-15T15:14:40.493404+0000","last_became_peered":"2024-04-15T15:14:40.493404+0000","last_unstale":"2024-04-15T15:16:01.690588+0000","last_undegraded":"2024-04-15T15:16:01.690588+0000","last_fullsized":"2024-04-15T15:16:01.690588+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:39.418508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:39.418508+0000","last_clean_scrub_stamp":"2024-04-15T15:14:39.418508+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:01:51.789246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343856,"statfs":{"total":96636764160,"available":96608108544,"internally_reserved":0,"allocated":983040,"data_stored":753669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":163301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27412,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":389120,"data_stored":163301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27412,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":389120,"data_stored":163301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27408,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":389120,"data_stored":163301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":163301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":163301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27996,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343844,"statfs":{"total":96636764160,"available":96608096256,"internally_reserved":0,"allocated":983040,"data_stored":753669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:16:10.690 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:16:10.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.826+0000 7fd4891f4640 1 Processor -- start 2024-04-15T15:16:10.829 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.826+0000 7fd4891f4640 1 -- start start 2024-04-15T15:16:10.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.827+0000 7fd4891f4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd48413f3c0 0x7fd48406f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.827+0000 7fd4891f4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd484141f60 con 0x7fd4840b49e0 2024-04-15T15:16:10.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.827+0000 7fd4891f4640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd484142240 con 0x7fd48413f7a0 2024-04-15T15:16:10.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.827+0000 7fd4891f4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fd484141d10 con 0x7fd48413f3c0 2024-04-15T15:16:10.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.827+0000 7fd482d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd48413f3c0 0x7fd48406f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.827+0000 7fd482d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd48413f3c0 0x7fd48406f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42260/0 (socket says 172.21.15.72:42260) 2024-04-15T15:16:10.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.827+0000 7fd482d9d640 1 -- 172.21.15.72:0/2675115685 learned_addr learned my addr 172.21.15.72:0/2675115685 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:10.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.827+0000 7fd48359e640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fd4840b49e0 0x7fd48415cb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:58876/0 (socket says 172.21.15.72:58876) 2024-04-15T15:16:10.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.828+0000 7fd482d9d640 1 -- 172.21.15.72:0/2675115685 >> v1:172.21.15.175:6789/0 conn(0x7fd48413f7a0 legacy=0x7fd4840b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:10.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.828+0000 7fd482d9d640 1 -- 172.21.15.72:0/2675115685 >> v1:172.21.15.72:6789/0 conn(0x7fd4840b49e0 legacy=0x7fd48415cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.828+0000 7fd482d9d640 1 -- 172.21.15.72:0/2675115685 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd484142520 con 0x7fd48413f3c0 2024-04-15T15:16:10.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.828+0000 7fd482d9d640 1 --2- 172.21.15.72:0/2675115685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd48413f3c0 0x7fd48406f510 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fd474009e50 tx=0x7fd47402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20ae7c683c19d2e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:10.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.828+0000 7fd481d9b640 1 -- 172.21.15.72:0/2675115685 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd47402ca80 con 0x7fd48413f3c0 2024-04-15T15:16:10.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.828+0000 7fd481d9b640 1 -- 172.21.15.72:0/2675115685 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd47402cc20 con 0x7fd48413f3c0 2024-04-15T15:16:10.832 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.828+0000 7fd481d9b640 1 -- 172.21.15.72:0/2675115685 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4740343f0 con 0x7fd48413f3c0 2024-04-15T15:16:10.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.829+0000 7fd4891f4640 1 -- 172.21.15.72:0/2675115685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd48413f3c0 msgr2=0x7fd48406f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.829+0000 7fd4891f4640 1 --2- 172.21.15.72:0/2675115685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd48413f3c0 0x7fd48406f510 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fd474009e50 tx=0x7fd47402bee0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.829+0000 7fd4891f4640 1 -- 172.21.15.72:0/2675115685 shutdown_connections 2024-04-15T15:16:10.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.829+0000 7fd4891f4640 1 --2- 172.21.15.72:0/2675115685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd48413f3c0 0x7fd48406f510 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.829+0000 7fd4891f4640 1 -- 172.21.15.72:0/2675115685 >> 172.21.15.72:0/2675115685 conn(0x7fd4840cdd20 msgr2=0x7fd4840ce100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:10.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.829+0000 7fd4891f4640 1 -- 172.21.15.72:0/2675115685 shutdown_connections 2024-04-15T15:16:10.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.829+0000 7fd4891f4640 1 -- 172.21.15.72:0/2675115685 wait complete. 2024-04-15T15:16:10.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.831+0000 7fd4891f4640 1 Processor -- start 2024-04-15T15:16:10.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.831+0000 7fd4891f4640 1 -- start start 2024-04-15T15:16:10.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.831+0000 7fd4891f4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4840b49e0 0x7fd4840aff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.831+0000 7fd4891f4640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd48413f3c0 0x7fd4840aaf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.831+0000 7fd4891f4640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd48413f7a0 0x7fd48414cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.831+0000 7fd4891f4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd48406fbd0 con 0x7fd4840b49e0 2024-04-15T15:16:10.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.831+0000 7fd4891f4640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fd48406fa50 con 0x7fd48413f7a0 2024-04-15T15:16:10.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.831+0000 7fd4891f4640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fd48406fd50 con 0x7fd48413f3c0 2024-04-15T15:16:10.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.831+0000 7fd482d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4840b49e0 0x7fd4840aff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.831+0000 7fd482d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4840b49e0 0x7fd4840aff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42272/0 (socket says 172.21.15.72:42272) 2024-04-15T15:16:10.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.831+0000 7fd482d9d640 1 -- 172.21.15.72:0/1587703773 learned_addr learned my addr 172.21.15.72:0/1587703773 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:10.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.832+0000 7fd482d9d640 1 -- 172.21.15.72:0/1587703773 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd48413f3c0 msgr2=0x7fd4840aaf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.832+0000 7fd482d9d640 1 --2- 172.21.15.72:0/1587703773 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd48413f3c0 0x7fd4840aaf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.832+0000 7fd482d9d640 1 -- 172.21.15.72:0/1587703773 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd48413f7a0 msgr2=0x7fd48414cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.832+0000 7fd482d9d640 1 --2- 172.21.15.72:0/1587703773 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd48413f7a0 0x7fd48414cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.832+0000 7fd482d9d640 1 -- 172.21.15.72:0/1587703773 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4840ab460 con 0x7fd4840b49e0 2024-04-15T15:16:10.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.832+0000 7fd482d9d640 1 --2- 172.21.15.72:0/1587703773 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4840b49e0 0x7fd4840aff90 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fd474036040 tx=0x7fd474004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:10.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.832+0000 7fd46b7fe640 1 -- 172.21.15.72:0/1587703773 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd474043020 con 0x7fd4840b49e0 2024-04-15T15:16:10.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.832+0000 7fd46b7fe640 1 -- 172.21.15.72:0/1587703773 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd47403e450 con 0x7fd4840b49e0 2024-04-15T15:16:10.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.832+0000 7fd46b7fe640 1 -- 172.21.15.72:0/1587703773 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd474009510 con 0x7fd4840b49e0 2024-04-15T15:16:10.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.832+0000 7fd4891f4640 1 -- 172.21.15.72:0/1587703773 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd48414d000 con 0x7fd4840b49e0 2024-04-15T15:16:10.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.832+0000 7fd4891f4640 1 -- 172.21.15.72:0/1587703773 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4840aba10 con 0x7fd4840b49e0 2024-04-15T15:16:10.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.833+0000 7fd4891f4640 1 -- 172.21.15.72:0/1587703773 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4840690f0 con 0x7fd4840b49e0 2024-04-15T15:16:10.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.833+0000 7fd46b7fe640 1 -- 172.21.15.72:0/1587703773 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd474046020 con 0x7fd4840b49e0 2024-04-15T15:16:10.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.834+0000 7fd46b7fe640 1 --2- 172.21.15.72:0/1587703773 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd47006c8f0 0x7fd47006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.834+0000 7fd46b7fe640 1 -- 172.21.15.72:0/1587703773 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fd4740ae660 con 0x7fd4840b49e0 2024-04-15T15:16:10.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.834+0000 7fd48259c640 1 --2- 172.21.15.72:0/1587703773 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd47006c8f0 0x7fd47006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.834+0000 7fd48259c640 1 --2- 172.21.15.72:0/1587703773 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd47006c8f0 0x7fd47006edb0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd484156b70 tx=0x7fd478007990 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:10.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:10.838+0000 7fd46b7fe640 1 -- 172.21.15.72:0/1587703773 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd47404d030 con 0x7fd4840b49e0 2024-04-15T15:16:11.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.010+0000 7fd4891f4640 1 -- 172.21.15.72:0/1587703773 --> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd484070fd0 con 0x7fd47006c8f0 2024-04-15T15:16:11.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.015+0000 7fd46b7fe640 1 -- 172.21.15.72:0/1587703773 <== mgr.4101 v2:172.21.15.72:6832/3503807805 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+237200 (secure 0 0 0) 0x7fd484070fd0 con 0x7fd47006c8f0 2024-04-15T15:16:11.020 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:16:11.021 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-04-15T15:16:11.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.018+0000 7fd4891f4640 1 -- 172.21.15.72:0/1587703773 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd47006c8f0 msgr2=0x7fd47006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.018+0000 7fd4891f4640 1 --2- 172.21.15.72:0/1587703773 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd47006c8f0 0x7fd47006edb0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd484156b70 tx=0x7fd478007990 comp rx=0 tx=0).stop 2024-04-15T15:16:11.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.018+0000 7fd4891f4640 1 -- 172.21.15.72:0/1587703773 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4840b49e0 msgr2=0x7fd4840aff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.018+0000 7fd4891f4640 1 --2- 172.21.15.72:0/1587703773 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4840b49e0 0x7fd4840aff90 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fd474036040 tx=0x7fd474004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:11.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.018+0000 7fd4891f4640 1 -- 172.21.15.72:0/1587703773 shutdown_connections 2024-04-15T15:16:11.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.018+0000 7fd4891f4640 1 --2- 172.21.15.72:0/1587703773 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd47006c8f0 0x7fd47006edb0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.018+0000 7fd4891f4640 1 --2- 172.21.15.72:0/1587703773 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd48413f7a0 0x7fd48414cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.018+0000 7fd4891f4640 1 --2- 172.21.15.72:0/1587703773 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd48413f3c0 0x7fd4840aaf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.018+0000 7fd4891f4640 1 --2- 172.21.15.72:0/1587703773 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd4840b49e0 0x7fd4840aff90 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.018+0000 7fd4891f4640 1 -- 172.21.15.72:0/1587703773 >> 172.21.15.72:0/1587703773 conn(0x7fd4840cdd20 msgr2=0x7fd48413ded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:11.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.018+0000 7fd4891f4640 1 -- 172.21.15.72:0/1587703773 shutdown_connections 2024-04-15T15:16:11.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.018+0000 7fd4891f4640 1 -- 172.21.15.72:0/1587703773 wait complete. 2024-04-15T15:16:11.031 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-04-15T15:16:09.359072+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":66,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":220420,"kb_used_data":4200,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754300,"statfs":{"total":773094113280,"available":772868403200,"internally_reserved":0,"allocated":4300800,"data_stored":2487144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000702"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:06.681976+0000","last_change":"2024-04-15T15:16:06.681976+0000","last_active":"2024-04-15T15:16:06.681976+0000","last_peered":"2024-04-15T15:16:06.681976+0000","last_clean":"2024-04-15T15:16:06.681976+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:06.681976+0000","last_undegraded":"2024-04-15T15:16:06.681976+0000","last_fullsized":"2024-04-15T15:16:06.681976+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:05.677882+0000","last_change":"2024-04-15T15:16:05.677882+0000","last_active":"2024-04-15T15:16:05.677882+0000","last_peered":"2024-04-15T15:16:05.677882+0000","last_clean":"2024-04-15T15:16:05.677882+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:05.677882+0000","last_undegraded":"2024-04-15T15:16:05.677882+0000","last_fullsized":"2024-04-15T15:16:05.677882+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":194,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-04-15T15:16:01.690588+0000","last_change":"2024-04-15T15:14:40.494279+0000","last_active":"2024-04-15T15:16:01.690588+0000","last_peered":"2024-04-15T15:16:01.690588+0000","last_clean":"2024-04-15T15:16:01.690588+0000","last_became_active":"2024-04-15T15:14:40.493404+0000","last_became_peered":"2024-04-15T15:14:40.493404+0000","last_unstale":"2024-04-15T15:16:01.690588+0000","last_undegraded":"2024-04-15T15:16:01.690588+0000","last_fullsized":"2024-04-15T15:16:01.690588+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:39.418508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:39.418508+0000","last_clean_scrub_stamp":"2024-04-15T15:14:39.418508+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:01:51.789246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343856,"statfs":{"total":96636764160,"available":96608108544,"internally_reserved":0,"allocated":983040,"data_stored":753669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":163301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27412,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":389120,"data_stored":163301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27412,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":389120,"data_stored":163301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27408,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":389120,"data_stored":163301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":163301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":163301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27996,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343844,"statfs":{"total":96636764160,"available":96608096256,"internally_reserved":0,"allocated":983040,"data_stored":753669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:16:11.036 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:16:11.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.174+0000 7fe3cf51c640 1 Processor -- start 2024-04-15T15:16:11.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.174+0000 7fe3cf51c640 1 -- start start 2024-04-15T15:16:11.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.175+0000 7fe3cf51c640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe3c80d45b0 0x7fe3c80d4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.175+0000 7fe3cf51c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3c8162220 con 0x7fe3c8161e40 2024-04-15T15:16:11.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.175+0000 7fe3cf51c640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3c80d3070 con 0x7fe3c80d12b0 2024-04-15T15:16:11.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.175+0000 7fe3cf51c640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fe3c80b3f00 con 0x7fe3c80d45b0 2024-04-15T15:16:11.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.175+0000 7fe3cd2b8640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fe3c8161e40 0x7fe3c80b37f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:58884/0 (socket says 172.21.15.72:58884) 2024-04-15T15:16:11.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.175+0000 7fe3cd2b8640 1 -- 172.21.15.72:0/4227128023 learned_addr learned my addr 172.21.15.72:0/4227128023 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:11.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.176+0000 7fe3cdab9640 1 --2- 172.21.15.72:0/4227128023 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe3c80d45b0 0x7fe3c80d4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.176+0000 7fe3bffff640 1 -- 172.21.15.72:0/4227128023 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1105903730 0 0) 0x7fe3c8162220 con 0x7fe3c8161e40 2024-04-15T15:16:11.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.176+0000 7fe3bffff640 1 -- 172.21.15.72:0/4227128023 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe3b4003520 con 0x7fe3c8161e40 2024-04-15T15:16:11.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.176+0000 7fe3bffff640 1 -- 172.21.15.72:0/4227128023 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2053652864 0 0) 0x7fe3b4003520 con 0x7fe3c8161e40 2024-04-15T15:16:11.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.176+0000 7fe3bffff640 1 -- 172.21.15.72:0/4227128023 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe3c80d45b0 msgr2=0x7fe3c80d4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:11.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.176+0000 7fe3bffff640 1 --2- 172.21.15.72:0/4227128023 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe3c80d45b0 0x7fe3c80d4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.176+0000 7fe3bffff640 1 -- 172.21.15.72:0/4227128023 >> v1:172.21.15.175:6789/0 conn(0x7fe3c80d12b0 legacy=0x7fe3c80d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.176+0000 7fe3bffff640 1 -- 172.21.15.72:0/4227128023 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3c80d1820 con 0x7fe3c8161e40 2024-04-15T15:16:11.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.176+0000 7fe3bffff640 1 -- 172.21.15.72:0/4227128023 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7fe3c4001670 con 0x7fe3c8161e40 2024-04-15T15:16:11.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.176+0000 7fe3bffff640 1 -- 172.21.15.72:0/4227128023 >> v1:172.21.15.72:6789/0 conn(0x7fe3c8161e40 legacy=0x7fe3c80b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.176+0000 7fe3bffff640 1 --2- 172.21.15.72:0/4227128023 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe3c80d45b0 0x7fe3b40039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.176+0000 7fe3bffff640 1 --2- 172.21.15.72:0/4227128023 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3b4003f60 0x7fe3b4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.176+0000 7fe3bffff640 1 --2- 172.21.15.72:0/4227128023 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3b4004d40 0x7fe3b40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.176+0000 7fe3bffff640 1 -- 172.21.15.72:0/4227128023 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c80b3f00 con 0x7fe3b4003f60 2024-04-15T15:16:11.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.176+0000 7fe3bffff640 1 -- 172.21.15.72:0/4227128023 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fe3b400bb50 con 0x7fe3b4004d40 2024-04-15T15:16:11.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.176+0000 7fe3bffff640 1 -- 172.21.15.72:0/4227128023 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fe3b400bd50 con 0x7fe3c80d45b0 2024-04-15T15:16:11.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.176+0000 7fe3ccab7640 1 --2- 172.21.15.72:0/4227128023 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3b4003f60 0x7fe3b4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.176+0000 7fe3cd2b8640 1 --2- 172.21.15.72:0/4227128023 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe3c80d45b0 0x7fe3b40039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.176+0000 7fe3cdab9640 1 --2- 172.21.15.72:0/4227128023 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3b4004d40 0x7fe3b40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.176+0000 7fe3ccab7640 1 -- 172.21.15.72:0/4227128023 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe3c80d45b0 msgr2=0x7fe3b40039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:11.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.176+0000 7fe3ccab7640 1 --2- 172.21.15.72:0/4227128023 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe3c80d45b0 0x7fe3b40039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.176+0000 7fe3ccab7640 1 -- 172.21.15.72:0/4227128023 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3b4004d40 msgr2=0x7fe3b40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:11.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.176+0000 7fe3ccab7640 1 --2- 172.21.15.72:0/4227128023 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3b4004d40 0x7fe3b40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.176+0000 7fe3ccab7640 1 -- 172.21.15.72:0/4227128023 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3b4003520 con 0x7fe3b4003f60 2024-04-15T15:16:11.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.176+0000 7fe3cdab9640 1 --2- 172.21.15.72:0/4227128023 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3b4004d40 0x7fe3b40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:11.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.177+0000 7fe3ccab7640 1 --2- 172.21.15.72:0/4227128023 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3b4003f60 0x7fe3b4004800 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8001750 tx=0x7fe3b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cd84c02056c6469 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:11.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.177+0000 7fe3bffff640 1 -- 172.21.15.72:0/4227128023 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b8001370 con 0x7fe3b4003f60 2024-04-15T15:16:11.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.177+0000 7fe3bffff640 1 -- 172.21.15.72:0/4227128023 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3b80052d0 con 0x7fe3b4003f60 2024-04-15T15:16:11.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.177+0000 7fe3bffff640 1 -- 172.21.15.72:0/4227128023 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b8003830 con 0x7fe3b4003f60 2024-04-15T15:16:11.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.178+0000 7fe3cf51c640 1 -- 172.21.15.72:0/4227128023 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3b4003f60 msgr2=0x7fe3b4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.178+0000 7fe3cf51c640 1 --2- 172.21.15.72:0/4227128023 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3b4003f60 0x7fe3b4004800 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8001750 tx=0x7fe3b4005830 comp rx=0 tx=0).stop 2024-04-15T15:16:11.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.178+0000 7fe3cdab9640 1 -- 172.21.15.72:0/4227128023 reap_dead start 2024-04-15T15:16:11.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.178+0000 7fe3cf51c640 1 -- 172.21.15.72:0/4227128023 shutdown_connections 2024-04-15T15:16:11.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.178+0000 7fe3cf51c640 1 -- 172.21.15.72:0/4227128023 >> 172.21.15.72:0/4227128023 conn(0x7fe3c809d970 msgr2=0x7fe3c80b95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:11.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.178+0000 7fe3cf51c640 1 -- 172.21.15.72:0/4227128023 shutdown_connections 2024-04-15T15:16:11.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.178+0000 7fe3cf51c640 1 -- 172.21.15.72:0/4227128023 wait complete. 2024-04-15T15:16:11.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.179+0000 7fe3cf51c640 1 Processor -- start 2024-04-15T15:16:11.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.179+0000 7fe3cf51c640 1 -- start start 2024-04-15T15:16:11.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.180+0000 7fe3cf51c640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3b4003f60 0x7fe3c81f7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.180+0000 7fe3cf51c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3c80b2580 0x7fe3c80b2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.180+0000 7fe3cf51c640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe3c81f8040 0x7fe3c81f84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.180+0000 7fe3cf51c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c80d21f0 con 0x7fe3c80b2580 2024-04-15T15:16:11.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.180+0000 7fe3cf51c640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c80d4f60 con 0x7fe3b4003f60 2024-04-15T15:16:11.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.180+0000 7fe3cf51c640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fe3c80d1de0 con 0x7fe3c81f8040 2024-04-15T15:16:11.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.180+0000 7fe3ccab7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3c80b2580 0x7fe3c80b2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.180+0000 7fe3ccab7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3c80b2580 0x7fe3c80b2a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42302/0 (socket says 172.21.15.72:42302) 2024-04-15T15:16:11.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.180+0000 7fe3ccab7640 1 -- 172.21.15.72:0/1575665122 learned_addr learned my addr 172.21.15.72:0/1575665122 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:11.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.180+0000 7fe3cd2b8640 1 --2- 172.21.15.72:0/1575665122 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3b4003f60 0x7fe3c81f7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.180+0000 7fe3cdab9640 1 --2- 172.21.15.72:0/1575665122 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe3c81f8040 0x7fe3c81f84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.180+0000 7fe3ccab7640 1 -- 172.21.15.72:0/1575665122 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe3c81f8040 msgr2=0x7fe3c81f84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.180+0000 7fe3ccab7640 1 --2- 172.21.15.72:0/1575665122 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe3c81f8040 0x7fe3c81f84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.180+0000 7fe3ccab7640 1 -- 172.21.15.72:0/1575665122 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3b4003f60 msgr2=0x7fe3c81f7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.180+0000 7fe3ccab7640 1 --2- 172.21.15.72:0/1575665122 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3b4003f60 0x7fe3c81f7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.180+0000 7fe3ccab7640 1 -- 172.21.15.72:0/1575665122 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3c81f49e0 con 0x7fe3c80b2580 2024-04-15T15:16:11.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.180+0000 7fe3cd2b8640 1 --2- 172.21.15.72:0/1575665122 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3b4003f60 0x7fe3c81f7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:11.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.180+0000 7fe3cdab9640 1 --2- 172.21.15.72:0/1575665122 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe3c81f8040 0x7fe3c81f84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:11.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.180+0000 7fe3ccab7640 1 --2- 172.21.15.72:0/1575665122 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3c80b2580 0x7fe3c80b2a00 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8008fd0 tx=0x7fe3b8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:11.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.180+0000 7fe3bdffb640 1 -- 172.21.15.72:0/1575665122 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b8041070 con 0x7fe3c80b2580 2024-04-15T15:16:11.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.181+0000 7fe3bdffb640 1 -- 172.21.15.72:0/1575665122 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3b8005480 con 0x7fe3c80b2580 2024-04-15T15:16:11.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.181+0000 7fe3cf51c640 1 -- 172.21.15.72:0/1575665122 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3c81f4cc0 con 0x7fe3c80b2580 2024-04-15T15:16:11.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.181+0000 7fe3bdffb640 1 -- 172.21.15.72:0/1575665122 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b8001370 con 0x7fe3c80b2580 2024-04-15T15:16:11.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.181+0000 7fe3cf51c640 1 -- 172.21.15.72:0/1575665122 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3c81f5280 con 0x7fe3c80b2580 2024-04-15T15:16:11.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.182+0000 7fe3cf51c640 1 -- 172.21.15.72:0/1575665122 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3c80689e0 con 0x7fe3c80b2580 2024-04-15T15:16:11.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.182+0000 7fe3bdffb640 1 -- 172.21.15.72:0/1575665122 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe3b8044070 con 0x7fe3c80b2580 2024-04-15T15:16:11.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.182+0000 7fe3bdffb640 1 --2- 172.21.15.72:0/1575665122 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe3ac06c920 0x7fe3ac06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.183+0000 7fe3bdffb640 1 -- 172.21.15.72:0/1575665122 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fe3b80ad740 con 0x7fe3c80b2580 2024-04-15T15:16:11.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.183+0000 7fe3cd2b8640 1 --2- 172.21.15.72:0/1575665122 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe3ac06c920 0x7fe3ac06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.183+0000 7fe3cd2b8640 1 --2- 172.21.15.72:0/1575665122 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe3ac06c920 0x7fe3ac06ede0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4001eb0 tx=0x7fe3c4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:11.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.187+0000 7fe3bdffb640 1 -- 172.21.15.72:0/1575665122 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe3b807bb00 con 0x7fe3c80b2580 2024-04-15T15:16:11.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.361+0000 7fe3cf51c640 1 -- 172.21.15.72:0/1575665122 --> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe3c80d51b0 con 0x7fe3ac06c920 2024-04-15T15:16:11.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.367+0000 7fe3bdffb640 1 -- 172.21.15.72:0/1575665122 <== mgr.4101 v2:172.21.15.72:6832/3503807805 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251944 (secure 0 0 0) 0x7fe3c80d51b0 con 0x7fe3ac06c920 2024-04-15T15:16:11.372 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:16:11.373 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-04-15T15:16:11.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.370+0000 7fe3cf51c640 1 -- 172.21.15.72:0/1575665122 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe3ac06c920 msgr2=0x7fe3ac06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.370+0000 7fe3cf51c640 1 --2- 172.21.15.72:0/1575665122 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe3ac06c920 0x7fe3ac06ede0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4001eb0 tx=0x7fe3c4009040 comp rx=0 tx=0).stop 2024-04-15T15:16:11.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.370+0000 7fe3cf51c640 1 -- 172.21.15.72:0/1575665122 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3c80b2580 msgr2=0x7fe3c80b2a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.370+0000 7fe3cf51c640 1 --2- 172.21.15.72:0/1575665122 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3c80b2580 0x7fe3c80b2a00 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8008fd0 tx=0x7fe3b8001120 comp rx=0 tx=0).stop 2024-04-15T15:16:11.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.370+0000 7fe3cf51c640 1 -- 172.21.15.72:0/1575665122 shutdown_connections 2024-04-15T15:16:11.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.370+0000 7fe3cf51c640 1 --2- 172.21.15.72:0/1575665122 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe3ac06c920 0x7fe3ac06ede0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.370+0000 7fe3cf51c640 1 --2- 172.21.15.72:0/1575665122 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe3c81f8040 0x7fe3c81f84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.370+0000 7fe3cf51c640 1 --2- 172.21.15.72:0/1575665122 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3c80b2580 0x7fe3c80b2a00 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.370+0000 7fe3cf51c640 1 --2- 172.21.15.72:0/1575665122 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3b4003f60 0x7fe3c81f7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.370+0000 7fe3cf51c640 1 -- 172.21.15.72:0/1575665122 >> 172.21.15.72:0/1575665122 conn(0x7fe3c809d970 msgr2=0x7fe3c80b1e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:11.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.370+0000 7fe3cf51c640 1 -- 172.21.15.72:0/1575665122 shutdown_connections 2024-04-15T15:16:11.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.370+0000 7fe3cf51c640 1 -- 172.21.15.72:0/1575665122 wait complete. 2024-04-15T15:16:11.383 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":77,"stamp":"2024-04-15T15:16:11.359547+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":159,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220756,"kb_used_data":4576,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753964,"statfs":{"total":773094113280,"available":772868059136,"internally_reserved":0,"allocated":4685824,"data_stored":2665692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.677172"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973935+0000","last_change":"2024-04-15T15:16:07.714016+0000","last_active":"2024-04-15T15:16:08.973935+0000","last_peered":"2024-04-15T15:16:08.973935+0000","last_clean":"2024-04-15T15:16:08.973935+0000","last_became_active":"2024-04-15T15:16:07.713760+0000","last_became_peered":"2024-04-15T15:16:07.713760+0000","last_unstale":"2024-04-15T15:16:08.973935+0000","last_undegraded":"2024-04-15T15:16:08.973935+0000","last_fullsized":"2024-04-15T15:16:08.973935+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:26:10.169823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973663+0000","last_change":"2024-04-15T15:16:07.716357+0000","last_active":"2024-04-15T15:16:08.973663+0000","last_peered":"2024-04-15T15:16:08.973663+0000","last_clean":"2024-04-15T15:16:08.973663+0000","last_became_active":"2024-04-15T15:16:07.715891+0000","last_became_peered":"2024-04-15T15:16:07.715891+0000","last_unstale":"2024-04-15T15:16:08.973663+0000","last_undegraded":"2024-04-15T15:16:08.973663+0000","last_fullsized":"2024-04-15T15:16:08.973663+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:32:24.986373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.715319+0000","last_change":"2024-04-15T15:16:07.706936+0000","last_active":"2024-04-15T15:16:07.715319+0000","last_peered":"2024-04-15T15:16:07.715319+0000","last_clean":"2024-04-15T15:16:07.715319+0000","last_became_active":"2024-04-15T15:16:07.706670+0000","last_became_peered":"2024-04-15T15:16:07.706670+0000","last_unstale":"2024-04-15T15:16:07.715319+0000","last_undegraded":"2024-04-15T15:16:07.715319+0000","last_fullsized":"2024-04-15T15:16:07.715319+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:51:04.229786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.708922+0000","last_change":"2024-04-15T15:16:07.708400+0000","last_active":"2024-04-15T15:16:08.708922+0000","last_peered":"2024-04-15T15:16:08.708922+0000","last_clean":"2024-04-15T15:16:08.708922+0000","last_became_active":"2024-04-15T15:16:07.707958+0000","last_became_peered":"2024-04-15T15:16:07.707958+0000","last_unstale":"2024-04-15T15:16:08.708922+0000","last_undegraded":"2024-04-15T15:16:08.708922+0000","last_fullsized":"2024-04-15T15:16:08.708922+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:18:13.186969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.708673+0000","last_change":"2024-04-15T15:16:07.712681+0000","last_active":"2024-04-15T15:16:08.708673+0000","last_peered":"2024-04-15T15:16:08.708673+0000","last_clean":"2024-04-15T15:16:08.708673+0000","last_became_active":"2024-04-15T15:16:07.712430+0000","last_became_peered":"2024-04-15T15:16:07.712430+0000","last_unstale":"2024-04-15T15:16:08.708673+0000","last_undegraded":"2024-04-15T15:16:08.708673+0000","last_fullsized":"2024-04-15T15:16:08.708673+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:19:44.813755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.718802+0000","last_change":"2024-04-15T15:16:07.709386+0000","last_active":"2024-04-15T15:16:07.718802+0000","last_peered":"2024-04-15T15:16:07.718802+0000","last_clean":"2024-04-15T15:16:07.718802+0000","last_became_active":"2024-04-15T15:16:07.709113+0000","last_became_peered":"2024-04-15T15:16:07.709113+0000","last_unstale":"2024-04-15T15:16:07.718802+0000","last_undegraded":"2024-04-15T15:16:07.718802+0000","last_fullsized":"2024-04-15T15:16:07.718802+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:01:01.910545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718270+0000","last_change":"2024-04-15T15:16:07.711815+0000","last_active":"2024-04-15T15:16:08.718270+0000","last_peered":"2024-04-15T15:16:08.718270+0000","last_clean":"2024-04-15T15:16:08.718270+0000","last_became_active":"2024-04-15T15:16:07.711561+0000","last_became_peered":"2024-04-15T15:16:07.711561+0000","last_unstale":"2024-04-15T15:16:08.718270+0000","last_undegraded":"2024-04-15T15:16:08.718270+0000","last_fullsized":"2024-04-15T15:16:08.718270+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:09:09.330985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973403+0000","last_change":"2024-04-15T15:16:07.711991+0000","last_active":"2024-04-15T15:16:08.973403+0000","last_peered":"2024-04-15T15:16:08.973403+0000","last_clean":"2024-04-15T15:16:08.973403+0000","last_became_active":"2024-04-15T15:16:07.711753+0000","last_became_peered":"2024-04-15T15:16:07.711753+0000","last_unstale":"2024-04-15T15:16:08.973403+0000","last_undegraded":"2024-04-15T15:16:08.973403+0000","last_fullsized":"2024-04-15T15:16:08.973403+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:00:49.742979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.716270+0000","last_change":"2024-04-15T15:16:07.711991+0000","last_active":"2024-04-15T15:16:07.716270+0000","last_peered":"2024-04-15T15:16:07.716270+0000","last_clean":"2024-04-15T15:16:07.716270+0000","last_became_active":"2024-04-15T15:16:07.711634+0000","last_became_peered":"2024-04-15T15:16:07.711634+0000","last_unstale":"2024-04-15T15:16:07.716270+0000","last_undegraded":"2024-04-15T15:16:07.716270+0000","last_fullsized":"2024-04-15T15:16:07.716270+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:14:28.406716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718646+0000","last_change":"2024-04-15T15:16:07.714191+0000","last_active":"2024-04-15T15:16:08.718646+0000","last_peered":"2024-04-15T15:16:08.718646+0000","last_clean":"2024-04-15T15:16:08.718646+0000","last_became_active":"2024-04-15T15:16:07.713121+0000","last_became_peered":"2024-04-15T15:16:07.713121+0000","last_unstale":"2024-04-15T15:16:08.718646+0000","last_undegraded":"2024-04-15T15:16:08.718646+0000","last_fullsized":"2024-04-15T15:16:08.718646+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:32:24.986373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718831+0000","last_change":"2024-04-15T15:16:07.714129+0000","last_active":"2024-04-15T15:16:08.718831+0000","last_peered":"2024-04-15T15:16:08.718831+0000","last_clean":"2024-04-15T15:16:08.718831+0000","last_became_active":"2024-04-15T15:16:07.713746+0000","last_became_peered":"2024-04-15T15:16:07.713746+0000","last_unstale":"2024-04-15T15:16:08.718831+0000","last_undegraded":"2024-04-15T15:16:08.718831+0000","last_fullsized":"2024-04-15T15:16:08.718831+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:41:28.911237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.713119+0000","last_change":"2024-04-15T15:16:07.710622+0000","last_active":"2024-04-15T15:16:07.713119+0000","last_peered":"2024-04-15T15:16:07.713119+0000","last_clean":"2024-04-15T15:16:07.713119+0000","last_became_active":"2024-04-15T15:16:07.710280+0000","last_became_peered":"2024-04-15T15:16:07.710280+0000","last_unstale":"2024-04-15T15:16:07.713119+0000","last_undegraded":"2024-04-15T15:16:07.713119+0000","last_fullsized":"2024-04-15T15:16:07.713119+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:33:10.314944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718413+0000","last_change":"2024-04-15T15:16:07.713369+0000","last_active":"2024-04-15T15:16:08.718413+0000","last_peered":"2024-04-15T15:16:08.718413+0000","last_clean":"2024-04-15T15:16:08.718413+0000","last_became_active":"2024-04-15T15:16:07.713109+0000","last_became_peered":"2024-04-15T15:16:07.713109+0000","last_unstale":"2024-04-15T15:16:08.718413+0000","last_undegraded":"2024-04-15T15:16:08.718413+0000","last_fullsized":"2024-04-15T15:16:08.718413+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:00:49.742979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.358927+0000","last_change":"2024-04-15T15:16:07.709278+0000","last_active":"2024-04-15T15:16:09.358927+0000","last_peered":"2024-04-15T15:16:09.358927+0000","last_clean":"2024-04-15T15:16:09.358927+0000","last_became_active":"2024-04-15T15:16:07.708829+0000","last_became_peered":"2024-04-15T15:16:07.708829+0000","last_unstale":"2024-04-15T15:16:09.358927+0000","last_undegraded":"2024-04-15T15:16:09.358927+0000","last_fullsized":"2024-04-15T15:16:09.358927+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:51:17.043733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718403+0000","last_change":"2024-04-15T15:16:07.700808+0000","last_active":"2024-04-15T15:16:08.718403+0000","last_peered":"2024-04-15T15:16:08.718403+0000","last_clean":"2024-04-15T15:16:08.718403+0000","last_became_active":"2024-04-15T15:16:07.700484+0000","last_became_peered":"2024-04-15T15:16:07.700484+0000","last_unstale":"2024-04-15T15:16:08.718403+0000","last_undegraded":"2024-04-15T15:16:08.718403+0000","last_fullsized":"2024-04-15T15:16:08.718403+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:14:45.605613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709813+0000","last_change":"2024-04-15T15:16:07.713960+0000","last_active":"2024-04-15T15:16:08.709813+0000","last_peered":"2024-04-15T15:16:08.709813+0000","last_clean":"2024-04-15T15:16:08.709813+0000","last_became_active":"2024-04-15T15:16:07.713665+0000","last_became_peered":"2024-04-15T15:16:07.713665+0000","last_unstale":"2024-04-15T15:16:08.709813+0000","last_undegraded":"2024-04-15T15:16:08.709813+0000","last_fullsized":"2024-04-15T15:16:08.709813+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:54:41.375909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.710290+0000","last_change":"2024-04-15T15:16:07.714486+0000","last_active":"2024-04-15T15:16:08.710290+0000","last_peered":"2024-04-15T15:16:08.710290+0000","last_clean":"2024-04-15T15:16:08.710290+0000","last_became_active":"2024-04-15T15:16:07.714191+0000","last_became_peered":"2024-04-15T15:16:07.714191+0000","last_unstale":"2024-04-15T15:16:08.710290+0000","last_undegraded":"2024-04-15T15:16:08.710290+0000","last_fullsized":"2024-04-15T15:16:08.710290+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:09:09.330985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709650+0000","last_change":"2024-04-15T15:16:07.711021+0000","last_active":"2024-04-15T15:16:08.709650+0000","last_peered":"2024-04-15T15:16:08.709650+0000","last_clean":"2024-04-15T15:16:08.709650+0000","last_became_active":"2024-04-15T15:16:07.710709+0000","last_became_peered":"2024-04-15T15:16:07.710709+0000","last_unstale":"2024-04-15T15:16:08.709650+0000","last_undegraded":"2024-04-15T15:16:08.709650+0000","last_fullsized":"2024-04-15T15:16:08.709650+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:51:17.043733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.708951+0000","last_change":"2024-04-15T15:16:07.712960+0000","last_active":"2024-04-15T15:16:08.708951+0000","last_peered":"2024-04-15T15:16:08.708951+0000","last_clean":"2024-04-15T15:16:08.708951+0000","last_became_active":"2024-04-15T15:16:07.712636+0000","last_became_peered":"2024-04-15T15:16:07.712636+0000","last_unstale":"2024-04-15T15:16:08.708951+0000","last_undegraded":"2024-04-15T15:16:08.708951+0000","last_fullsized":"2024-04-15T15:16:08.708951+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:02:51.429249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709437+0000","last_change":"2024-04-15T15:16:07.712946+0000","last_active":"2024-04-15T15:16:08.709437+0000","last_peered":"2024-04-15T15:16:08.709437+0000","last_clean":"2024-04-15T15:16:08.709437+0000","last_became_active":"2024-04-15T15:16:07.712677+0000","last_became_peered":"2024-04-15T15:16:07.712677+0000","last_unstale":"2024-04-15T15:16:08.709437+0000","last_undegraded":"2024-04-15T15:16:08.709437+0000","last_fullsized":"2024-04-15T15:16:08.709437+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:32:44.859092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718447+0000","last_change":"2024-04-15T15:16:07.708983+0000","last_active":"2024-04-15T15:16:08.718447+0000","last_peered":"2024-04-15T15:16:08.718447+0000","last_clean":"2024-04-15T15:16:08.718447+0000","last_became_active":"2024-04-15T15:16:07.708612+0000","last_became_peered":"2024-04-15T15:16:07.708612+0000","last_unstale":"2024-04-15T15:16:08.718447+0000","last_undegraded":"2024-04-15T15:16:08.718447+0000","last_fullsized":"2024-04-15T15:16:08.718447+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:12:03.446783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.359328+0000","last_change":"2024-04-15T15:16:07.710990+0000","last_active":"2024-04-15T15:16:09.359328+0000","last_peered":"2024-04-15T15:16:09.359328+0000","last_clean":"2024-04-15T15:16:09.359328+0000","last_became_active":"2024-04-15T15:16:07.710763+0000","last_became_peered":"2024-04-15T15:16:07.710763+0000","last_unstale":"2024-04-15T15:16:09.359328+0000","last_undegraded":"2024-04-15T15:16:09.359328+0000","last_fullsized":"2024-04-15T15:16:09.359328+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:39:05.235149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709573+0000","last_change":"2024-04-15T15:16:07.718348+0000","last_active":"2024-04-15T15:16:08.709573+0000","last_peered":"2024-04-15T15:16:08.709573+0000","last_clean":"2024-04-15T15:16:08.709573+0000","last_became_active":"2024-04-15T15:16:07.717954+0000","last_became_peered":"2024-04-15T15:16:07.717954+0000","last_unstale":"2024-04-15T15:16:08.709573+0000","last_undegraded":"2024-04-15T15:16:08.709573+0000","last_fullsized":"2024-04-15T15:16:08.709573+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:32:26.516274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.708677+0000","last_change":"2024-04-15T15:16:07.714768+0000","last_active":"2024-04-15T15:16:08.708677+0000","last_peered":"2024-04-15T15:16:08.708677+0000","last_clean":"2024-04-15T15:16:08.708677+0000","last_became_active":"2024-04-15T15:16:07.714512+0000","last_became_peered":"2024-04-15T15:16:07.714512+0000","last_unstale":"2024-04-15T15:16:08.708677+0000","last_undegraded":"2024-04-15T15:16:08.708677+0000","last_fullsized":"2024-04-15T15:16:08.708677+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:42:44.933220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.710013+0000","last_change":"2024-04-15T15:16:07.714668+0000","last_active":"2024-04-15T15:16:08.710013+0000","last_peered":"2024-04-15T15:16:08.710013+0000","last_clean":"2024-04-15T15:16:08.710013+0000","last_became_active":"2024-04-15T15:16:07.714354+0000","last_became_peered":"2024-04-15T15:16:07.714354+0000","last_unstale":"2024-04-15T15:16:08.710013+0000","last_undegraded":"2024-04-15T15:16:08.710013+0000","last_fullsized":"2024-04-15T15:16:08.710013+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:08:22.249305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.708863+0000","last_change":"2024-04-15T15:16:07.708195+0000","last_active":"2024-04-15T15:16:07.708863+0000","last_peered":"2024-04-15T15:16:07.708863+0000","last_clean":"2024-04-15T15:16:07.708863+0000","last_became_active":"2024-04-15T15:16:07.707897+0000","last_became_peered":"2024-04-15T15:16:07.707897+0000","last_unstale":"2024-04-15T15:16:07.708863+0000","last_undegraded":"2024-04-15T15:16:07.708863+0000","last_fullsized":"2024-04-15T15:16:07.708863+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:14:28.406716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709808+0000","last_change":"2024-04-15T15:16:07.707076+0000","last_active":"2024-04-15T15:16:08.709808+0000","last_peered":"2024-04-15T15:16:08.709808+0000","last_clean":"2024-04-15T15:16:08.709808+0000","last_became_active":"2024-04-15T15:16:07.706810+0000","last_became_peered":"2024-04-15T15:16:07.706810+0000","last_unstale":"2024-04-15T15:16:08.709808+0000","last_undegraded":"2024-04-15T15:16:08.709808+0000","last_fullsized":"2024-04-15T15:16:08.709808+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:45:03.154342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718597+0000","last_change":"2024-04-15T15:16:07.717296+0000","last_active":"2024-04-15T15:16:08.718597+0000","last_peered":"2024-04-15T15:16:08.718597+0000","last_clean":"2024-04-15T15:16:08.718597+0000","last_became_active":"2024-04-15T15:16:07.716737+0000","last_became_peered":"2024-04-15T15:16:07.716737+0000","last_unstale":"2024-04-15T15:16:08.718597+0000","last_undegraded":"2024-04-15T15:16:08.718597+0000","last_fullsized":"2024-04-15T15:16:08.718597+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:38:41.698007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709781+0000","last_change":"2024-04-15T15:16:07.708652+0000","last_active":"2024-04-15T15:16:08.709781+0000","last_peered":"2024-04-15T15:16:08.709781+0000","last_clean":"2024-04-15T15:16:08.709781+0000","last_became_active":"2024-04-15T15:16:07.708305+0000","last_became_peered":"2024-04-15T15:16:07.708305+0000","last_unstale":"2024-04-15T15:16:08.709781+0000","last_undegraded":"2024-04-15T15:16:08.709781+0000","last_fullsized":"2024-04-15T15:16:08.709781+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:57:26.177964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709498+0000","last_change":"2024-04-15T15:16:07.709602+0000","last_active":"2024-04-15T15:16:08.709498+0000","last_peered":"2024-04-15T15:16:08.709498+0000","last_clean":"2024-04-15T15:16:08.709498+0000","last_became_active":"2024-04-15T15:16:07.709332+0000","last_became_peered":"2024-04-15T15:16:07.709332+0000","last_unstale":"2024-04-15T15:16:08.709498+0000","last_undegraded":"2024-04-15T15:16:08.709498+0000","last_fullsized":"2024-04-15T15:16:08.709498+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:03:12.877648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709730+0000","last_change":"2024-04-15T15:16:07.717993+0000","last_active":"2024-04-15T15:16:08.709730+0000","last_peered":"2024-04-15T15:16:08.709730+0000","last_clean":"2024-04-15T15:16:08.709730+0000","last_became_active":"2024-04-15T15:16:07.717663+0000","last_became_peered":"2024-04-15T15:16:07.717663+0000","last_unstale":"2024-04-15T15:16:08.709730+0000","last_undegraded":"2024-04-15T15:16:08.709730+0000","last_fullsized":"2024-04-15T15:16:08.709730+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:42:46.979916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.722682+0000","last_change":"2024-04-15T15:16:07.719599+0000","last_active":"2024-04-15T15:16:07.722682+0000","last_peered":"2024-04-15T15:16:07.722682+0000","last_clean":"2024-04-15T15:16:07.722682+0000","last_became_active":"2024-04-15T15:16:07.719303+0000","last_became_peered":"2024-04-15T15:16:07.719303+0000","last_unstale":"2024-04-15T15:16:07.722682+0000","last_undegraded":"2024-04-15T15:16:07.722682+0000","last_fullsized":"2024-04-15T15:16:07.722682+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:32:44.859092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709700+0000","last_change":"2024-04-15T15:16:07.719675+0000","last_active":"2024-04-15T15:16:08.709700+0000","last_peered":"2024-04-15T15:16:08.709700+0000","last_clean":"2024-04-15T15:16:08.709700+0000","last_became_active":"2024-04-15T15:16:07.719391+0000","last_became_peered":"2024-04-15T15:16:07.719391+0000","last_unstale":"2024-04-15T15:16:08.709700+0000","last_undegraded":"2024-04-15T15:16:08.709700+0000","last_fullsized":"2024-04-15T15:16:08.709700+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:14:39.023105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"37'4","reported_seq":18,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.710178+0000","last_change":"2024-04-15T15:16:06.702682+0000","last_active":"2024-04-15T15:16:08.710178+0000","last_peered":"2024-04-15T15:16:08.710178+0000","last_clean":"2024-04-15T15:16:08.710178+0000","last_became_active":"2024-04-15T15:16:06.702428+0000","last_became_peered":"2024-04-15T15:16:06.702428+0000","last_unstale":"2024-04-15T15:16:08.710178+0000","last_undegraded":"2024-04-15T15:16:08.710178+0000","last_fullsized":"2024-04-15T15:16:08.710178+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:35:01.354434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.718799+0000","last_change":"2024-04-15T15:16:07.712184+0000","last_active":"2024-04-15T15:16:07.718799+0000","last_peered":"2024-04-15T15:16:07.718799+0000","last_clean":"2024-04-15T15:16:07.718799+0000","last_became_active":"2024-04-15T15:16:07.711833+0000","last_became_peered":"2024-04-15T15:16:07.711833+0000","last_unstale":"2024-04-15T15:16:07.718799+0000","last_undegraded":"2024-04-15T15:16:07.718799+0000","last_fullsized":"2024-04-15T15:16:07.718799+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:10:57.528229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.727968+0000","last_change":"2024-04-15T15:16:06.705264+0000","last_active":"2024-04-15T15:16:07.727968+0000","last_peered":"2024-04-15T15:16:07.727968+0000","last_clean":"2024-04-15T15:16:07.727968+0000","last_became_active":"2024-04-15T15:16:06.704985+0000","last_became_peered":"2024-04-15T15:16:06.704985+0000","last_unstale":"2024-04-15T15:16:07.727968+0000","last_undegraded":"2024-04-15T15:16:07.727968+0000","last_fullsized":"2024-04-15T15:16:07.727968+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:02:50.425156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.359305+0000","last_change":"2024-04-15T15:16:07.710774+0000","last_active":"2024-04-15T15:16:09.359305+0000","last_peered":"2024-04-15T15:16:09.359305+0000","last_clean":"2024-04-15T15:16:09.359305+0000","last_became_active":"2024-04-15T15:16:07.710468+0000","last_became_peered":"2024-04-15T15:16:07.710468+0000","last_unstale":"2024-04-15T15:16:09.359305+0000","last_undegraded":"2024-04-15T15:16:09.359305+0000","last_fullsized":"2024-04-15T15:16:09.359305+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:56:02.237631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.733126+0000","last_change":"2024-04-15T15:16:06.706399+0000","last_active":"2024-04-15T15:16:07.733126+0000","last_peered":"2024-04-15T15:16:07.733126+0000","last_clean":"2024-04-15T15:16:07.733126+0000","last_became_active":"2024-04-15T15:16:06.706141+0000","last_became_peered":"2024-04-15T15:16:06.706141+0000","last_unstale":"2024-04-15T15:16:07.733126+0000","last_undegraded":"2024-04-15T15:16:07.733126+0000","last_fullsized":"2024-04-15T15:16:07.733126+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:33:09.310850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973814+0000","last_change":"2024-04-15T15:16:07.710742+0000","last_active":"2024-04-15T15:16:08.973814+0000","last_peered":"2024-04-15T15:16:08.973814+0000","last_clean":"2024-04-15T15:16:08.973814+0000","last_became_active":"2024-04-15T15:16:07.710455+0000","last_became_peered":"2024-04-15T15:16:07.710455+0000","last_unstale":"2024-04-15T15:16:08.973814+0000","last_undegraded":"2024-04-15T15:16:08.973814+0000","last_fullsized":"2024-04-15T15:16:08.973814+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:41:28.911237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.359266+0000","last_change":"2024-04-15T15:16:06.704244+0000","last_active":"2024-04-15T15:16:09.359266+0000","last_peered":"2024-04-15T15:16:09.359266+0000","last_clean":"2024-04-15T15:16:09.359266+0000","last_became_active":"2024-04-15T15:16:06.703919+0000","last_became_peered":"2024-04-15T15:16:06.703919+0000","last_unstale":"2024-04-15T15:16:09.359266+0000","last_undegraded":"2024-04-15T15:16:09.359266+0000","last_fullsized":"2024-04-15T15:16:09.359266+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:51:03.225693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.716037+0000","last_change":"2024-04-15T15:16:07.712901+0000","last_active":"2024-04-15T15:16:07.716037+0000","last_peered":"2024-04-15T15:16:07.716037+0000","last_clean":"2024-04-15T15:16:07.716037+0000","last_became_active":"2024-04-15T15:16:07.712444+0000","last_became_peered":"2024-04-15T15:16:07.712444+0000","last_unstale":"2024-04-15T15:16:07.716037+0000","last_undegraded":"2024-04-15T15:16:07.716037+0000","last_fullsized":"2024-04-15T15:16:07.716037+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:38:29.844648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718599+0000","last_change":"2024-04-15T15:16:06.706424+0000","last_active":"2024-04-15T15:16:08.718599+0000","last_peered":"2024-04-15T15:16:08.718599+0000","last_clean":"2024-04-15T15:16:08.718599+0000","last_became_active":"2024-04-15T15:16:06.706138+0000","last_became_peered":"2024-04-15T15:16:06.706138+0000","last_unstale":"2024-04-15T15:16:08.718599+0000","last_undegraded":"2024-04-15T15:16:08.718599+0000","last_fullsized":"2024-04-15T15:16:08.718599+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:01:00.906451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718514+0000","last_change":"2024-04-15T15:16:07.709849+0000","last_active":"2024-04-15T15:16:08.718514+0000","last_peered":"2024-04-15T15:16:08.718514+0000","last_clean":"2024-04-15T15:16:08.718514+0000","last_became_active":"2024-04-15T15:16:07.709582+0000","last_became_peered":"2024-04-15T15:16:07.709582+0000","last_unstale":"2024-04-15T15:16:08.718514+0000","last_undegraded":"2024-04-15T15:16:08.718514+0000","last_fullsized":"2024-04-15T15:16:08.718514+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:04:27.874758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.710385+0000","last_change":"2024-04-15T15:16:06.704215+0000","last_active":"2024-04-15T15:16:08.710385+0000","last_peered":"2024-04-15T15:16:08.710385+0000","last_clean":"2024-04-15T15:16:08.710385+0000","last_became_active":"2024-04-15T15:16:06.703429+0000","last_became_peered":"2024-04-15T15:16:06.703429+0000","last_unstale":"2024-04-15T15:16:08.710385+0000","last_undegraded":"2024-04-15T15:16:08.710385+0000","last_fullsized":"2024-04-15T15:16:08.710385+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:51:03.225693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.359185+0000","last_change":"2024-04-15T15:16:07.709067+0000","last_active":"2024-04-15T15:16:09.359185+0000","last_peered":"2024-04-15T15:16:09.359185+0000","last_clean":"2024-04-15T15:16:09.359185+0000","last_became_active":"2024-04-15T15:16:07.708491+0000","last_became_peered":"2024-04-15T15:16:07.708491+0000","last_unstale":"2024-04-15T15:16:09.359185+0000","last_undegraded":"2024-04-15T15:16:09.359185+0000","last_fullsized":"2024-04-15T15:16:09.359185+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:45:03.154342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.702105+0000","last_change":"2024-04-15T15:16:06.707235+0000","last_active":"2024-04-15T15:16:07.702105+0000","last_peered":"2024-04-15T15:16:07.702105+0000","last_clean":"2024-04-15T15:16:07.702105+0000","last_became_active":"2024-04-15T15:16:06.706918+0000","last_became_peered":"2024-04-15T15:16:06.706918+0000","last_unstale":"2024-04-15T15:16:07.702105+0000","last_undegraded":"2024-04-15T15:16:07.702105+0000","last_fullsized":"2024-04-15T15:16:07.702105+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:25:09.602767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973566+0000","last_change":"2024-04-15T15:16:07.715092+0000","last_active":"2024-04-15T15:16:08.973566+0000","last_peered":"2024-04-15T15:16:08.973566+0000","last_clean":"2024-04-15T15:16:08.973566+0000","last_became_active":"2024-04-15T15:16:07.714832+0000","last_became_peered":"2024-04-15T15:16:07.714832+0000","last_unstale":"2024-04-15T15:16:08.973566+0000","last_undegraded":"2024-04-15T15:16:08.973566+0000","last_fullsized":"2024-04-15T15:16:08.973566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:30:18.481379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.358856+0000","last_change":"2024-04-15T15:16:06.699694+0000","last_active":"2024-04-15T15:16:09.358856+0000","last_peered":"2024-04-15T15:16:09.358856+0000","last_clean":"2024-04-15T15:16:09.358856+0000","last_became_active":"2024-04-15T15:16:06.699412+0000","last_became_peered":"2024-04-15T15:16:06.699412+0000","last_unstale":"2024-04-15T15:16:09.358856+0000","last_undegraded":"2024-04-15T15:16:09.358856+0000","last_fullsized":"2024-04-15T15:16:09.358856+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:02:50.425156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.708969+0000","last_change":"2024-04-15T15:16:07.708203+0000","last_active":"2024-04-15T15:16:08.708969+0000","last_peered":"2024-04-15T15:16:08.708969+0000","last_clean":"2024-04-15T15:16:08.708969+0000","last_became_active":"2024-04-15T15:16:07.707642+0000","last_became_peered":"2024-04-15T15:16:07.707642+0000","last_unstale":"2024-04-15T15:16:08.708969+0000","last_undegraded":"2024-04-15T15:16:08.708969+0000","last_fullsized":"2024-04-15T15:16:08.708969+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:35:02.358527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.974013+0000","last_change":"2024-04-15T15:16:06.702867+0000","last_active":"2024-04-15T15:16:08.974013+0000","last_peered":"2024-04-15T15:16:08.974013+0000","last_clean":"2024-04-15T15:16:08.974013+0000","last_became_active":"2024-04-15T15:16:06.701891+0000","last_became_peered":"2024-04-15T15:16:06.701891+0000","last_unstale":"2024-04-15T15:16:08.974013+0000","last_undegraded":"2024-04-15T15:16:08.974013+0000","last_fullsized":"2024-04-15T15:16:08.974013+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:51:16.039639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.710590+0000","last_change":"2024-04-15T15:16:07.718269+0000","last_active":"2024-04-15T15:16:08.710590+0000","last_peered":"2024-04-15T15:16:08.710590+0000","last_clean":"2024-04-15T15:16:08.710590+0000","last_became_active":"2024-04-15T15:16:07.717880+0000","last_became_peered":"2024-04-15T15:16:07.717880+0000","last_unstale":"2024-04-15T15:16:08.710590+0000","last_undegraded":"2024-04-15T15:16:08.710590+0000","last_fullsized":"2024-04-15T15:16:08.710590+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:00:49.742979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718797+0000","last_change":"2024-04-15T15:16:06.704693+0000","last_active":"2024-04-15T15:16:08.718797+0000","last_peered":"2024-04-15T15:16:08.718797+0000","last_clean":"2024-04-15T15:16:08.718797+0000","last_became_active":"2024-04-15T15:16:06.704416+0000","last_became_peered":"2024-04-15T15:16:06.704416+0000","last_unstale":"2024-04-15T15:16:08.718797+0000","last_undegraded":"2024-04-15T15:16:08.718797+0000","last_fullsized":"2024-04-15T15:16:08.718797+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:32:43.854999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.723438+0000","last_change":"2024-04-15T15:16:07.716298+0000","last_active":"2024-04-15T15:16:07.723438+0000","last_peered":"2024-04-15T15:16:07.723438+0000","last_clean":"2024-04-15T15:16:07.723438+0000","last_became_active":"2024-04-15T15:16:07.715865+0000","last_became_peered":"2024-04-15T15:16:07.715865+0000","last_unstale":"2024-04-15T15:16:07.723438+0000","last_undegraded":"2024-04-15T15:16:07.723438+0000","last_fullsized":"2024-04-15T15:16:07.723438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:10:57.528229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"37'3","reported_seq":17,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718918+0000","last_change":"2024-04-15T15:16:06.702716+0000","last_active":"2024-04-15T15:16:08.718918+0000","last_peered":"2024-04-15T15:16:08.718918+0000","last_clean":"2024-04-15T15:16:08.718918+0000","last_became_active":"2024-04-15T15:16:06.702416+0000","last_became_peered":"2024-04-15T15:16:06.702416+0000","last_unstale":"2024-04-15T15:16:08.718918+0000","last_undegraded":"2024-04-15T15:16:08.718918+0000","last_fullsized":"2024-04-15T15:16:08.718918+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:38:28.840555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.359401+0000","last_change":"2024-04-15T15:16:07.711716+0000","last_active":"2024-04-15T15:16:09.359401+0000","last_peered":"2024-04-15T15:16:09.359401+0000","last_clean":"2024-04-15T15:16:09.359401+0000","last_became_active":"2024-04-15T15:16:07.711459+0000","last_became_peered":"2024-04-15T15:16:07.711459+0000","last_unstale":"2024-04-15T15:16:09.359401+0000","last_undegraded":"2024-04-15T15:16:09.359401+0000","last_fullsized":"2024-04-15T15:16:09.359401+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:32:26.516274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"37'6","reported_seq":20,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973672+0000","last_change":"2024-04-15T15:16:06.707247+0000","last_active":"2024-04-15T15:16:08.973672+0000","last_peered":"2024-04-15T15:16:08.973672+0000","last_clean":"2024-04-15T15:16:08.973672+0000","last_became_active":"2024-04-15T15:16:06.706166+0000","last_became_peered":"2024-04-15T15:16:06.706166+0000","last_unstale":"2024-04-15T15:16:08.973672+0000","last_undegraded":"2024-04-15T15:16:08.973672+0000","last_fullsized":"2024-04-15T15:16:08.973672+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:57:25.173871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.716014+0000","last_change":"2024-04-15T15:16:07.712212+0000","last_active":"2024-04-15T15:16:07.716014+0000","last_peered":"2024-04-15T15:16:07.716014+0000","last_clean":"2024-04-15T15:16:07.716014+0000","last_became_active":"2024-04-15T15:16:07.711327+0000","last_became_peered":"2024-04-15T15:16:07.711327+0000","last_unstale":"2024-04-15T15:16:07.716014+0000","last_undegraded":"2024-04-15T15:16:07.716014+0000","last_fullsized":"2024-04-15T15:16:07.716014+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:01:01.910545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973690+0000","last_change":"2024-04-15T15:16:06.704498+0000","last_active":"2024-04-15T15:16:08.973690+0000","last_peered":"2024-04-15T15:16:08.973690+0000","last_clean":"2024-04-15T15:16:08.973690+0000","last_became_active":"2024-04-15T15:16:06.704230+0000","last_became_peered":"2024-04-15T15:16:06.704230+0000","last_unstale":"2024-04-15T15:16:08.973690+0000","last_undegraded":"2024-04-15T15:16:08.973690+0000","last_fullsized":"2024-04-15T15:16:08.973690+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:32:43.854999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973429+0000","last_change":"2024-04-15T15:16:07.712780+0000","last_active":"2024-04-15T15:16:08.973429+0000","last_peered":"2024-04-15T15:16:08.973429+0000","last_clean":"2024-04-15T15:16:08.973429+0000","last_became_active":"2024-04-15T15:16:07.712534+0000","last_became_peered":"2024-04-15T15:16:07.712534+0000","last_unstale":"2024-04-15T15:16:08.973429+0000","last_undegraded":"2024-04-15T15:16:08.973429+0000","last_fullsized":"2024-04-15T15:16:08.973429+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:12:03.446783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.701926+0000","last_change":"2024-04-15T15:16:06.707538+0000","last_active":"2024-04-15T15:16:07.701926+0000","last_peered":"2024-04-15T15:16:07.701926+0000","last_clean":"2024-04-15T15:16:07.701926+0000","last_became_active":"2024-04-15T15:16:06.707264+0000","last_became_peered":"2024-04-15T15:16:06.707264+0000","last_unstale":"2024-04-15T15:16:07.701926+0000","last_undegraded":"2024-04-15T15:16:07.701926+0000","last_fullsized":"2024-04-15T15:16:07.701926+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:35:01.354434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.710310+0000","last_change":"2024-04-15T15:16:07.716245+0000","last_active":"2024-04-15T15:16:08.710310+0000","last_peered":"2024-04-15T15:16:08.710310+0000","last_clean":"2024-04-15T15:16:08.710310+0000","last_became_active":"2024-04-15T15:16:07.715993+0000","last_became_peered":"2024-04-15T15:16:07.715993+0000","last_unstale":"2024-04-15T15:16:08.710310+0000","last_undegraded":"2024-04-15T15:16:08.710310+0000","last_fullsized":"2024-04-15T15:16:08.710310+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:14:45.605613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709283+0000","last_change":"2024-04-15T15:16:06.700246+0000","last_active":"2024-04-15T15:16:08.709283+0000","last_peered":"2024-04-15T15:16:08.709283+0000","last_clean":"2024-04-15T15:16:08.709283+0000","last_became_active":"2024-04-15T15:16:06.699972+0000","last_became_peered":"2024-04-15T15:16:06.699972+0000","last_unstale":"2024-04-15T15:16:08.709283+0000","last_undegraded":"2024-04-15T15:16:08.709283+0000","last_fullsized":"2024-04-15T15:16:08.709283+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:02:50.425156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973578+0000","last_change":"2024-04-15T15:16:07.713401+0000","last_active":"2024-04-15T15:16:08.973578+0000","last_peered":"2024-04-15T15:16:08.973578+0000","last_clean":"2024-04-15T15:16:08.973578+0000","last_became_active":"2024-04-15T15:16:07.713157+0000","last_became_peered":"2024-04-15T15:16:07.713157+0000","last_unstale":"2024-04-15T15:16:08.973578+0000","last_undegraded":"2024-04-15T15:16:08.973578+0000","last_fullsized":"2024-04-15T15:16:08.973578+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:04:27.874758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"37'8","reported_seq":22,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709937+0000","last_change":"2024-04-15T15:16:06.703005+0000","last_active":"2024-04-15T15:16:08.709937+0000","last_peered":"2024-04-15T15:16:08.709937+0000","last_clean":"2024-04-15T15:16:08.709937+0000","last_became_active":"2024-04-15T15:16:06.702719+0000","last_became_peered":"2024-04-15T15:16:06.702719+0000","last_unstale":"2024-04-15T15:16:08.709937+0000","last_undegraded":"2024-04-15T15:16:08.709937+0000","last_fullsized":"2024-04-15T15:16:08.709937+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:33:09.310850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973624+0000","last_change":"2024-04-15T15:16:07.716082+0000","last_active":"2024-04-15T15:16:08.973624+0000","last_peered":"2024-04-15T15:16:08.973624+0000","last_clean":"2024-04-15T15:16:08.973624+0000","last_became_active":"2024-04-15T15:16:07.715802+0000","last_became_peered":"2024-04-15T15:16:07.715802+0000","last_unstale":"2024-04-15T15:16:08.973624+0000","last_undegraded":"2024-04-15T15:16:08.973624+0000","last_fullsized":"2024-04-15T15:16:08.973624+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:38:41.698007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718834+0000","last_change":"2024-04-15T15:16:06.699687+0000","last_active":"2024-04-15T15:16:08.718834+0000","last_peered":"2024-04-15T15:16:08.718834+0000","last_clean":"2024-04-15T15:16:08.718834+0000","last_became_active":"2024-04-15T15:16:06.699403+0000","last_became_peered":"2024-04-15T15:16:06.699403+0000","last_unstale":"2024-04-15T15:16:08.718834+0000","last_undegraded":"2024-04-15T15:16:08.718834+0000","last_fullsized":"2024-04-15T15:16:08.718834+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:14:27.402622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718805+0000","last_change":"2024-04-15T15:16:07.720614+0000","last_active":"2024-04-15T15:16:08.718805+0000","last_peered":"2024-04-15T15:16:08.718805+0000","last_clean":"2024-04-15T15:16:08.718805+0000","last_became_active":"2024-04-15T15:16:07.720327+0000","last_became_peered":"2024-04-15T15:16:07.720327+0000","last_unstale":"2024-04-15T15:16:08.718805+0000","last_undegraded":"2024-04-15T15:16:08.718805+0000","last_fullsized":"2024-04-15T15:16:08.718805+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:57:37.374559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.359381+0000","last_change":"2024-04-15T15:16:06.703119+0000","last_active":"2024-04-15T15:16:09.359381+0000","last_peered":"2024-04-15T15:16:09.359381+0000","last_clean":"2024-04-15T15:16:09.359381+0000","last_became_active":"2024-04-15T15:16:06.702423+0000","last_became_peered":"2024-04-15T15:16:06.702423+0000","last_unstale":"2024-04-15T15:16:09.359381+0000","last_undegraded":"2024-04-15T15:16:09.359381+0000","last_fullsized":"2024-04-15T15:16:09.359381+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:25:09.602767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.359209+0000","last_change":"2024-04-15T15:16:07.713465+0000","last_active":"2024-04-15T15:16:09.359209+0000","last_peered":"2024-04-15T15:16:09.359209+0000","last_clean":"2024-04-15T15:16:09.359209+0000","last_became_active":"2024-04-15T15:16:07.713154+0000","last_became_peered":"2024-04-15T15:16:07.713154+0000","last_unstale":"2024-04-15T15:16:09.359209+0000","last_undegraded":"2024-04-15T15:16:09.359209+0000","last_fullsized":"2024-04-15T15:16:09.359209+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:08:22.249305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709042+0000","last_change":"2024-04-15T15:16:06.692920+0000","last_active":"2024-04-15T15:16:08.709042+0000","last_peered":"2024-04-15T15:16:08.709042+0000","last_clean":"2024-04-15T15:16:08.709042+0000","last_became_active":"2024-04-15T15:16:06.692641+0000","last_became_peered":"2024-04-15T15:16:06.692641+0000","last_unstale":"2024-04-15T15:16:08.709042+0000","last_undegraded":"2024-04-15T15:16:08.709042+0000","last_fullsized":"2024-04-15T15:16:08.709042+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:38:20.537537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709058+0000","last_change":"2024-04-15T15:16:07.723443+0000","last_active":"2024-04-15T15:16:08.709058+0000","last_peered":"2024-04-15T15:16:08.709058+0000","last_clean":"2024-04-15T15:16:08.709058+0000","last_became_active":"2024-04-15T15:16:07.723186+0000","last_became_peered":"2024-04-15T15:16:07.723186+0000","last_unstale":"2024-04-15T15:16:08.709058+0000","last_undegraded":"2024-04-15T15:16:08.709058+0000","last_fullsized":"2024-04-15T15:16:08.709058+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:25:10.606860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.702351+0000","last_change":"2024-04-15T15:16:06.707391+0000","last_active":"2024-04-15T15:16:07.702351+0000","last_peered":"2024-04-15T15:16:07.702351+0000","last_clean":"2024-04-15T15:16:07.702351+0000","last_became_active":"2024-04-15T15:16:06.707125+0000","last_became_peered":"2024-04-15T15:16:06.707125+0000","last_unstale":"2024-04-15T15:16:07.702351+0000","last_undegraded":"2024-04-15T15:16:07.702351+0000","last_fullsized":"2024-04-15T15:16:07.702351+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:51:03.225693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.710592+0000","last_change":"2024-04-15T15:16:07.719157+0000","last_active":"2024-04-15T15:16:08.710592+0000","last_peered":"2024-04-15T15:16:08.710592+0000","last_clean":"2024-04-15T15:16:08.710592+0000","last_became_active":"2024-04-15T15:16:07.718892+0000","last_became_peered":"2024-04-15T15:16:07.718892+0000","last_unstale":"2024-04-15T15:16:08.710592+0000","last_undegraded":"2024-04-15T15:16:08.710592+0000","last_fullsized":"2024-04-15T15:16:08.710592+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:30:18.481379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.708740+0000","last_change":"2024-04-15T15:16:06.706044+0000","last_active":"2024-04-15T15:16:08.708740+0000","last_peered":"2024-04-15T15:16:08.708740+0000","last_clean":"2024-04-15T15:16:08.708740+0000","last_became_active":"2024-04-15T15:16:06.705532+0000","last_became_peered":"2024-04-15T15:16:06.705532+0000","last_unstale":"2024-04-15T15:16:08.708740+0000","last_undegraded":"2024-04-15T15:16:08.708740+0000","last_fullsized":"2024-04-15T15:16:08.708740+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:40:09.944851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.720383+0000","last_change":"2024-04-15T15:16:07.716379+0000","last_active":"2024-04-15T15:16:07.720383+0000","last_peered":"2024-04-15T15:16:07.720383+0000","last_clean":"2024-04-15T15:16:07.720383+0000","last_became_active":"2024-04-15T15:16:07.715437+0000","last_became_peered":"2024-04-15T15:16:07.715437+0000","last_unstale":"2024-04-15T15:16:07.720383+0000","last_undegraded":"2024-04-15T15:16:07.720383+0000","last_fullsized":"2024-04-15T15:16:07.720383+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:32:44.859092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.708758+0000","last_change":"2024-04-15T15:16:06.698041+0000","last_active":"2024-04-15T15:16:08.708758+0000","last_peered":"2024-04-15T15:16:08.708758+0000","last_clean":"2024-04-15T15:16:08.708758+0000","last_became_active":"2024-04-15T15:16:06.697754+0000","last_became_peered":"2024-04-15T15:16:06.697754+0000","last_unstale":"2024-04-15T15:16:08.708758+0000","last_undegraded":"2024-04-15T15:16:08.708758+0000","last_fullsized":"2024-04-15T15:16:08.708758+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:10:07.758669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718456+0000","last_change":"2024-04-15T15:16:07.701259+0000","last_active":"2024-04-15T15:16:08.718456+0000","last_peered":"2024-04-15T15:16:08.718456+0000","last_clean":"2024-04-15T15:16:08.718456+0000","last_became_active":"2024-04-15T15:16:07.700984+0000","last_became_peered":"2024-04-15T15:16:07.700984+0000","last_unstale":"2024-04-15T15:16:08.718456+0000","last_undegraded":"2024-04-15T15:16:08.718456+0000","last_fullsized":"2024-04-15T15:16:08.718456+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:30:18.481379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709831+0000","last_change":"2024-04-15T15:16:06.699270+0000","last_active":"2024-04-15T15:16:08.709831+0000","last_peered":"2024-04-15T15:16:08.709831+0000","last_clean":"2024-04-15T15:16:08.709831+0000","last_became_active":"2024-04-15T15:16:06.698992+0000","last_became_peered":"2024-04-15T15:16:06.698992+0000","last_unstale":"2024-04-15T15:16:08.709831+0000","last_undegraded":"2024-04-15T15:16:08.709831+0000","last_fullsized":"2024-04-15T15:16:08.709831+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:18:33.749347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973763+0000","last_change":"2024-04-15T15:16:07.714324+0000","last_active":"2024-04-15T15:16:08.973763+0000","last_peered":"2024-04-15T15:16:08.973763+0000","last_clean":"2024-04-15T15:16:08.973763+0000","last_became_active":"2024-04-15T15:16:07.714069+0000","last_became_peered":"2024-04-15T15:16:07.714069+0000","last_unstale":"2024-04-15T15:16:08.973763+0000","last_undegraded":"2024-04-15T15:16:08.973763+0000","last_fullsized":"2024-04-15T15:16:08.973763+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:57:37.374559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.708615+0000","last_change":"2024-04-15T15:16:06.704254+0000","last_active":"2024-04-15T15:16:08.708615+0000","last_peered":"2024-04-15T15:16:08.708615+0000","last_clean":"2024-04-15T15:16:08.708615+0000","last_became_active":"2024-04-15T15:16:06.703962+0000","last_became_peered":"2024-04-15T15:16:06.703962+0000","last_unstale":"2024-04-15T15:16:08.708615+0000","last_undegraded":"2024-04-15T15:16:08.708615+0000","last_fullsized":"2024-04-15T15:16:08.708615+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:34:42.463356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.708978+0000","last_change":"2024-04-15T15:16:07.707273+0000","last_active":"2024-04-15T15:16:07.708978+0000","last_peered":"2024-04-15T15:16:07.708978+0000","last_clean":"2024-04-15T15:16:07.708978+0000","last_became_active":"2024-04-15T15:16:07.707005+0000","last_became_peered":"2024-04-15T15:16:07.707005+0000","last_unstale":"2024-04-15T15:16:07.708978+0000","last_undegraded":"2024-04-15T15:16:07.708978+0000","last_fullsized":"2024-04-15T15:16:07.708978+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:18:34.753440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718915+0000","last_change":"2024-04-15T15:16:06.702827+0000","last_active":"2024-04-15T15:16:08.718915+0000","last_peered":"2024-04-15T15:16:08.718915+0000","last_clean":"2024-04-15T15:16:08.718915+0000","last_became_active":"2024-04-15T15:16:06.702558+0000","last_became_peered":"2024-04-15T15:16:06.702558+0000","last_unstale":"2024-04-15T15:16:08.718915+0000","last_undegraded":"2024-04-15T15:16:08.718915+0000","last_fullsized":"2024-04-15T15:16:08.718915+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:10:56.524136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.710593+0000","last_change":"2024-04-15T15:16:07.717216+0000","last_active":"2024-04-15T15:16:08.710593+0000","last_peered":"2024-04-15T15:16:08.710593+0000","last_clean":"2024-04-15T15:16:08.710593+0000","last_became_active":"2024-04-15T15:16:07.716939+0000","last_became_peered":"2024-04-15T15:16:07.716939+0000","last_unstale":"2024-04-15T15:16:08.710593+0000","last_undegraded":"2024-04-15T15:16:08.710593+0000","last_fullsized":"2024-04-15T15:16:08.710593+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:12:03.446783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"38'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.717703+0000","last_change":"2024-04-15T15:16:06.702772+0000","last_active":"2024-04-15T15:16:08.717703+0000","last_peered":"2024-04-15T15:16:08.717703+0000","last_clean":"2024-04-15T15:16:08.717703+0000","last_became_active":"2024-04-15T15:16:06.702503+0000","last_became_peered":"2024-04-15T15:16:06.702503+0000","last_unstale":"2024-04-15T15:16:08.717703+0000","last_undegraded":"2024-04-15T15:16:08.717703+0000","last_fullsized":"2024-04-15T15:16:08.717703+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:19:47.422670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973693+0000","last_change":"2024-04-15T15:16:07.716142+0000","last_active":"2024-04-15T15:16:08.973693+0000","last_peered":"2024-04-15T15:16:08.973693+0000","last_clean":"2024-04-15T15:16:08.973693+0000","last_became_active":"2024-04-15T15:16:07.715562+0000","last_became_peered":"2024-04-15T15:16:07.715562+0000","last_unstale":"2024-04-15T15:16:08.973693+0000","last_undegraded":"2024-04-15T15:16:08.973693+0000","last_fullsized":"2024-04-15T15:16:08.973693+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:51:12.622012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.701542+0000","last_change":"2024-04-15T15:16:06.706530+0000","last_active":"2024-04-15T15:16:07.701542+0000","last_peered":"2024-04-15T15:16:07.701542+0000","last_clean":"2024-04-15T15:16:07.701542+0000","last_became_active":"2024-04-15T15:16:06.706214+0000","last_became_peered":"2024-04-15T15:16:06.706214+0000","last_unstale":"2024-04-15T15:16:07.701542+0000","last_undegraded":"2024-04-15T15:16:07.701542+0000","last_fullsized":"2024-04-15T15:16:07.701542+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:25:09.602767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.713028+0000","last_change":"2024-04-15T15:16:07.710663+0000","last_active":"2024-04-15T15:16:07.713028+0000","last_peered":"2024-04-15T15:16:07.713028+0000","last_clean":"2024-04-15T15:16:07.713028+0000","last_became_active":"2024-04-15T15:16:07.710388+0000","last_became_peered":"2024-04-15T15:16:07.710388+0000","last_unstale":"2024-04-15T15:16:07.713028+0000","last_undegraded":"2024-04-15T15:16:07.713028+0000","last_fullsized":"2024-04-15T15:16:07.713028+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:18:34.753440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.701317+0000","last_change":"2024-04-15T15:16:06.706722+0000","last_active":"2024-04-15T15:16:07.701317+0000","last_peered":"2024-04-15T15:16:07.701317+0000","last_clean":"2024-04-15T15:16:07.701317+0000","last_became_active":"2024-04-15T15:16:06.706041+0000","last_became_peered":"2024-04-15T15:16:06.706041+0000","last_unstale":"2024-04-15T15:16:07.701317+0000","last_undegraded":"2024-04-15T15:16:07.701317+0000","last_fullsized":"2024-04-15T15:16:07.701317+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:35:01.354434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.718765+0000","last_change":"2024-04-15T15:16:07.709909+0000","last_active":"2024-04-15T15:16:07.718765+0000","last_peered":"2024-04-15T15:16:07.718765+0000","last_clean":"2024-04-15T15:16:07.718765+0000","last_became_active":"2024-04-15T15:16:07.709672+0000","last_became_peered":"2024-04-15T15:16:07.709672+0000","last_unstale":"2024-04-15T15:16:07.718765+0000","last_undegraded":"2024-04-15T15:16:07.718765+0000","last_fullsized":"2024-04-15T15:16:07.718765+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:38:29.844648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709503+0000","last_change":"2024-04-15T15:16:06.704950+0000","last_active":"2024-04-15T15:16:08.709503+0000","last_peered":"2024-04-15T15:16:08.709503+0000","last_clean":"2024-04-15T15:16:08.709503+0000","last_became_active":"2024-04-15T15:16:06.704688+0000","last_became_peered":"2024-04-15T15:16:06.704688+0000","last_unstale":"2024-04-15T15:16:08.709503+0000","last_undegraded":"2024-04-15T15:16:08.709503+0000","last_fullsized":"2024-04-15T15:16:08.709503+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:25:09.602767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.358983+0000","last_change":"2024-04-15T15:16:07.709428+0000","last_active":"2024-04-15T15:16:09.358983+0000","last_peered":"2024-04-15T15:16:09.358983+0000","last_clean":"2024-04-15T15:16:09.358983+0000","last_became_active":"2024-04-15T15:16:07.709164+0000","last_became_peered":"2024-04-15T15:16:07.709164+0000","last_unstale":"2024-04-15T15:16:09.358983+0000","last_undegraded":"2024-04-15T15:16:09.358983+0000","last_fullsized":"2024-04-15T15:16:09.358983+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:57:26.177964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.358898+0000","last_change":"2024-04-15T15:16:06.697149+0000","last_active":"2024-04-15T15:16:09.358898+0000","last_peered":"2024-04-15T15:16:09.358898+0000","last_clean":"2024-04-15T15:16:09.358898+0000","last_became_active":"2024-04-15T15:16:06.696879+0000","last_became_peered":"2024-04-15T15:16:06.696879+0000","last_unstale":"2024-04-15T15:16:09.358898+0000","last_undegraded":"2024-04-15T15:16:09.358898+0000","last_fullsized":"2024-04-15T15:16:09.358898+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:35:01.354434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.710256+0000","last_change":"2024-04-15T15:16:07.717293+0000","last_active":"2024-04-15T15:16:08.710256+0000","last_peered":"2024-04-15T15:16:08.710256+0000","last_clean":"2024-04-15T15:16:08.710256+0000","last_became_active":"2024-04-15T15:16:07.716988+0000","last_became_peered":"2024-04-15T15:16:07.716988+0000","last_unstale":"2024-04-15T15:16:08.710256+0000","last_undegraded":"2024-04-15T15:16:08.710256+0000","last_fullsized":"2024-04-15T15:16:08.710256+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:09:06.021853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973388+0000","last_change":"2024-04-15T15:16:06.702502+0000","last_active":"2024-04-15T15:16:08.973388+0000","last_peered":"2024-04-15T15:16:08.973388+0000","last_clean":"2024-04-15T15:16:08.973388+0000","last_became_active":"2024-04-15T15:16:06.701301+0000","last_became_peered":"2024-04-15T15:16:06.701301+0000","last_unstale":"2024-04-15T15:16:08.973388+0000","last_undegraded":"2024-04-15T15:16:08.973388+0000","last_fullsized":"2024-04-15T15:16:08.973388+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:03:11.873554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718815+0000","last_change":"2024-04-15T15:16:07.712592+0000","last_active":"2024-04-15T15:16:08.718815+0000","last_peered":"2024-04-15T15:16:08.718815+0000","last_clean":"2024-04-15T15:16:08.718815+0000","last_became_active":"2024-04-15T15:16:07.712213+0000","last_became_peered":"2024-04-15T15:16:07.712213+0000","last_unstale":"2024-04-15T15:16:08.718815+0000","last_undegraded":"2024-04-15T15:16:08.718815+0000","last_fullsized":"2024-04-15T15:16:08.718815+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:51:12.622012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709805+0000","last_change":"2024-04-15T15:16:06.700821+0000","last_active":"2024-04-15T15:16:08.709805+0000","last_peered":"2024-04-15T15:16:08.709805+0000","last_clean":"2024-04-15T15:16:08.709805+0000","last_became_active":"2024-04-15T15:16:06.700436+0000","last_became_peered":"2024-04-15T15:16:06.700436+0000","last_unstale":"2024-04-15T15:16:08.709805+0000","last_undegraded":"2024-04-15T15:16:08.709805+0000","last_fullsized":"2024-04-15T15:16:08.709805+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:14:27.402622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":202,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.710011+0000","last_change":"2024-04-15T15:14:40.494279+0000","last_active":"2024-04-15T15:16:08.710011+0000","last_peered":"2024-04-15T15:16:08.710011+0000","last_clean":"2024-04-15T15:16:08.710011+0000","last_became_active":"2024-04-15T15:14:40.493404+0000","last_became_peered":"2024-04-15T15:14:40.493404+0000","last_unstale":"2024-04-15T15:16:08.710011+0000","last_undegraded":"2024-04-15T15:16:08.710011+0000","last_fullsized":"2024-04-15T15:16:08.710011+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:39.418508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:39.418508+0000","last_clean_scrub_stamp":"2024-04-15T15:14:39.418508+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:01:51.789246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28052,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343788,"statfs":{"total":96636764160,"available":96608038912,"internally_reserved":0,"allocated":1048576,"data_stored":782086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574870,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":430080,"data_stored":183926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27460,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344380,"statfs":{"total":96636764160,"available":96608645120,"internally_reserved":0,"allocated":446464,"data_stored":190148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27444,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344396,"statfs":{"total":96636764160,"available":96608661504,"internally_reserved":0,"allocated":434176,"data_stored":183948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":442368,"data_stored":184990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":430080,"data_stored":183414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":430080,"data_stored":183434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343816,"statfs":{"total":96636764160,"available":96608067584,"internally_reserved":0,"allocated":1024000,"data_stored":773746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:16:11.388 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-15T15:16:11.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.524+0000 7f8deee7b640 1 Processor -- start 2024-04-15T15:16:11.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.524+0000 7f8deee7b640 1 -- start start 2024-04-15T15:16:11.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.525+0000 7f8deee7b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8de8155c30 0x7f8de806e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.525+0000 7f8deee7b640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8de80b4b30 0x7f8de806e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.525+0000 7f8deee7b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8de806f1e0 con 0x7f8de8155850 2024-04-15T15:16:11.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.525+0000 7f8deee7b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f8de806fd60 con 0x7f8de8155c30 2024-04-15T15:16:11.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.525+0000 7f8deee7b640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f8de806f4c0 con 0x7f8de80b4b30 2024-04-15T15:16:11.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.525+0000 7f8ddffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8de8155c30 0x7f8de806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.525+0000 7f8ddffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8de8155c30 0x7f8de806e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42306/0 (socket says 172.21.15.72:42306) 2024-04-15T15:16:11.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.525+0000 7f8decc17640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f8de8155850 0x7f8de80b4420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:58894/0 (socket says 172.21.15.72:58894) 2024-04-15T15:16:11.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.525+0000 7f8ddffff640 1 -- 172.21.15.72:0/1273268355 learned_addr learned my addr 172.21.15.72:0/1273268355 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:11.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.525+0000 7f8ded418640 1 --2- 172.21.15.72:0/1273268355 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8de80b4b30 0x7f8de806e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.525+0000 7f8ddffff640 1 -- 172.21.15.72:0/1273268355 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8de80b4b30 msgr2=0x7f8de806e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:11.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.525+0000 7f8ddffff640 1 --2- 172.21.15.72:0/1273268355 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8de80b4b30 0x7f8de806e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.525+0000 7f8ddffff640 1 -- 172.21.15.72:0/1273268355 >> v1:172.21.15.72:6789/0 conn(0x7f8de8155850 legacy=0x7f8de80b4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.525+0000 7f8ddf7fe640 1 -- 172.21.15.72:0/1273268355 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 458961057 0 0) 0x7f8de806f1e0 con 0x7f8de8155850 2024-04-15T15:16:11.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.525+0000 7f8ddffff640 1 -- 172.21.15.72:0/1273268355 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8de806f7a0 con 0x7f8de8155c30 2024-04-15T15:16:11.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.526+0000 7f8ddffff640 1 --2- 172.21.15.72:0/1273268355 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8de8155c30 0x7f8de806e470 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f8dd8009870 tx=0x7f8dd802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fff346e2799c3da6 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:11.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.526+0000 7f8ddf7fe640 1 -- 172.21.15.72:0/1273268355 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd8035020 con 0x7f8de8155c30 2024-04-15T15:16:11.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.526+0000 7f8ddf7fe640 1 -- 172.21.15.72:0/1273268355 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8dd8031640 con 0x7f8de8155c30 2024-04-15T15:16:11.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.526+0000 7f8ddf7fe640 1 -- 172.21.15.72:0/1273268355 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd802d040 con 0x7f8de8155c30 2024-04-15T15:16:11.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.527+0000 7f8deee7b640 1 -- 172.21.15.72:0/1273268355 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8de8155c30 msgr2=0x7f8de806e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.527+0000 7f8deee7b640 1 --2- 172.21.15.72:0/1273268355 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8de8155c30 0x7f8de806e470 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f8dd8009870 tx=0x7f8dd802b8f0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.527+0000 7f8deee7b640 1 -- 172.21.15.72:0/1273268355 shutdown_connections 2024-04-15T15:16:11.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.527+0000 7f8deee7b640 1 --2- 172.21.15.72:0/1273268355 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8de80b4b30 0x7f8de806e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.527+0000 7f8deee7b640 1 --2- 172.21.15.72:0/1273268355 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8de8155c30 0x7f8de806e470 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.527+0000 7f8deee7b640 1 -- 172.21.15.72:0/1273268355 >> 172.21.15.72:0/1273268355 conn(0x7f8de80b31b0 msgr2=0x7f8de80d0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:11.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.527+0000 7f8deee7b640 1 -- 172.21.15.72:0/1273268355 shutdown_connections 2024-04-15T15:16:11.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.527+0000 7f8deee7b640 1 -- 172.21.15.72:0/1273268355 wait complete. 2024-04-15T15:16:11.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.528+0000 7f8deee7b640 1 Processor -- start 2024-04-15T15:16:11.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.528+0000 7f8deee7b640 1 -- start start 2024-04-15T15:16:11.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.529+0000 7f8deee7b640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8de80b4b30 0x7f8de806c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.529+0000 7f8deee7b640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8de8155850 0x7f8de8160e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.529+0000 7f8deee7b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8de81613a0 0x7f8de815b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.529+0000 7f8deee7b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8de806bf00 con 0x7f8de81613a0 2024-04-15T15:16:11.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.529+0000 7f8deee7b640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f8de806eef0 con 0x7f8de80b4b30 2024-04-15T15:16:11.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.529+0000 7f8deee7b640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f8de806c080 con 0x7f8de8155850 2024-04-15T15:16:11.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.529+0000 7f8ded418640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8de81613a0 0x7f8de815b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.529+0000 7f8ded418640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8de81613a0 0x7f8de815b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42318/0 (socket says 172.21.15.72:42318) 2024-04-15T15:16:11.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.529+0000 7f8ded418640 1 -- 172.21.15.72:0/2285914160 learned_addr learned my addr 172.21.15.72:0/2285914160 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:11.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.529+0000 7f8ddffff640 1 --2- 172.21.15.72:0/2285914160 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8de8155850 0x7f8de8160e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.529+0000 7f8decc17640 1 --2- 172.21.15.72:0/2285914160 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8de80b4b30 0x7f8de806c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.529+0000 7f8ded418640 1 -- 172.21.15.72:0/2285914160 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8de8155850 msgr2=0x7f8de8160e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.529+0000 7f8ded418640 1 --2- 172.21.15.72:0/2285914160 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8de8155850 0x7f8de8160e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.529+0000 7f8ded418640 1 -- 172.21.15.72:0/2285914160 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8de80b4b30 msgr2=0x7f8de806c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.529+0000 7f8ded418640 1 --2- 172.21.15.72:0/2285914160 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8de80b4b30 0x7f8de806c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.529+0000 7f8ded418640 1 -- 172.21.15.72:0/2285914160 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8de815abe0 con 0x7f8de81613a0 2024-04-15T15:16:11.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.530+0000 7f8ded418640 1 --2- 172.21.15.72:0/2285914160 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8de81613a0 0x7f8de815b2e0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f8de0007e40 tx=0x7f8de000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:11.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.530+0000 7f8ddd7fa640 1 -- 172.21.15.72:0/2285914160 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de0014620 con 0x7f8de81613a0 2024-04-15T15:16:11.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.530+0000 7f8ddd7fa640 1 -- 172.21.15.72:0/2285914160 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8de0019070 con 0x7f8de81613a0 2024-04-15T15:16:11.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.530+0000 7f8deee7b640 1 -- 172.21.15.72:0/2285914160 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8de81619d0 con 0x7f8de81613a0 2024-04-15T15:16:11.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.530+0000 7f8ddd7fa640 1 -- 172.21.15.72:0/2285914160 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de0014a20 con 0x7f8de81613a0 2024-04-15T15:16:11.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.530+0000 7f8deee7b640 1 -- 172.21.15.72:0/2285914160 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8de8161f90 con 0x7f8de81613a0 2024-04-15T15:16:11.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.531+0000 7f8deee7b640 1 -- 172.21.15.72:0/2285914160 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8de8162190 con 0x7f8de81613a0 2024-04-15T15:16:11.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.531+0000 7f8ddd7fa640 1 -- 172.21.15.72:0/2285914160 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8de0014bc0 con 0x7f8de81613a0 2024-04-15T15:16:11.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.532+0000 7f8ddd7fa640 1 --2- 172.21.15.72:0/2285914160 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8dd406c920 0x7f8dd406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.532+0000 7f8decc17640 1 --2- 172.21.15.72:0/2285914160 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8dd406c920 0x7f8dd406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.532+0000 7f8ddd7fa640 1 -- 172.21.15.72:0/2285914160 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f8de008c410 con 0x7f8de81613a0 2024-04-15T15:16:11.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.532+0000 7f8decc17640 1 --2- 172.21.15.72:0/2285914160 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8dd406c920 0x7f8dd406ede0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8de4000f60 tx=0x7f8de4007d50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:11.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.536+0000 7f8ddd7fa640 1 -- 172.21.15.72:0/2285914160 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8de005a060 con 0x7f8de81613a0 2024-04-15T15:16:11.677 INFO:tasks.ceph.mds.a.smithi072.stderr:2024-04-15T15:16:11.675+0000 7f9aed3fd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-15T15:16:11.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.784+0000 7f8deee7b640 1 -- 172.21.15.72:0/2285914160 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f8de80b0310 con 0x7f8de81613a0 2024-04-15T15:16:11.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.785+0000 7f8ddd7fa640 1 -- 172.21.15.72:0/2285914160 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1013 (secure 0 0 0) 0x7f8de005e360 con 0x7f8de81613a0 2024-04-15T15:16:11.788 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:16:11.788 INFO:teuthology.orchestra.run.smithi072.stdout:{"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":104,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":8,"num_up_osds":8,"osd_up_since":1713194078,"num_in_osds":8,"osd_in_since":1713194069,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":24,"data_bytes":592778,"bytes_used":226054144,"bytes_avail":772868059136,"bytes_total":773094113280,"write_bytes_sec":2525,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":16,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"a","status":"up:active","gid":5559}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-04-15T15:15:57.357224+0000","services":{}},"progress_events":{}} 2024-04-15T15:16:11.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.787+0000 7f8deee7b640 1 -- 172.21.15.72:0/2285914160 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8dd406c920 msgr2=0x7f8dd406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.787+0000 7f8deee7b640 1 --2- 172.21.15.72:0/2285914160 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8dd406c920 0x7f8dd406ede0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8de4000f60 tx=0x7f8de4007d50 comp rx=0 tx=0).stop 2024-04-15T15:16:11.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.787+0000 7f8deee7b640 1 -- 172.21.15.72:0/2285914160 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8de81613a0 msgr2=0x7f8de815b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.787+0000 7f8deee7b640 1 --2- 172.21.15.72:0/2285914160 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8de81613a0 0x7f8de815b2e0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f8de0007e40 tx=0x7f8de000e4b0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.787+0000 7f8deee7b640 1 -- 172.21.15.72:0/2285914160 shutdown_connections 2024-04-15T15:16:11.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.787+0000 7f8deee7b640 1 --2- 172.21.15.72:0/2285914160 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8dd406c920 0x7f8dd406ede0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.787+0000 7f8deee7b640 1 --2- 172.21.15.72:0/2285914160 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8de81613a0 0x7f8de815b2e0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.787+0000 7f8deee7b640 1 --2- 172.21.15.72:0/2285914160 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8de8155850 0x7f8de8160e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.787+0000 7f8deee7b640 1 --2- 172.21.15.72:0/2285914160 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8de80b4b30 0x7f8de806c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.787+0000 7f8deee7b640 1 -- 172.21.15.72:0/2285914160 >> 172.21.15.72:0/2285914160 conn(0x7f8de80b31b0 msgr2=0x7f8de80b0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:11.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.787+0000 7f8deee7b640 1 -- 172.21.15.72:0/2285914160 shutdown_connections 2024-04-15T15:16:11.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.787+0000 7f8deee7b640 1 -- 172.21.15.72:0/2285914160 wait complete. 2024-04-15T15:16:11.800 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-04-15T15:16:11.801 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:16:11.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.900+0000 7faa4e0d9640 1 Processor -- start 2024-04-15T15:16:11.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.900+0000 7faa4e0d9640 1 -- start start 2024-04-15T15:16:11.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.901+0000 7faa4e0d9640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7faa480af400 0x7faa480ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.901+0000 7faa4e0d9640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7faa480af860 0x7faa480cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.901+0000 7faa4e0d9640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa4806f8f0 con 0x7faa4815eba0 2024-04-15T15:16:11.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.901+0000 7faa4e0d9640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7faa4815ef80 con 0x7faa480af400 2024-04-15T15:16:11.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.901+0000 7faa4e0d9640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7faa480ce2b0 con 0x7faa480af860 2024-04-15T15:16:11.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.901+0000 7faa46ffd640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7faa480af400 0x7faa480ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.901+0000 7faa477fe640 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7faa4815eba0 0x7faa4806f1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.72:49478/0 (socket says 172.21.15.72:49478) 2024-04-15T15:16:11.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.901+0000 7faa477fe640 1 -- 172.21.15.72:0/1829693767 learned_addr learned my addr 172.21.15.72:0/1829693767 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:11.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.901+0000 7faa46ffd640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7faa480af400 0x7faa480ccb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.72:42770/0 (socket says 172.21.15.72:42770) 2024-04-15T15:16:11.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.901+0000 7faa47fff640 1 --2- 172.21.15.72:0/1829693767 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7faa480af860 0x7faa480cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.901+0000 7faa46ffd640 1 -- 172.21.15.72:0/1829693767 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7faa480af860 msgr2=0x7faa480cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:11.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.901+0000 7faa46ffd640 1 --2- 172.21.15.72:0/1829693767 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7faa480af860 0x7faa480cd0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.901+0000 7faa46ffd640 1 -- 172.21.15.72:0/1829693767 >> v1:172.21.15.175:6789/0 conn(0x7faa4815eba0 legacy=0x7faa4806f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.901+0000 7faa46ffd640 1 -- 172.21.15.72:0/1829693767 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa480ce590 con 0x7faa480af400 2024-04-15T15:16:11.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.901+0000 7faa467fc640 1 -- 172.21.15.72:0/1829693767 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1721053359 0 0) 0x7faa4806f8f0 con 0x7faa4815eba0 2024-04-15T15:16:11.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.902+0000 7faa46ffd640 1 --2- 172.21.15.72:0/1829693767 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faa480af400 0x7faa480ccb90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7faa3c0097a0 tx=0x7faa3c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=670f63a2aa72306 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:11.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.902+0000 7faa467fc640 1 -- 172.21.15.72:0/1829693767 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3c0315a0 con 0x7faa480af400 2024-04-15T15:16:11.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.902+0000 7faa467fc640 1 -- 172.21.15.72:0/1829693767 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa3c037070 con 0x7faa480af400 2024-04-15T15:16:11.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.902+0000 7faa467fc640 1 -- 172.21.15.72:0/1829693767 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3c031960 con 0x7faa480af400 2024-04-15T15:16:11.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.903+0000 7faa4e0d9640 1 -- 172.21.15.72:0/1829693767 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faa480af400 msgr2=0x7faa480ccb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.903+0000 7faa4e0d9640 1 --2- 172.21.15.72:0/1829693767 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faa480af400 0x7faa480ccb90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7faa3c0097a0 tx=0x7faa3c02b840 comp rx=0 tx=0).stop 2024-04-15T15:16:11.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.903+0000 7faa4e0d9640 1 -- 172.21.15.72:0/1829693767 shutdown_connections 2024-04-15T15:16:11.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.903+0000 7faa4e0d9640 1 --2- 172.21.15.72:0/1829693767 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7faa480af860 0x7faa480cd0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.903+0000 7faa4e0d9640 1 --2- 172.21.15.72:0/1829693767 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faa480af400 0x7faa480ccb90 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.903+0000 7faa4e0d9640 1 -- 172.21.15.72:0/1829693767 >> 172.21.15.72:0/1829693767 conn(0x7faa480b1dd0 msgr2=0x7faa480b21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:11.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.903+0000 7faa4e0d9640 1 -- 172.21.15.72:0/1829693767 shutdown_connections 2024-04-15T15:16:11.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.903+0000 7faa4e0d9640 1 -- 172.21.15.72:0/1829693767 wait complete. 2024-04-15T15:16:11.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.904+0000 7faa4e0d9640 1 Processor -- start 2024-04-15T15:16:11.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.904+0000 7faa4e0d9640 1 -- start start 2024-04-15T15:16:11.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.905+0000 7faa4e0d9640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faa480af400 0x7faa4815b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.905+0000 7faa4e0d9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa480af860 0x7faa4813c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.905+0000 7faa4e0d9640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7faa4815eba0 0x7faa48151750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.905+0000 7faa4e0d9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7faa480b2b80 con 0x7faa480af860 2024-04-15T15:16:11.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.905+0000 7faa4e0d9640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7faa4806ecc0 con 0x7faa480af400 2024-04-15T15:16:11.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.905+0000 7faa4e0d9640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7faa480ceb50 con 0x7faa4815eba0 2024-04-15T15:16:11.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.905+0000 7faa46ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa480af860 0x7faa4813c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.905+0000 7faa46ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa480af860 0x7faa4813c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56170/0 (socket says 172.21.15.72:56170) 2024-04-15T15:16:11.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.905+0000 7faa46ffd640 1 -- 172.21.15.72:0/2143616515 learned_addr learned my addr 172.21.15.72:0/2143616515 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:11.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.905+0000 7faa477fe640 1 --2- 172.21.15.72:0/2143616515 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faa480af400 0x7faa4815b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.905+0000 7faa47fff640 1 --2- 172.21.15.72:0/2143616515 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7faa4815eba0 0x7faa48151750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.905+0000 7faa46ffd640 1 -- 172.21.15.72:0/2143616515 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7faa4815eba0 msgr2=0x7faa48151750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.905+0000 7faa46ffd640 1 --2- 172.21.15.72:0/2143616515 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7faa4815eba0 0x7faa48151750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.905+0000 7faa46ffd640 1 -- 172.21.15.72:0/2143616515 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faa480af400 msgr2=0x7faa4815b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.905+0000 7faa46ffd640 1 --2- 172.21.15.72:0/2143616515 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faa480af400 0x7faa4815b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.905+0000 7faa46ffd640 1 -- 172.21.15.72:0/2143616515 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa4813cba0 con 0x7faa480af860 2024-04-15T15:16:11.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.905+0000 7faa477fe640 1 --2- 172.21.15.72:0/2143616515 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faa480af400 0x7faa4815b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:11.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.906+0000 7faa46ffd640 1 --2- 172.21.15.72:0/2143616515 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa480af860 0x7faa4813c370 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7faa3c0097a0 tx=0x7faa3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:11.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.906+0000 7faa2ffff640 1 -- 172.21.15.72:0/2143616515 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3c004230 con 0x7faa480af860 2024-04-15T15:16:11.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.906+0000 7faa2ffff640 1 -- 172.21.15.72:0/2143616515 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa3c0043d0 con 0x7faa480af860 2024-04-15T15:16:11.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.906+0000 7faa2ffff640 1 -- 172.21.15.72:0/2143616515 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3c003850 con 0x7faa480af860 2024-04-15T15:16:11.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.906+0000 7faa4e0d9640 1 -- 172.21.15.72:0/2143616515 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa4813ce80 con 0x7faa480af860 2024-04-15T15:16:11.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.906+0000 7faa4e0d9640 1 -- 172.21.15.72:0/2143616515 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa4813a2e0 con 0x7faa480af860 2024-04-15T15:16:11.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.907+0000 7faa4e0d9640 1 -- 172.21.15.72:0/2143616515 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa480cf0f0 con 0x7faa480af860 2024-04-15T15:16:11.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.907+0000 7faa2ffff640 1 -- 172.21.15.72:0/2143616515 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7faa3c0039f0 con 0x7faa480af860 2024-04-15T15:16:11.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.908+0000 7faa2ffff640 1 --2- 172.21.15.72:0/2143616515 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7faa3406c8f0 0x7faa3406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.908+0000 7faa477fe640 1 --2- 172.21.15.72:0/2143616515 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7faa3406c8f0 0x7faa3406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.908+0000 7faa2ffff640 1 -- 172.21.15.72:0/2143616515 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7faa3c02f030 con 0x7faa480af860 2024-04-15T15:16:11.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.908+0000 7faa477fe640 1 --2- 172.21.15.72:0/2143616515 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7faa3406c8f0 0x7faa3406edb0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7faa38000f60 tx=0x7faa38007d50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:11.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:11.912+0000 7faa2ffff640 1 -- 172.21.15.72:0/2143616515 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7faa480cf0f0 con 0x7faa480af860 2024-04-15T15:16:12.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:12.083+0000 7faa4e0d9640 1 -- 172.21.15.72:0/2143616515 --> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7faa4806d180 con 0x7faa3406c8f0 2024-04-15T15:16:12.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:12.089+0000 7faa2ffff640 1 -- 172.21.15.72:0/2143616515 <== mgr.4101 v2:172.21.15.72:6832/3503807805 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251944 (secure 0 0 0) 0x7faa4806d180 con 0x7faa3406c8f0 2024-04-15T15:16:12.094 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:16:12.096 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-04-15T15:16:12.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:12.092+0000 7faa4e0d9640 1 -- 172.21.15.72:0/2143616515 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7faa3406c8f0 msgr2=0x7faa3406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:12.097 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:12.092+0000 7faa4e0d9640 1 --2- 172.21.15.72:0/2143616515 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7faa3406c8f0 0x7faa3406edb0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7faa38000f60 tx=0x7faa38007d50 comp rx=0 tx=0).stop 2024-04-15T15:16:12.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:12.092+0000 7faa4e0d9640 1 -- 172.21.15.72:0/2143616515 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa480af860 msgr2=0x7faa4813c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:12.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:12.092+0000 7faa4e0d9640 1 --2- 172.21.15.72:0/2143616515 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa480af860 0x7faa4813c370 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7faa3c0097a0 tx=0x7faa3c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:12.092+0000 7faa4e0d9640 1 -- 172.21.15.72:0/2143616515 shutdown_connections 2024-04-15T15:16:12.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:12.092+0000 7faa4e0d9640 1 --2- 172.21.15.72:0/2143616515 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7faa3406c8f0 0x7faa3406edb0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:12.092+0000 7faa4e0d9640 1 --2- 172.21.15.72:0/2143616515 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7faa4815eba0 0x7faa48151750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:12.092+0000 7faa4e0d9640 1 --2- 172.21.15.72:0/2143616515 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa480af860 0x7faa4813c370 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:12.092+0000 7faa4e0d9640 1 --2- 172.21.15.72:0/2143616515 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faa480af400 0x7faa4815b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:12.092+0000 7faa4e0d9640 1 -- 172.21.15.72:0/2143616515 >> 172.21.15.72:0/2143616515 conn(0x7faa480b1dd0 msgr2=0x7faa4815c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:12.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:12.092+0000 7faa4e0d9640 1 -- 172.21.15.72:0/2143616515 shutdown_connections 2024-04-15T15:16:12.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:12.092+0000 7faa4e0d9640 1 -- 172.21.15.72:0/2143616515 wait complete. 2024-04-15T15:16:12.105 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":77,"stamp":"2024-04-15T15:16:11.359547+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":159,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220756,"kb_used_data":4576,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753964,"statfs":{"total":773094113280,"available":772868059136,"internally_reserved":0,"allocated":4685824,"data_stored":2665692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.677172"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973935+0000","last_change":"2024-04-15T15:16:07.714016+0000","last_active":"2024-04-15T15:16:08.973935+0000","last_peered":"2024-04-15T15:16:08.973935+0000","last_clean":"2024-04-15T15:16:08.973935+0000","last_became_active":"2024-04-15T15:16:07.713760+0000","last_became_peered":"2024-04-15T15:16:07.713760+0000","last_unstale":"2024-04-15T15:16:08.973935+0000","last_undegraded":"2024-04-15T15:16:08.973935+0000","last_fullsized":"2024-04-15T15:16:08.973935+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:26:10.169823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973663+0000","last_change":"2024-04-15T15:16:07.716357+0000","last_active":"2024-04-15T15:16:08.973663+0000","last_peered":"2024-04-15T15:16:08.973663+0000","last_clean":"2024-04-15T15:16:08.973663+0000","last_became_active":"2024-04-15T15:16:07.715891+0000","last_became_peered":"2024-04-15T15:16:07.715891+0000","last_unstale":"2024-04-15T15:16:08.973663+0000","last_undegraded":"2024-04-15T15:16:08.973663+0000","last_fullsized":"2024-04-15T15:16:08.973663+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:32:24.986373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.715319+0000","last_change":"2024-04-15T15:16:07.706936+0000","last_active":"2024-04-15T15:16:07.715319+0000","last_peered":"2024-04-15T15:16:07.715319+0000","last_clean":"2024-04-15T15:16:07.715319+0000","last_became_active":"2024-04-15T15:16:07.706670+0000","last_became_peered":"2024-04-15T15:16:07.706670+0000","last_unstale":"2024-04-15T15:16:07.715319+0000","last_undegraded":"2024-04-15T15:16:07.715319+0000","last_fullsized":"2024-04-15T15:16:07.715319+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:51:04.229786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.708922+0000","last_change":"2024-04-15T15:16:07.708400+0000","last_active":"2024-04-15T15:16:08.708922+0000","last_peered":"2024-04-15T15:16:08.708922+0000","last_clean":"2024-04-15T15:16:08.708922+0000","last_became_active":"2024-04-15T15:16:07.707958+0000","last_became_peered":"2024-04-15T15:16:07.707958+0000","last_unstale":"2024-04-15T15:16:08.708922+0000","last_undegraded":"2024-04-15T15:16:08.708922+0000","last_fullsized":"2024-04-15T15:16:08.708922+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:18:13.186969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.708673+0000","last_change":"2024-04-15T15:16:07.712681+0000","last_active":"2024-04-15T15:16:08.708673+0000","last_peered":"2024-04-15T15:16:08.708673+0000","last_clean":"2024-04-15T15:16:08.708673+0000","last_became_active":"2024-04-15T15:16:07.712430+0000","last_became_peered":"2024-04-15T15:16:07.712430+0000","last_unstale":"2024-04-15T15:16:08.708673+0000","last_undegraded":"2024-04-15T15:16:08.708673+0000","last_fullsized":"2024-04-15T15:16:08.708673+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:19:44.813755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.718802+0000","last_change":"2024-04-15T15:16:07.709386+0000","last_active":"2024-04-15T15:16:07.718802+0000","last_peered":"2024-04-15T15:16:07.718802+0000","last_clean":"2024-04-15T15:16:07.718802+0000","last_became_active":"2024-04-15T15:16:07.709113+0000","last_became_peered":"2024-04-15T15:16:07.709113+0000","last_unstale":"2024-04-15T15:16:07.718802+0000","last_undegraded":"2024-04-15T15:16:07.718802+0000","last_fullsized":"2024-04-15T15:16:07.718802+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:01:01.910545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718270+0000","last_change":"2024-04-15T15:16:07.711815+0000","last_active":"2024-04-15T15:16:08.718270+0000","last_peered":"2024-04-15T15:16:08.718270+0000","last_clean":"2024-04-15T15:16:08.718270+0000","last_became_active":"2024-04-15T15:16:07.711561+0000","last_became_peered":"2024-04-15T15:16:07.711561+0000","last_unstale":"2024-04-15T15:16:08.718270+0000","last_undegraded":"2024-04-15T15:16:08.718270+0000","last_fullsized":"2024-04-15T15:16:08.718270+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:09:09.330985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973403+0000","last_change":"2024-04-15T15:16:07.711991+0000","last_active":"2024-04-15T15:16:08.973403+0000","last_peered":"2024-04-15T15:16:08.973403+0000","last_clean":"2024-04-15T15:16:08.973403+0000","last_became_active":"2024-04-15T15:16:07.711753+0000","last_became_peered":"2024-04-15T15:16:07.711753+0000","last_unstale":"2024-04-15T15:16:08.973403+0000","last_undegraded":"2024-04-15T15:16:08.973403+0000","last_fullsized":"2024-04-15T15:16:08.973403+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:00:49.742979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.716270+0000","last_change":"2024-04-15T15:16:07.711991+0000","last_active":"2024-04-15T15:16:07.716270+0000","last_peered":"2024-04-15T15:16:07.716270+0000","last_clean":"2024-04-15T15:16:07.716270+0000","last_became_active":"2024-04-15T15:16:07.711634+0000","last_became_peered":"2024-04-15T15:16:07.711634+0000","last_unstale":"2024-04-15T15:16:07.716270+0000","last_undegraded":"2024-04-15T15:16:07.716270+0000","last_fullsized":"2024-04-15T15:16:07.716270+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:14:28.406716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718646+0000","last_change":"2024-04-15T15:16:07.714191+0000","last_active":"2024-04-15T15:16:08.718646+0000","last_peered":"2024-04-15T15:16:08.718646+0000","last_clean":"2024-04-15T15:16:08.718646+0000","last_became_active":"2024-04-15T15:16:07.713121+0000","last_became_peered":"2024-04-15T15:16:07.713121+0000","last_unstale":"2024-04-15T15:16:08.718646+0000","last_undegraded":"2024-04-15T15:16:08.718646+0000","last_fullsized":"2024-04-15T15:16:08.718646+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:32:24.986373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718831+0000","last_change":"2024-04-15T15:16:07.714129+0000","last_active":"2024-04-15T15:16:08.718831+0000","last_peered":"2024-04-15T15:16:08.718831+0000","last_clean":"2024-04-15T15:16:08.718831+0000","last_became_active":"2024-04-15T15:16:07.713746+0000","last_became_peered":"2024-04-15T15:16:07.713746+0000","last_unstale":"2024-04-15T15:16:08.718831+0000","last_undegraded":"2024-04-15T15:16:08.718831+0000","last_fullsized":"2024-04-15T15:16:08.718831+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:41:28.911237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.713119+0000","last_change":"2024-04-15T15:16:07.710622+0000","last_active":"2024-04-15T15:16:07.713119+0000","last_peered":"2024-04-15T15:16:07.713119+0000","last_clean":"2024-04-15T15:16:07.713119+0000","last_became_active":"2024-04-15T15:16:07.710280+0000","last_became_peered":"2024-04-15T15:16:07.710280+0000","last_unstale":"2024-04-15T15:16:07.713119+0000","last_undegraded":"2024-04-15T15:16:07.713119+0000","last_fullsized":"2024-04-15T15:16:07.713119+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:33:10.314944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718413+0000","last_change":"2024-04-15T15:16:07.713369+0000","last_active":"2024-04-15T15:16:08.718413+0000","last_peered":"2024-04-15T15:16:08.718413+0000","last_clean":"2024-04-15T15:16:08.718413+0000","last_became_active":"2024-04-15T15:16:07.713109+0000","last_became_peered":"2024-04-15T15:16:07.713109+0000","last_unstale":"2024-04-15T15:16:08.718413+0000","last_undegraded":"2024-04-15T15:16:08.718413+0000","last_fullsized":"2024-04-15T15:16:08.718413+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:00:49.742979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.358927+0000","last_change":"2024-04-15T15:16:07.709278+0000","last_active":"2024-04-15T15:16:09.358927+0000","last_peered":"2024-04-15T15:16:09.358927+0000","last_clean":"2024-04-15T15:16:09.358927+0000","last_became_active":"2024-04-15T15:16:07.708829+0000","last_became_peered":"2024-04-15T15:16:07.708829+0000","last_unstale":"2024-04-15T15:16:09.358927+0000","last_undegraded":"2024-04-15T15:16:09.358927+0000","last_fullsized":"2024-04-15T15:16:09.358927+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:51:17.043733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718403+0000","last_change":"2024-04-15T15:16:07.700808+0000","last_active":"2024-04-15T15:16:08.718403+0000","last_peered":"2024-04-15T15:16:08.718403+0000","last_clean":"2024-04-15T15:16:08.718403+0000","last_became_active":"2024-04-15T15:16:07.700484+0000","last_became_peered":"2024-04-15T15:16:07.700484+0000","last_unstale":"2024-04-15T15:16:08.718403+0000","last_undegraded":"2024-04-15T15:16:08.718403+0000","last_fullsized":"2024-04-15T15:16:08.718403+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:14:45.605613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709813+0000","last_change":"2024-04-15T15:16:07.713960+0000","last_active":"2024-04-15T15:16:08.709813+0000","last_peered":"2024-04-15T15:16:08.709813+0000","last_clean":"2024-04-15T15:16:08.709813+0000","last_became_active":"2024-04-15T15:16:07.713665+0000","last_became_peered":"2024-04-15T15:16:07.713665+0000","last_unstale":"2024-04-15T15:16:08.709813+0000","last_undegraded":"2024-04-15T15:16:08.709813+0000","last_fullsized":"2024-04-15T15:16:08.709813+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:54:41.375909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.710290+0000","last_change":"2024-04-15T15:16:07.714486+0000","last_active":"2024-04-15T15:16:08.710290+0000","last_peered":"2024-04-15T15:16:08.710290+0000","last_clean":"2024-04-15T15:16:08.710290+0000","last_became_active":"2024-04-15T15:16:07.714191+0000","last_became_peered":"2024-04-15T15:16:07.714191+0000","last_unstale":"2024-04-15T15:16:08.710290+0000","last_undegraded":"2024-04-15T15:16:08.710290+0000","last_fullsized":"2024-04-15T15:16:08.710290+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:09:09.330985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709650+0000","last_change":"2024-04-15T15:16:07.711021+0000","last_active":"2024-04-15T15:16:08.709650+0000","last_peered":"2024-04-15T15:16:08.709650+0000","last_clean":"2024-04-15T15:16:08.709650+0000","last_became_active":"2024-04-15T15:16:07.710709+0000","last_became_peered":"2024-04-15T15:16:07.710709+0000","last_unstale":"2024-04-15T15:16:08.709650+0000","last_undegraded":"2024-04-15T15:16:08.709650+0000","last_fullsized":"2024-04-15T15:16:08.709650+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:51:17.043733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.708951+0000","last_change":"2024-04-15T15:16:07.712960+0000","last_active":"2024-04-15T15:16:08.708951+0000","last_peered":"2024-04-15T15:16:08.708951+0000","last_clean":"2024-04-15T15:16:08.708951+0000","last_became_active":"2024-04-15T15:16:07.712636+0000","last_became_peered":"2024-04-15T15:16:07.712636+0000","last_unstale":"2024-04-15T15:16:08.708951+0000","last_undegraded":"2024-04-15T15:16:08.708951+0000","last_fullsized":"2024-04-15T15:16:08.708951+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:02:51.429249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709437+0000","last_change":"2024-04-15T15:16:07.712946+0000","last_active":"2024-04-15T15:16:08.709437+0000","last_peered":"2024-04-15T15:16:08.709437+0000","last_clean":"2024-04-15T15:16:08.709437+0000","last_became_active":"2024-04-15T15:16:07.712677+0000","last_became_peered":"2024-04-15T15:16:07.712677+0000","last_unstale":"2024-04-15T15:16:08.709437+0000","last_undegraded":"2024-04-15T15:16:08.709437+0000","last_fullsized":"2024-04-15T15:16:08.709437+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:32:44.859092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718447+0000","last_change":"2024-04-15T15:16:07.708983+0000","last_active":"2024-04-15T15:16:08.718447+0000","last_peered":"2024-04-15T15:16:08.718447+0000","last_clean":"2024-04-15T15:16:08.718447+0000","last_became_active":"2024-04-15T15:16:07.708612+0000","last_became_peered":"2024-04-15T15:16:07.708612+0000","last_unstale":"2024-04-15T15:16:08.718447+0000","last_undegraded":"2024-04-15T15:16:08.718447+0000","last_fullsized":"2024-04-15T15:16:08.718447+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:12:03.446783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.359328+0000","last_change":"2024-04-15T15:16:07.710990+0000","last_active":"2024-04-15T15:16:09.359328+0000","last_peered":"2024-04-15T15:16:09.359328+0000","last_clean":"2024-04-15T15:16:09.359328+0000","last_became_active":"2024-04-15T15:16:07.710763+0000","last_became_peered":"2024-04-15T15:16:07.710763+0000","last_unstale":"2024-04-15T15:16:09.359328+0000","last_undegraded":"2024-04-15T15:16:09.359328+0000","last_fullsized":"2024-04-15T15:16:09.359328+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:39:05.235149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709573+0000","last_change":"2024-04-15T15:16:07.718348+0000","last_active":"2024-04-15T15:16:08.709573+0000","last_peered":"2024-04-15T15:16:08.709573+0000","last_clean":"2024-04-15T15:16:08.709573+0000","last_became_active":"2024-04-15T15:16:07.717954+0000","last_became_peered":"2024-04-15T15:16:07.717954+0000","last_unstale":"2024-04-15T15:16:08.709573+0000","last_undegraded":"2024-04-15T15:16:08.709573+0000","last_fullsized":"2024-04-15T15:16:08.709573+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:32:26.516274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.708677+0000","last_change":"2024-04-15T15:16:07.714768+0000","last_active":"2024-04-15T15:16:08.708677+0000","last_peered":"2024-04-15T15:16:08.708677+0000","last_clean":"2024-04-15T15:16:08.708677+0000","last_became_active":"2024-04-15T15:16:07.714512+0000","last_became_peered":"2024-04-15T15:16:07.714512+0000","last_unstale":"2024-04-15T15:16:08.708677+0000","last_undegraded":"2024-04-15T15:16:08.708677+0000","last_fullsized":"2024-04-15T15:16:08.708677+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:42:44.933220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.710013+0000","last_change":"2024-04-15T15:16:07.714668+0000","last_active":"2024-04-15T15:16:08.710013+0000","last_peered":"2024-04-15T15:16:08.710013+0000","last_clean":"2024-04-15T15:16:08.710013+0000","last_became_active":"2024-04-15T15:16:07.714354+0000","last_became_peered":"2024-04-15T15:16:07.714354+0000","last_unstale":"2024-04-15T15:16:08.710013+0000","last_undegraded":"2024-04-15T15:16:08.710013+0000","last_fullsized":"2024-04-15T15:16:08.710013+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:08:22.249305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.708863+0000","last_change":"2024-04-15T15:16:07.708195+0000","last_active":"2024-04-15T15:16:07.708863+0000","last_peered":"2024-04-15T15:16:07.708863+0000","last_clean":"2024-04-15T15:16:07.708863+0000","last_became_active":"2024-04-15T15:16:07.707897+0000","last_became_peered":"2024-04-15T15:16:07.707897+0000","last_unstale":"2024-04-15T15:16:07.708863+0000","last_undegraded":"2024-04-15T15:16:07.708863+0000","last_fullsized":"2024-04-15T15:16:07.708863+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:14:28.406716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709808+0000","last_change":"2024-04-15T15:16:07.707076+0000","last_active":"2024-04-15T15:16:08.709808+0000","last_peered":"2024-04-15T15:16:08.709808+0000","last_clean":"2024-04-15T15:16:08.709808+0000","last_became_active":"2024-04-15T15:16:07.706810+0000","last_became_peered":"2024-04-15T15:16:07.706810+0000","last_unstale":"2024-04-15T15:16:08.709808+0000","last_undegraded":"2024-04-15T15:16:08.709808+0000","last_fullsized":"2024-04-15T15:16:08.709808+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:45:03.154342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718597+0000","last_change":"2024-04-15T15:16:07.717296+0000","last_active":"2024-04-15T15:16:08.718597+0000","last_peered":"2024-04-15T15:16:08.718597+0000","last_clean":"2024-04-15T15:16:08.718597+0000","last_became_active":"2024-04-15T15:16:07.716737+0000","last_became_peered":"2024-04-15T15:16:07.716737+0000","last_unstale":"2024-04-15T15:16:08.718597+0000","last_undegraded":"2024-04-15T15:16:08.718597+0000","last_fullsized":"2024-04-15T15:16:08.718597+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:38:41.698007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709781+0000","last_change":"2024-04-15T15:16:07.708652+0000","last_active":"2024-04-15T15:16:08.709781+0000","last_peered":"2024-04-15T15:16:08.709781+0000","last_clean":"2024-04-15T15:16:08.709781+0000","last_became_active":"2024-04-15T15:16:07.708305+0000","last_became_peered":"2024-04-15T15:16:07.708305+0000","last_unstale":"2024-04-15T15:16:08.709781+0000","last_undegraded":"2024-04-15T15:16:08.709781+0000","last_fullsized":"2024-04-15T15:16:08.709781+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:57:26.177964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709498+0000","last_change":"2024-04-15T15:16:07.709602+0000","last_active":"2024-04-15T15:16:08.709498+0000","last_peered":"2024-04-15T15:16:08.709498+0000","last_clean":"2024-04-15T15:16:08.709498+0000","last_became_active":"2024-04-15T15:16:07.709332+0000","last_became_peered":"2024-04-15T15:16:07.709332+0000","last_unstale":"2024-04-15T15:16:08.709498+0000","last_undegraded":"2024-04-15T15:16:08.709498+0000","last_fullsized":"2024-04-15T15:16:08.709498+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:03:12.877648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709730+0000","last_change":"2024-04-15T15:16:07.717993+0000","last_active":"2024-04-15T15:16:08.709730+0000","last_peered":"2024-04-15T15:16:08.709730+0000","last_clean":"2024-04-15T15:16:08.709730+0000","last_became_active":"2024-04-15T15:16:07.717663+0000","last_became_peered":"2024-04-15T15:16:07.717663+0000","last_unstale":"2024-04-15T15:16:08.709730+0000","last_undegraded":"2024-04-15T15:16:08.709730+0000","last_fullsized":"2024-04-15T15:16:08.709730+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:42:46.979916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.722682+0000","last_change":"2024-04-15T15:16:07.719599+0000","last_active":"2024-04-15T15:16:07.722682+0000","last_peered":"2024-04-15T15:16:07.722682+0000","last_clean":"2024-04-15T15:16:07.722682+0000","last_became_active":"2024-04-15T15:16:07.719303+0000","last_became_peered":"2024-04-15T15:16:07.719303+0000","last_unstale":"2024-04-15T15:16:07.722682+0000","last_undegraded":"2024-04-15T15:16:07.722682+0000","last_fullsized":"2024-04-15T15:16:07.722682+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:32:44.859092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709700+0000","last_change":"2024-04-15T15:16:07.719675+0000","last_active":"2024-04-15T15:16:08.709700+0000","last_peered":"2024-04-15T15:16:08.709700+0000","last_clean":"2024-04-15T15:16:08.709700+0000","last_became_active":"2024-04-15T15:16:07.719391+0000","last_became_peered":"2024-04-15T15:16:07.719391+0000","last_unstale":"2024-04-15T15:16:08.709700+0000","last_undegraded":"2024-04-15T15:16:08.709700+0000","last_fullsized":"2024-04-15T15:16:08.709700+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:14:39.023105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"37'4","reported_seq":18,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.710178+0000","last_change":"2024-04-15T15:16:06.702682+0000","last_active":"2024-04-15T15:16:08.710178+0000","last_peered":"2024-04-15T15:16:08.710178+0000","last_clean":"2024-04-15T15:16:08.710178+0000","last_became_active":"2024-04-15T15:16:06.702428+0000","last_became_peered":"2024-04-15T15:16:06.702428+0000","last_unstale":"2024-04-15T15:16:08.710178+0000","last_undegraded":"2024-04-15T15:16:08.710178+0000","last_fullsized":"2024-04-15T15:16:08.710178+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:35:01.354434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.718799+0000","last_change":"2024-04-15T15:16:07.712184+0000","last_active":"2024-04-15T15:16:07.718799+0000","last_peered":"2024-04-15T15:16:07.718799+0000","last_clean":"2024-04-15T15:16:07.718799+0000","last_became_active":"2024-04-15T15:16:07.711833+0000","last_became_peered":"2024-04-15T15:16:07.711833+0000","last_unstale":"2024-04-15T15:16:07.718799+0000","last_undegraded":"2024-04-15T15:16:07.718799+0000","last_fullsized":"2024-04-15T15:16:07.718799+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:10:57.528229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"37'1","reported_seq":13,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.727968+0000","last_change":"2024-04-15T15:16:06.705264+0000","last_active":"2024-04-15T15:16:07.727968+0000","last_peered":"2024-04-15T15:16:07.727968+0000","last_clean":"2024-04-15T15:16:07.727968+0000","last_became_active":"2024-04-15T15:16:06.704985+0000","last_became_peered":"2024-04-15T15:16:06.704985+0000","last_unstale":"2024-04-15T15:16:07.727968+0000","last_undegraded":"2024-04-15T15:16:07.727968+0000","last_fullsized":"2024-04-15T15:16:07.727968+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:02:50.425156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.359305+0000","last_change":"2024-04-15T15:16:07.710774+0000","last_active":"2024-04-15T15:16:09.359305+0000","last_peered":"2024-04-15T15:16:09.359305+0000","last_clean":"2024-04-15T15:16:09.359305+0000","last_became_active":"2024-04-15T15:16:07.710468+0000","last_became_peered":"2024-04-15T15:16:07.710468+0000","last_unstale":"2024-04-15T15:16:09.359305+0000","last_undegraded":"2024-04-15T15:16:09.359305+0000","last_fullsized":"2024-04-15T15:16:09.359305+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:56:02.237631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"37'2","reported_seq":14,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.733126+0000","last_change":"2024-04-15T15:16:06.706399+0000","last_active":"2024-04-15T15:16:07.733126+0000","last_peered":"2024-04-15T15:16:07.733126+0000","last_clean":"2024-04-15T15:16:07.733126+0000","last_became_active":"2024-04-15T15:16:06.706141+0000","last_became_peered":"2024-04-15T15:16:06.706141+0000","last_unstale":"2024-04-15T15:16:07.733126+0000","last_undegraded":"2024-04-15T15:16:07.733126+0000","last_fullsized":"2024-04-15T15:16:07.733126+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:33:09.310850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973814+0000","last_change":"2024-04-15T15:16:07.710742+0000","last_active":"2024-04-15T15:16:08.973814+0000","last_peered":"2024-04-15T15:16:08.973814+0000","last_clean":"2024-04-15T15:16:08.973814+0000","last_became_active":"2024-04-15T15:16:07.710455+0000","last_became_peered":"2024-04-15T15:16:07.710455+0000","last_unstale":"2024-04-15T15:16:08.973814+0000","last_undegraded":"2024-04-15T15:16:08.973814+0000","last_fullsized":"2024-04-15T15:16:08.973814+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:41:28.911237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.359266+0000","last_change":"2024-04-15T15:16:06.704244+0000","last_active":"2024-04-15T15:16:09.359266+0000","last_peered":"2024-04-15T15:16:09.359266+0000","last_clean":"2024-04-15T15:16:09.359266+0000","last_became_active":"2024-04-15T15:16:06.703919+0000","last_became_peered":"2024-04-15T15:16:06.703919+0000","last_unstale":"2024-04-15T15:16:09.359266+0000","last_undegraded":"2024-04-15T15:16:09.359266+0000","last_fullsized":"2024-04-15T15:16:09.359266+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:51:03.225693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.716037+0000","last_change":"2024-04-15T15:16:07.712901+0000","last_active":"2024-04-15T15:16:07.716037+0000","last_peered":"2024-04-15T15:16:07.716037+0000","last_clean":"2024-04-15T15:16:07.716037+0000","last_became_active":"2024-04-15T15:16:07.712444+0000","last_became_peered":"2024-04-15T15:16:07.712444+0000","last_unstale":"2024-04-15T15:16:07.716037+0000","last_undegraded":"2024-04-15T15:16:07.716037+0000","last_fullsized":"2024-04-15T15:16:07.716037+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:38:29.844648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718599+0000","last_change":"2024-04-15T15:16:06.706424+0000","last_active":"2024-04-15T15:16:08.718599+0000","last_peered":"2024-04-15T15:16:08.718599+0000","last_clean":"2024-04-15T15:16:08.718599+0000","last_became_active":"2024-04-15T15:16:06.706138+0000","last_became_peered":"2024-04-15T15:16:06.706138+0000","last_unstale":"2024-04-15T15:16:08.718599+0000","last_undegraded":"2024-04-15T15:16:08.718599+0000","last_fullsized":"2024-04-15T15:16:08.718599+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:01:00.906451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718514+0000","last_change":"2024-04-15T15:16:07.709849+0000","last_active":"2024-04-15T15:16:08.718514+0000","last_peered":"2024-04-15T15:16:08.718514+0000","last_clean":"2024-04-15T15:16:08.718514+0000","last_became_active":"2024-04-15T15:16:07.709582+0000","last_became_peered":"2024-04-15T15:16:07.709582+0000","last_unstale":"2024-04-15T15:16:08.718514+0000","last_undegraded":"2024-04-15T15:16:08.718514+0000","last_fullsized":"2024-04-15T15:16:08.718514+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:04:27.874758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.710385+0000","last_change":"2024-04-15T15:16:06.704215+0000","last_active":"2024-04-15T15:16:08.710385+0000","last_peered":"2024-04-15T15:16:08.710385+0000","last_clean":"2024-04-15T15:16:08.710385+0000","last_became_active":"2024-04-15T15:16:06.703429+0000","last_became_peered":"2024-04-15T15:16:06.703429+0000","last_unstale":"2024-04-15T15:16:08.710385+0000","last_undegraded":"2024-04-15T15:16:08.710385+0000","last_fullsized":"2024-04-15T15:16:08.710385+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:51:03.225693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.359185+0000","last_change":"2024-04-15T15:16:07.709067+0000","last_active":"2024-04-15T15:16:09.359185+0000","last_peered":"2024-04-15T15:16:09.359185+0000","last_clean":"2024-04-15T15:16:09.359185+0000","last_became_active":"2024-04-15T15:16:07.708491+0000","last_became_peered":"2024-04-15T15:16:07.708491+0000","last_unstale":"2024-04-15T15:16:09.359185+0000","last_undegraded":"2024-04-15T15:16:09.359185+0000","last_fullsized":"2024-04-15T15:16:09.359185+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:45:03.154342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.702105+0000","last_change":"2024-04-15T15:16:06.707235+0000","last_active":"2024-04-15T15:16:07.702105+0000","last_peered":"2024-04-15T15:16:07.702105+0000","last_clean":"2024-04-15T15:16:07.702105+0000","last_became_active":"2024-04-15T15:16:06.706918+0000","last_became_peered":"2024-04-15T15:16:06.706918+0000","last_unstale":"2024-04-15T15:16:07.702105+0000","last_undegraded":"2024-04-15T15:16:07.702105+0000","last_fullsized":"2024-04-15T15:16:07.702105+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:25:09.602767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973566+0000","last_change":"2024-04-15T15:16:07.715092+0000","last_active":"2024-04-15T15:16:08.973566+0000","last_peered":"2024-04-15T15:16:08.973566+0000","last_clean":"2024-04-15T15:16:08.973566+0000","last_became_active":"2024-04-15T15:16:07.714832+0000","last_became_peered":"2024-04-15T15:16:07.714832+0000","last_unstale":"2024-04-15T15:16:08.973566+0000","last_undegraded":"2024-04-15T15:16:08.973566+0000","last_fullsized":"2024-04-15T15:16:08.973566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:30:18.481379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.358856+0000","last_change":"2024-04-15T15:16:06.699694+0000","last_active":"2024-04-15T15:16:09.358856+0000","last_peered":"2024-04-15T15:16:09.358856+0000","last_clean":"2024-04-15T15:16:09.358856+0000","last_became_active":"2024-04-15T15:16:06.699412+0000","last_became_peered":"2024-04-15T15:16:06.699412+0000","last_unstale":"2024-04-15T15:16:09.358856+0000","last_undegraded":"2024-04-15T15:16:09.358856+0000","last_fullsized":"2024-04-15T15:16:09.358856+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:02:50.425156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.708969+0000","last_change":"2024-04-15T15:16:07.708203+0000","last_active":"2024-04-15T15:16:08.708969+0000","last_peered":"2024-04-15T15:16:08.708969+0000","last_clean":"2024-04-15T15:16:08.708969+0000","last_became_active":"2024-04-15T15:16:07.707642+0000","last_became_peered":"2024-04-15T15:16:07.707642+0000","last_unstale":"2024-04-15T15:16:08.708969+0000","last_undegraded":"2024-04-15T15:16:08.708969+0000","last_fullsized":"2024-04-15T15:16:08.708969+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:35:02.358527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.974013+0000","last_change":"2024-04-15T15:16:06.702867+0000","last_active":"2024-04-15T15:16:08.974013+0000","last_peered":"2024-04-15T15:16:08.974013+0000","last_clean":"2024-04-15T15:16:08.974013+0000","last_became_active":"2024-04-15T15:16:06.701891+0000","last_became_peered":"2024-04-15T15:16:06.701891+0000","last_unstale":"2024-04-15T15:16:08.974013+0000","last_undegraded":"2024-04-15T15:16:08.974013+0000","last_fullsized":"2024-04-15T15:16:08.974013+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:51:16.039639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.710590+0000","last_change":"2024-04-15T15:16:07.718269+0000","last_active":"2024-04-15T15:16:08.710590+0000","last_peered":"2024-04-15T15:16:08.710590+0000","last_clean":"2024-04-15T15:16:08.710590+0000","last_became_active":"2024-04-15T15:16:07.717880+0000","last_became_peered":"2024-04-15T15:16:07.717880+0000","last_unstale":"2024-04-15T15:16:08.710590+0000","last_undegraded":"2024-04-15T15:16:08.710590+0000","last_fullsized":"2024-04-15T15:16:08.710590+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:00:49.742979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718797+0000","last_change":"2024-04-15T15:16:06.704693+0000","last_active":"2024-04-15T15:16:08.718797+0000","last_peered":"2024-04-15T15:16:08.718797+0000","last_clean":"2024-04-15T15:16:08.718797+0000","last_became_active":"2024-04-15T15:16:06.704416+0000","last_became_peered":"2024-04-15T15:16:06.704416+0000","last_unstale":"2024-04-15T15:16:08.718797+0000","last_undegraded":"2024-04-15T15:16:08.718797+0000","last_fullsized":"2024-04-15T15:16:08.718797+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:32:43.854999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.723438+0000","last_change":"2024-04-15T15:16:07.716298+0000","last_active":"2024-04-15T15:16:07.723438+0000","last_peered":"2024-04-15T15:16:07.723438+0000","last_clean":"2024-04-15T15:16:07.723438+0000","last_became_active":"2024-04-15T15:16:07.715865+0000","last_became_peered":"2024-04-15T15:16:07.715865+0000","last_unstale":"2024-04-15T15:16:07.723438+0000","last_undegraded":"2024-04-15T15:16:07.723438+0000","last_fullsized":"2024-04-15T15:16:07.723438+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:10:57.528229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"37'3","reported_seq":17,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718918+0000","last_change":"2024-04-15T15:16:06.702716+0000","last_active":"2024-04-15T15:16:08.718918+0000","last_peered":"2024-04-15T15:16:08.718918+0000","last_clean":"2024-04-15T15:16:08.718918+0000","last_became_active":"2024-04-15T15:16:06.702416+0000","last_became_peered":"2024-04-15T15:16:06.702416+0000","last_unstale":"2024-04-15T15:16:08.718918+0000","last_undegraded":"2024-04-15T15:16:08.718918+0000","last_fullsized":"2024-04-15T15:16:08.718918+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:38:28.840555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.359401+0000","last_change":"2024-04-15T15:16:07.711716+0000","last_active":"2024-04-15T15:16:09.359401+0000","last_peered":"2024-04-15T15:16:09.359401+0000","last_clean":"2024-04-15T15:16:09.359401+0000","last_became_active":"2024-04-15T15:16:07.711459+0000","last_became_peered":"2024-04-15T15:16:07.711459+0000","last_unstale":"2024-04-15T15:16:09.359401+0000","last_undegraded":"2024-04-15T15:16:09.359401+0000","last_fullsized":"2024-04-15T15:16:09.359401+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:32:26.516274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"37'6","reported_seq":20,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973672+0000","last_change":"2024-04-15T15:16:06.707247+0000","last_active":"2024-04-15T15:16:08.973672+0000","last_peered":"2024-04-15T15:16:08.973672+0000","last_clean":"2024-04-15T15:16:08.973672+0000","last_became_active":"2024-04-15T15:16:06.706166+0000","last_became_peered":"2024-04-15T15:16:06.706166+0000","last_unstale":"2024-04-15T15:16:08.973672+0000","last_undegraded":"2024-04-15T15:16:08.973672+0000","last_fullsized":"2024-04-15T15:16:08.973672+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:57:25.173871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.716014+0000","last_change":"2024-04-15T15:16:07.712212+0000","last_active":"2024-04-15T15:16:07.716014+0000","last_peered":"2024-04-15T15:16:07.716014+0000","last_clean":"2024-04-15T15:16:07.716014+0000","last_became_active":"2024-04-15T15:16:07.711327+0000","last_became_peered":"2024-04-15T15:16:07.711327+0000","last_unstale":"2024-04-15T15:16:07.716014+0000","last_undegraded":"2024-04-15T15:16:07.716014+0000","last_fullsized":"2024-04-15T15:16:07.716014+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:01:01.910545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973690+0000","last_change":"2024-04-15T15:16:06.704498+0000","last_active":"2024-04-15T15:16:08.973690+0000","last_peered":"2024-04-15T15:16:08.973690+0000","last_clean":"2024-04-15T15:16:08.973690+0000","last_became_active":"2024-04-15T15:16:06.704230+0000","last_became_peered":"2024-04-15T15:16:06.704230+0000","last_unstale":"2024-04-15T15:16:08.973690+0000","last_undegraded":"2024-04-15T15:16:08.973690+0000","last_fullsized":"2024-04-15T15:16:08.973690+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:32:43.854999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973429+0000","last_change":"2024-04-15T15:16:07.712780+0000","last_active":"2024-04-15T15:16:08.973429+0000","last_peered":"2024-04-15T15:16:08.973429+0000","last_clean":"2024-04-15T15:16:08.973429+0000","last_became_active":"2024-04-15T15:16:07.712534+0000","last_became_peered":"2024-04-15T15:16:07.712534+0000","last_unstale":"2024-04-15T15:16:08.973429+0000","last_undegraded":"2024-04-15T15:16:08.973429+0000","last_fullsized":"2024-04-15T15:16:08.973429+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:12:03.446783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.701926+0000","last_change":"2024-04-15T15:16:06.707538+0000","last_active":"2024-04-15T15:16:07.701926+0000","last_peered":"2024-04-15T15:16:07.701926+0000","last_clean":"2024-04-15T15:16:07.701926+0000","last_became_active":"2024-04-15T15:16:06.707264+0000","last_became_peered":"2024-04-15T15:16:06.707264+0000","last_unstale":"2024-04-15T15:16:07.701926+0000","last_undegraded":"2024-04-15T15:16:07.701926+0000","last_fullsized":"2024-04-15T15:16:07.701926+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:35:01.354434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.710310+0000","last_change":"2024-04-15T15:16:07.716245+0000","last_active":"2024-04-15T15:16:08.710310+0000","last_peered":"2024-04-15T15:16:08.710310+0000","last_clean":"2024-04-15T15:16:08.710310+0000","last_became_active":"2024-04-15T15:16:07.715993+0000","last_became_peered":"2024-04-15T15:16:07.715993+0000","last_unstale":"2024-04-15T15:16:08.710310+0000","last_undegraded":"2024-04-15T15:16:08.710310+0000","last_fullsized":"2024-04-15T15:16:08.710310+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:14:45.605613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709283+0000","last_change":"2024-04-15T15:16:06.700246+0000","last_active":"2024-04-15T15:16:08.709283+0000","last_peered":"2024-04-15T15:16:08.709283+0000","last_clean":"2024-04-15T15:16:08.709283+0000","last_became_active":"2024-04-15T15:16:06.699972+0000","last_became_peered":"2024-04-15T15:16:06.699972+0000","last_unstale":"2024-04-15T15:16:08.709283+0000","last_undegraded":"2024-04-15T15:16:08.709283+0000","last_fullsized":"2024-04-15T15:16:08.709283+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:02:50.425156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973578+0000","last_change":"2024-04-15T15:16:07.713401+0000","last_active":"2024-04-15T15:16:08.973578+0000","last_peered":"2024-04-15T15:16:08.973578+0000","last_clean":"2024-04-15T15:16:08.973578+0000","last_became_active":"2024-04-15T15:16:07.713157+0000","last_became_peered":"2024-04-15T15:16:07.713157+0000","last_unstale":"2024-04-15T15:16:08.973578+0000","last_undegraded":"2024-04-15T15:16:08.973578+0000","last_fullsized":"2024-04-15T15:16:08.973578+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:04:27.874758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"37'8","reported_seq":22,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709937+0000","last_change":"2024-04-15T15:16:06.703005+0000","last_active":"2024-04-15T15:16:08.709937+0000","last_peered":"2024-04-15T15:16:08.709937+0000","last_clean":"2024-04-15T15:16:08.709937+0000","last_became_active":"2024-04-15T15:16:06.702719+0000","last_became_peered":"2024-04-15T15:16:06.702719+0000","last_unstale":"2024-04-15T15:16:08.709937+0000","last_undegraded":"2024-04-15T15:16:08.709937+0000","last_fullsized":"2024-04-15T15:16:08.709937+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:33:09.310850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973624+0000","last_change":"2024-04-15T15:16:07.716082+0000","last_active":"2024-04-15T15:16:08.973624+0000","last_peered":"2024-04-15T15:16:08.973624+0000","last_clean":"2024-04-15T15:16:08.973624+0000","last_became_active":"2024-04-15T15:16:07.715802+0000","last_became_peered":"2024-04-15T15:16:07.715802+0000","last_unstale":"2024-04-15T15:16:08.973624+0000","last_undegraded":"2024-04-15T15:16:08.973624+0000","last_fullsized":"2024-04-15T15:16:08.973624+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:38:41.698007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718834+0000","last_change":"2024-04-15T15:16:06.699687+0000","last_active":"2024-04-15T15:16:08.718834+0000","last_peered":"2024-04-15T15:16:08.718834+0000","last_clean":"2024-04-15T15:16:08.718834+0000","last_became_active":"2024-04-15T15:16:06.699403+0000","last_became_peered":"2024-04-15T15:16:06.699403+0000","last_unstale":"2024-04-15T15:16:08.718834+0000","last_undegraded":"2024-04-15T15:16:08.718834+0000","last_fullsized":"2024-04-15T15:16:08.718834+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:14:27.402622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718805+0000","last_change":"2024-04-15T15:16:07.720614+0000","last_active":"2024-04-15T15:16:08.718805+0000","last_peered":"2024-04-15T15:16:08.718805+0000","last_clean":"2024-04-15T15:16:08.718805+0000","last_became_active":"2024-04-15T15:16:07.720327+0000","last_became_peered":"2024-04-15T15:16:07.720327+0000","last_unstale":"2024-04-15T15:16:08.718805+0000","last_undegraded":"2024-04-15T15:16:08.718805+0000","last_fullsized":"2024-04-15T15:16:08.718805+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:57:37.374559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.359381+0000","last_change":"2024-04-15T15:16:06.703119+0000","last_active":"2024-04-15T15:16:09.359381+0000","last_peered":"2024-04-15T15:16:09.359381+0000","last_clean":"2024-04-15T15:16:09.359381+0000","last_became_active":"2024-04-15T15:16:06.702423+0000","last_became_peered":"2024-04-15T15:16:06.702423+0000","last_unstale":"2024-04-15T15:16:09.359381+0000","last_undegraded":"2024-04-15T15:16:09.359381+0000","last_fullsized":"2024-04-15T15:16:09.359381+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:25:09.602767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.359209+0000","last_change":"2024-04-15T15:16:07.713465+0000","last_active":"2024-04-15T15:16:09.359209+0000","last_peered":"2024-04-15T15:16:09.359209+0000","last_clean":"2024-04-15T15:16:09.359209+0000","last_became_active":"2024-04-15T15:16:07.713154+0000","last_became_peered":"2024-04-15T15:16:07.713154+0000","last_unstale":"2024-04-15T15:16:09.359209+0000","last_undegraded":"2024-04-15T15:16:09.359209+0000","last_fullsized":"2024-04-15T15:16:09.359209+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:08:22.249305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709042+0000","last_change":"2024-04-15T15:16:06.692920+0000","last_active":"2024-04-15T15:16:08.709042+0000","last_peered":"2024-04-15T15:16:08.709042+0000","last_clean":"2024-04-15T15:16:08.709042+0000","last_became_active":"2024-04-15T15:16:06.692641+0000","last_became_peered":"2024-04-15T15:16:06.692641+0000","last_unstale":"2024-04-15T15:16:08.709042+0000","last_undegraded":"2024-04-15T15:16:08.709042+0000","last_fullsized":"2024-04-15T15:16:08.709042+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:38:20.537537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709058+0000","last_change":"2024-04-15T15:16:07.723443+0000","last_active":"2024-04-15T15:16:08.709058+0000","last_peered":"2024-04-15T15:16:08.709058+0000","last_clean":"2024-04-15T15:16:08.709058+0000","last_became_active":"2024-04-15T15:16:07.723186+0000","last_became_peered":"2024-04-15T15:16:07.723186+0000","last_unstale":"2024-04-15T15:16:08.709058+0000","last_undegraded":"2024-04-15T15:16:08.709058+0000","last_fullsized":"2024-04-15T15:16:08.709058+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:25:10.606860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.702351+0000","last_change":"2024-04-15T15:16:06.707391+0000","last_active":"2024-04-15T15:16:07.702351+0000","last_peered":"2024-04-15T15:16:07.702351+0000","last_clean":"2024-04-15T15:16:07.702351+0000","last_became_active":"2024-04-15T15:16:06.707125+0000","last_became_peered":"2024-04-15T15:16:06.707125+0000","last_unstale":"2024-04-15T15:16:07.702351+0000","last_undegraded":"2024-04-15T15:16:07.702351+0000","last_fullsized":"2024-04-15T15:16:07.702351+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:51:03.225693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.710592+0000","last_change":"2024-04-15T15:16:07.719157+0000","last_active":"2024-04-15T15:16:08.710592+0000","last_peered":"2024-04-15T15:16:08.710592+0000","last_clean":"2024-04-15T15:16:08.710592+0000","last_became_active":"2024-04-15T15:16:07.718892+0000","last_became_peered":"2024-04-15T15:16:07.718892+0000","last_unstale":"2024-04-15T15:16:08.710592+0000","last_undegraded":"2024-04-15T15:16:08.710592+0000","last_fullsized":"2024-04-15T15:16:08.710592+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:30:18.481379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.708740+0000","last_change":"2024-04-15T15:16:06.706044+0000","last_active":"2024-04-15T15:16:08.708740+0000","last_peered":"2024-04-15T15:16:08.708740+0000","last_clean":"2024-04-15T15:16:08.708740+0000","last_became_active":"2024-04-15T15:16:06.705532+0000","last_became_peered":"2024-04-15T15:16:06.705532+0000","last_unstale":"2024-04-15T15:16:08.708740+0000","last_undegraded":"2024-04-15T15:16:08.708740+0000","last_fullsized":"2024-04-15T15:16:08.708740+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:40:09.944851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.720383+0000","last_change":"2024-04-15T15:16:07.716379+0000","last_active":"2024-04-15T15:16:07.720383+0000","last_peered":"2024-04-15T15:16:07.720383+0000","last_clean":"2024-04-15T15:16:07.720383+0000","last_became_active":"2024-04-15T15:16:07.715437+0000","last_became_peered":"2024-04-15T15:16:07.715437+0000","last_unstale":"2024-04-15T15:16:07.720383+0000","last_undegraded":"2024-04-15T15:16:07.720383+0000","last_fullsized":"2024-04-15T15:16:07.720383+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:32:44.859092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.708758+0000","last_change":"2024-04-15T15:16:06.698041+0000","last_active":"2024-04-15T15:16:08.708758+0000","last_peered":"2024-04-15T15:16:08.708758+0000","last_clean":"2024-04-15T15:16:08.708758+0000","last_became_active":"2024-04-15T15:16:06.697754+0000","last_became_peered":"2024-04-15T15:16:06.697754+0000","last_unstale":"2024-04-15T15:16:08.708758+0000","last_undegraded":"2024-04-15T15:16:08.708758+0000","last_fullsized":"2024-04-15T15:16:08.708758+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:10:07.758669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718456+0000","last_change":"2024-04-15T15:16:07.701259+0000","last_active":"2024-04-15T15:16:08.718456+0000","last_peered":"2024-04-15T15:16:08.718456+0000","last_clean":"2024-04-15T15:16:08.718456+0000","last_became_active":"2024-04-15T15:16:07.700984+0000","last_became_peered":"2024-04-15T15:16:07.700984+0000","last_unstale":"2024-04-15T15:16:08.718456+0000","last_undegraded":"2024-04-15T15:16:08.718456+0000","last_fullsized":"2024-04-15T15:16:08.718456+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:30:18.481379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709831+0000","last_change":"2024-04-15T15:16:06.699270+0000","last_active":"2024-04-15T15:16:08.709831+0000","last_peered":"2024-04-15T15:16:08.709831+0000","last_clean":"2024-04-15T15:16:08.709831+0000","last_became_active":"2024-04-15T15:16:06.698992+0000","last_became_peered":"2024-04-15T15:16:06.698992+0000","last_unstale":"2024-04-15T15:16:08.709831+0000","last_undegraded":"2024-04-15T15:16:08.709831+0000","last_fullsized":"2024-04-15T15:16:08.709831+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:18:33.749347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973763+0000","last_change":"2024-04-15T15:16:07.714324+0000","last_active":"2024-04-15T15:16:08.973763+0000","last_peered":"2024-04-15T15:16:08.973763+0000","last_clean":"2024-04-15T15:16:08.973763+0000","last_became_active":"2024-04-15T15:16:07.714069+0000","last_became_peered":"2024-04-15T15:16:07.714069+0000","last_unstale":"2024-04-15T15:16:08.973763+0000","last_undegraded":"2024-04-15T15:16:08.973763+0000","last_fullsized":"2024-04-15T15:16:08.973763+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:57:37.374559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.708615+0000","last_change":"2024-04-15T15:16:06.704254+0000","last_active":"2024-04-15T15:16:08.708615+0000","last_peered":"2024-04-15T15:16:08.708615+0000","last_clean":"2024-04-15T15:16:08.708615+0000","last_became_active":"2024-04-15T15:16:06.703962+0000","last_became_peered":"2024-04-15T15:16:06.703962+0000","last_unstale":"2024-04-15T15:16:08.708615+0000","last_undegraded":"2024-04-15T15:16:08.708615+0000","last_fullsized":"2024-04-15T15:16:08.708615+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:34:42.463356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.708978+0000","last_change":"2024-04-15T15:16:07.707273+0000","last_active":"2024-04-15T15:16:07.708978+0000","last_peered":"2024-04-15T15:16:07.708978+0000","last_clean":"2024-04-15T15:16:07.708978+0000","last_became_active":"2024-04-15T15:16:07.707005+0000","last_became_peered":"2024-04-15T15:16:07.707005+0000","last_unstale":"2024-04-15T15:16:07.708978+0000","last_undegraded":"2024-04-15T15:16:07.708978+0000","last_fullsized":"2024-04-15T15:16:07.708978+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:18:34.753440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718915+0000","last_change":"2024-04-15T15:16:06.702827+0000","last_active":"2024-04-15T15:16:08.718915+0000","last_peered":"2024-04-15T15:16:08.718915+0000","last_clean":"2024-04-15T15:16:08.718915+0000","last_became_active":"2024-04-15T15:16:06.702558+0000","last_became_peered":"2024-04-15T15:16:06.702558+0000","last_unstale":"2024-04-15T15:16:08.718915+0000","last_undegraded":"2024-04-15T15:16:08.718915+0000","last_fullsized":"2024-04-15T15:16:08.718915+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:10:56.524136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.710593+0000","last_change":"2024-04-15T15:16:07.717216+0000","last_active":"2024-04-15T15:16:08.710593+0000","last_peered":"2024-04-15T15:16:08.710593+0000","last_clean":"2024-04-15T15:16:08.710593+0000","last_became_active":"2024-04-15T15:16:07.716939+0000","last_became_peered":"2024-04-15T15:16:07.716939+0000","last_unstale":"2024-04-15T15:16:08.710593+0000","last_undegraded":"2024-04-15T15:16:08.710593+0000","last_fullsized":"2024-04-15T15:16:08.710593+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:12:03.446783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"38'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.717703+0000","last_change":"2024-04-15T15:16:06.702772+0000","last_active":"2024-04-15T15:16:08.717703+0000","last_peered":"2024-04-15T15:16:08.717703+0000","last_clean":"2024-04-15T15:16:08.717703+0000","last_became_active":"2024-04-15T15:16:06.702503+0000","last_became_peered":"2024-04-15T15:16:06.702503+0000","last_unstale":"2024-04-15T15:16:08.717703+0000","last_undegraded":"2024-04-15T15:16:08.717703+0000","last_fullsized":"2024-04-15T15:16:08.717703+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:19:47.422670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973693+0000","last_change":"2024-04-15T15:16:07.716142+0000","last_active":"2024-04-15T15:16:08.973693+0000","last_peered":"2024-04-15T15:16:08.973693+0000","last_clean":"2024-04-15T15:16:08.973693+0000","last_became_active":"2024-04-15T15:16:07.715562+0000","last_became_peered":"2024-04-15T15:16:07.715562+0000","last_unstale":"2024-04-15T15:16:08.973693+0000","last_undegraded":"2024-04-15T15:16:08.973693+0000","last_fullsized":"2024-04-15T15:16:08.973693+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:51:12.622012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.701542+0000","last_change":"2024-04-15T15:16:06.706530+0000","last_active":"2024-04-15T15:16:07.701542+0000","last_peered":"2024-04-15T15:16:07.701542+0000","last_clean":"2024-04-15T15:16:07.701542+0000","last_became_active":"2024-04-15T15:16:06.706214+0000","last_became_peered":"2024-04-15T15:16:06.706214+0000","last_unstale":"2024-04-15T15:16:07.701542+0000","last_undegraded":"2024-04-15T15:16:07.701542+0000","last_fullsized":"2024-04-15T15:16:07.701542+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:25:09.602767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.713028+0000","last_change":"2024-04-15T15:16:07.710663+0000","last_active":"2024-04-15T15:16:07.713028+0000","last_peered":"2024-04-15T15:16:07.713028+0000","last_clean":"2024-04-15T15:16:07.713028+0000","last_became_active":"2024-04-15T15:16:07.710388+0000","last_became_peered":"2024-04-15T15:16:07.710388+0000","last_unstale":"2024-04-15T15:16:07.713028+0000","last_undegraded":"2024-04-15T15:16:07.713028+0000","last_fullsized":"2024-04-15T15:16:07.713028+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:18:34.753440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.701317+0000","last_change":"2024-04-15T15:16:06.706722+0000","last_active":"2024-04-15T15:16:07.701317+0000","last_peered":"2024-04-15T15:16:07.701317+0000","last_clean":"2024-04-15T15:16:07.701317+0000","last_became_active":"2024-04-15T15:16:06.706041+0000","last_became_peered":"2024-04-15T15:16:06.706041+0000","last_unstale":"2024-04-15T15:16:07.701317+0000","last_undegraded":"2024-04-15T15:16:07.701317+0000","last_fullsized":"2024-04-15T15:16:07.701317+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:35:01.354434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-04-15T15:16:07.718765+0000","last_change":"2024-04-15T15:16:07.709909+0000","last_active":"2024-04-15T15:16:07.718765+0000","last_peered":"2024-04-15T15:16:07.718765+0000","last_clean":"2024-04-15T15:16:07.718765+0000","last_became_active":"2024-04-15T15:16:07.709672+0000","last_became_peered":"2024-04-15T15:16:07.709672+0000","last_unstale":"2024-04-15T15:16:07.718765+0000","last_undegraded":"2024-04-15T15:16:07.718765+0000","last_fullsized":"2024-04-15T15:16:07.718765+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:38:29.844648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709503+0000","last_change":"2024-04-15T15:16:06.704950+0000","last_active":"2024-04-15T15:16:08.709503+0000","last_peered":"2024-04-15T15:16:08.709503+0000","last_clean":"2024-04-15T15:16:08.709503+0000","last_became_active":"2024-04-15T15:16:06.704688+0000","last_became_peered":"2024-04-15T15:16:06.704688+0000","last_unstale":"2024-04-15T15:16:08.709503+0000","last_undegraded":"2024-04-15T15:16:08.709503+0000","last_fullsized":"2024-04-15T15:16:08.709503+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:25:09.602767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.358983+0000","last_change":"2024-04-15T15:16:07.709428+0000","last_active":"2024-04-15T15:16:09.358983+0000","last_peered":"2024-04-15T15:16:09.358983+0000","last_clean":"2024-04-15T15:16:09.358983+0000","last_became_active":"2024-04-15T15:16:07.709164+0000","last_became_peered":"2024-04-15T15:16:07.709164+0000","last_unstale":"2024-04-15T15:16:09.358983+0000","last_undegraded":"2024-04-15T15:16:09.358983+0000","last_fullsized":"2024-04-15T15:16:09.358983+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:57:26.177964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.358898+0000","last_change":"2024-04-15T15:16:06.697149+0000","last_active":"2024-04-15T15:16:09.358898+0000","last_peered":"2024-04-15T15:16:09.358898+0000","last_clean":"2024-04-15T15:16:09.358898+0000","last_became_active":"2024-04-15T15:16:06.696879+0000","last_became_peered":"2024-04-15T15:16:06.696879+0000","last_unstale":"2024-04-15T15:16:09.358898+0000","last_undegraded":"2024-04-15T15:16:09.358898+0000","last_fullsized":"2024-04-15T15:16:09.358898+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:35:01.354434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.710256+0000","last_change":"2024-04-15T15:16:07.717293+0000","last_active":"2024-04-15T15:16:08.710256+0000","last_peered":"2024-04-15T15:16:08.710256+0000","last_clean":"2024-04-15T15:16:08.710256+0000","last_became_active":"2024-04-15T15:16:07.716988+0000","last_became_peered":"2024-04-15T15:16:07.716988+0000","last_unstale":"2024-04-15T15:16:08.710256+0000","last_undegraded":"2024-04-15T15:16:08.710256+0000","last_fullsized":"2024-04-15T15:16:08.710256+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:09:06.021853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973388+0000","last_change":"2024-04-15T15:16:06.702502+0000","last_active":"2024-04-15T15:16:08.973388+0000","last_peered":"2024-04-15T15:16:08.973388+0000","last_clean":"2024-04-15T15:16:08.973388+0000","last_became_active":"2024-04-15T15:16:06.701301+0000","last_became_peered":"2024-04-15T15:16:06.701301+0000","last_unstale":"2024-04-15T15:16:08.973388+0000","last_undegraded":"2024-04-15T15:16:08.973388+0000","last_fullsized":"2024-04-15T15:16:08.973388+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:03:11.873554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718815+0000","last_change":"2024-04-15T15:16:07.712592+0000","last_active":"2024-04-15T15:16:08.718815+0000","last_peered":"2024-04-15T15:16:08.718815+0000","last_clean":"2024-04-15T15:16:08.718815+0000","last_became_active":"2024-04-15T15:16:07.712213+0000","last_became_peered":"2024-04-15T15:16:07.712213+0000","last_unstale":"2024-04-15T15:16:08.718815+0000","last_undegraded":"2024-04-15T15:16:08.718815+0000","last_fullsized":"2024-04-15T15:16:08.718815+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:51:12.622012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709805+0000","last_change":"2024-04-15T15:16:06.700821+0000","last_active":"2024-04-15T15:16:08.709805+0000","last_peered":"2024-04-15T15:16:08.709805+0000","last_clean":"2024-04-15T15:16:08.709805+0000","last_became_active":"2024-04-15T15:16:06.700436+0000","last_became_peered":"2024-04-15T15:16:06.700436+0000","last_unstale":"2024-04-15T15:16:08.709805+0000","last_undegraded":"2024-04-15T15:16:08.709805+0000","last_fullsized":"2024-04-15T15:16:08.709805+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:14:27.402622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":202,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.710011+0000","last_change":"2024-04-15T15:14:40.494279+0000","last_active":"2024-04-15T15:16:08.710011+0000","last_peered":"2024-04-15T15:16:08.710011+0000","last_clean":"2024-04-15T15:16:08.710011+0000","last_became_active":"2024-04-15T15:14:40.493404+0000","last_became_peered":"2024-04-15T15:14:40.493404+0000","last_unstale":"2024-04-15T15:16:08.710011+0000","last_undegraded":"2024-04-15T15:16:08.710011+0000","last_fullsized":"2024-04-15T15:16:08.710011+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:39.418508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:39.418508+0000","last_clean_scrub_stamp":"2024-04-15T15:14:39.418508+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:01:51.789246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28052,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343788,"statfs":{"total":96636764160,"available":96608038912,"internally_reserved":0,"allocated":1048576,"data_stored":782086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574870,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":430080,"data_stored":183926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27460,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344380,"statfs":{"total":96636764160,"available":96608645120,"internally_reserved":0,"allocated":446464,"data_stored":190148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27444,"kb_used_data":424,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344396,"statfs":{"total":96636764160,"available":96608661504,"internally_reserved":0,"allocated":434176,"data_stored":183948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":442368,"data_stored":184990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":430080,"data_stored":183414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27440,"kb_used_data":420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":430080,"data_stored":183434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343816,"statfs":{"total":96636764160,"available":96608067584,"internally_reserved":0,"allocated":1024000,"data_stored":773746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:16:15.111 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:16:15.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.223+0000 7f1a15a86640 1 Processor -- start 2024-04-15T15:16:15.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.223+0000 7f1a15a86640 1 -- start start 2024-04-15T15:16:15.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.224+0000 7f1a15a86640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1a100c9570 0x7f1a10162b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:15.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.224+0000 7f1a15a86640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a100ccd40 con 0x7f1a10161fb0 2024-04-15T15:16:15.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.224+0000 7f1a15a86640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a100cd020 con 0x7f1a100d12f0 2024-04-15T15:16:15.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.224+0000 7f1a15a86640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f1a100ccaf0 con 0x7f1a100c9570 2024-04-15T15:16:15.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.224+0000 7f1a0effd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f1a10161fb0 0x7f1a10162450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:34208/0 (socket says 172.21.15.72:34208) 2024-04-15T15:16:15.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.224+0000 7f1a0effd640 1 -- 172.21.15.72:0/2433619995 learned_addr learned my addr 172.21.15.72:0/2433619995 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:15.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.224+0000 7f1a0ffff640 1 --2- 172.21.15.72:0/2433619995 >> v2:172.21.15.175:3300/0 conn(0x7f1a100c9570 0x7f1a10162b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:15.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.224+0000 7f1a0e7fc640 1 -- 172.21.15.72:0/2433619995 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1588230503 0 0) 0x7f1a100ccd40 con 0x7f1a10161fb0 2024-04-15T15:16:15.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.224+0000 7f1a0e7fc640 1 -- 172.21.15.72:0/2433619995 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f19f8003520 con 0x7f1a10161fb0 2024-04-15T15:16:15.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.225+0000 7f1a0e7fc640 1 -- 172.21.15.72:0/2433619995 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 890773023 0 0) 0x7f1a100cd020 con 0x7f1a100d12f0 2024-04-15T15:16:15.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.225+0000 7f1a0e7fc640 1 -- 172.21.15.72:0/2433619995 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a100ccd40 con 0x7f1a100d12f0 2024-04-15T15:16:15.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.225+0000 7f1a0e7fc640 1 -- 172.21.15.72:0/2433619995 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 41477511 0 0) 0x7f19f8003520 con 0x7f1a10161fb0 2024-04-15T15:16:15.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.225+0000 7f1a0e7fc640 1 -- 172.21.15.72:0/2433619995 >> v2:172.21.15.175:3300/0 conn(0x7f1a100c9570 msgr2=0x7f1a10162b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:15.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.225+0000 7f1a0e7fc640 1 --2- 172.21.15.72:0/2433619995 >> v2:172.21.15.175:3300/0 conn(0x7f1a100c9570 0x7f1a10162b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.225+0000 7f1a0e7fc640 1 -- 172.21.15.72:0/2433619995 >> v1:172.21.15.175:6789/0 conn(0x7f1a100d12f0 legacy=0x7f1a101618a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:15.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.225+0000 7f1a0e7fc640 1 -- 172.21.15.72:0/2433619995 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a100cd300 con 0x7f1a10161fb0 2024-04-15T15:16:15.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.225+0000 7f1a0e7fc640 1 -- 172.21.15.72:0/2433619995 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f1a04001670 con 0x7f1a10161fb0 2024-04-15T15:16:15.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.225+0000 7f1a0e7fc640 1 -- 172.21.15.72:0/2433619995 >> v1:172.21.15.72:6789/0 conn(0x7f1a10161fb0 legacy=0x7f1a10162450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:15.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.225+0000 7f1a0e7fc640 1 --2- 172.21.15.72:0/2433619995 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f19f8003b40 0x7f19f8003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:15.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.225+0000 7f1a0e7fc640 1 --2- 172.21.15.72:0/2433619995 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19f8004800 0x7f19f8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:15.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.225+0000 7f1a0e7fc640 1 --2- 172.21.15.72:0/2433619995 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19f80051c0 0x7f19f800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:15.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.225+0000 7f1a0e7fc640 1 -- 172.21.15.72:0/2433619995 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f1a100ccaf0 con 0x7f19f8004800 2024-04-15T15:16:15.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.225+0000 7f1a0e7fc640 1 -- 172.21.15.72:0/2433619995 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f19f800c040 con 0x7f19f80051c0 2024-04-15T15:16:15.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.225+0000 7f1a0e7fc640 1 -- 172.21.15.72:0/2433619995 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f19f800c240 con 0x7f19f8003b40 2024-04-15T15:16:15.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.225+0000 7f1a0effd640 1 --2- 172.21.15.72:0/2433619995 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19f8004800 0x7f19f8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:15.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.225+0000 7f1a0ffff640 1 --2- 172.21.15.72:0/2433619995 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19f80051c0 0x7f19f800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:15.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.225+0000 7f1a0f7fe640 1 --2- 172.21.15.72:0/2433619995 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f19f8003b40 0x7f19f8003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:15.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.225+0000 7f1a0effd640 1 -- 172.21.15.72:0/2433619995 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f19f8003b40 msgr2=0x7f19f8003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:15.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.225+0000 7f1a0effd640 1 --2- 172.21.15.72:0/2433619995 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f19f8003b40 0x7f19f8003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.225+0000 7f1a0effd640 1 -- 172.21.15.72:0/2433619995 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19f80051c0 msgr2=0x7f19f800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:15.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.225+0000 7f1a0effd640 1 --2- 172.21.15.72:0/2433619995 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f19f80051c0 0x7f19f800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.225+0000 7f1a0effd640 1 -- 172.21.15.72:0/2433619995 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19f8003520 con 0x7f19f8004800 2024-04-15T15:16:15.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.225+0000 7f1a0ffff640 1 -- 172.21.15.72:0/2433619995 reap_dead start 2024-04-15T15:16:15.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.226+0000 7f1a0effd640 1 --2- 172.21.15.72:0/2433619995 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19f8004800 0x7f19f8004c80 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f1a0400c160 tx=0x7f19f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64282dfa832628e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:15.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.226+0000 7f1a0e7fc640 1 -- 172.21.15.72:0/2433619995 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a04007cc0 con 0x7f19f8004800 2024-04-15T15:16:15.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.226+0000 7f1a0e7fc640 1 -- 172.21.15.72:0/2433619995 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a04030070 con 0x7f19f8004800 2024-04-15T15:16:15.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.226+0000 7f1a0e7fc640 1 -- 172.21.15.72:0/2433619995 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a04005d70 con 0x7f19f8004800 2024-04-15T15:16:15.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.227+0000 7f1a15a86640 1 -- 172.21.15.72:0/2433619995 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19f8004800 msgr2=0x7f19f8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:15.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.227+0000 7f1a15a86640 1 --2- 172.21.15.72:0/2433619995 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19f8004800 0x7f19f8004c80 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f1a0400c160 tx=0x7f19f8005830 comp rx=0 tx=0).stop 2024-04-15T15:16:15.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.227+0000 7f1a15a86640 1 -- 172.21.15.72:0/2433619995 shutdown_connections 2024-04-15T15:16:15.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.227+0000 7f1a15a86640 1 --2- 172.21.15.72:0/2433619995 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f19f8004800 0x7f19f8004c80 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.227+0000 7f1a15a86640 1 -- 172.21.15.72:0/2433619995 >> 172.21.15.72:0/2433619995 conn(0x7f1a100b3f90 msgr2=0x7f1a100b43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:15.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.227+0000 7f1a15a86640 1 -- 172.21.15.72:0/2433619995 shutdown_connections 2024-04-15T15:16:15.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.227+0000 7f1a15a86640 1 -- 172.21.15.72:0/2433619995 wait complete. 2024-04-15T15:16:15.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.228+0000 7f1a15a86640 1 Processor -- start 2024-04-15T15:16:15.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.228+0000 7f1a15a86640 1 -- start start 2024-04-15T15:16:15.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.229+0000 7f1a15a86640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f19f8004800 0x7f1a101471b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:15.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.229+0000 7f1a15a86640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a10147a50 0x7f1a1015ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:15.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.229+0000 7f1a15a86640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1a1015b1c0 0x7f1a1015ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:15.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.229+0000 7f1a15a86640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f1a100d2210 con 0x7f1a10147a50 2024-04-15T15:16:15.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.229+0000 7f1a15a86640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f1a100d2090 con 0x7f1a1015b1c0 2024-04-15T15:16:15.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.229+0000 7f1a15a86640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f1a100b68e0 con 0x7f19f8004800 2024-04-15T15:16:15.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.229+0000 7f1a0effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a10147a50 0x7f1a1015ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:15.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.229+0000 7f1a0effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a10147a50 0x7f1a1015ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56186/0 (socket says 172.21.15.72:56186) 2024-04-15T15:16:15.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.229+0000 7f1a0effd640 1 -- 172.21.15.72:0/1689484366 learned_addr learned my addr 172.21.15.72:0/1689484366 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:15.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.229+0000 7f1a0ffff640 1 --2- 172.21.15.72:0/1689484366 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1a1015b1c0 0x7f1a1015ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:15.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.229+0000 7f1a0f7fe640 1 --2- 172.21.15.72:0/1689484366 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f19f8004800 0x7f1a101471b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:15.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.229+0000 7f1a0effd640 1 -- 172.21.15.72:0/1689484366 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f19f8004800 msgr2=0x7f1a101471b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:15.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.229+0000 7f1a0effd640 1 --2- 172.21.15.72:0/1689484366 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f19f8004800 0x7f1a101471b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.229+0000 7f1a0effd640 1 -- 172.21.15.72:0/1689484366 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1a1015b1c0 msgr2=0x7f1a1015ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:15.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.229+0000 7f1a0effd640 1 --2- 172.21.15.72:0/1689484366 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1a1015b1c0 0x7f1a1015ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.229+0000 7f1a0effd640 1 -- 172.21.15.72:0/1689484366 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a1015dbf0 con 0x7f1a10147a50 2024-04-15T15:16:15.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.229+0000 7f1a0ffff640 1 --2- 172.21.15.72:0/1689484366 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1a1015b1c0 0x7f1a1015ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:15.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.229+0000 7f1a0effd640 1 --2- 172.21.15.72:0/1689484366 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a10147a50 0x7f1a1015ac80 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f1a0400c160 tx=0x7f1a040019c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:15.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.230+0000 7f19f7fff640 1 -- 172.21.15.72:0/1689484366 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a04001c00 con 0x7f1a10147a50 2024-04-15T15:16:15.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.230+0000 7f19f7fff640 1 -- 172.21.15.72:0/1689484366 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a04007480 con 0x7f1a10147a50 2024-04-15T15:16:15.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.230+0000 7f1a15a86640 1 -- 172.21.15.72:0/1689484366 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a1015ded0 con 0x7f1a10147a50 2024-04-15T15:16:15.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.230+0000 7f19f7fff640 1 -- 172.21.15.72:0/1689484366 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0402c740 con 0x7f1a10147a50 2024-04-15T15:16:15.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.230+0000 7f1a15a86640 1 -- 172.21.15.72:0/1689484366 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a100c7300 con 0x7f1a10147a50 2024-04-15T15:16:15.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.231+0000 7f1a15a86640 1 -- 172.21.15.72:0/1689484366 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a100b34a0 con 0x7f1a10147a50 2024-04-15T15:16:15.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.231+0000 7f19f7fff640 1 -- 172.21.15.72:0/1689484366 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1a04048020 con 0x7f1a10147a50 2024-04-15T15:16:15.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.232+0000 7f19f7fff640 1 --2- 172.21.15.72:0/1689484366 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f19fc06c8f0 0x7f19fc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:15.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.232+0000 7f19f7fff640 1 -- 172.21.15.72:0/1689484366 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f1a040ad0a0 con 0x7f1a10147a50 2024-04-15T15:16:15.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.232+0000 7f1a0f7fe640 1 --2- 172.21.15.72:0/1689484366 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f19fc06c8f0 0x7f19fc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:15.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.232+0000 7f1a0f7fe640 1 --2- 172.21.15.72:0/1689484366 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f19fc06c8f0 0x7f19fc06edb0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1a100c8ed0 tx=0x7f1a00009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:15.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.236+0000 7f19f7fff640 1 -- 172.21.15.72:0/1689484366 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1a100b34a0 con 0x7f1a10147a50 2024-04-15T15:16:15.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.410+0000 7f1a15a86640 1 -- 172.21.15.72:0/1689484366 --> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1a100689e0 con 0x7f19fc06c8f0 2024-04-15T15:16:15.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.415+0000 7f19f7fff640 1 -- 172.21.15.72:0/1689484366 <== mgr.4101 v2:172.21.15.72:6832/3503807805 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251847 (secure 0 0 0) 0x7f1a100689e0 con 0x7f19fc06c8f0 2024-04-15T15:16:15.420 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:16:15.421 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-04-15T15:16:15.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.418+0000 7f1a15a86640 1 -- 172.21.15.72:0/1689484366 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f19fc06c8f0 msgr2=0x7f19fc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:15.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.418+0000 7f1a15a86640 1 --2- 172.21.15.72:0/1689484366 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f19fc06c8f0 0x7f19fc06edb0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1a100c8ed0 tx=0x7f1a00009040 comp rx=0 tx=0).stop 2024-04-15T15:16:15.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.418+0000 7f1a15a86640 1 -- 172.21.15.72:0/1689484366 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a10147a50 msgr2=0x7f1a1015ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:15.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.418+0000 7f1a15a86640 1 --2- 172.21.15.72:0/1689484366 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a10147a50 0x7f1a1015ac80 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f1a0400c160 tx=0x7f1a040019c0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.418+0000 7f1a15a86640 1 -- 172.21.15.72:0/1689484366 shutdown_connections 2024-04-15T15:16:15.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.418+0000 7f1a15a86640 1 --2- 172.21.15.72:0/1689484366 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f19fc06c8f0 0x7f19fc06edb0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.418+0000 7f1a15a86640 1 --2- 172.21.15.72:0/1689484366 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1a1015b1c0 0x7f1a1015ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.418+0000 7f1a15a86640 1 --2- 172.21.15.72:0/1689484366 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1a10147a50 0x7f1a1015ac80 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.418+0000 7f1a15a86640 1 --2- 172.21.15.72:0/1689484366 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f19f8004800 0x7f1a101471b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.418+0000 7f1a15a86640 1 -- 172.21.15.72:0/1689484366 >> 172.21.15.72:0/1689484366 conn(0x7f1a100b3f90 msgr2=0x7f1a100b36a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:15.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.418+0000 7f1a15a86640 1 -- 172.21.15.72:0/1689484366 shutdown_connections 2024-04-15T15:16:15.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.418+0000 7f1a15a86640 1 -- 172.21.15.72:0/1689484366 wait complete. 2024-04-15T15:16:15.432 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2024-04-15T15:16:15.360413+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220836,"kb_used_data":4648,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753884,"statfs":{"total":773094113280,"available":772867977216,"internally_reserved":0,"allocated":4759552,"data_stored":2705844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001772"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973935+0000","last_change":"2024-04-15T15:16:07.714016+0000","last_active":"2024-04-15T15:16:08.973935+0000","last_peered":"2024-04-15T15:16:08.973935+0000","last_clean":"2024-04-15T15:16:08.973935+0000","last_became_active":"2024-04-15T15:16:07.713760+0000","last_became_peered":"2024-04-15T15:16:07.713760+0000","last_unstale":"2024-04-15T15:16:08.973935+0000","last_undegraded":"2024-04-15T15:16:08.973935+0000","last_fullsized":"2024-04-15T15:16:08.973935+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:26:10.169823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973663+0000","last_change":"2024-04-15T15:16:07.716357+0000","last_active":"2024-04-15T15:16:08.973663+0000","last_peered":"2024-04-15T15:16:08.973663+0000","last_clean":"2024-04-15T15:16:08.973663+0000","last_became_active":"2024-04-15T15:16:07.715891+0000","last_became_peered":"2024-04-15T15:16:07.715891+0000","last_unstale":"2024-04-15T15:16:08.973663+0000","last_undegraded":"2024-04-15T15:16:08.973663+0000","last_fullsized":"2024-04-15T15:16:08.973663+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:32:24.986373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.462099+0000","last_change":"2024-04-15T15:16:07.706936+0000","last_active":"2024-04-15T15:16:09.462099+0000","last_peered":"2024-04-15T15:16:09.462099+0000","last_clean":"2024-04-15T15:16:09.462099+0000","last_became_active":"2024-04-15T15:16:07.706670+0000","last_became_peered":"2024-04-15T15:16:07.706670+0000","last_unstale":"2024-04-15T15:16:09.462099+0000","last_undegraded":"2024-04-15T15:16:09.462099+0000","last_fullsized":"2024-04-15T15:16:09.462099+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:39:05.235149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.708922+0000","last_change":"2024-04-15T15:16:07.708400+0000","last_active":"2024-04-15T15:16:08.708922+0000","last_peered":"2024-04-15T15:16:08.708922+0000","last_clean":"2024-04-15T15:16:08.708922+0000","last_became_active":"2024-04-15T15:16:07.707958+0000","last_became_peered":"2024-04-15T15:16:07.707958+0000","last_unstale":"2024-04-15T15:16:08.708922+0000","last_undegraded":"2024-04-15T15:16:08.708922+0000","last_fullsized":"2024-04-15T15:16:08.708922+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:18:13.186969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.708673+0000","last_change":"2024-04-15T15:16:07.712681+0000","last_active":"2024-04-15T15:16:08.708673+0000","last_peered":"2024-04-15T15:16:08.708673+0000","last_clean":"2024-04-15T15:16:08.708673+0000","last_became_active":"2024-04-15T15:16:07.712430+0000","last_became_peered":"2024-04-15T15:16:07.712430+0000","last_unstale":"2024-04-15T15:16:08.708673+0000","last_undegraded":"2024-04-15T15:16:08.708673+0000","last_fullsized":"2024-04-15T15:16:08.708673+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:19:44.813755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.461400+0000","last_change":"2024-04-15T15:16:07.709386+0000","last_active":"2024-04-15T15:16:09.461400+0000","last_peered":"2024-04-15T15:16:09.461400+0000","last_clean":"2024-04-15T15:16:09.461400+0000","last_became_active":"2024-04-15T15:16:07.709113+0000","last_became_peered":"2024-04-15T15:16:07.709113+0000","last_unstale":"2024-04-15T15:16:09.461400+0000","last_undegraded":"2024-04-15T15:16:09.461400+0000","last_fullsized":"2024-04-15T15:16:09.461400+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:51:17.043733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718270+0000","last_change":"2024-04-15T15:16:07.711815+0000","last_active":"2024-04-15T15:16:08.718270+0000","last_peered":"2024-04-15T15:16:08.718270+0000","last_clean":"2024-04-15T15:16:08.718270+0000","last_became_active":"2024-04-15T15:16:07.711561+0000","last_became_peered":"2024-04-15T15:16:07.711561+0000","last_unstale":"2024-04-15T15:16:08.718270+0000","last_undegraded":"2024-04-15T15:16:08.718270+0000","last_fullsized":"2024-04-15T15:16:08.718270+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:09:09.330985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973403+0000","last_change":"2024-04-15T15:16:07.711991+0000","last_active":"2024-04-15T15:16:08.973403+0000","last_peered":"2024-04-15T15:16:08.973403+0000","last_clean":"2024-04-15T15:16:08.973403+0000","last_became_active":"2024-04-15T15:16:07.711753+0000","last_became_peered":"2024-04-15T15:16:07.711753+0000","last_unstale":"2024-04-15T15:16:08.973403+0000","last_undegraded":"2024-04-15T15:16:08.973403+0000","last_fullsized":"2024-04-15T15:16:08.973403+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:00:49.742979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.461754+0000","last_change":"2024-04-15T15:16:07.711991+0000","last_active":"2024-04-15T15:16:09.461754+0000","last_peered":"2024-04-15T15:16:09.461754+0000","last_clean":"2024-04-15T15:16:09.461754+0000","last_became_active":"2024-04-15T15:16:07.711634+0000","last_became_peered":"2024-04-15T15:16:07.711634+0000","last_unstale":"2024-04-15T15:16:09.461754+0000","last_undegraded":"2024-04-15T15:16:09.461754+0000","last_fullsized":"2024-04-15T15:16:09.461754+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:08:22.249305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718646+0000","last_change":"2024-04-15T15:16:07.714191+0000","last_active":"2024-04-15T15:16:08.718646+0000","last_peered":"2024-04-15T15:16:08.718646+0000","last_clean":"2024-04-15T15:16:08.718646+0000","last_became_active":"2024-04-15T15:16:07.713121+0000","last_became_peered":"2024-04-15T15:16:07.713121+0000","last_unstale":"2024-04-15T15:16:08.718646+0000","last_undegraded":"2024-04-15T15:16:08.718646+0000","last_fullsized":"2024-04-15T15:16:08.718646+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:32:24.986373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718831+0000","last_change":"2024-04-15T15:16:07.714129+0000","last_active":"2024-04-15T15:16:08.718831+0000","last_peered":"2024-04-15T15:16:08.718831+0000","last_clean":"2024-04-15T15:16:08.718831+0000","last_became_active":"2024-04-15T15:16:07.713746+0000","last_became_peered":"2024-04-15T15:16:07.713746+0000","last_unstale":"2024-04-15T15:16:08.718831+0000","last_undegraded":"2024-04-15T15:16:08.718831+0000","last_fullsized":"2024-04-15T15:16:08.718831+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:41:28.911237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.462112+0000","last_change":"2024-04-15T15:16:07.710622+0000","last_active":"2024-04-15T15:16:09.462112+0000","last_peered":"2024-04-15T15:16:09.462112+0000","last_clean":"2024-04-15T15:16:09.462112+0000","last_became_active":"2024-04-15T15:16:07.710280+0000","last_became_peered":"2024-04-15T15:16:07.710280+0000","last_unstale":"2024-04-15T15:16:09.462112+0000","last_undegraded":"2024-04-15T15:16:09.462112+0000","last_fullsized":"2024-04-15T15:16:09.462112+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:32:26.516274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718413+0000","last_change":"2024-04-15T15:16:07.713369+0000","last_active":"2024-04-15T15:16:08.718413+0000","last_peered":"2024-04-15T15:16:08.718413+0000","last_clean":"2024-04-15T15:16:08.718413+0000","last_became_active":"2024-04-15T15:16:07.713109+0000","last_became_peered":"2024-04-15T15:16:07.713109+0000","last_unstale":"2024-04-15T15:16:08.718413+0000","last_undegraded":"2024-04-15T15:16:08.718413+0000","last_fullsized":"2024-04-15T15:16:08.718413+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:00:49.742979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.358927+0000","last_change":"2024-04-15T15:16:07.709278+0000","last_active":"2024-04-15T15:16:09.358927+0000","last_peered":"2024-04-15T15:16:09.358927+0000","last_clean":"2024-04-15T15:16:09.358927+0000","last_became_active":"2024-04-15T15:16:07.708829+0000","last_became_peered":"2024-04-15T15:16:07.708829+0000","last_unstale":"2024-04-15T15:16:09.358927+0000","last_undegraded":"2024-04-15T15:16:09.358927+0000","last_fullsized":"2024-04-15T15:16:09.358927+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:51:17.043733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718403+0000","last_change":"2024-04-15T15:16:07.700808+0000","last_active":"2024-04-15T15:16:08.718403+0000","last_peered":"2024-04-15T15:16:08.718403+0000","last_clean":"2024-04-15T15:16:08.718403+0000","last_became_active":"2024-04-15T15:16:07.700484+0000","last_became_peered":"2024-04-15T15:16:07.700484+0000","last_unstale":"2024-04-15T15:16:08.718403+0000","last_undegraded":"2024-04-15T15:16:08.718403+0000","last_fullsized":"2024-04-15T15:16:08.718403+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:14:45.605613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709813+0000","last_change":"2024-04-15T15:16:07.713960+0000","last_active":"2024-04-15T15:16:08.709813+0000","last_peered":"2024-04-15T15:16:08.709813+0000","last_clean":"2024-04-15T15:16:08.709813+0000","last_became_active":"2024-04-15T15:16:07.713665+0000","last_became_peered":"2024-04-15T15:16:07.713665+0000","last_unstale":"2024-04-15T15:16:08.709813+0000","last_undegraded":"2024-04-15T15:16:08.709813+0000","last_fullsized":"2024-04-15T15:16:08.709813+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:54:41.375909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.710290+0000","last_change":"2024-04-15T15:16:07.714486+0000","last_active":"2024-04-15T15:16:08.710290+0000","last_peered":"2024-04-15T15:16:08.710290+0000","last_clean":"2024-04-15T15:16:08.710290+0000","last_became_active":"2024-04-15T15:16:07.714191+0000","last_became_peered":"2024-04-15T15:16:07.714191+0000","last_unstale":"2024-04-15T15:16:08.710290+0000","last_undegraded":"2024-04-15T15:16:08.710290+0000","last_fullsized":"2024-04-15T15:16:08.710290+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:09:09.330985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709650+0000","last_change":"2024-04-15T15:16:07.711021+0000","last_active":"2024-04-15T15:16:08.709650+0000","last_peered":"2024-04-15T15:16:08.709650+0000","last_clean":"2024-04-15T15:16:08.709650+0000","last_became_active":"2024-04-15T15:16:07.710709+0000","last_became_peered":"2024-04-15T15:16:07.710709+0000","last_unstale":"2024-04-15T15:16:08.709650+0000","last_undegraded":"2024-04-15T15:16:08.709650+0000","last_fullsized":"2024-04-15T15:16:08.709650+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:51:17.043733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.708951+0000","last_change":"2024-04-15T15:16:07.712960+0000","last_active":"2024-04-15T15:16:08.708951+0000","last_peered":"2024-04-15T15:16:08.708951+0000","last_clean":"2024-04-15T15:16:08.708951+0000","last_became_active":"2024-04-15T15:16:07.712636+0000","last_became_peered":"2024-04-15T15:16:07.712636+0000","last_unstale":"2024-04-15T15:16:08.708951+0000","last_undegraded":"2024-04-15T15:16:08.708951+0000","last_fullsized":"2024-04-15T15:16:08.708951+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:02:51.429249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709437+0000","last_change":"2024-04-15T15:16:07.712946+0000","last_active":"2024-04-15T15:16:08.709437+0000","last_peered":"2024-04-15T15:16:08.709437+0000","last_clean":"2024-04-15T15:16:08.709437+0000","last_became_active":"2024-04-15T15:16:07.712677+0000","last_became_peered":"2024-04-15T15:16:07.712677+0000","last_unstale":"2024-04-15T15:16:08.709437+0000","last_undegraded":"2024-04-15T15:16:08.709437+0000","last_fullsized":"2024-04-15T15:16:08.709437+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:32:44.859092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718447+0000","last_change":"2024-04-15T15:16:07.708983+0000","last_active":"2024-04-15T15:16:08.718447+0000","last_peered":"2024-04-15T15:16:08.718447+0000","last_clean":"2024-04-15T15:16:08.718447+0000","last_became_active":"2024-04-15T15:16:07.708612+0000","last_became_peered":"2024-04-15T15:16:07.708612+0000","last_unstale":"2024-04-15T15:16:08.718447+0000","last_undegraded":"2024-04-15T15:16:08.718447+0000","last_fullsized":"2024-04-15T15:16:08.718447+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:12:03.446783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.359328+0000","last_change":"2024-04-15T15:16:07.710990+0000","last_active":"2024-04-15T15:16:09.359328+0000","last_peered":"2024-04-15T15:16:09.359328+0000","last_clean":"2024-04-15T15:16:09.359328+0000","last_became_active":"2024-04-15T15:16:07.710763+0000","last_became_peered":"2024-04-15T15:16:07.710763+0000","last_unstale":"2024-04-15T15:16:09.359328+0000","last_undegraded":"2024-04-15T15:16:09.359328+0000","last_fullsized":"2024-04-15T15:16:09.359328+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:39:05.235149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709573+0000","last_change":"2024-04-15T15:16:07.718348+0000","last_active":"2024-04-15T15:16:08.709573+0000","last_peered":"2024-04-15T15:16:08.709573+0000","last_clean":"2024-04-15T15:16:08.709573+0000","last_became_active":"2024-04-15T15:16:07.717954+0000","last_became_peered":"2024-04-15T15:16:07.717954+0000","last_unstale":"2024-04-15T15:16:08.709573+0000","last_undegraded":"2024-04-15T15:16:08.709573+0000","last_fullsized":"2024-04-15T15:16:08.709573+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:32:26.516274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.708677+0000","last_change":"2024-04-15T15:16:07.714768+0000","last_active":"2024-04-15T15:16:08.708677+0000","last_peered":"2024-04-15T15:16:08.708677+0000","last_clean":"2024-04-15T15:16:08.708677+0000","last_became_active":"2024-04-15T15:16:07.714512+0000","last_became_peered":"2024-04-15T15:16:07.714512+0000","last_unstale":"2024-04-15T15:16:08.708677+0000","last_undegraded":"2024-04-15T15:16:08.708677+0000","last_fullsized":"2024-04-15T15:16:08.708677+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:42:44.933220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.710013+0000","last_change":"2024-04-15T15:16:07.714668+0000","last_active":"2024-04-15T15:16:08.710013+0000","last_peered":"2024-04-15T15:16:08.710013+0000","last_clean":"2024-04-15T15:16:08.710013+0000","last_became_active":"2024-04-15T15:16:07.714354+0000","last_became_peered":"2024-04-15T15:16:07.714354+0000","last_unstale":"2024-04-15T15:16:08.710013+0000","last_undegraded":"2024-04-15T15:16:08.710013+0000","last_fullsized":"2024-04-15T15:16:08.710013+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:08:22.249305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.461563+0000","last_change":"2024-04-15T15:16:07.708195+0000","last_active":"2024-04-15T15:16:09.461563+0000","last_peered":"2024-04-15T15:16:09.461563+0000","last_clean":"2024-04-15T15:16:09.461563+0000","last_became_active":"2024-04-15T15:16:07.707897+0000","last_became_peered":"2024-04-15T15:16:07.707897+0000","last_unstale":"2024-04-15T15:16:09.461563+0000","last_undegraded":"2024-04-15T15:16:09.461563+0000","last_fullsized":"2024-04-15T15:16:09.461563+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:57:26.177964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709808+0000","last_change":"2024-04-15T15:16:07.707076+0000","last_active":"2024-04-15T15:16:08.709808+0000","last_peered":"2024-04-15T15:16:08.709808+0000","last_clean":"2024-04-15T15:16:08.709808+0000","last_became_active":"2024-04-15T15:16:07.706810+0000","last_became_peered":"2024-04-15T15:16:07.706810+0000","last_unstale":"2024-04-15T15:16:08.709808+0000","last_undegraded":"2024-04-15T15:16:08.709808+0000","last_fullsized":"2024-04-15T15:16:08.709808+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:45:03.154342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718597+0000","last_change":"2024-04-15T15:16:07.717296+0000","last_active":"2024-04-15T15:16:08.718597+0000","last_peered":"2024-04-15T15:16:08.718597+0000","last_clean":"2024-04-15T15:16:08.718597+0000","last_became_active":"2024-04-15T15:16:07.716737+0000","last_became_peered":"2024-04-15T15:16:07.716737+0000","last_unstale":"2024-04-15T15:16:08.718597+0000","last_undegraded":"2024-04-15T15:16:08.718597+0000","last_fullsized":"2024-04-15T15:16:08.718597+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:38:41.698007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709781+0000","last_change":"2024-04-15T15:16:07.708652+0000","last_active":"2024-04-15T15:16:08.709781+0000","last_peered":"2024-04-15T15:16:08.709781+0000","last_clean":"2024-04-15T15:16:08.709781+0000","last_became_active":"2024-04-15T15:16:07.708305+0000","last_became_peered":"2024-04-15T15:16:07.708305+0000","last_unstale":"2024-04-15T15:16:08.709781+0000","last_undegraded":"2024-04-15T15:16:08.709781+0000","last_fullsized":"2024-04-15T15:16:08.709781+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:57:26.177964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709498+0000","last_change":"2024-04-15T15:16:07.709602+0000","last_active":"2024-04-15T15:16:08.709498+0000","last_peered":"2024-04-15T15:16:08.709498+0000","last_clean":"2024-04-15T15:16:08.709498+0000","last_became_active":"2024-04-15T15:16:07.709332+0000","last_became_peered":"2024-04-15T15:16:07.709332+0000","last_unstale":"2024-04-15T15:16:08.709498+0000","last_undegraded":"2024-04-15T15:16:08.709498+0000","last_fullsized":"2024-04-15T15:16:08.709498+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:03:12.877648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709730+0000","last_change":"2024-04-15T15:16:07.717993+0000","last_active":"2024-04-15T15:16:08.709730+0000","last_peered":"2024-04-15T15:16:08.709730+0000","last_clean":"2024-04-15T15:16:08.709730+0000","last_became_active":"2024-04-15T15:16:07.717663+0000","last_became_peered":"2024-04-15T15:16:07.717663+0000","last_unstale":"2024-04-15T15:16:08.709730+0000","last_undegraded":"2024-04-15T15:16:08.709730+0000","last_fullsized":"2024-04-15T15:16:08.709730+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:42:46.979916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.461305+0000","last_change":"2024-04-15T15:16:07.719599+0000","last_active":"2024-04-15T15:16:09.461305+0000","last_peered":"2024-04-15T15:16:09.461305+0000","last_clean":"2024-04-15T15:16:09.461305+0000","last_became_active":"2024-04-15T15:16:07.719303+0000","last_became_peered":"2024-04-15T15:16:07.719303+0000","last_unstale":"2024-04-15T15:16:09.461305+0000","last_undegraded":"2024-04-15T15:16:09.461305+0000","last_fullsized":"2024-04-15T15:16:09.461305+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:03:12.877648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709700+0000","last_change":"2024-04-15T15:16:07.719675+0000","last_active":"2024-04-15T15:16:08.709700+0000","last_peered":"2024-04-15T15:16:08.709700+0000","last_clean":"2024-04-15T15:16:08.709700+0000","last_became_active":"2024-04-15T15:16:07.719391+0000","last_became_peered":"2024-04-15T15:16:07.719391+0000","last_unstale":"2024-04-15T15:16:08.709700+0000","last_undegraded":"2024-04-15T15:16:08.709700+0000","last_fullsized":"2024-04-15T15:16:08.709700+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:14:39.023105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"37'4","reported_seq":18,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.710178+0000","last_change":"2024-04-15T15:16:06.702682+0000","last_active":"2024-04-15T15:16:08.710178+0000","last_peered":"2024-04-15T15:16:08.710178+0000","last_clean":"2024-04-15T15:16:08.710178+0000","last_became_active":"2024-04-15T15:16:06.702428+0000","last_became_peered":"2024-04-15T15:16:06.702428+0000","last_unstale":"2024-04-15T15:16:08.710178+0000","last_undegraded":"2024-04-15T15:16:08.710178+0000","last_fullsized":"2024-04-15T15:16:08.710178+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:35:01.354434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.461763+0000","last_change":"2024-04-15T15:16:07.712184+0000","last_active":"2024-04-15T15:16:09.461763+0000","last_peered":"2024-04-15T15:16:09.461763+0000","last_clean":"2024-04-15T15:16:09.461763+0000","last_became_active":"2024-04-15T15:16:07.711833+0000","last_became_peered":"2024-04-15T15:16:07.711833+0000","last_unstale":"2024-04-15T15:16:09.461763+0000","last_undegraded":"2024-04-15T15:16:09.461763+0000","last_fullsized":"2024-04-15T15:16:09.461763+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:08:22.249305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.461626+0000","last_change":"2024-04-15T15:16:06.705264+0000","last_active":"2024-04-15T15:16:09.461626+0000","last_peered":"2024-04-15T15:16:09.461626+0000","last_clean":"2024-04-15T15:16:09.461626+0000","last_became_active":"2024-04-15T15:16:06.704985+0000","last_became_peered":"2024-04-15T15:16:06.704985+0000","last_unstale":"2024-04-15T15:16:09.461626+0000","last_undegraded":"2024-04-15T15:16:09.461626+0000","last_fullsized":"2024-04-15T15:16:09.461626+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:02:50.425156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.359305+0000","last_change":"2024-04-15T15:16:07.710774+0000","last_active":"2024-04-15T15:16:09.359305+0000","last_peered":"2024-04-15T15:16:09.359305+0000","last_clean":"2024-04-15T15:16:09.359305+0000","last_became_active":"2024-04-15T15:16:07.710468+0000","last_became_peered":"2024-04-15T15:16:07.710468+0000","last_unstale":"2024-04-15T15:16:09.359305+0000","last_undegraded":"2024-04-15T15:16:09.359305+0000","last_fullsized":"2024-04-15T15:16:09.359305+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:56:02.237631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.461902+0000","last_change":"2024-04-15T15:16:06.706399+0000","last_active":"2024-04-15T15:16:09.461902+0000","last_peered":"2024-04-15T15:16:09.461902+0000","last_clean":"2024-04-15T15:16:09.461902+0000","last_became_active":"2024-04-15T15:16:06.706141+0000","last_became_peered":"2024-04-15T15:16:06.706141+0000","last_unstale":"2024-04-15T15:16:09.461902+0000","last_undegraded":"2024-04-15T15:16:09.461902+0000","last_fullsized":"2024-04-15T15:16:09.461902+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:33:09.310850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973814+0000","last_change":"2024-04-15T15:16:07.710742+0000","last_active":"2024-04-15T15:16:08.973814+0000","last_peered":"2024-04-15T15:16:08.973814+0000","last_clean":"2024-04-15T15:16:08.973814+0000","last_became_active":"2024-04-15T15:16:07.710455+0000","last_became_peered":"2024-04-15T15:16:07.710455+0000","last_unstale":"2024-04-15T15:16:08.973814+0000","last_undegraded":"2024-04-15T15:16:08.973814+0000","last_fullsized":"2024-04-15T15:16:08.973814+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:41:28.911237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.359266+0000","last_change":"2024-04-15T15:16:06.704244+0000","last_active":"2024-04-15T15:16:09.359266+0000","last_peered":"2024-04-15T15:16:09.359266+0000","last_clean":"2024-04-15T15:16:09.359266+0000","last_became_active":"2024-04-15T15:16:06.703919+0000","last_became_peered":"2024-04-15T15:16:06.703919+0000","last_unstale":"2024-04-15T15:16:09.359266+0000","last_undegraded":"2024-04-15T15:16:09.359266+0000","last_fullsized":"2024-04-15T15:16:09.359266+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:51:03.225693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.461422+0000","last_change":"2024-04-15T15:16:07.712901+0000","last_active":"2024-04-15T15:16:09.461422+0000","last_peered":"2024-04-15T15:16:09.461422+0000","last_clean":"2024-04-15T15:16:09.461422+0000","last_became_active":"2024-04-15T15:16:07.712444+0000","last_became_peered":"2024-04-15T15:16:07.712444+0000","last_unstale":"2024-04-15T15:16:09.461422+0000","last_undegraded":"2024-04-15T15:16:09.461422+0000","last_fullsized":"2024-04-15T15:16:09.461422+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:51:17.043733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718599+0000","last_change":"2024-04-15T15:16:06.706424+0000","last_active":"2024-04-15T15:16:08.718599+0000","last_peered":"2024-04-15T15:16:08.718599+0000","last_clean":"2024-04-15T15:16:08.718599+0000","last_became_active":"2024-04-15T15:16:06.706138+0000","last_became_peered":"2024-04-15T15:16:06.706138+0000","last_unstale":"2024-04-15T15:16:08.718599+0000","last_undegraded":"2024-04-15T15:16:08.718599+0000","last_fullsized":"2024-04-15T15:16:08.718599+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:01:00.906451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718514+0000","last_change":"2024-04-15T15:16:07.709849+0000","last_active":"2024-04-15T15:16:08.718514+0000","last_peered":"2024-04-15T15:16:08.718514+0000","last_clean":"2024-04-15T15:16:08.718514+0000","last_became_active":"2024-04-15T15:16:07.709582+0000","last_became_peered":"2024-04-15T15:16:07.709582+0000","last_unstale":"2024-04-15T15:16:08.718514+0000","last_undegraded":"2024-04-15T15:16:08.718514+0000","last_fullsized":"2024-04-15T15:16:08.718514+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:04:27.874758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.710385+0000","last_change":"2024-04-15T15:16:06.704215+0000","last_active":"2024-04-15T15:16:08.710385+0000","last_peered":"2024-04-15T15:16:08.710385+0000","last_clean":"2024-04-15T15:16:08.710385+0000","last_became_active":"2024-04-15T15:16:06.703429+0000","last_became_peered":"2024-04-15T15:16:06.703429+0000","last_unstale":"2024-04-15T15:16:08.710385+0000","last_undegraded":"2024-04-15T15:16:08.710385+0000","last_fullsized":"2024-04-15T15:16:08.710385+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:51:03.225693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.359185+0000","last_change":"2024-04-15T15:16:07.709067+0000","last_active":"2024-04-15T15:16:09.359185+0000","last_peered":"2024-04-15T15:16:09.359185+0000","last_clean":"2024-04-15T15:16:09.359185+0000","last_became_active":"2024-04-15T15:16:07.708491+0000","last_became_peered":"2024-04-15T15:16:07.708491+0000","last_unstale":"2024-04-15T15:16:09.359185+0000","last_undegraded":"2024-04-15T15:16:09.359185+0000","last_fullsized":"2024-04-15T15:16:09.359185+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:45:03.154342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.461946+0000","last_change":"2024-04-15T15:16:06.707235+0000","last_active":"2024-04-15T15:16:09.461946+0000","last_peered":"2024-04-15T15:16:09.461946+0000","last_clean":"2024-04-15T15:16:09.461946+0000","last_became_active":"2024-04-15T15:16:06.706918+0000","last_became_peered":"2024-04-15T15:16:06.706918+0000","last_unstale":"2024-04-15T15:16:09.461946+0000","last_undegraded":"2024-04-15T15:16:09.461946+0000","last_fullsized":"2024-04-15T15:16:09.461946+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:25:09.602767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973566+0000","last_change":"2024-04-15T15:16:07.715092+0000","last_active":"2024-04-15T15:16:08.973566+0000","last_peered":"2024-04-15T15:16:08.973566+0000","last_clean":"2024-04-15T15:16:08.973566+0000","last_became_active":"2024-04-15T15:16:07.714832+0000","last_became_peered":"2024-04-15T15:16:07.714832+0000","last_unstale":"2024-04-15T15:16:08.973566+0000","last_undegraded":"2024-04-15T15:16:08.973566+0000","last_fullsized":"2024-04-15T15:16:08.973566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:30:18.481379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.358856+0000","last_change":"2024-04-15T15:16:06.699694+0000","last_active":"2024-04-15T15:16:09.358856+0000","last_peered":"2024-04-15T15:16:09.358856+0000","last_clean":"2024-04-15T15:16:09.358856+0000","last_became_active":"2024-04-15T15:16:06.699412+0000","last_became_peered":"2024-04-15T15:16:06.699412+0000","last_unstale":"2024-04-15T15:16:09.358856+0000","last_undegraded":"2024-04-15T15:16:09.358856+0000","last_fullsized":"2024-04-15T15:16:09.358856+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:02:50.425156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.708969+0000","last_change":"2024-04-15T15:16:07.708203+0000","last_active":"2024-04-15T15:16:08.708969+0000","last_peered":"2024-04-15T15:16:08.708969+0000","last_clean":"2024-04-15T15:16:08.708969+0000","last_became_active":"2024-04-15T15:16:07.707642+0000","last_became_peered":"2024-04-15T15:16:07.707642+0000","last_unstale":"2024-04-15T15:16:08.708969+0000","last_undegraded":"2024-04-15T15:16:08.708969+0000","last_fullsized":"2024-04-15T15:16:08.708969+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:35:02.358527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.974013+0000","last_change":"2024-04-15T15:16:06.702867+0000","last_active":"2024-04-15T15:16:08.974013+0000","last_peered":"2024-04-15T15:16:08.974013+0000","last_clean":"2024-04-15T15:16:08.974013+0000","last_became_active":"2024-04-15T15:16:06.701891+0000","last_became_peered":"2024-04-15T15:16:06.701891+0000","last_unstale":"2024-04-15T15:16:08.974013+0000","last_undegraded":"2024-04-15T15:16:08.974013+0000","last_fullsized":"2024-04-15T15:16:08.974013+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:51:16.039639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.710590+0000","last_change":"2024-04-15T15:16:07.718269+0000","last_active":"2024-04-15T15:16:08.710590+0000","last_peered":"2024-04-15T15:16:08.710590+0000","last_clean":"2024-04-15T15:16:08.710590+0000","last_became_active":"2024-04-15T15:16:07.717880+0000","last_became_peered":"2024-04-15T15:16:07.717880+0000","last_unstale":"2024-04-15T15:16:08.710590+0000","last_undegraded":"2024-04-15T15:16:08.710590+0000","last_fullsized":"2024-04-15T15:16:08.710590+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:00:49.742979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718797+0000","last_change":"2024-04-15T15:16:06.704693+0000","last_active":"2024-04-15T15:16:08.718797+0000","last_peered":"2024-04-15T15:16:08.718797+0000","last_clean":"2024-04-15T15:16:08.718797+0000","last_became_active":"2024-04-15T15:16:06.704416+0000","last_became_peered":"2024-04-15T15:16:06.704416+0000","last_unstale":"2024-04-15T15:16:08.718797+0000","last_undegraded":"2024-04-15T15:16:08.718797+0000","last_fullsized":"2024-04-15T15:16:08.718797+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:32:43.854999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.461533+0000","last_change":"2024-04-15T15:16:07.716298+0000","last_active":"2024-04-15T15:16:09.461533+0000","last_peered":"2024-04-15T15:16:09.461533+0000","last_clean":"2024-04-15T15:16:09.461533+0000","last_became_active":"2024-04-15T15:16:07.715865+0000","last_became_peered":"2024-04-15T15:16:07.715865+0000","last_unstale":"2024-04-15T15:16:09.461533+0000","last_undegraded":"2024-04-15T15:16:09.461533+0000","last_fullsized":"2024-04-15T15:16:09.461533+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:45:03.154342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"37'3","reported_seq":17,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718918+0000","last_change":"2024-04-15T15:16:06.702716+0000","last_active":"2024-04-15T15:16:08.718918+0000","last_peered":"2024-04-15T15:16:08.718918+0000","last_clean":"2024-04-15T15:16:08.718918+0000","last_became_active":"2024-04-15T15:16:06.702416+0000","last_became_peered":"2024-04-15T15:16:06.702416+0000","last_unstale":"2024-04-15T15:16:08.718918+0000","last_undegraded":"2024-04-15T15:16:08.718918+0000","last_fullsized":"2024-04-15T15:16:08.718918+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:38:28.840555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.359401+0000","last_change":"2024-04-15T15:16:07.711716+0000","last_active":"2024-04-15T15:16:09.359401+0000","last_peered":"2024-04-15T15:16:09.359401+0000","last_clean":"2024-04-15T15:16:09.359401+0000","last_became_active":"2024-04-15T15:16:07.711459+0000","last_became_peered":"2024-04-15T15:16:07.711459+0000","last_unstale":"2024-04-15T15:16:09.359401+0000","last_undegraded":"2024-04-15T15:16:09.359401+0000","last_fullsized":"2024-04-15T15:16:09.359401+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:32:26.516274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"37'6","reported_seq":20,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973672+0000","last_change":"2024-04-15T15:16:06.707247+0000","last_active":"2024-04-15T15:16:08.973672+0000","last_peered":"2024-04-15T15:16:08.973672+0000","last_clean":"2024-04-15T15:16:08.973672+0000","last_became_active":"2024-04-15T15:16:06.706166+0000","last_became_peered":"2024-04-15T15:16:06.706166+0000","last_unstale":"2024-04-15T15:16:08.973672+0000","last_undegraded":"2024-04-15T15:16:08.973672+0000","last_fullsized":"2024-04-15T15:16:08.973672+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:57:25.173871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.461302+0000","last_change":"2024-04-15T15:16:07.712212+0000","last_active":"2024-04-15T15:16:09.461302+0000","last_peered":"2024-04-15T15:16:09.461302+0000","last_clean":"2024-04-15T15:16:09.461302+0000","last_became_active":"2024-04-15T15:16:07.711327+0000","last_became_peered":"2024-04-15T15:16:07.711327+0000","last_unstale":"2024-04-15T15:16:09.461302+0000","last_undegraded":"2024-04-15T15:16:09.461302+0000","last_fullsized":"2024-04-15T15:16:09.461302+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:03:12.877648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973690+0000","last_change":"2024-04-15T15:16:06.704498+0000","last_active":"2024-04-15T15:16:08.973690+0000","last_peered":"2024-04-15T15:16:08.973690+0000","last_clean":"2024-04-15T15:16:08.973690+0000","last_became_active":"2024-04-15T15:16:06.704230+0000","last_became_peered":"2024-04-15T15:16:06.704230+0000","last_unstale":"2024-04-15T15:16:08.973690+0000","last_undegraded":"2024-04-15T15:16:08.973690+0000","last_fullsized":"2024-04-15T15:16:08.973690+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:32:43.854999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973429+0000","last_change":"2024-04-15T15:16:07.712780+0000","last_active":"2024-04-15T15:16:08.973429+0000","last_peered":"2024-04-15T15:16:08.973429+0000","last_clean":"2024-04-15T15:16:08.973429+0000","last_became_active":"2024-04-15T15:16:07.712534+0000","last_became_peered":"2024-04-15T15:16:07.712534+0000","last_unstale":"2024-04-15T15:16:08.973429+0000","last_undegraded":"2024-04-15T15:16:08.973429+0000","last_fullsized":"2024-04-15T15:16:08.973429+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:12:03.446783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.461498+0000","last_change":"2024-04-15T15:16:06.707538+0000","last_active":"2024-04-15T15:16:09.461498+0000","last_peered":"2024-04-15T15:16:09.461498+0000","last_clean":"2024-04-15T15:16:09.461498+0000","last_became_active":"2024-04-15T15:16:06.707264+0000","last_became_peered":"2024-04-15T15:16:06.707264+0000","last_unstale":"2024-04-15T15:16:09.461498+0000","last_undegraded":"2024-04-15T15:16:09.461498+0000","last_fullsized":"2024-04-15T15:16:09.461498+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:35:01.354434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.710310+0000","last_change":"2024-04-15T15:16:07.716245+0000","last_active":"2024-04-15T15:16:08.710310+0000","last_peered":"2024-04-15T15:16:08.710310+0000","last_clean":"2024-04-15T15:16:08.710310+0000","last_became_active":"2024-04-15T15:16:07.715993+0000","last_became_peered":"2024-04-15T15:16:07.715993+0000","last_unstale":"2024-04-15T15:16:08.710310+0000","last_undegraded":"2024-04-15T15:16:08.710310+0000","last_fullsized":"2024-04-15T15:16:08.710310+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:14:45.605613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709283+0000","last_change":"2024-04-15T15:16:06.700246+0000","last_active":"2024-04-15T15:16:08.709283+0000","last_peered":"2024-04-15T15:16:08.709283+0000","last_clean":"2024-04-15T15:16:08.709283+0000","last_became_active":"2024-04-15T15:16:06.699972+0000","last_became_peered":"2024-04-15T15:16:06.699972+0000","last_unstale":"2024-04-15T15:16:08.709283+0000","last_undegraded":"2024-04-15T15:16:08.709283+0000","last_fullsized":"2024-04-15T15:16:08.709283+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:02:50.425156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973578+0000","last_change":"2024-04-15T15:16:07.713401+0000","last_active":"2024-04-15T15:16:08.973578+0000","last_peered":"2024-04-15T15:16:08.973578+0000","last_clean":"2024-04-15T15:16:08.973578+0000","last_became_active":"2024-04-15T15:16:07.713157+0000","last_became_peered":"2024-04-15T15:16:07.713157+0000","last_unstale":"2024-04-15T15:16:08.973578+0000","last_undegraded":"2024-04-15T15:16:08.973578+0000","last_fullsized":"2024-04-15T15:16:08.973578+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:04:27.874758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"37'8","reported_seq":22,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709937+0000","last_change":"2024-04-15T15:16:06.703005+0000","last_active":"2024-04-15T15:16:08.709937+0000","last_peered":"2024-04-15T15:16:08.709937+0000","last_clean":"2024-04-15T15:16:08.709937+0000","last_became_active":"2024-04-15T15:16:06.702719+0000","last_became_peered":"2024-04-15T15:16:06.702719+0000","last_unstale":"2024-04-15T15:16:08.709937+0000","last_undegraded":"2024-04-15T15:16:08.709937+0000","last_fullsized":"2024-04-15T15:16:08.709937+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:33:09.310850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973624+0000","last_change":"2024-04-15T15:16:07.716082+0000","last_active":"2024-04-15T15:16:08.973624+0000","last_peered":"2024-04-15T15:16:08.973624+0000","last_clean":"2024-04-15T15:16:08.973624+0000","last_became_active":"2024-04-15T15:16:07.715802+0000","last_became_peered":"2024-04-15T15:16:07.715802+0000","last_unstale":"2024-04-15T15:16:08.973624+0000","last_undegraded":"2024-04-15T15:16:08.973624+0000","last_fullsized":"2024-04-15T15:16:08.973624+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:38:41.698007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718834+0000","last_change":"2024-04-15T15:16:06.699687+0000","last_active":"2024-04-15T15:16:08.718834+0000","last_peered":"2024-04-15T15:16:08.718834+0000","last_clean":"2024-04-15T15:16:08.718834+0000","last_became_active":"2024-04-15T15:16:06.699403+0000","last_became_peered":"2024-04-15T15:16:06.699403+0000","last_unstale":"2024-04-15T15:16:08.718834+0000","last_undegraded":"2024-04-15T15:16:08.718834+0000","last_fullsized":"2024-04-15T15:16:08.718834+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:14:27.402622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718805+0000","last_change":"2024-04-15T15:16:07.720614+0000","last_active":"2024-04-15T15:16:08.718805+0000","last_peered":"2024-04-15T15:16:08.718805+0000","last_clean":"2024-04-15T15:16:08.718805+0000","last_became_active":"2024-04-15T15:16:07.720327+0000","last_became_peered":"2024-04-15T15:16:07.720327+0000","last_unstale":"2024-04-15T15:16:08.718805+0000","last_undegraded":"2024-04-15T15:16:08.718805+0000","last_fullsized":"2024-04-15T15:16:08.718805+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:57:37.374559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.359381+0000","last_change":"2024-04-15T15:16:06.703119+0000","last_active":"2024-04-15T15:16:09.359381+0000","last_peered":"2024-04-15T15:16:09.359381+0000","last_clean":"2024-04-15T15:16:09.359381+0000","last_became_active":"2024-04-15T15:16:06.702423+0000","last_became_peered":"2024-04-15T15:16:06.702423+0000","last_unstale":"2024-04-15T15:16:09.359381+0000","last_undegraded":"2024-04-15T15:16:09.359381+0000","last_fullsized":"2024-04-15T15:16:09.359381+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:25:09.602767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.359209+0000","last_change":"2024-04-15T15:16:07.713465+0000","last_active":"2024-04-15T15:16:09.359209+0000","last_peered":"2024-04-15T15:16:09.359209+0000","last_clean":"2024-04-15T15:16:09.359209+0000","last_became_active":"2024-04-15T15:16:07.713154+0000","last_became_peered":"2024-04-15T15:16:07.713154+0000","last_unstale":"2024-04-15T15:16:09.359209+0000","last_undegraded":"2024-04-15T15:16:09.359209+0000","last_fullsized":"2024-04-15T15:16:09.359209+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:08:22.249305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709042+0000","last_change":"2024-04-15T15:16:06.692920+0000","last_active":"2024-04-15T15:16:08.709042+0000","last_peered":"2024-04-15T15:16:08.709042+0000","last_clean":"2024-04-15T15:16:08.709042+0000","last_became_active":"2024-04-15T15:16:06.692641+0000","last_became_peered":"2024-04-15T15:16:06.692641+0000","last_unstale":"2024-04-15T15:16:08.709042+0000","last_undegraded":"2024-04-15T15:16:08.709042+0000","last_fullsized":"2024-04-15T15:16:08.709042+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:38:20.537537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709058+0000","last_change":"2024-04-15T15:16:07.723443+0000","last_active":"2024-04-15T15:16:08.709058+0000","last_peered":"2024-04-15T15:16:08.709058+0000","last_clean":"2024-04-15T15:16:08.709058+0000","last_became_active":"2024-04-15T15:16:07.723186+0000","last_became_peered":"2024-04-15T15:16:07.723186+0000","last_unstale":"2024-04-15T15:16:08.709058+0000","last_undegraded":"2024-04-15T15:16:08.709058+0000","last_fullsized":"2024-04-15T15:16:08.709058+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:25:10.606860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.461740+0000","last_change":"2024-04-15T15:16:06.707391+0000","last_active":"2024-04-15T15:16:09.461740+0000","last_peered":"2024-04-15T15:16:09.461740+0000","last_clean":"2024-04-15T15:16:09.461740+0000","last_became_active":"2024-04-15T15:16:06.707125+0000","last_became_peered":"2024-04-15T15:16:06.707125+0000","last_unstale":"2024-04-15T15:16:09.461740+0000","last_undegraded":"2024-04-15T15:16:09.461740+0000","last_fullsized":"2024-04-15T15:16:09.461740+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:51:03.225693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.710592+0000","last_change":"2024-04-15T15:16:07.719157+0000","last_active":"2024-04-15T15:16:08.710592+0000","last_peered":"2024-04-15T15:16:08.710592+0000","last_clean":"2024-04-15T15:16:08.710592+0000","last_became_active":"2024-04-15T15:16:07.718892+0000","last_became_peered":"2024-04-15T15:16:07.718892+0000","last_unstale":"2024-04-15T15:16:08.710592+0000","last_undegraded":"2024-04-15T15:16:08.710592+0000","last_fullsized":"2024-04-15T15:16:08.710592+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:30:18.481379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.708740+0000","last_change":"2024-04-15T15:16:06.706044+0000","last_active":"2024-04-15T15:16:08.708740+0000","last_peered":"2024-04-15T15:16:08.708740+0000","last_clean":"2024-04-15T15:16:08.708740+0000","last_became_active":"2024-04-15T15:16:06.705532+0000","last_became_peered":"2024-04-15T15:16:06.705532+0000","last_unstale":"2024-04-15T15:16:08.708740+0000","last_undegraded":"2024-04-15T15:16:08.708740+0000","last_fullsized":"2024-04-15T15:16:08.708740+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:40:09.944851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.461829+0000","last_change":"2024-04-15T15:16:07.716379+0000","last_active":"2024-04-15T15:16:09.461829+0000","last_peered":"2024-04-15T15:16:09.461829+0000","last_clean":"2024-04-15T15:16:09.461829+0000","last_became_active":"2024-04-15T15:16:07.715437+0000","last_became_peered":"2024-04-15T15:16:07.715437+0000","last_unstale":"2024-04-15T15:16:09.461829+0000","last_undegraded":"2024-04-15T15:16:09.461829+0000","last_fullsized":"2024-04-15T15:16:09.461829+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:56:02.237631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.708758+0000","last_change":"2024-04-15T15:16:06.698041+0000","last_active":"2024-04-15T15:16:08.708758+0000","last_peered":"2024-04-15T15:16:08.708758+0000","last_clean":"2024-04-15T15:16:08.708758+0000","last_became_active":"2024-04-15T15:16:06.697754+0000","last_became_peered":"2024-04-15T15:16:06.697754+0000","last_unstale":"2024-04-15T15:16:08.708758+0000","last_undegraded":"2024-04-15T15:16:08.708758+0000","last_fullsized":"2024-04-15T15:16:08.708758+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:10:07.758669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718456+0000","last_change":"2024-04-15T15:16:07.701259+0000","last_active":"2024-04-15T15:16:08.718456+0000","last_peered":"2024-04-15T15:16:08.718456+0000","last_clean":"2024-04-15T15:16:08.718456+0000","last_became_active":"2024-04-15T15:16:07.700984+0000","last_became_peered":"2024-04-15T15:16:07.700984+0000","last_unstale":"2024-04-15T15:16:08.718456+0000","last_undegraded":"2024-04-15T15:16:08.718456+0000","last_fullsized":"2024-04-15T15:16:08.718456+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:30:18.481379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709831+0000","last_change":"2024-04-15T15:16:06.699270+0000","last_active":"2024-04-15T15:16:08.709831+0000","last_peered":"2024-04-15T15:16:08.709831+0000","last_clean":"2024-04-15T15:16:08.709831+0000","last_became_active":"2024-04-15T15:16:06.698992+0000","last_became_peered":"2024-04-15T15:16:06.698992+0000","last_unstale":"2024-04-15T15:16:08.709831+0000","last_undegraded":"2024-04-15T15:16:08.709831+0000","last_fullsized":"2024-04-15T15:16:08.709831+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:18:33.749347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973763+0000","last_change":"2024-04-15T15:16:07.714324+0000","last_active":"2024-04-15T15:16:08.973763+0000","last_peered":"2024-04-15T15:16:08.973763+0000","last_clean":"2024-04-15T15:16:08.973763+0000","last_became_active":"2024-04-15T15:16:07.714069+0000","last_became_peered":"2024-04-15T15:16:07.714069+0000","last_unstale":"2024-04-15T15:16:08.973763+0000","last_undegraded":"2024-04-15T15:16:08.973763+0000","last_fullsized":"2024-04-15T15:16:08.973763+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:57:37.374559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.708615+0000","last_change":"2024-04-15T15:16:06.704254+0000","last_active":"2024-04-15T15:16:08.708615+0000","last_peered":"2024-04-15T15:16:08.708615+0000","last_clean":"2024-04-15T15:16:08.708615+0000","last_became_active":"2024-04-15T15:16:06.703962+0000","last_became_peered":"2024-04-15T15:16:06.703962+0000","last_unstale":"2024-04-15T15:16:08.708615+0000","last_undegraded":"2024-04-15T15:16:08.708615+0000","last_fullsized":"2024-04-15T15:16:08.708615+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:34:42.463356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.461856+0000","last_change":"2024-04-15T15:16:07.707273+0000","last_active":"2024-04-15T15:16:09.461856+0000","last_peered":"2024-04-15T15:16:09.461856+0000","last_clean":"2024-04-15T15:16:09.461856+0000","last_became_active":"2024-04-15T15:16:07.707005+0000","last_became_peered":"2024-04-15T15:16:07.707005+0000","last_unstale":"2024-04-15T15:16:09.461856+0000","last_undegraded":"2024-04-15T15:16:09.461856+0000","last_fullsized":"2024-04-15T15:16:09.461856+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:56:02.237631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718915+0000","last_change":"2024-04-15T15:16:06.702827+0000","last_active":"2024-04-15T15:16:08.718915+0000","last_peered":"2024-04-15T15:16:08.718915+0000","last_clean":"2024-04-15T15:16:08.718915+0000","last_became_active":"2024-04-15T15:16:06.702558+0000","last_became_peered":"2024-04-15T15:16:06.702558+0000","last_unstale":"2024-04-15T15:16:08.718915+0000","last_undegraded":"2024-04-15T15:16:08.718915+0000","last_fullsized":"2024-04-15T15:16:08.718915+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:10:56.524136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.710593+0000","last_change":"2024-04-15T15:16:07.717216+0000","last_active":"2024-04-15T15:16:08.710593+0000","last_peered":"2024-04-15T15:16:08.710593+0000","last_clean":"2024-04-15T15:16:08.710593+0000","last_became_active":"2024-04-15T15:16:07.716939+0000","last_became_peered":"2024-04-15T15:16:07.716939+0000","last_unstale":"2024-04-15T15:16:08.710593+0000","last_undegraded":"2024-04-15T15:16:08.710593+0000","last_fullsized":"2024-04-15T15:16:08.710593+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:12:03.446783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"38'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.717703+0000","last_change":"2024-04-15T15:16:06.702772+0000","last_active":"2024-04-15T15:16:08.717703+0000","last_peered":"2024-04-15T15:16:08.717703+0000","last_clean":"2024-04-15T15:16:08.717703+0000","last_became_active":"2024-04-15T15:16:06.702503+0000","last_became_peered":"2024-04-15T15:16:06.702503+0000","last_unstale":"2024-04-15T15:16:08.717703+0000","last_undegraded":"2024-04-15T15:16:08.717703+0000","last_fullsized":"2024-04-15T15:16:08.717703+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:19:47.422670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973693+0000","last_change":"2024-04-15T15:16:07.716142+0000","last_active":"2024-04-15T15:16:08.973693+0000","last_peered":"2024-04-15T15:16:08.973693+0000","last_clean":"2024-04-15T15:16:08.973693+0000","last_became_active":"2024-04-15T15:16:07.715562+0000","last_became_peered":"2024-04-15T15:16:07.715562+0000","last_unstale":"2024-04-15T15:16:08.973693+0000","last_undegraded":"2024-04-15T15:16:08.973693+0000","last_fullsized":"2024-04-15T15:16:08.973693+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:51:12.622012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.462094+0000","last_change":"2024-04-15T15:16:06.706530+0000","last_active":"2024-04-15T15:16:09.462094+0000","last_peered":"2024-04-15T15:16:09.462094+0000","last_clean":"2024-04-15T15:16:09.462094+0000","last_became_active":"2024-04-15T15:16:06.706214+0000","last_became_peered":"2024-04-15T15:16:06.706214+0000","last_unstale":"2024-04-15T15:16:09.462094+0000","last_undegraded":"2024-04-15T15:16:09.462094+0000","last_fullsized":"2024-04-15T15:16:09.462094+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:25:09.602767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.461454+0000","last_change":"2024-04-15T15:16:07.710663+0000","last_active":"2024-04-15T15:16:09.461454+0000","last_peered":"2024-04-15T15:16:09.461454+0000","last_clean":"2024-04-15T15:16:09.461454+0000","last_became_active":"2024-04-15T15:16:07.710388+0000","last_became_peered":"2024-04-15T15:16:07.710388+0000","last_unstale":"2024-04-15T15:16:09.461454+0000","last_undegraded":"2024-04-15T15:16:09.461454+0000","last_fullsized":"2024-04-15T15:16:09.461454+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:57:26.177964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.461525+0000","last_change":"2024-04-15T15:16:06.706722+0000","last_active":"2024-04-15T15:16:09.461525+0000","last_peered":"2024-04-15T15:16:09.461525+0000","last_clean":"2024-04-15T15:16:09.461525+0000","last_became_active":"2024-04-15T15:16:06.706041+0000","last_became_peered":"2024-04-15T15:16:06.706041+0000","last_unstale":"2024-04-15T15:16:09.461525+0000","last_undegraded":"2024-04-15T15:16:09.461525+0000","last_fullsized":"2024-04-15T15:16:09.461525+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:35:01.354434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.461610+0000","last_change":"2024-04-15T15:16:07.709909+0000","last_active":"2024-04-15T15:16:09.461610+0000","last_peered":"2024-04-15T15:16:09.461610+0000","last_clean":"2024-04-15T15:16:09.461610+0000","last_became_active":"2024-04-15T15:16:07.709672+0000","last_became_peered":"2024-04-15T15:16:07.709672+0000","last_unstale":"2024-04-15T15:16:09.461610+0000","last_undegraded":"2024-04-15T15:16:09.461610+0000","last_fullsized":"2024-04-15T15:16:09.461610+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:45:03.154342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709503+0000","last_change":"2024-04-15T15:16:06.704950+0000","last_active":"2024-04-15T15:16:08.709503+0000","last_peered":"2024-04-15T15:16:08.709503+0000","last_clean":"2024-04-15T15:16:08.709503+0000","last_became_active":"2024-04-15T15:16:06.704688+0000","last_became_peered":"2024-04-15T15:16:06.704688+0000","last_unstale":"2024-04-15T15:16:08.709503+0000","last_undegraded":"2024-04-15T15:16:08.709503+0000","last_fullsized":"2024-04-15T15:16:08.709503+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:25:09.602767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.358983+0000","last_change":"2024-04-15T15:16:07.709428+0000","last_active":"2024-04-15T15:16:09.358983+0000","last_peered":"2024-04-15T15:16:09.358983+0000","last_clean":"2024-04-15T15:16:09.358983+0000","last_became_active":"2024-04-15T15:16:07.709164+0000","last_became_peered":"2024-04-15T15:16:07.709164+0000","last_unstale":"2024-04-15T15:16:09.358983+0000","last_undegraded":"2024-04-15T15:16:09.358983+0000","last_fullsized":"2024-04-15T15:16:09.358983+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:57:26.177964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:09.358898+0000","last_change":"2024-04-15T15:16:06.697149+0000","last_active":"2024-04-15T15:16:09.358898+0000","last_peered":"2024-04-15T15:16:09.358898+0000","last_clean":"2024-04-15T15:16:09.358898+0000","last_became_active":"2024-04-15T15:16:06.696879+0000","last_became_peered":"2024-04-15T15:16:06.696879+0000","last_unstale":"2024-04-15T15:16:09.358898+0000","last_undegraded":"2024-04-15T15:16:09.358898+0000","last_fullsized":"2024-04-15T15:16:09.358898+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:35:01.354434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.710256+0000","last_change":"2024-04-15T15:16:07.717293+0000","last_active":"2024-04-15T15:16:08.710256+0000","last_peered":"2024-04-15T15:16:08.710256+0000","last_clean":"2024-04-15T15:16:08.710256+0000","last_became_active":"2024-04-15T15:16:07.716988+0000","last_became_peered":"2024-04-15T15:16:07.716988+0000","last_unstale":"2024-04-15T15:16:08.710256+0000","last_undegraded":"2024-04-15T15:16:08.710256+0000","last_fullsized":"2024-04-15T15:16:08.710256+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:09:06.021853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.973388+0000","last_change":"2024-04-15T15:16:06.702502+0000","last_active":"2024-04-15T15:16:08.973388+0000","last_peered":"2024-04-15T15:16:08.973388+0000","last_clean":"2024-04-15T15:16:08.973388+0000","last_became_active":"2024-04-15T15:16:06.701301+0000","last_became_peered":"2024-04-15T15:16:06.701301+0000","last_unstale":"2024-04-15T15:16:08.973388+0000","last_undegraded":"2024-04-15T15:16:08.973388+0000","last_fullsized":"2024-04-15T15:16:08.973388+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:03:11.873554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.718815+0000","last_change":"2024-04-15T15:16:07.712592+0000","last_active":"2024-04-15T15:16:08.718815+0000","last_peered":"2024-04-15T15:16:08.718815+0000","last_clean":"2024-04-15T15:16:08.718815+0000","last_became_active":"2024-04-15T15:16:07.712213+0000","last_became_peered":"2024-04-15T15:16:07.712213+0000","last_unstale":"2024-04-15T15:16:08.718815+0000","last_undegraded":"2024-04-15T15:16:08.718815+0000","last_fullsized":"2024-04-15T15:16:08.718815+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:06.681976+0000","last_clean_scrub_stamp":"2024-04-15T15:16:06.681976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:51:12.622012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.709805+0000","last_change":"2024-04-15T15:16:06.700821+0000","last_active":"2024-04-15T15:16:08.709805+0000","last_peered":"2024-04-15T15:16:08.709805+0000","last_clean":"2024-04-15T15:16:08.709805+0000","last_became_active":"2024-04-15T15:16:06.700436+0000","last_became_peered":"2024-04-15T15:16:06.700436+0000","last_unstale":"2024-04-15T15:16:08.709805+0000","last_undegraded":"2024-04-15T15:16:08.709805+0000","last_fullsized":"2024-04-15T15:16:08.709805+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:05.677882+0000","last_clean_scrub_stamp":"2024-04-15T15:16:05.677882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:14:27.402622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":202,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:08.710011+0000","last_change":"2024-04-15T15:14:40.494279+0000","last_active":"2024-04-15T15:16:08.710011+0000","last_peered":"2024-04-15T15:16:08.710011+0000","last_clean":"2024-04-15T15:16:08.710011+0000","last_became_active":"2024-04-15T15:14:40.493404+0000","last_became_peered":"2024-04-15T15:14:40.493404+0000","last_unstale":"2024-04-15T15:16:08.710011+0000","last_undegraded":"2024-04-15T15:16:08.710011+0000","last_fullsized":"2024-04-15T15:16:08.710011+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:39.418508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:39.418508+0000","last_clean_scrub_stamp":"2024-04-15T15:14:39.418508+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:01:51.789246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28052,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343788,"statfs":{"total":96636764160,"available":96608038912,"internally_reserved":0,"allocated":1048576,"data_stored":782086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":442368,"data_stored":190618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27460,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344380,"statfs":{"total":96636764160,"available":96608645120,"internally_reserved":0,"allocated":446464,"data_stored":190148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27460,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344380,"statfs":{"total":96636764160,"available":96608645120,"internally_reserved":0,"allocated":446464,"data_stored":190640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574871,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27464,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344376,"statfs":{"total":96636764160,"available":96608641024,"internally_reserved":0,"allocated":454656,"data_stored":191682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574870,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":442368,"data_stored":190106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":442368,"data_stored":190126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343808,"statfs":{"total":96636764160,"available":96608059392,"internally_reserved":0,"allocated":1036288,"data_stored":780438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:16:15.436 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-15T15:16:15.436 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-15T15:16:15.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.580+0000 7fa5a907b640 1 Processor -- start 2024-04-15T15:16:15.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.580+0000 7fa5a907b640 1 -- start start 2024-04-15T15:16:15.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.580+0000 7fa5a907b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa5a415c650 0x7fa5a40b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:15.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.580+0000 7fa5a907b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5a40d02d0 con 0x7fa5a40d0a60 2024-04-15T15:16:15.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.581+0000 7fa5a907b640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5a415ca30 con 0x7fa5a40b36f0 2024-04-15T15:16:15.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.581+0000 7fa5a907b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fa5a40b26b0 con 0x7fa5a415c650 2024-04-15T15:16:15.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.581+0000 7fa5a2d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa5a415c650 0x7fa5a40b1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:15.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.581+0000 7fa5a2d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa5a415c650 0x7fa5a40b1d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56192/0 (socket says 172.21.15.72:56192) 2024-04-15T15:16:15.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.581+0000 7fa5a359e640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fa5a40d0a60 0x7fa5a40cca90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:34218/0 (socket says 172.21.15.72:34218) 2024-04-15T15:16:15.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.581+0000 7fa5a2d9d640 1 -- 172.21.15.72:0/4281072583 learned_addr learned my addr 172.21.15.72:0/4281072583 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:15.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.581+0000 7fa5a2d9d640 1 -- 172.21.15.72:0/4281072583 >> v1:172.21.15.175:6789/0 conn(0x7fa5a40b36f0 legacy=0x7fa5a40b5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:15.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.581+0000 7fa5a2d9d640 1 -- 172.21.15.72:0/4281072583 >> v1:172.21.15.72:6789/0 conn(0x7fa5a40d0a60 legacy=0x7fa5a40cca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:15.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.581+0000 7fa5a2d9d640 1 -- 172.21.15.72:0/4281072583 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5a40b2990 con 0x7fa5a415c650 2024-04-15T15:16:15.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.581+0000 7fa5a1d9b640 1 -- 172.21.15.72:0/4281072583 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 73394736 0 0) 0x7fa5a40d02d0 con 0x7fa5a40d0a60 2024-04-15T15:16:15.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.582+0000 7fa5a2d9d640 1 --2- 172.21.15.72:0/4281072583 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5a415c650 0x7fa5a40b1d40 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fa594009870 tx=0x7fa59402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a99a575f016ebb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:15.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.582+0000 7fa5a1d9b640 1 -- 172.21.15.72:0/4281072583 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa59402f840 con 0x7fa5a415c650 2024-04-15T15:16:15.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.582+0000 7fa5a1d9b640 1 -- 172.21.15.72:0/4281072583 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa59402f9e0 con 0x7fa5a415c650 2024-04-15T15:16:15.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.582+0000 7fa5a1d9b640 1 -- 172.21.15.72:0/4281072583 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa59402fdb0 con 0x7fa5a415c650 2024-04-15T15:16:15.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.582+0000 7fa5a907b640 1 -- 172.21.15.72:0/4281072583 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5a415c650 msgr2=0x7fa5a40b1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:15.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.582+0000 7fa5a907b640 1 --2- 172.21.15.72:0/4281072583 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5a415c650 0x7fa5a40b1d40 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fa594009870 tx=0x7fa59402ba20 comp rx=0 tx=0).stop 2024-04-15T15:16:15.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.582+0000 7fa5a907b640 1 -- 172.21.15.72:0/4281072583 shutdown_connections 2024-04-15T15:16:15.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.582+0000 7fa5a907b640 1 --2- 172.21.15.72:0/4281072583 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5a415c650 0x7fa5a40b1d40 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.582+0000 7fa5a907b640 1 -- 172.21.15.72:0/4281072583 >> 172.21.15.72:0/4281072583 conn(0x7fa5a40ceec0 msgr2=0x7fa5a40cf2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:15.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.582+0000 7fa5a907b640 1 -- 172.21.15.72:0/4281072583 shutdown_connections 2024-04-15T15:16:15.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.582+0000 7fa5a907b640 1 -- 172.21.15.72:0/4281072583 wait complete. 2024-04-15T15:16:15.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.584+0000 7fa5a907b640 1 Processor -- start 2024-04-15T15:16:15.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.584+0000 7fa5a907b640 1 -- start start 2024-04-15T15:16:15.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.585+0000 7fa5a907b640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa5a40b36f0 0x7fa5a414c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:15.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.585+0000 7fa5a907b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5a40d0a60 0x7fa5a40b6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:15.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.585+0000 7fa5a907b640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa5a415c650 0x7fa5a40b70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:15.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.585+0000 7fa5a907b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a40b59a0 con 0x7fa5a40d0a60 2024-04-15T15:16:15.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.585+0000 7fa5a907b640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fa5a40b2c70 con 0x7fa5a415c650 2024-04-15T15:16:15.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.585+0000 7fa5a907b640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fa5a40b2280 con 0x7fa5a40b36f0 2024-04-15T15:16:15.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.585+0000 7fa5a259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5a40d0a60 0x7fa5a40b6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:15.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.585+0000 7fa5a259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5a40d0a60 0x7fa5a40b6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56206/0 (socket says 172.21.15.72:56206) 2024-04-15T15:16:15.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.585+0000 7fa5a259c640 1 -- 172.21.15.72:0/2900765652 learned_addr learned my addr 172.21.15.72:0/2900765652 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:15.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.585+0000 7fa5a259c640 1 -- 172.21.15.72:0/2900765652 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa5a40b36f0 msgr2=0x7fa5a414c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:15.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.585+0000 7fa5a259c640 1 --2- 172.21.15.72:0/2900765652 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa5a40b36f0 0x7fa5a414c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.585+0000 7fa5a259c640 1 -- 172.21.15.72:0/2900765652 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa5a415c650 msgr2=0x7fa5a40b70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:15.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.585+0000 7fa5a359e640 1 --2- 172.21.15.72:0/2900765652 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa5a415c650 0x7fa5a40b70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:15.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.585+0000 7fa5a259c640 1 --2- 172.21.15.72:0/2900765652 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa5a415c650 0x7fa5a40b70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.585+0000 7fa5a259c640 1 -- 172.21.15.72:0/2900765652 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5a40ca0c0 con 0x7fa5a40d0a60 2024-04-15T15:16:15.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.585+0000 7fa5a359e640 1 --2- 172.21.15.72:0/2900765652 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa5a415c650 0x7fa5a40b70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:15.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.585+0000 7fa5a259c640 1 --2- 172.21.15.72:0/2900765652 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5a40d0a60 0x7fa5a40b6b70 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fa59800c400 tx=0x7fa59800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:15.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.585+0000 7fa58b7fe640 1 -- 172.21.15.72:0/2900765652 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa598017070 con 0x7fa5a40d0a60 2024-04-15T15:16:15.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.585+0000 7fa58b7fe640 1 -- 172.21.15.72:0/2900765652 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5980052d0 con 0x7fa5a40d0a60 2024-04-15T15:16:15.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.585+0000 7fa5a907b640 1 -- 172.21.15.72:0/2900765652 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5a40ca3a0 con 0x7fa5a40d0a60 2024-04-15T15:16:15.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.585+0000 7fa58b7fe640 1 -- 172.21.15.72:0/2900765652 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa598003830 con 0x7fa5a40d0a60 2024-04-15T15:16:15.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.585+0000 7fa5a907b640 1 -- 172.21.15.72:0/2900765652 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5a40ca960 con 0x7fa5a40d0a60 2024-04-15T15:16:15.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.586+0000 7fa5a907b640 1 -- 172.21.15.72:0/2900765652 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5a4075c50 con 0x7fa5a40d0a60 2024-04-15T15:16:15.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.587+0000 7fa58b7fe640 1 -- 172.21.15.72:0/2900765652 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa59801a070 con 0x7fa5a40d0a60 2024-04-15T15:16:15.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.587+0000 7fa58b7fe640 1 --2- 172.21.15.72:0/2900765652 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa59006c8f0 0x7fa59006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:15.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.588+0000 7fa5a2d9d640 1 --2- 172.21.15.72:0/2900765652 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa59006c8f0 0x7fa59006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:15.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.588+0000 7fa58b7fe640 1 -- 172.21.15.72:0/2900765652 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fa59808b020 con 0x7fa5a40d0a60 2024-04-15T15:16:15.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.588+0000 7fa5a2d9d640 1 --2- 172.21.15.72:0/2900765652 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa59006c8f0 0x7fa59006edb0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa594036000 tx=0x7fa594039040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:15.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.592+0000 7fa58b7fe640 1 -- 172.21.15.72:0/2900765652 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa5980592c0 con 0x7fa5a40d0a60 2024-04-15T15:16:15.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.839+0000 7fa5a907b640 1 -- 172.21.15.72:0/2900765652 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fa5a40d0010 con 0x7fa5a40d0a60 2024-04-15T15:16:15.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.841+0000 7fa58b7fe640 1 -- 172.21.15.72:0/2900765652 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) ==== 190+0+147 (secure 0 0 0) 0x7fa598058bc0 con 0x7fa5a40d0a60 2024-04-15T15:16:15.843 INFO:teuthology.orchestra.run.smithi072.stdout:[client.0] 2024-04-15T15:16:15.844 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQAxRB1mycl8LBAA87k203Ws8lFHsc/7l5rqbg== 2024-04-15T15:16:15.844 INFO:teuthology.orchestra.run.smithi072.stdout: caps mds = "allow" 2024-04-15T15:16:15.844 INFO:teuthology.orchestra.run.smithi072.stdout: caps mon = "allow r" 2024-04-15T15:16:15.844 INFO:teuthology.orchestra.run.smithi072.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-15T15:16:15.844 INFO:teuthology.orchestra.run.smithi072.stderr:updated caps for client.0 2024-04-15T15:16:15.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.843+0000 7fa5a907b640 1 -- 172.21.15.72:0/2900765652 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa59006c8f0 msgr2=0x7fa59006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:15.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.843+0000 7fa5a907b640 1 --2- 172.21.15.72:0/2900765652 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa59006c8f0 0x7fa59006edb0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa594036000 tx=0x7fa594039040 comp rx=0 tx=0).stop 2024-04-15T15:16:15.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.843+0000 7fa5a907b640 1 -- 172.21.15.72:0/2900765652 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5a40d0a60 msgr2=0x7fa5a40b6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:15.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.843+0000 7fa5a907b640 1 --2- 172.21.15.72:0/2900765652 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5a40d0a60 0x7fa5a40b6b70 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fa59800c400 tx=0x7fa59800c8c0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.843+0000 7fa5a907b640 1 -- 172.21.15.72:0/2900765652 shutdown_connections 2024-04-15T15:16:15.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.843+0000 7fa5a907b640 1 --2- 172.21.15.72:0/2900765652 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa59006c8f0 0x7fa59006edb0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.843+0000 7fa5a907b640 1 --2- 172.21.15.72:0/2900765652 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa5a415c650 0x7fa5a40b70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.843+0000 7fa5a907b640 1 --2- 172.21.15.72:0/2900765652 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5a40d0a60 0x7fa5a40b6b70 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.843+0000 7fa5a907b640 1 --2- 172.21.15.72:0/2900765652 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa5a40b36f0 0x7fa5a414c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.843+0000 7fa5a907b640 1 -- 172.21.15.72:0/2900765652 >> 172.21.15.72:0/2900765652 conn(0x7fa5a40ceec0 msgr2=0x7fa5a415b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:15.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.843+0000 7fa5a907b640 1 -- 172.21.15.72:0/2900765652 shutdown_connections 2024-04-15T15:16:15.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.843+0000 7fa5a907b640 1 -- 172.21.15.72:0/2900765652 wait complete. 2024-04-15T15:16:15.857 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:16:15.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.957+0000 7f6ade115640 1 Processor -- start 2024-04-15T15:16:15.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.957+0000 7f6ade115640 1 -- start start 2024-04-15T15:16:15.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.958+0000 7f6ade115640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6ad815c120 0x7f6ad815f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:15.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.958+0000 7f6ade115640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f6ad815c500 0x7f6ad80cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:15.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.958+0000 7f6ade115640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ad81624e0 con 0x7f6ad8070760 2024-04-15T15:16:15.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.958+0000 7f6ade115640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f6ad81627c0 con 0x7f6ad815c500 2024-04-15T15:16:15.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.958+0000 7f6ade115640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f6ad8162aa0 con 0x7f6ad815c120 2024-04-15T15:16:15.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.958+0000 7f6ad7fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f6ad8070760 0x7f6ad80c68d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:34234/0 (socket says 172.21.15.72:34234) 2024-04-15T15:16:15.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.958+0000 7f6ad7fff640 1 -- 172.21.15.72:0/1650416379 learned_addr learned my addr 172.21.15.72:0/1650416379 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:15.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.958+0000 7f6ad67fc640 1 -- 172.21.15.72:0/1650416379 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2316149721 0 0) 0x7f6ad81624e0 con 0x7f6ad8070760 2024-04-15T15:16:15.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.958+0000 7f6ad67fc640 1 -- 172.21.15.72:0/1650416379 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ac0003520 con 0x7f6ad8070760 2024-04-15T15:16:15.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.958+0000 7f6ad77fe640 1 --2- 172.21.15.72:0/1650416379 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6ad815c120 0x7f6ad815f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:15.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.958+0000 7f6ad6ffd640 1 --2- 172.21.15.72:0/1650416379 >> v2:172.21.15.175:3300/0 conn(0x7f6ad815c500 0x7f6ad80cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:15.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.959+0000 7f6ad67fc640 1 -- 172.21.15.72:0/1650416379 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1837929647 0 0) 0x7f6ac0003520 con 0x7f6ad8070760 2024-04-15T15:16:15.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.959+0000 7f6ad67fc640 1 -- 172.21.15.72:0/1650416379 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6ad815c120 msgr2=0x7f6ad815f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:15.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.959+0000 7f6ad67fc640 1 --2- 172.21.15.72:0/1650416379 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6ad815c120 0x7f6ad815f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.959+0000 7f6ad67fc640 1 -- 172.21.15.72:0/1650416379 >> v2:172.21.15.175:3300/0 conn(0x7f6ad815c500 msgr2=0x7f6ad80cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:15.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.959+0000 7f6ad67fc640 1 --2- 172.21.15.72:0/1650416379 >> v2:172.21.15.175:3300/0 conn(0x7f6ad815c500 0x7f6ad80cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.959+0000 7f6ad67fc640 1 -- 172.21.15.72:0/1650416379 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ad8162d80 con 0x7f6ad8070760 2024-04-15T15:16:15.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.959+0000 7f6ad67fc640 1 -- 172.21.15.72:0/1650416379 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f6ad0001670 con 0x7f6ad8070760 2024-04-15T15:16:15.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.959+0000 7f6ad67fc640 1 -- 172.21.15.72:0/1650416379 >> v1:172.21.15.72:6789/0 conn(0x7f6ad8070760 legacy=0x7f6ad80c68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:15.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.959+0000 7f6ad67fc640 1 --2- 172.21.15.72:0/1650416379 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ac00039b0 0x7f6ac0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:15.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.959+0000 7f6ad67fc640 1 --2- 172.21.15.72:0/1650416379 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ac0004800 0x7f6ac0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:15.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.959+0000 7f6ad67fc640 1 --2- 172.21.15.72:0/1650416379 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6ac00051c0 0x7f6ac000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:15.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.959+0000 7f6ad67fc640 1 -- 172.21.15.72:0/1650416379 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad81627c0 con 0x7f6ac0004800 2024-04-15T15:16:15.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.959+0000 7f6ad67fc640 1 -- 172.21.15.72:0/1650416379 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad8162aa0 con 0x7f6ac00039b0 2024-04-15T15:16:15.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.959+0000 7f6ad67fc640 1 -- 172.21.15.72:0/1650416379 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f6ac000c000 con 0x7f6ac00051c0 2024-04-15T15:16:15.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.959+0000 7f6ad77fe640 1 --2- 172.21.15.72:0/1650416379 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6ad815c120 0x7f6ad815f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:15.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.959+0000 7f6ad6ffd640 1 --2- 172.21.15.72:0/1650416379 >> v2:172.21.15.175:3300/0 conn(0x7f6ad815c500 0x7f6ad80cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:15.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.959+0000 7f6ad6ffd640 1 --2- 172.21.15.72:0/1650416379 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ac0004800 0x7f6ac0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:15.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.959+0000 7f6ad7fff640 1 --2- 172.21.15.72:0/1650416379 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6ac00051c0 0x7f6ac000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:15.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.959+0000 7f6ad77fe640 1 --2- 172.21.15.72:0/1650416379 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ac00039b0 0x7f6ac0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:15.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.959+0000 7f6ad6ffd640 1 -- 172.21.15.72:0/1650416379 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6ac00051c0 msgr2=0x7f6ac000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:15.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.959+0000 7f6ad6ffd640 1 --2- 172.21.15.72:0/1650416379 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6ac00051c0 0x7f6ac000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.959+0000 7f6ad6ffd640 1 -- 172.21.15.72:0/1650416379 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ac00039b0 msgr2=0x7f6ac0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:15.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.959+0000 7f6ad6ffd640 1 --2- 172.21.15.72:0/1650416379 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ac00039b0 0x7f6ac0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.959+0000 7f6ad6ffd640 1 -- 172.21.15.72:0/1650416379 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ac0003520 con 0x7f6ac0004800 2024-04-15T15:16:15.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.960+0000 7f6ad6ffd640 1 --2- 172.21.15.72:0/1650416379 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ac0004800 0x7f6ac0004c80 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f6acc0098d0 tx=0x7f6ac0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa12bf9e617d51f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:15.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.960+0000 7f6ad67fc640 1 -- 172.21.15.72:0/1650416379 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6acc034070 con 0x7f6ac0004800 2024-04-15T15:16:15.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.960+0000 7f6ad67fc640 1 -- 172.21.15.72:0/1650416379 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6acc030680 con 0x7f6ac0004800 2024-04-15T15:16:15.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.960+0000 7f6ad67fc640 1 -- 172.21.15.72:0/1650416379 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6acc00f040 con 0x7f6ac0004800 2024-04-15T15:16:15.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.961+0000 7f6ade115640 1 -- 172.21.15.72:0/1650416379 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ac0004800 msgr2=0x7f6ac0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:15.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.961+0000 7f6ade115640 1 --2- 172.21.15.72:0/1650416379 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ac0004800 0x7f6ac0004c80 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f6acc0098d0 tx=0x7f6ac0005830 comp rx=0 tx=0).stop 2024-04-15T15:16:15.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.961+0000 7f6ad7fff640 1 -- 172.21.15.72:0/1650416379 reap_dead start 2024-04-15T15:16:15.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.961+0000 7f6ade115640 1 -- 172.21.15.72:0/1650416379 shutdown_connections 2024-04-15T15:16:15.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.961+0000 7f6ade115640 1 -- 172.21.15.72:0/1650416379 >> 172.21.15.72:0/1650416379 conn(0x7f6ad80cd4a0 msgr2=0x7f6ad80cd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:15.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.961+0000 7f6ade115640 1 -- 172.21.15.72:0/1650416379 shutdown_connections 2024-04-15T15:16:15.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.961+0000 7f6ade115640 1 -- 172.21.15.72:0/1650416379 wait complete. 2024-04-15T15:16:15.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.962+0000 7f6ade115640 1 Processor -- start 2024-04-15T15:16:15.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.962+0000 7f6ade115640 1 -- start start 2024-04-15T15:16:15.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.963+0000 7f6ade115640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ac0004800 0x7f6ad80c7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:15.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.963+0000 7f6ade115640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6ad81f5030 0x7f6ad81593a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:15.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.963+0000 7f6ade115640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ad80b4ec0 0x7f6ad806f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:15.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.963+0000 7f6ade115640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad80cc060 con 0x7f6ac0004800 2024-04-15T15:16:15.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.963+0000 7f6ade115640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f6ad80cbee0 con 0x7f6ad80b4ec0 2024-04-15T15:16:15.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.963+0000 7f6ade115640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f6ad80cc1e0 con 0x7f6ad81f5030 2024-04-15T15:16:15.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.963+0000 7f6ad77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ac0004800 0x7f6ad80c7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:15.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.963+0000 7f6ad77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ac0004800 0x7f6ad80c7e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56230/0 (socket says 172.21.15.72:56230) 2024-04-15T15:16:15.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.963+0000 7f6ad77fe640 1 -- 172.21.15.72:0/594043673 learned_addr learned my addr 172.21.15.72:0/594043673 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:15.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.963+0000 7f6ad77fe640 1 -- 172.21.15.72:0/594043673 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6ad81f5030 msgr2=0x7f6ad81593a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:15.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.963+0000 7f6ad77fe640 1 --2- 172.21.15.72:0/594043673 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6ad81f5030 0x7f6ad81593a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.963+0000 7f6ad77fe640 1 -- 172.21.15.72:0/594043673 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ad80b4ec0 msgr2=0x7f6ad806f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:15.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.963+0000 7f6ad7fff640 1 --2- 172.21.15.72:0/594043673 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ad80b4ec0 0x7f6ad806f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:15.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.963+0000 7f6ad77fe640 1 --2- 172.21.15.72:0/594043673 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ad80b4ec0 0x7f6ad806f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.963+0000 7f6ad77fe640 1 -- 172.21.15.72:0/594043673 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ad81598e0 con 0x7f6ac0004800 2024-04-15T15:16:15.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.963+0000 7f6ad77fe640 1 --2- 172.21.15.72:0/594043673 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ac0004800 0x7f6ad80c7e50 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f6ac8004670 tx=0x7f6ac8007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:15.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.963+0000 7f6abffff640 1 -- 172.21.15.72:0/594043673 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac8015680 con 0x7f6ac0004800 2024-04-15T15:16:15.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.963+0000 7f6abffff640 1 -- 172.21.15.72:0/594043673 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ac801a070 con 0x7f6ac0004800 2024-04-15T15:16:15.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.963+0000 7f6ade115640 1 -- 172.21.15.72:0/594043673 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ad80c8510 con 0x7f6ac0004800 2024-04-15T15:16:15.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.963+0000 7f6abffff640 1 -- 172.21.15.72:0/594043673 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac8015a40 con 0x7f6ac0004800 2024-04-15T15:16:15.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.964+0000 7f6ade115640 1 -- 172.21.15.72:0/594043673 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ad80b3970 con 0x7f6ac0004800 2024-04-15T15:16:15.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.965+0000 7f6ade115640 1 -- 172.21.15.72:0/594043673 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ad80689e0 con 0x7f6ac0004800 2024-04-15T15:16:15.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.965+0000 7f6abffff640 1 -- 172.21.15.72:0/594043673 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6ac8015be0 con 0x7f6ac0004800 2024-04-15T15:16:15.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.965+0000 7f6abffff640 1 --2- 172.21.15.72:0/594043673 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6ac406c920 0x7f6ac406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:15.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.965+0000 7f6abffff640 1 -- 172.21.15.72:0/594043673 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f6ac808ca70 con 0x7f6ac0004800 2024-04-15T15:16:15.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.966+0000 7f6ad6ffd640 1 --2- 172.21.15.72:0/594043673 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6ac406c920 0x7f6ac406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:15.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.966+0000 7f6ad6ffd640 1 --2- 172.21.15.72:0/594043673 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6ac406c920 0x7f6ac406ede0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6acc036000 tx=0x7f6acc030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:15.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:15.970+0000 7f6abffff640 1 -- 172.21.15.72:0/594043673 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6ac8010050 con 0x7f6ac0004800 2024-04-15T15:16:16.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.181+0000 7f6ade115640 1 -- 172.21.15.72:0/594043673 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ad80cec90 con 0x7f6ac0004800 2024-04-15T15:16:16.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.181+0000 7f6abffff640 1 -- 172.21.15.72:0/594043673 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5184 (secure 0 0 0) 0x7f6ac805ad10 con 0x7f6ac0004800 2024-04-15T15:16:16.184 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:16:16.184 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4883,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3363272836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3363272836},{"type":"v1","addr":"172.21.15.175:6834","nonce":3363272836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5134,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6835/1059187505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":1059187505},{"type":"v1","addr":"172.21.15.175:6835","nonce":1059187505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5556,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/690132275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":690132275},{"type":"v1","addr":"172.21.15.72:6837","nonce":690132275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:16:07.050078+0000","modified":"2024-04-15T15:16:09.052120+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5559},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/4270793613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4270793613},{"type":"v1","addr":"172.21.15.72:6836","nonce":4270793613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5559,"qdb_cluster":[5559]},"id":2}]} 2024-04-15T15:16:16.184 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 16 2024-04-15T15:16:16.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.183+0000 7f6ade115640 1 -- 172.21.15.72:0/594043673 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6ac406c920 msgr2=0x7f6ac406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:16.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.183+0000 7f6ade115640 1 --2- 172.21.15.72:0/594043673 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6ac406c920 0x7f6ac406ede0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6acc036000 tx=0x7f6acc030000 comp rx=0 tx=0).stop 2024-04-15T15:16:16.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.183+0000 7f6ade115640 1 -- 172.21.15.72:0/594043673 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ac0004800 msgr2=0x7f6ad80c7e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:16.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.183+0000 7f6ade115640 1 --2- 172.21.15.72:0/594043673 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ac0004800 0x7f6ad80c7e50 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f6ac8004670 tx=0x7f6ac8007700 comp rx=0 tx=0).stop 2024-04-15T15:16:16.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.183+0000 7f6ade115640 1 -- 172.21.15.72:0/594043673 shutdown_connections 2024-04-15T15:16:16.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.183+0000 7f6ade115640 1 --2- 172.21.15.72:0/594043673 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6ac406c920 0x7f6ac406ede0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.183+0000 7f6ade115640 1 --2- 172.21.15.72:0/594043673 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6ad80b4ec0 0x7f6ad806f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.183+0000 7f6ade115640 1 --2- 172.21.15.72:0/594043673 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6ad81f5030 0x7f6ad81593a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.183+0000 7f6ade115640 1 --2- 172.21.15.72:0/594043673 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6ac0004800 0x7f6ad80c7e50 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.183+0000 7f6ade115640 1 -- 172.21.15.72:0/594043673 >> 172.21.15.72:0/594043673 conn(0x7f6ad80cd4a0 msgr2=0x7f6ad80b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:16.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.184+0000 7f6ade115640 1 -- 172.21.15.72:0/594043673 shutdown_connections 2024-04-15T15:16:16.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.184+0000 7f6ade115640 1 -- 172.21.15.72:0/594043673 wait complete. 2024-04-15T15:16:16.600 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-15T15:16:07.050078+0000', 'modified': '2024-04-15T15:16:09.052120+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5559}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5559': {'gid': 5559, 'name': 'a', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.72:6836/4270793613', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 4270793613}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 4270793613}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5559, 'qdb_cluster': [5559]} 2024-04-15T15:16:16.601 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-15T15:16:16.601 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-15T15:16:16.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.704+0000 7f353a84a640 1 Processor -- start 2024-04-15T15:16:16.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.704+0000 7f353a84a640 1 -- start start 2024-04-15T15:16:16.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.704+0000 7f353a84a640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f35340ce2f0 0x7f353415c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:16.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.704+0000 7f353a84a640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f35340d08f0 0x7f35340cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:16.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.705+0000 7f353a84a640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f353406bbb0 con 0x7f35340ce6d0 2024-04-15T15:16:16.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.705+0000 7f353a84a640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f353406b960 con 0x7f35340ce2f0 2024-04-15T15:16:16.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.705+0000 7f353a84a640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f353406be90 con 0x7f35340d08f0 2024-04-15T15:16:16.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.705+0000 7f35337fe640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f35340d08f0 0x7f35340cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:16.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.705+0000 7f3538de7640 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f35340ce6d0 0x7f3534155600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.72:49502/0 (socket says 172.21.15.72:49502) 2024-04-15T15:16:16.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.705+0000 7f3538de7640 1 -- 172.21.15.72:0/33486470 learned_addr learned my addr 172.21.15.72:0/33486470 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:16.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.705+0000 7f35337fe640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f35340d08f0 0x7f35340cc0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3301/0 says I am v2:172.21.15.72:47644/0 (socket says 172.21.15.72:47644) 2024-04-15T15:16:16.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.705+0000 7f3533fff640 1 --2- 172.21.15.72:0/33486470 >> v2:172.21.15.175:3300/0 conn(0x7f35340ce2f0 0x7f353415c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:16.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.705+0000 7f35337fe640 1 -- 172.21.15.72:0/33486470 >> v2:172.21.15.175:3300/0 conn(0x7f35340ce2f0 msgr2=0x7f353415c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:16.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.705+0000 7f35337fe640 1 --2- 172.21.15.72:0/33486470 >> v2:172.21.15.175:3300/0 conn(0x7f35340ce2f0 0x7f353415c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.705+0000 7f35337fe640 1 -- 172.21.15.72:0/33486470 >> v1:172.21.15.175:6789/0 conn(0x7f35340ce6d0 legacy=0x7f3534155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:16.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.705+0000 7f35337fe640 1 -- 172.21.15.72:0/33486470 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f353406c170 con 0x7f35340d08f0 2024-04-15T15:16:16.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.705+0000 7f3533fff640 1 --2- 172.21.15.72:0/33486470 >> v2:172.21.15.175:3300/0 conn(0x7f35340ce2f0 0x7f353415c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:16.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.706+0000 7f35337fe640 1 --2- 172.21.15.72:0/33486470 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f35340d08f0 0x7f35340cc0b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f35280097a0 tx=0x7f352802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c53af1c3acc447cf server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:16.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.706+0000 7f3532ffd640 1 -- 172.21.15.72:0/33486470 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3528035020 con 0x7f35340d08f0 2024-04-15T15:16:16.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.706+0000 7f3532ffd640 1 -- 172.21.15.72:0/33486470 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3528031810 con 0x7f35340d08f0 2024-04-15T15:16:16.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.706+0000 7f3532ffd640 1 -- 172.21.15.72:0/33486470 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3528031be0 con 0x7f35340d08f0 2024-04-15T15:16:16.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.707+0000 7f353a84a640 1 -- 172.21.15.72:0/33486470 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f35340d08f0 msgr2=0x7f35340cc0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:16.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.707+0000 7f353a84a640 1 --2- 172.21.15.72:0/33486470 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f35340d08f0 0x7f35340cc0b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f35280097a0 tx=0x7f352802b8b0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.707+0000 7f353a84a640 1 -- 172.21.15.72:0/33486470 shutdown_connections 2024-04-15T15:16:16.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.707+0000 7f353a84a640 1 --2- 172.21.15.72:0/33486470 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f35340d08f0 0x7f35340cc0b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.707+0000 7f353a84a640 1 --2- 172.21.15.72:0/33486470 >> v2:172.21.15.175:3300/0 conn(0x7f35340ce2f0 0x7f353415c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.707+0000 7f353a84a640 1 -- 172.21.15.72:0/33486470 >> 172.21.15.72:0/33486470 conn(0x7f35340b31d0 msgr2=0x7f35340b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:16.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.707+0000 7f353a84a640 1 -- 172.21.15.72:0/33486470 shutdown_connections 2024-04-15T15:16:16.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.707+0000 7f353a84a640 1 -- 172.21.15.72:0/33486470 wait complete. 2024-04-15T15:16:16.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.708+0000 7f353a84a640 1 Processor -- start 2024-04-15T15:16:16.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.708+0000 7f353a84a640 1 -- start start 2024-04-15T15:16:16.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.709+0000 7f353a84a640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f35340ce2f0 0x7f3534151e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:16.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.709+0000 7f353a84a640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35340ce6d0 0x7f35340af800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:16.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.709+0000 7f353a84a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35340d08f0 0x7f35340afd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:16.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.709+0000 7f353a84a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f3534155d10 con 0x7f35340d08f0 2024-04-15T15:16:16.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.709+0000 7f353a84a640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f353415ca70 con 0x7f35340ce6d0 2024-04-15T15:16:16.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.709+0000 7f353a84a640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f3534155e90 con 0x7f35340ce2f0 2024-04-15T15:16:16.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.709+0000 7f3538de7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35340d08f0 0x7f35340afd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:16.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.709+0000 7f3538de7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35340d08f0 0x7f35340afd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56232/0 (socket says 172.21.15.72:56232) 2024-04-15T15:16:16.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.709+0000 7f3538de7640 1 -- 172.21.15.72:0/853562672 learned_addr learned my addr 172.21.15.72:0/853562672 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:16.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.709+0000 7f3533fff640 1 --2- 172.21.15.72:0/853562672 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f35340ce2f0 0x7f3534151e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:16.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.709+0000 7f35337fe640 1 --2- 172.21.15.72:0/853562672 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35340ce6d0 0x7f35340af800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:16.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.709+0000 7f3538de7640 1 -- 172.21.15.72:0/853562672 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f35340ce2f0 msgr2=0x7f3534151e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:16.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.710+0000 7f3538de7640 1 --2- 172.21.15.72:0/853562672 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f35340ce2f0 0x7f3534151e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.710+0000 7f3538de7640 1 -- 172.21.15.72:0/853562672 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35340ce6d0 msgr2=0x7f35340af800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:16.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.710+0000 7f3538de7640 1 --2- 172.21.15.72:0/853562672 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35340ce6d0 0x7f35340af800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.710+0000 7f3538de7640 1 -- 172.21.15.72:0/853562672 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3534153fc0 con 0x7f35340d08f0 2024-04-15T15:16:16.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.710+0000 7f3533fff640 1 --2- 172.21.15.72:0/853562672 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f35340ce2f0 0x7f3534151e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:16.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.710+0000 7f35337fe640 1 --2- 172.21.15.72:0/853562672 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35340ce6d0 0x7f35340af800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:16.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.710+0000 7f3538de7640 1 --2- 172.21.15.72:0/853562672 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35340d08f0 0x7f35340afd40 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f352c00cbc0 tx=0x7f352c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:16.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.710+0000 7f3530ff9640 1 -- 172.21.15.72:0/853562672 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352c017070 con 0x7f35340d08f0 2024-04-15T15:16:16.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.710+0000 7f3530ff9640 1 -- 172.21.15.72:0/853562672 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f352c00ad40 con 0x7f35340d08f0 2024-04-15T15:16:16.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.710+0000 7f353a84a640 1 -- 172.21.15.72:0/853562672 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35341542a0 con 0x7f35340d08f0 2024-04-15T15:16:16.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.710+0000 7f3530ff9640 1 -- 172.21.15.72:0/853562672 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352c012600 con 0x7f35340d08f0 2024-04-15T15:16:16.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.710+0000 7f353a84a640 1 -- 172.21.15.72:0/853562672 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3534154860 con 0x7f35340d08f0 2024-04-15T15:16:16.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.711+0000 7f3530ff9640 1 -- 172.21.15.72:0/853562672 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f352c01a070 con 0x7f35340d08f0 2024-04-15T15:16:16.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.712+0000 7f3530ff9640 1 --2- 172.21.15.72:0/853562672 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f352006c920 0x7f352006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:16.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.712+0000 7f3530ff9640 1 -- 172.21.15.72:0/853562672 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f352c08b890 con 0x7f35340d08f0 2024-04-15T15:16:16.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.712+0000 7f3533fff640 1 --2- 172.21.15.72:0/853562672 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f352006c920 0x7f352006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:16.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.712+0000 7f3533fff640 1 --2- 172.21.15.72:0/853562672 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f352006c920 0x7f352006ede0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3524009ac0 tx=0x7f3524008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:16.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.713+0000 7f353d437740 1 Processor -- start 2024-04-15T15:16:16.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.713+0000 7f353d437740 1 -- start start 2024-04-15T15:16:16.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.714+0000 7f353d437740 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x555dbd372990 0x555dbd370550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:16.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.714+0000 7f353d437740 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x555dbd3521d0 0x555dbd329ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:16.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.714+0000 7f353d437740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555dbd32a400 0x555dbd354ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:16.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.714+0000 7f353d437740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x555dbd28f9c0 con 0x555dbd32a400 2024-04-15T15:16:16.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.714+0000 7f353d437740 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x555dbd236650 con 0x555dbd3521d0 2024-04-15T15:16:16.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.714+0000 7f353d437740 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x555dbd28b980 con 0x555dbd372990 2024-04-15T15:16:16.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.714+0000 7f3538de7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555dbd32a400 0x555dbd354ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:16.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.714+0000 7f3538de7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555dbd32a400 0x555dbd354ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56246/0 (socket says 172.21.15.72:56246) 2024-04-15T15:16:16.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.714+0000 7f3538de7640 1 -- 172.21.15.72:0/2261209972 learned_addr learned my addr 172.21.15.72:0/2261209972 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:16.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.714+0000 7f35337fe640 1 --2- 172.21.15.72:0/2261209972 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x555dbd3521d0 0x555dbd329ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:16.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.714+0000 7f3533fff640 1 --2- 172.21.15.72:0/2261209972 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x555dbd372990 0x555dbd370550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:16.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.714+0000 7f3538de7640 1 -- 172.21.15.72:0/2261209972 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x555dbd372990 msgr2=0x555dbd370550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:16.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.714+0000 7f3538de7640 1 --2- 172.21.15.72:0/2261209972 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x555dbd372990 0x555dbd370550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.714+0000 7f3538de7640 1 -- 172.21.15.72:0/2261209972 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x555dbd3521d0 msgr2=0x555dbd329ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:16.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.714+0000 7f3538de7640 1 --2- 172.21.15.72:0/2261209972 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x555dbd3521d0 0x555dbd329ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.714+0000 7f3538de7640 1 -- 172.21.15.72:0/2261209972 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555dbcfa22d0 con 0x555dbd32a400 2024-04-15T15:16:16.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.714+0000 7f3538de7640 1 --2- 172.21.15.72:0/2261209972 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555dbd32a400 0x555dbd354ba0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f352c00afd0 tx=0x7f352c05d870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f99e89ac484ae38c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:16.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.714+0000 7f3519ffb640 1 -- 172.21.15.72:0/2261209972 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352c057870 con 0x555dbd32a400 2024-04-15T15:16:16.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.714+0000 7f3519ffb640 1 -- 172.21.15.72:0/2261209972 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f352c055be0 con 0x555dbd32a400 2024-04-15T15:16:16.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.714+0000 7f3519ffb640 1 -- 172.21.15.72:0/2261209972 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352c061e10 con 0x555dbd32a400 2024-04-15T15:16:16.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.715+0000 7f353d437740 1 -- 172.21.15.72:0/2261209972 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555dbd32a400 msgr2=0x555dbd354ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:16.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.715+0000 7f353d437740 1 --2- 172.21.15.72:0/2261209972 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555dbd32a400 0x555dbd354ba0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f352c00afd0 tx=0x7f352c05d870 comp rx=0 tx=0).stop 2024-04-15T15:16:16.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.715+0000 7f353d437740 1 -- 172.21.15.72:0/2261209972 shutdown_connections 2024-04-15T15:16:16.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.715+0000 7f353d437740 1 --2- 172.21.15.72:0/2261209972 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555dbd32a400 0x555dbd354ba0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.715+0000 7f353d437740 1 --2- 172.21.15.72:0/2261209972 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x555dbd3521d0 0x555dbd329ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.715+0000 7f353d437740 1 --2- 172.21.15.72:0/2261209972 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x555dbd372990 0x555dbd370550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.715+0000 7f353d437740 1 -- 172.21.15.72:0/2261209972 >> 172.21.15.72:0/2261209972 conn(0x555dbd327020 msgr2=0x555dbd36fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:16.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.715+0000 7f353d437740 1 -- 172.21.15.72:0/2261209972 shutdown_connections 2024-04-15T15:16:16.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.715+0000 7f353d437740 1 -- 172.21.15.72:0/2261209972 wait complete. 2024-04-15T15:16:16.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.715+0000 7f353d437740 1 -- start start 2024-04-15T15:16:16.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.716+0000 7f353d437740 1 Processor -- start 2024-04-15T15:16:16.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.716+0000 7f353d437740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555dbd3521d0 0x555dbd350e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:16.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.716+0000 7f353d437740 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x555dbd372990 0x555dbd37be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:16.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.716+0000 7f353d437740 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x555dbd384fe0 0x555dbd376650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:16.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.716+0000 7f353d437740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x555dbd329320 con 0x555dbd3521d0 2024-04-15T15:16:16.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.716+0000 7f353d437740 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x555dbd249a10 con 0x555dbd372990 2024-04-15T15:16:16.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.716+0000 7f353d437740 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x555dbd325650 con 0x555dbd384fe0 2024-04-15T15:16:16.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.717+0000 7f353d437740 10 client.0 _ll_register_callbacks cb 0x555dbd285250 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:16:16.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.717+0000 7f3533fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555dbd3521d0 0x555dbd350e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:16.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.717+0000 7f3533fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555dbd3521d0 0x555dbd350e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56256/0 (socket says 172.21.15.72:56256) 2024-04-15T15:16:16.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.717+0000 7f3533fff640 1 -- 172.21.15.72:0/3572588465 learned_addr learned my addr 172.21.15.72:0/3572588465 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:16.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.717+0000 7f3538de7640 1 --2- 172.21.15.72:0/3572588465 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x555dbd384fe0 0x555dbd376650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:16.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.717+0000 7f35337fe640 1 --2- 172.21.15.72:0/3572588465 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x555dbd372990 0x555dbd37be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:16.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.717+0000 7f3533fff640 1 -- 172.21.15.72:0/3572588465 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x555dbd384fe0 msgr2=0x555dbd376650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:16.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.717+0000 7f3533fff640 1 --2- 172.21.15.72:0/3572588465 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x555dbd384fe0 0x555dbd376650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.717+0000 7f3533fff640 1 -- 172.21.15.72:0/3572588465 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x555dbd372990 msgr2=0x555dbd37be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:16.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.717+0000 7f3533fff640 1 --2- 172.21.15.72:0/3572588465 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x555dbd372990 0x555dbd37be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.717+0000 7f3533fff640 1 -- 172.21.15.72:0/3572588465 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555dbd06c6b0 con 0x555dbd3521d0 2024-04-15T15:16:16.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.717+0000 7f35337fe640 1 --2- 172.21.15.72:0/3572588465 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x555dbd372990 0x555dbd37be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:16.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.717+0000 7f3538de7640 1 --2- 172.21.15.72:0/3572588465 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x555dbd384fe0 0x555dbd376650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:16.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.717+0000 7f3533fff640 1 --2- 172.21.15.72:0/3572588465 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555dbd3521d0 0x555dbd350e90 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f3524015a20 tx=0x7f3524004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31dc07030131464f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:16.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.717+0000 7f3512ffd640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-04-15T15:16:16.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.717+0000 7f3512ffd640 1 -- 172.21.15.72:0/3572588465 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352401c070 con 0x555dbd3521d0 2024-04-15T15:16:16.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.717+0000 7f3512ffd640 1 -- 172.21.15.72:0/3572588465 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f3524016db0 con 0x555dbd3521d0 2024-04-15T15:16:16.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.717+0000 7f353d437740 1 -- 172.21.15.72:0/3572588465 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f352c055be0 con 0x555dbd3521d0 2024-04-15T15:16:16.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.717+0000 7f3512ffd640 1 -- 172.21.15.72:0/3572588465 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f352402f050 con 0x555dbd3521d0 2024-04-15T15:16:16.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.717+0000 7f353d437740 10 client.5760 fetch_fsmap learned FSMap version 16 2024-04-15T15:16:16.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.718+0000 7f353d437740 1 -- 172.21.15.72:0/3572588465 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x555dbd351a70 con 0x555dbd3521d0 2024-04-15T15:16:16.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.718+0000 7f3512ffd640 1 -- 172.21.15.72:0/3572588465 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f352c055be0 con 0x555dbd3521d0 2024-04-15T15:16:16.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.718+0000 7f3512ffd640 1 -- 172.21.15.72:0/3572588465 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 16) ==== 3494+0+0 (secure 0 0 0) 0x7f3524032020 con 0x555dbd3521d0 2024-04-15T15:16:16.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.718+0000 7f353d437740 10 client.5760 fetch_fsmap finished waiting for FSMap version 16 2024-04-15T15:16:16.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.718+0000 7f353d437740 10 client.5760 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-15T15:16:16.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.718+0000 7f353d437740 20 client.5760 populate_metadata read hostname 'smithi072' 2024-04-15T15:16:16.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.718+0000 7f353d437740 1 --2- 172.21.15.72:0/3572588465 >> [v2:172.21.15.72:6834/4270793613,v1:172.21.15.72:6836/4270793613] conn(0x555dbd387f80 0x555dbd3751a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:16.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.718+0000 7f353d437740 4 client.5760 mds_command: new command op to 5559 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-15T15:16:16.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.718+0000 7f353d437740 1 -- 172.21.15.72:0/3572588465 --> [v2:172.21.15.72:6834/4270793613,v1:172.21.15.72:6836/4270793613] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555dbd29d3e0 con 0x555dbd387f80 2024-04-15T15:16:16.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.718+0000 7f35337fe640 1 --2- 172.21.15.72:0/3572588465 >> [v2:172.21.15.72:6834/4270793613,v1:172.21.15.72:6836/4270793613] conn(0x555dbd387f80 0x555dbd3751a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:16.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.718+0000 7f35337fe640 1 --2- 172.21.15.72:0/3572588465 >> [v2:172.21.15.72:6834/4270793613,v1:172.21.15.72:6836/4270793613] conn(0x555dbd387f80 0x555dbd3751a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24c462e61f60f17f server_cookie=d801a1f5465cdf86 in_seq=0 out_seq=0 2024-04-15T15:16:16.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.718+0000 7f3512ffd640 10 client.5760 ms_handle_connect on v2:172.21.15.72:6834/4270793613 2024-04-15T15:16:16.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.719+0000 7f3512ffd640 1 -- 172.21.15.72:0/3572588465 <== mds.0 v2:172.21.15.72:6834/4270793613 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555dbd29d3e0 con 0x555dbd387f80 2024-04-15T15:16:16.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.719+0000 7f3512ffd640 10 client.5760 handle_command_reply: tid=0 2024-04-15T15:16:16.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.719+0000 7f353d437740 1 client.5760 shutdown 2024-04-15T15:16:16.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.719+0000 7f353d437740 1 -- 172.21.15.72:0/3572588465 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555dbd3521d0 msgr2=0x555dbd350e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:16.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.719+0000 7f353d437740 1 --2- 172.21.15.72:0/3572588465 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555dbd3521d0 0x555dbd350e90 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f3524015a20 tx=0x7f3524004730 comp rx=0 tx=0).stop 2024-04-15T15:16:16.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.719+0000 7f353d437740 1 -- 172.21.15.72:0/3572588465 shutdown_connections 2024-04-15T15:16:16.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.719+0000 7f353d437740 1 --2- 172.21.15.72:0/3572588465 >> [v2:172.21.15.72:6834/4270793613,v1:172.21.15.72:6836/4270793613] conn(0x555dbd387f80 0x555dbd3751a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.719+0000 7f353d437740 1 --2- 172.21.15.72:0/3572588465 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x555dbd384fe0 0x555dbd376650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.719+0000 7f353d437740 1 --2- 172.21.15.72:0/3572588465 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x555dbd372990 0x555dbd37be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.719+0000 7f3512ffd640 0 client.5760 ms_handle_reset on v2:172.21.15.72:6834/4270793613 2024-04-15T15:16:16.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.719+0000 7f353d437740 1 --2- 172.21.15.72:0/3572588465 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555dbd3521d0 0x555dbd350e90 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.730 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.719+0000 7f353d437740 1 -- 172.21.15.72:0/3572588465 >> 172.21.15.72:0/3572588465 conn(0x555dbd32a400 msgr2=0x555dbd32a820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:16.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.719+0000 7f353d437740 1 -- 172.21.15.72:0/3572588465 shutdown_connections 2024-04-15T15:16:16.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.719+0000 7f353d437740 1 -- 172.21.15.72:0/3572588465 wait complete. 2024-04-15T15:16:16.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.719+0000 7f353d437740 20 client.5760 trim_cache size 0 max 16384 2024-04-15T15:16:16.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.738+0000 7f353d437740 1 Processor -- start 2024-04-15T15:16:16.740 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.738+0000 7f353d437740 1 -- start start 2024-04-15T15:16:16.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.738+0000 7f353d437740 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x555dbd3521d0 0x555dbd371e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:16.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.738+0000 7f353d437740 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x555dbd372990 0x555dbd377fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:16.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.738+0000 7f353d437740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555dbd384fe0 0x555dbd36ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:16.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.738+0000 7f353d437740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f35341fc1a0 con 0x555dbd384fe0 2024-04-15T15:16:16.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.738+0000 7f353d437740 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f35340b2990 con 0x555dbd372990 2024-04-15T15:16:16.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.738+0000 7f353d437740 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f35341fbb50 con 0x555dbd3521d0 2024-04-15T15:16:16.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.739+0000 7f3538de7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555dbd384fe0 0x555dbd36ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:16.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.739+0000 7f3538de7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555dbd384fe0 0x555dbd36ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56264/0 (socket says 172.21.15.72:56264) 2024-04-15T15:16:16.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.739+0000 7f3538de7640 1 -- 172.21.15.72:0/1529070175 learned_addr learned my addr 172.21.15.72:0/1529070175 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:16.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.739+0000 7f3538de7640 1 -- 172.21.15.72:0/1529070175 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x555dbd3521d0 msgr2=0x555dbd371e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:16.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.739+0000 7f3538de7640 1 --2- 172.21.15.72:0/1529070175 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x555dbd3521d0 0x555dbd371e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.739+0000 7f3538de7640 1 -- 172.21.15.72:0/1529070175 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x555dbd372990 msgr2=0x555dbd377fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:16.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.739+0000 7f3538de7640 1 --2- 172.21.15.72:0/1529070175 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x555dbd372990 0x555dbd377fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.739+0000 7f3538de7640 1 -- 172.21.15.72:0/1529070175 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555dbd355a90 con 0x555dbd384fe0 2024-04-15T15:16:16.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.739+0000 7f3538de7640 1 --2- 172.21.15.72:0/1529070175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555dbd384fe0 0x555dbd36ed90 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f352c089d10 tx=0x7f352c01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c2f720f8d169e32 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:16.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.739+0000 7f35197fa640 1 -- 172.21.15.72:0/1529070175 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352c059340 con 0x555dbd384fe0 2024-04-15T15:16:16.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.739+0000 7f35197fa640 1 -- 172.21.15.72:0/1529070175 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f352c055be0 con 0x555dbd384fe0 2024-04-15T15:16:16.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.739+0000 7f35197fa640 1 -- 172.21.15.72:0/1529070175 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352c061e50 con 0x555dbd384fe0 2024-04-15T15:16:16.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.740+0000 7f353d437740 1 -- 172.21.15.72:0/1529070175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555dbd384fe0 msgr2=0x555dbd36ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:16.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.740+0000 7f353d437740 1 --2- 172.21.15.72:0/1529070175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555dbd384fe0 0x555dbd36ed90 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f352c089d10 tx=0x7f352c01d000 comp rx=0 tx=0).stop 2024-04-15T15:16:16.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.740+0000 7f353d437740 1 -- 172.21.15.72:0/1529070175 shutdown_connections 2024-04-15T15:16:16.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.740+0000 7f353d437740 1 --2- 172.21.15.72:0/1529070175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555dbd384fe0 0x555dbd36ed90 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.740+0000 7f353d437740 1 --2- 172.21.15.72:0/1529070175 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x555dbd372990 0x555dbd377fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.740+0000 7f353d437740 1 --2- 172.21.15.72:0/1529070175 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x555dbd3521d0 0x555dbd371e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.740+0000 7f353d437740 1 -- 172.21.15.72:0/1529070175 >> 172.21.15.72:0/1529070175 conn(0x555dbd32a400 msgr2=0x555dbd376f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:16.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.740+0000 7f353d437740 1 -- 172.21.15.72:0/1529070175 shutdown_connections 2024-04-15T15:16:16.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.740+0000 7f353d437740 1 -- 172.21.15.72:0/1529070175 wait complete. 2024-04-15T15:16:16.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.740+0000 7f353d437740 1 -- start start 2024-04-15T15:16:16.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.741+0000 7f353d437740 1 Processor -- start 2024-04-15T15:16:16.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.741+0000 7f353d437740 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x555dbd3521d0 0x555dbd356cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:16.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.741+0000 7f353d437740 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x555dbd372990 0x555dbd3d8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:16.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.741+0000 7f353d437740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555dbd384fe0 0x555dbd372d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:16.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.741+0000 7f353d437740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x555dbd387e00 con 0x555dbd384fe0 2024-04-15T15:16:16.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.741+0000 7f353d437740 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x555dbd351d50 con 0x555dbd372990 2024-04-15T15:16:16.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.741+0000 7f353d437740 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x555dbd388450 con 0x555dbd3521d0 2024-04-15T15:16:16.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.741+0000 7f353d437740 10 client.0 _ll_register_callbacks cb 0x7f3524010910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:16:16.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.741+0000 7f3538de7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555dbd384fe0 0x555dbd372d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:16.749 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.742+0000 7f3538de7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555dbd384fe0 0x555dbd372d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56272/0 (socket says 172.21.15.72:56272) 2024-04-15T15:16:16.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.742+0000 7f3538de7640 1 -- 172.21.15.72:0/3663733612 learned_addr learned my addr 172.21.15.72:0/3663733612 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:16.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.742+0000 7f3533fff640 1 --2- 172.21.15.72:0/3663733612 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x555dbd3521d0 0x555dbd356cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:16.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.742+0000 7f35337fe640 1 --2- 172.21.15.72:0/3663733612 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x555dbd372990 0x555dbd3d8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:16.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.742+0000 7f3538de7640 1 -- 172.21.15.72:0/3663733612 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x555dbd3521d0 msgr2=0x555dbd356cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:16.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.742+0000 7f3538de7640 1 --2- 172.21.15.72:0/3663733612 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x555dbd3521d0 0x555dbd356cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.742+0000 7f3538de7640 1 -- 172.21.15.72:0/3663733612 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x555dbd372990 msgr2=0x555dbd3d8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:16.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.742+0000 7f3538de7640 1 --2- 172.21.15.72:0/3663733612 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x555dbd372990 0x555dbd3d8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.742+0000 7f3538de7640 1 -- 172.21.15.72:0/3663733612 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555dbd378370 con 0x555dbd384fe0 2024-04-15T15:16:16.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.742+0000 7f35337fe640 1 --2- 172.21.15.72:0/3663733612 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x555dbd372990 0x555dbd3d8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:16.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.742+0000 7f3538de7640 1 --2- 172.21.15.72:0/3663733612 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555dbd384fe0 0x555dbd372d70 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f352c089d10 tx=0x7f352c001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8509e1fe83bd5af server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:16.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.742+0000 7f3512ffd640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-04-15T15:16:16.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.742+0000 7f3512ffd640 1 -- 172.21.15.72:0/3663733612 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f352c075070 con 0x555dbd384fe0 2024-04-15T15:16:16.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.742+0000 7f3512ffd640 1 -- 172.21.15.72:0/3663733612 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f352c004d60 con 0x555dbd384fe0 2024-04-15T15:16:16.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.742+0000 7f353d437740 1 -- 172.21.15.72:0/3663733612 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f352c055be0 con 0x555dbd384fe0 2024-04-15T15:16:16.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.742+0000 7f3512ffd640 1 -- 172.21.15.72:0/3663733612 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f352c059340 con 0x555dbd384fe0 2024-04-15T15:16:16.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.742+0000 7f353d437740 10 client.5766 fetch_fsmap learned FSMap version 16 2024-04-15T15:16:16.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.742+0000 7f353d437740 1 -- 172.21.15.72:0/3663733612 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x555dbd3738b0 con 0x555dbd384fe0 2024-04-15T15:16:16.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.742+0000 7f3512ffd640 1 -- 172.21.15.72:0/3663733612 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f352c055be0 con 0x555dbd384fe0 2024-04-15T15:16:16.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.742+0000 7f3512ffd640 1 -- 172.21.15.72:0/3663733612 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 16) ==== 3494+0+0 (secure 0 0 0) 0x7f352c0589c0 con 0x555dbd384fe0 2024-04-15T15:16:16.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.742+0000 7f353d437740 10 client.5766 fetch_fsmap finished waiting for FSMap version 16 2024-04-15T15:16:16.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.742+0000 7f353d437740 10 client.5766 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-15T15:16:16.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.742+0000 7f353d437740 20 client.5766 populate_metadata read hostname 'smithi072' 2024-04-15T15:16:16.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.743+0000 7f353d437740 1 --2- 172.21.15.72:0/3663733612 >> [v2:172.21.15.72:6834/4270793613,v1:172.21.15.72:6836/4270793613] conn(0x555dbd37a820 0x555dbd37ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:16.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.743+0000 7f353d437740 4 client.5766 mds_command: new command op to 5559 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-15T15:16:16.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.743+0000 7f353d437740 1 -- 172.21.15.72:0/3663733612 --> [v2:172.21.15.72:6834/4270793613,v1:172.21.15.72:6836/4270793613] -- command(tid 0: {"prefix": "status"}) -- 0x555dbd297250 con 0x555dbd37a820 2024-04-15T15:16:16.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.743+0000 7f3533fff640 1 --2- 172.21.15.72:0/3663733612 >> [v2:172.21.15.72:6834/4270793613,v1:172.21.15.72:6836/4270793613] conn(0x555dbd37a820 0x555dbd37ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:16.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.743+0000 7f3512ffd640 10 client.5766 ms_handle_connect on v2:172.21.15.72:6834/4270793613 2024-04-15T15:16:16.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.743+0000 7f3533fff640 1 --2- 172.21.15.72:0/3663733612 >> [v2:172.21.15.72:6834/4270793613,v1:172.21.15.72:6836/4270793613] conn(0x555dbd37a820 0x555dbd37ac00 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6e1b02c5a667b25 server_cookie=8f7f4ad51e18ff4e in_seq=0 out_seq=0 2024-04-15T15:16:16.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.743+0000 7f3512ffd640 1 -- 172.21.15.72:0/3663733612 <== mds.0 v2:172.21.15.72:6834/4270793613 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x555dbd297250 con 0x555dbd37a820 2024-04-15T15:16:16.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.743+0000 7f3512ffd640 10 client.5766 handle_command_reply: tid=0 2024-04-15T15:16:16.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.743+0000 7f353d437740 1 client.5766 shutdown 2024-04-15T15:16:16.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.743+0000 7f353d437740 1 -- 172.21.15.72:0/3663733612 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555dbd384fe0 msgr2=0x555dbd372d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:16.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.743+0000 7f353d437740 1 --2- 172.21.15.72:0/3663733612 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555dbd384fe0 0x555dbd372d70 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f352c089d10 tx=0x7f352c001250 comp rx=0 tx=0).stop 2024-04-15T15:16:16.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.743+0000 7f353d437740 1 -- 172.21.15.72:0/3663733612 shutdown_connections 2024-04-15T15:16:16.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.743+0000 7f353d437740 1 --2- 172.21.15.72:0/3663733612 >> [v2:172.21.15.72:6834/4270793613,v1:172.21.15.72:6836/4270793613] conn(0x555dbd37a820 0x555dbd37ac00 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.743+0000 7f353d437740 1 --2- 172.21.15.72:0/3663733612 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x555dbd384fe0 0x555dbd372d70 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.743+0000 7f353d437740 1 --2- 172.21.15.72:0/3663733612 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x555dbd372990 0x555dbd3d8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.743+0000 7f3512ffd640 0 client.5766 ms_handle_reset on v2:172.21.15.72:6834/4270793613 2024-04-15T15:16:16.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.743+0000 7f353d437740 1 --2- 172.21.15.72:0/3663733612 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x555dbd3521d0 0x555dbd356cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.743+0000 7f353d437740 1 -- 172.21.15.72:0/3663733612 >> 172.21.15.72:0/3663733612 conn(0x555dbd32a400 msgr2=0x555dbd37b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:16.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.743+0000 7f353d437740 1 -- 172.21.15.72:0/3663733612 shutdown_connections 2024-04-15T15:16:16.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.743+0000 7f353d437740 1 -- 172.21.15.72:0/3663733612 wait complete. 2024-04-15T15:16:16.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.744+0000 7f353d437740 20 client.5766 trim_cache size 0 max 16384 2024-04-15T15:16:16.753 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-04-15T15:16:16.753 INFO:teuthology.orchestra.run.smithi072.stdout: "cluster_fsid": "05091e39-b18a-4545-8b68-c0ae4699e116", 2024-04-15T15:16:16.753 INFO:teuthology.orchestra.run.smithi072.stdout: "whoami": 0, 2024-04-15T15:16:16.753 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 5559, 2024-04-15T15:16:16.753 INFO:teuthology.orchestra.run.smithi072.stdout: "want_state": "up:active", 2024-04-15T15:16:16.753 INFO:teuthology.orchestra.run.smithi072.stdout: "state": "up:active", 2024-04-15T15:16:16.753 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs", 2024-04-15T15:16:16.753 INFO:teuthology.orchestra.run.smithi072.stdout: "rank_uptime": 9.0409452340000005, 2024-04-15T15:16:16.753 INFO:teuthology.orchestra.run.smithi072.stdout: "mdsmap_epoch": 16, 2024-04-15T15:16:16.753 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch": 38, 2024-04-15T15:16:16.753 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch_barrier": 38, 2024-04-15T15:16:16.753 INFO:teuthology.orchestra.run.smithi072.stdout: "uptime": 15.935275857000001 2024-04-15T15:16:16.754 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-04-15T15:16:16.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.744+0000 7f353a84a640 1 -- 172.21.15.72:0/853562672 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f352006c920 msgr2=0x7f352006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:16.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.744+0000 7f353a84a640 1 --2- 172.21.15.72:0/853562672 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f352006c920 0x7f352006ede0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3524009ac0 tx=0x7f3524008040 comp rx=0 tx=0).stop 2024-04-15T15:16:16.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.744+0000 7f353a84a640 1 -- 172.21.15.72:0/853562672 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35340d08f0 msgr2=0x7f35340afd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:16.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.744+0000 7f353a84a640 1 --2- 172.21.15.72:0/853562672 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35340d08f0 0x7f35340afd40 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f352c00cbc0 tx=0x7f352c00a430 comp rx=0 tx=0).stop 2024-04-15T15:16:16.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.744+0000 7f353a84a640 1 -- 172.21.15.72:0/853562672 shutdown_connections 2024-04-15T15:16:16.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.744+0000 7f353a84a640 1 --2- 172.21.15.72:0/853562672 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f352006c920 0x7f352006ede0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.744+0000 7f353a84a640 1 --2- 172.21.15.72:0/853562672 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35340d08f0 0x7f35340afd40 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.744+0000 7f353a84a640 1 --2- 172.21.15.72:0/853562672 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35340ce6d0 0x7f35340af800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.744+0000 7f353a84a640 1 --2- 172.21.15.72:0/853562672 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f35340ce2f0 0x7f3534151e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.744+0000 7f353a84a640 1 -- 172.21.15.72:0/853562672 >> 172.21.15.72:0/853562672 conn(0x7f35340b31d0 msgr2=0x7f353415b020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:16.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.744+0000 7f353a84a640 1 -- 172.21.15.72:0/853562672 shutdown_connections 2024-04-15T15:16:16.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.744+0000 7f353a84a640 1 -- 172.21.15.72:0/853562672 wait complete. 2024-04-15T15:16:16.758 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:16:16.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.858+0000 7fa4b24a7640 1 Processor -- start 2024-04-15T15:16:16.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.858+0000 7fa4b24a7640 1 -- start start 2024-04-15T15:16:16.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.859+0000 7fa4b24a7640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa4ac156da0 0x7fa4ac0ccb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:16.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.859+0000 7fa4b24a7640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4ac0cbad0 con 0x7fa4ac0cd0b0 2024-04-15T15:16:16.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.859+0000 7fa4b24a7640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4ac0cbdb0 con 0x7fa4ac1569c0 2024-04-15T15:16:16.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.859+0000 7fa4b24a7640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fa4ac0cc090 con 0x7fa4ac156da0 2024-04-15T15:16:16.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.859+0000 7fa4b0a44640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fa4ac0cd0b0 0x7fa4ac06c370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:34242/0 (socket says 172.21.15.72:34242) 2024-04-15T15:16:16.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.859+0000 7fa4b0a44640 1 -- 172.21.15.72:0/4036169265 learned_addr learned my addr 172.21.15.72:0/4036169265 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:16.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.859+0000 7fa4ab7fe640 1 -- 172.21.15.72:0/4036169265 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2412428334 0 0) 0x7fa4ac0cbad0 con 0x7fa4ac0cd0b0 2024-04-15T15:16:16.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.859+0000 7fa4ab7fe640 1 -- 172.21.15.72:0/4036169265 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa498003520 con 0x7fa4ac0cd0b0 2024-04-15T15:16:16.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.859+0000 7fa4a3fff640 1 --2- 172.21.15.72:0/4036169265 >> v2:172.21.15.175:3300/0 conn(0x7fa4ac156da0 0x7fa4ac0ccb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:16.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.860+0000 7fa4ab7fe640 1 -- 172.21.15.72:0/4036169265 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1238041121 0 0) 0x7fa498003520 con 0x7fa4ac0cd0b0 2024-04-15T15:16:16.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.860+0000 7fa4ab7fe640 1 -- 172.21.15.72:0/4036169265 >> v2:172.21.15.175:3300/0 conn(0x7fa4ac156da0 msgr2=0x7fa4ac0ccb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:16.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.860+0000 7fa4ab7fe640 1 --2- 172.21.15.72:0/4036169265 >> v2:172.21.15.175:3300/0 conn(0x7fa4ac156da0 0x7fa4ac0ccb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.860+0000 7fa4ab7fe640 1 -- 172.21.15.72:0/4036169265 >> v1:172.21.15.175:6789/0 conn(0x7fa4ac1569c0 legacy=0x7fa4ac15d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:16.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.860+0000 7fa4ab7fe640 1 -- 172.21.15.72:0/4036169265 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4ac0cc370 con 0x7fa4ac0cd0b0 2024-04-15T15:16:16.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.860+0000 7fa4ab7fe640 1 -- 172.21.15.72:0/4036169265 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7fa4a40019f0 con 0x7fa4ac0cd0b0 2024-04-15T15:16:16.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.860+0000 7fa4a3fff640 1 --2- 172.21.15.72:0/4036169265 >> v2:172.21.15.175:3300/0 conn(0x7fa4ac156da0 0x7fa4ac0ccb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:16.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.860+0000 7fa4ab7fe640 1 -- 172.21.15.72:0/4036169265 >> v1:172.21.15.72:6789/0 conn(0x7fa4ac0cd0b0 legacy=0x7fa4ac06c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:16.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.860+0000 7fa4ab7fe640 1 --2- 172.21.15.72:0/4036169265 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa4980039b0 0x7fa498003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:16.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.860+0000 7fa4ab7fe640 1 --2- 172.21.15.72:0/4036169265 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa498004800 0x7fa498004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:16.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.860+0000 7fa4ab7fe640 1 --2- 172.21.15.72:0/4036169265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4980051c0 0x7fa49800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:16.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.860+0000 7fa4ab7fe640 1 -- 172.21.15.72:0/4036169265 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fa4ac0cc090 con 0x7fa4980051c0 2024-04-15T15:16:16.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.860+0000 7fa4ab7fe640 1 -- 172.21.15.72:0/4036169265 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fa49800c110 con 0x7fa4980039b0 2024-04-15T15:16:16.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.860+0000 7fa4ab7fe640 1 -- 172.21.15.72:0/4036169265 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fa49800c310 con 0x7fa498004800 2024-04-15T15:16:16.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.860+0000 7fa4b0a44640 1 --2- 172.21.15.72:0/4036169265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4980051c0 0x7fa49800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:16.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.860+0000 7fa4a3fff640 1 --2- 172.21.15.72:0/4036169265 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa498004800 0x7fa498004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:16.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.860+0000 7fa4abfff640 1 --2- 172.21.15.72:0/4036169265 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa4980039b0 0x7fa498003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:16.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.860+0000 7fa4b0a44640 1 -- 172.21.15.72:0/4036169265 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa498004800 msgr2=0x7fa498004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:16.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.860+0000 7fa4b0a44640 1 --2- 172.21.15.72:0/4036169265 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa498004800 0x7fa498004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.860+0000 7fa4b0a44640 1 -- 172.21.15.72:0/4036169265 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa4980039b0 msgr2=0x7fa498003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:16.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.860+0000 7fa4b0a44640 1 --2- 172.21.15.72:0/4036169265 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa4980039b0 0x7fa498003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.860+0000 7fa4b0a44640 1 -- 172.21.15.72:0/4036169265 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa498003520 con 0x7fa4980051c0 2024-04-15T15:16:16.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.860+0000 7fa4a3fff640 1 --2- 172.21.15.72:0/4036169265 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa498004800 0x7fa498004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:16.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.861+0000 7fa4b0a44640 1 -- 172.21.15.72:0/4036169265 reap_dead start 2024-04-15T15:16:16.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.861+0000 7fa4b0a44640 1 --2- 172.21.15.72:0/4036169265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4980051c0 0x7fa49800b9c0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fa4a400a840 tx=0x7fa498005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=493d8bbb2a4c66c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:16.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.861+0000 7fa4ab7fe640 1 -- 172.21.15.72:0/4036169265 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a400a460 con 0x7fa4980051c0 2024-04-15T15:16:16.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.861+0000 7fa4ab7fe640 1 -- 172.21.15.72:0/4036169265 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4a4034650 con 0x7fa4980051c0 2024-04-15T15:16:16.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.861+0000 7fa4ab7fe640 1 -- 172.21.15.72:0/4036169265 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a4034a40 con 0x7fa4980051c0 2024-04-15T15:16:16.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.861+0000 7fa4b24a7640 1 -- 172.21.15.72:0/4036169265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4980051c0 msgr2=0x7fa49800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:16.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.861+0000 7fa4b24a7640 1 --2- 172.21.15.72:0/4036169265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4980051c0 0x7fa49800b9c0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fa4a400a840 tx=0x7fa498005830 comp rx=0 tx=0).stop 2024-04-15T15:16:16.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.862+0000 7fa4b24a7640 1 -- 172.21.15.72:0/4036169265 shutdown_connections 2024-04-15T15:16:16.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.862+0000 7fa4b24a7640 1 --2- 172.21.15.72:0/4036169265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4980051c0 0x7fa49800b9c0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.862+0000 7fa4b24a7640 1 -- 172.21.15.72:0/4036169265 >> 172.21.15.72:0/4036169265 conn(0x7fa4ac0cd4a0 msgr2=0x7fa4ac0cd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:16.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.862+0000 7fa4b24a7640 1 -- 172.21.15.72:0/4036169265 shutdown_connections 2024-04-15T15:16:16.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.862+0000 7fa4b24a7640 1 -- 172.21.15.72:0/4036169265 wait complete. 2024-04-15T15:16:16.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.863+0000 7fa4b24a7640 1 Processor -- start 2024-04-15T15:16:16.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.863+0000 7fa4b24a7640 1 -- start start 2024-04-15T15:16:16.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.863+0000 7fa4b24a7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4980051c0 0x7fa4ac0c7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:16.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.863+0000 7fa4b24a7640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa4ac150920 0x7fa4ac14dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:16.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.863+0000 7fa4b24a7640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa4ac15b590 0x7fa4ac15ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:16.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.864+0000 7fa4b24a7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fa4ac0cb750 con 0x7fa4980051c0 2024-04-15T15:16:16.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.864+0000 7fa4b24a7640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fa4ac0b01a0 con 0x7fa4ac15b590 2024-04-15T15:16:16.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.864+0000 7fa4b24a7640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fa4ac0ccb70 con 0x7fa4ac150920 2024-04-15T15:16:16.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.864+0000 7fa4abfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4980051c0 0x7fa4ac0c7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:16.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.864+0000 7fa4abfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4980051c0 0x7fa4ac0c7420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56290/0 (socket says 172.21.15.72:56290) 2024-04-15T15:16:16.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.864+0000 7fa4abfff640 1 -- 172.21.15.72:0/1389850315 learned_addr learned my addr 172.21.15.72:0/1389850315 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:16.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.864+0000 7fa4abfff640 1 -- 172.21.15.72:0/1389850315 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa4ac150920 msgr2=0x7fa4ac14dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:16.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.864+0000 7fa4abfff640 1 --2- 172.21.15.72:0/1389850315 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa4ac150920 0x7fa4ac14dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.864+0000 7fa4abfff640 1 -- 172.21.15.72:0/1389850315 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa4ac15b590 msgr2=0x7fa4ac15ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:16.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.864+0000 7fa4abfff640 1 --2- 172.21.15.72:0/1389850315 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa4ac15b590 0x7fa4ac15ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.864+0000 7fa4abfff640 1 -- 172.21.15.72:0/1389850315 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4ac150da0 con 0x7fa4980051c0 2024-04-15T15:16:16.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.864+0000 7fa4abfff640 1 --2- 172.21.15.72:0/1389850315 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4980051c0 0x7fa4ac0c7420 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fa49c00e4b0 tx=0x7fa49c00e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:16.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.864+0000 7fa4a97fa640 1 -- 172.21.15.72:0/1389850315 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa49c0052e0 con 0x7fa4980051c0 2024-04-15T15:16:16.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.864+0000 7fa4a97fa640 1 -- 172.21.15.72:0/1389850315 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa49c018070 con 0x7fa4980051c0 2024-04-15T15:16:16.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.864+0000 7fa4b24a7640 1 -- 172.21.15.72:0/1389850315 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4ac151020 con 0x7fa4980051c0 2024-04-15T15:16:16.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.864+0000 7fa4a97fa640 1 -- 172.21.15.72:0/1389850315 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa49c003830 con 0x7fa4980051c0 2024-04-15T15:16:16.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.864+0000 7fa4b24a7640 1 -- 172.21.15.72:0/1389850315 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4ac0c8020 con 0x7fa4980051c0 2024-04-15T15:16:16.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.865+0000 7fa4b24a7640 1 -- 172.21.15.72:0/1389850315 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4ac15bf50 con 0x7fa4980051c0 2024-04-15T15:16:16.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.866+0000 7fa4a97fa640 1 -- 172.21.15.72:0/1389850315 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa49c0039d0 con 0x7fa4980051c0 2024-04-15T15:16:16.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.866+0000 7fa4a97fa640 1 --2- 172.21.15.72:0/1389850315 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa48c06c920 0x7fa48c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:16.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.866+0000 7fa4a97fa640 1 -- 172.21.15.72:0/1389850315 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fa49c08b840 con 0x7fa4980051c0 2024-04-15T15:16:16.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.866+0000 7fa4a3fff640 1 --2- 172.21.15.72:0/1389850315 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa48c06c920 0x7fa48c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:16.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.867+0000 7fa4a3fff640 1 --2- 172.21.15.72:0/1389850315 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa48c06c920 0x7fa48c06ede0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac0b0ae0 tx=0x7fa49400e1f0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:16.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:16.871+0000 7fa4a97fa640 1 -- 172.21.15.72:0/1389850315 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa4ac150da0 con 0x7fa4980051c0 2024-04-15T15:16:17.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:17.086+0000 7fa4b24a7640 1 -- 172.21.15.72:0/1389850315 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fa4ac0689e0 con 0x7fa4980051c0 2024-04-15T15:16:17.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:17.086+0000 7fa4a97fa640 1 -- 172.21.15.72:0/1389850315 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+73 (secure 0 0 0) 0x7fa49c059ad0 con 0x7fa4980051c0 2024-04-15T15:16:17.089 INFO:teuthology.orchestra.run.smithi072.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-15T15:16:17.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:17.088+0000 7fa4b24a7640 1 -- 172.21.15.72:0/1389850315 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa48c06c920 msgr2=0x7fa48c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:17.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:17.088+0000 7fa4b24a7640 1 --2- 172.21.15.72:0/1389850315 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa48c06c920 0x7fa48c06ede0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac0b0ae0 tx=0x7fa49400e1f0 comp rx=0 tx=0).stop 2024-04-15T15:16:17.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:17.088+0000 7fa4b24a7640 1 -- 172.21.15.72:0/1389850315 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4980051c0 msgr2=0x7fa4ac0c7420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:17.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:17.088+0000 7fa4b24a7640 1 --2- 172.21.15.72:0/1389850315 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4980051c0 0x7fa4ac0c7420 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fa49c00e4b0 tx=0x7fa49c00e970 comp rx=0 tx=0).stop 2024-04-15T15:16:17.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:17.088+0000 7fa4b24a7640 1 -- 172.21.15.72:0/1389850315 shutdown_connections 2024-04-15T15:16:17.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:17.088+0000 7fa4b24a7640 1 --2- 172.21.15.72:0/1389850315 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa48c06c920 0x7fa48c06ede0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:17.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:17.088+0000 7fa4b24a7640 1 --2- 172.21.15.72:0/1389850315 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa4ac15b590 0x7fa4ac15ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:17.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:17.088+0000 7fa4b24a7640 1 --2- 172.21.15.72:0/1389850315 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa4ac150920 0x7fa4ac14dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:17.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:17.088+0000 7fa4b24a7640 1 --2- 172.21.15.72:0/1389850315 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa4980051c0 0x7fa4ac0c7420 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:17.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:17.088+0000 7fa4b24a7640 1 -- 172.21.15.72:0/1389850315 >> 172.21.15.72:0/1389850315 conn(0x7fa4ac0cd4a0 msgr2=0x7fa4ac0cd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:17.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:17.088+0000 7fa4b24a7640 1 -- 172.21.15.72:0/1389850315 shutdown_connections 2024-04-15T15:16:17.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:17.088+0000 7fa4b24a7640 1 -- 172.21.15.72:0/1389850315 wait complete. 2024-04-15T15:16:17.101 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:16:17.101 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:16:17.101 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi072.front.sepia.ceph.com 2024-04-15T15:16:17.101 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-15T15:16:17.101 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-15T15:16:17.102 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:16:17.102 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-15T15:16:17.102 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:16:17.102 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-15T15:16:17.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:17.102 DEBUG:teuthology.orchestra.run.smithi072:> ip addr 2024-04-15T15:16:17.119 INFO:teuthology.orchestra.run.smithi072.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:16:17.119 INFO:teuthology.orchestra.run.smithi072.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:16:17.119 INFO:teuthology.orchestra.run.smithi072.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:16:17.119 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:17.119 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 ::1/128 scope host 2024-04-15T15:16:17.119 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:17.119 INFO:teuthology.orchestra.run.smithi072.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:17.120 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f4 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:17.120 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f0 2024-04-15T15:16:17.120 INFO:teuthology.orchestra.run.smithi072.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:17.120 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f5 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:17.120 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f1 2024-04-15T15:16:17.120 INFO:teuthology.orchestra.run.smithi072.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:17.120 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:64 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:17.120 INFO:teuthology.orchestra.run.smithi072.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:16:17.120 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:65 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:17.121 INFO:teuthology.orchestra.run.smithi072.stdout: inet 172.21.15.72/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-15T15:16:17.121 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft 42626sec preferred_lft 42626sec 2024-04-15T15:16:17.121 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::ec4:7aff:fe8f:b765/64 scope link 2024-04-15T15:16:17.121 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:17.121 INFO:teuthology.orchestra.run.smithi072.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:16:17.121 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 62:2f:60:9d:8a:ab brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:17.121 INFO:teuthology.orchestra.run.smithi072.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-15T15:16:17.121 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:17.122 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::6871:2cff:fe2a:4823/64 scope link 2024-04-15T15:16:17.122 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:17.122 INFO:teuthology.orchestra.run.smithi072.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:16:17.122 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether b6:ae:7a:2e:38:96 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:17.122 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::b4ae:7aff:fe2e:3896/64 scope link 2024-04-15T15:16:17.123 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:17.123 INFO:teuthology.orchestra.run.smithi072.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:16:17.123 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 62:2f:60:9d:8a:ab brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:16:17.123 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::602f:60ff:fe9d:8aab/64 scope link 2024-04-15T15:16:17.123 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:17.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:17.124 DEBUG:teuthology.orchestra.run.smithi072:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:16:17.187 INFO:teuthology.orchestra.run.smithi072.stdout:1 2024-04-15T15:16:17.188 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:17.188 DEBUG:teuthology.orchestra.run.smithi072:> route 2024-04-15T15:16:17.245 INFO:teuthology.orchestra.run.smithi072.stdout:Kernel IP routing table 2024-04-15T15:16:17.245 INFO:teuthology.orchestra.run.smithi072.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:16:17.245 INFO:teuthology.orchestra.run.smithi072.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-15T15:16:17.246 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-15T15:16:17.246 INFO:teuthology.orchestra.run.smithi072.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:16:17.246 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:16:17.246 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-04-15T15:16:17.246 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-15T15:16:17.246 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-15T15:16:17.246 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-15T15:16:17.246 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-04-15T15:16:17.334 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:16:17.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:17.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:17.385 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-04-15T15:16:17.400 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-15T15:16:17.400 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-15T15:16:17.400 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-15T15:16:17.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:17.400 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set brx.0 up 2024-04-15T15:16:17.469 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-15T15:16:17.470 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:16:17.470 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-15T15:16:17.525 INFO:teuthology.orchestra.run.smithi072.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-15T15:16:17.525 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:16:17.525 DEBUG:teuthology.orchestra.run.smithi072:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-15T15:16:17.580 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-04-15T15:16:17.650 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:16:17.707 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-04-15T15:16:17.707 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-04-15T15:16:17.707 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-04-15T15:16:17.707 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-04-15T15:16:17.707 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-04-15T15:16:17.707 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-04-15T15:16:17.707 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-04-15T15:16:17.707 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-04-15T15:16:17.707 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-04-15T15:16:17.707 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-04-15T15:16:17.708 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-04-15T15:16:17.708 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-04-15T15:16:17.708 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:16:17.708 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-04-15T15:16:17.708 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-04-15T15:16:17.708 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-04-15T15:16:17.708 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-04-15T15:16:17.708 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:16:17.708 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-04-15T15:16:17.708 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-04-15T15:16:17.708 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:16:17.708 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:16:17.708 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:16:17.708 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-04-15T15:16:17.708 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-04-15T15:16:17.709 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:16:17.709 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:16:17.709 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:16:17.709 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:16:17.709 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-04-15T15:16:17.709 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:17.709 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:17.709 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:16:17.709 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:16:17.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:17.710 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-04-15T15:16:17.763 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-15T15:16:17.763 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-15T15:16:17.806 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-04-15T15:16:17.834 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:16:17.890 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-04-15T15:16:17.891 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-04-15T15:16:17.891 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-04-15T15:16:17.891 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-04-15T15:16:17.891 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-04-15T15:16:17.891 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-04-15T15:16:17.891 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-04-15T15:16:17.891 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-04-15T15:16:17.891 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-04-15T15:16:17.891 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-04-15T15:16:17.892 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-04-15T15:16:17.892 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-04-15T15:16:17.892 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:16:17.892 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-04-15T15:16:17.892 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-04-15T15:16:17.892 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-04-15T15:16:17.892 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-04-15T15:16:17.892 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:16:17.892 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-04-15T15:16:17.892 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-04-15T15:16:17.892 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:16:17.892 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:16:17.892 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:16:17.893 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-04-15T15:16:17.893 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-04-15T15:16:17.893 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:16:17.893 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:16:17.893 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:16:17.893 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:16:17.893 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-04-15T15:16:17.893 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:17.893 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:17.893 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:16:17.894 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:16:17.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:17.894 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-04-15T15:16:17.895 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:2024-04-15T15:16:17.892+0000 7f018b5b8480 -1 init, newargv = 0x561e3a33b400 newargc=15 2024-04-15T15:16:17.895 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[54493]: starting ceph client 2024-04-15T15:16:17.899 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[54493]: starting fuse 2024-04-15T15:16:17.912 INFO:teuthology.orchestra.run.smithi072.stdout:43 2024-04-15T15:16:17.912 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-04-15T15:16:17.912 DEBUG:teuthology.orchestra.run.smithi072:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:16:17.912 DEBUG:teuthology.orchestra.run.smithi072:> import glob 2024-04-15T15:16:17.912 DEBUG:teuthology.orchestra.run.smithi072:> import re 2024-04-15T15:16:17.912 DEBUG:teuthology.orchestra.run.smithi072:> import os 2024-04-15T15:16:17.913 DEBUG:teuthology.orchestra.run.smithi072:> import subprocess 2024-04-15T15:16:17.913 DEBUG:teuthology.orchestra.run.smithi072:> 2024-04-15T15:16:17.913 DEBUG:teuthology.orchestra.run.smithi072:> def _find_admin_socket(client_name): 2024-04-15T15:16:17.913 DEBUG:teuthology.orchestra.run.smithi072:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-15T15:16:17.913 DEBUG:teuthology.orchestra.run.smithi072:> files = glob.glob(asok_path) 2024-04-15T15:16:17.913 DEBUG:teuthology.orchestra.run.smithi072:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-15T15:16:17.913 DEBUG:teuthology.orchestra.run.smithi072:> 2024-04-15T15:16:17.913 DEBUG:teuthology.orchestra.run.smithi072:> # Given a non-glob path, it better be there 2024-04-15T15:16:17.913 DEBUG:teuthology.orchestra.run.smithi072:> if "*" not in asok_path: 2024-04-15T15:16:17.913 DEBUG:teuthology.orchestra.run.smithi072:> assert(len(files) == 1) 2024-04-15T15:16:17.913 DEBUG:teuthology.orchestra.run.smithi072:> return files[0] 2024-04-15T15:16:17.913 DEBUG:teuthology.orchestra.run.smithi072:> 2024-04-15T15:16:17.913 DEBUG:teuthology.orchestra.run.smithi072:> for f in files: 2024-04-15T15:16:17.914 DEBUG:teuthology.orchestra.run.smithi072:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:16:17.914 DEBUG:teuthology.orchestra.run.smithi072:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:16:17.914 DEBUG:teuthology.orchestra.run.smithi072:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:16:17.914 DEBUG:teuthology.orchestra.run.smithi072:> contents = proc_f.read() 2024-04-15T15:16:17.914 DEBUG:teuthology.orchestra.run.smithi072:> if mountpoint in contents: 2024-04-15T15:16:17.914 DEBUG:teuthology.orchestra.run.smithi072:> return f 2024-04-15T15:16:17.914 DEBUG:teuthology.orchestra.run.smithi072:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:16:17.914 DEBUG:teuthology.orchestra.run.smithi072:> 2024-04-15T15:16:17.914 DEBUG:teuthology.orchestra.run.smithi072:> print(_find_admin_socket("client.0")) 2024-04-15T15:16:17.914 DEBUG:teuthology.orchestra.run.smithi072:> ' 2024-04-15T15:16:18.034 INFO:teuthology.orchestra.run.smithi072.stdout:/var/run/ceph/ceph-client.0.54493.asok 2024-04-15T15:16:18.037 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:18.044 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.54493.asok 2024-04-15T15:16:18.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:18.045 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.54493.asok status 2024-04-15T15:16:18.182 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-04-15T15:16:18.183 INFO:teuthology.orchestra.run.smithi072.stdout: "metadata": { 2024-04-15T15:16:18.183 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:16:18.183 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:16:18.183 INFO:teuthology.orchestra.run.smithi072.stdout: "entity_id": "0", 2024-04-15T15:16:18.183 INFO:teuthology.orchestra.run.smithi072.stdout: "hostname": "smithi072", 2024-04-15T15:16:18.183 INFO:teuthology.orchestra.run.smithi072.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-15T15:16:18.183 INFO:teuthology.orchestra.run.smithi072.stdout: "pid": "54493", 2024-04-15T15:16:18.183 INFO:teuthology.orchestra.run.smithi072.stdout: "root": "/" 2024-04-15T15:16:18.183 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:18.183 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_count": 0, 2024-04-15T15:16:18.183 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_pinned_count": 0, 2024-04-15T15:16:18.183 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 5778, 2024-04-15T15:16:18.183 INFO:teuthology.orchestra.run.smithi072.stdout: "inst": { 2024-04-15T15:16:18.184 INFO:teuthology.orchestra.run.smithi072.stdout: "name": { 2024-04-15T15:16:18.184 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "client", 2024-04-15T15:16:18.184 INFO:teuthology.orchestra.run.smithi072.stdout: "num": 5778 2024-04-15T15:16:18.184 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:18.184 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-04-15T15:16:18.184 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-04-15T15:16:18.184 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:16:18.184 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 3747870602 2024-04-15T15:16:18.184 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:16:18.184 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:18.184 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-04-15T15:16:18.184 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-04-15T15:16:18.184 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:16:18.184 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 3747870602 2024-04-15T15:16:18.184 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:18.184 INFO:teuthology.orchestra.run.smithi072.stdout: "inst_str": "client.5778 192.168.0.1:0/3747870602", 2024-04-15T15:16:18.185 INFO:teuthology.orchestra.run.smithi072.stdout: "addr_str": "192.168.0.1:0/3747870602", 2024-04-15T15:16:18.185 INFO:teuthology.orchestra.run.smithi072.stdout: "inode_count": 1, 2024-04-15T15:16:18.185 INFO:teuthology.orchestra.run.smithi072.stdout: "mds_epoch": 16, 2024-04-15T15:16:18.185 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch": 38, 2024-04-15T15:16:18.185 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:16:18.185 INFO:teuthology.orchestra.run.smithi072.stdout: "blocklisted": false, 2024-04-15T15:16:18.185 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs" 2024-04-15T15:16:18.185 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-04-15T15:16:18.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:18.191 DEBUG:teuthology.orchestra.run.smithi072:> stat --file-system '--printf=%T 2024-04-15T15:16:18.191 DEBUG:teuthology.orchestra.run.smithi072:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-15T15:16:18.208 INFO:teuthology.orchestra.run.smithi072.stdout:fuseblk 2024-04-15T15:16:18.208 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-15T15:16:18.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:18.208 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-15T15:16:18.278 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:16:18.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.419+0000 7f244d288640 1 Processor -- start 2024-04-15T15:16:18.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.419+0000 7f244d288640 1 -- start start 2024-04-15T15:16:18.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.419+0000 7f244d288640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2448161e40 0x7f24480b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:18.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.419+0000 7f244d288640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f24480d45b0 0x7f24480d4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:18.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.419+0000 7f244d288640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2448162220 con 0x7f24480d12b0 2024-04-15T15:16:18.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.419+0000 7f244d288640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f24480d4f60 con 0x7f2448161e40 2024-04-15T15:16:18.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.419+0000 7f244d288640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f24480d21f0 con 0x7f24480d45b0 2024-04-15T15:16:18.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.420+0000 7f2446ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2448161e40 0x7f24480b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:18.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.420+0000 7f24467fc640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f24480d12b0 0x7f24480d2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:34254/0 (socket says 172.21.15.72:34254) 2024-04-15T15:16:18.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.420+0000 7f24467fc640 1 -- 172.21.15.72:0/2148810583 learned_addr learned my addr 172.21.15.72:0/2148810583 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:18.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.420+0000 7f2446ffd640 1 -- 172.21.15.72:0/2148810583 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f24480d45b0 msgr2=0x7f24480d4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:18.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.420+0000 7f2446ffd640 1 --2- 172.21.15.72:0/2148810583 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f24480d45b0 0x7f24480d4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:18.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.420+0000 7f2446ffd640 1 -- 172.21.15.72:0/2148810583 >> v1:172.21.15.72:6789/0 conn(0x7f24480d12b0 legacy=0x7f24480d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:18.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.420+0000 7f2446ffd640 1 -- 172.21.15.72:0/2148810583 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24480d3070 con 0x7f2448161e40 2024-04-15T15:16:18.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.420+0000 7f2445ffb640 1 -- 172.21.15.72:0/2148810583 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3964958170 0 0) 0x7f2448162220 con 0x7f24480d12b0 2024-04-15T15:16:18.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.421+0000 7f2446ffd640 1 --2- 172.21.15.72:0/2148810583 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2448161e40 0x7f24480b37f0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f24380097d0 tx=0x7f243802b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1ab79585aacc4a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:18.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.421+0000 7f2445ffb640 1 -- 172.21.15.72:0/2148810583 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2438031770 con 0x7f2448161e40 2024-04-15T15:16:18.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.421+0000 7f2445ffb640 1 -- 172.21.15.72:0/2148810583 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2438037070 con 0x7f2448161e40 2024-04-15T15:16:18.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.421+0000 7f2445ffb640 1 -- 172.21.15.72:0/2148810583 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2438031b30 con 0x7f2448161e40 2024-04-15T15:16:18.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.421+0000 7f244d288640 1 -- 172.21.15.72:0/2148810583 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2448161e40 msgr2=0x7f24480b37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:18.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.421+0000 7f244d288640 1 --2- 172.21.15.72:0/2148810583 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2448161e40 0x7f24480b37f0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f24380097d0 tx=0x7f243802b810 comp rx=0 tx=0).stop 2024-04-15T15:16:18.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.421+0000 7f244d288640 1 -- 172.21.15.72:0/2148810583 shutdown_connections 2024-04-15T15:16:18.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.421+0000 7f244d288640 1 --2- 172.21.15.72:0/2148810583 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f24480d45b0 0x7f24480d4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:18.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.421+0000 7f244d288640 1 --2- 172.21.15.72:0/2148810583 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2448161e40 0x7f24480b37f0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:18.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.421+0000 7f244d288640 1 -- 172.21.15.72:0/2148810583 >> 172.21.15.72:0/2148810583 conn(0x7f244809d970 msgr2=0x7f24480b95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:18.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.421+0000 7f244d288640 1 -- 172.21.15.72:0/2148810583 shutdown_connections 2024-04-15T15:16:18.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.421+0000 7f244d288640 1 -- 172.21.15.72:0/2148810583 wait complete. 2024-04-15T15:16:18.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.423+0000 7f244d288640 1 Processor -- start 2024-04-15T15:16:18.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.423+0000 7f244d288640 1 -- start start 2024-04-15T15:16:18.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.423+0000 7f244d288640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24480d12b0 0x7f24481f41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:18.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.423+0000 7f244d288640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f24480d45b0 0x7f24480b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:18.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.423+0000 7f244d288640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2448161e40 0x7f24480b5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:18.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.423+0000 7f244d288640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f24480d1de0 con 0x7f24480d12b0 2024-04-15T15:16:18.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.423+0000 7f244d288640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f24480d1b00 con 0x7f24480d45b0 2024-04-15T15:16:18.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.423+0000 7f244d288640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f24480d2440 con 0x7f2448161e40 2024-04-15T15:16:18.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.423+0000 7f2446ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24480d12b0 0x7f24481f41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:18.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.423+0000 7f2446ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24480d12b0 0x7f24481f41a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56304/0 (socket says 172.21.15.72:56304) 2024-04-15T15:16:18.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.423+0000 7f2446ffd640 1 -- 172.21.15.72:0/843105398 learned_addr learned my addr 172.21.15.72:0/843105398 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:18.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.424+0000 7f2446ffd640 1 -- 172.21.15.72:0/843105398 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2448161e40 msgr2=0x7f24480b5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:18.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.424+0000 7f2446ffd640 1 --2- 172.21.15.72:0/843105398 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2448161e40 0x7f24480b5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:18.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.424+0000 7f2446ffd640 1 -- 172.21.15.72:0/843105398 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f24480d45b0 msgr2=0x7f24480b3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:18.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.424+0000 7f2446ffd640 1 --2- 172.21.15.72:0/843105398 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f24480d45b0 0x7f24480b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:18.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.424+0000 7f2446ffd640 1 -- 172.21.15.72:0/843105398 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24480b4800 con 0x7f24480d12b0 2024-04-15T15:16:18.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.424+0000 7f2446ffd640 1 --2- 172.21.15.72:0/843105398 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24480d12b0 0x7f24481f41a0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f24380097d0 tx=0x7f24380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:18.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.424+0000 7f242f7fe640 1 -- 172.21.15.72:0/843105398 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2438004230 con 0x7f24480d12b0 2024-04-15T15:16:18.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.424+0000 7f242f7fe640 1 -- 172.21.15.72:0/843105398 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24380043d0 con 0x7f24480d12b0 2024-04-15T15:16:18.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.424+0000 7f244d288640 1 -- 172.21.15.72:0/843105398 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24480b4ae0 con 0x7f24480d12b0 2024-04-15T15:16:18.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.424+0000 7f242f7fe640 1 -- 172.21.15.72:0/843105398 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2438003850 con 0x7f24480d12b0 2024-04-15T15:16:18.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.424+0000 7f244d288640 1 -- 172.21.15.72:0/843105398 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24480cbb70 con 0x7f24480d12b0 2024-04-15T15:16:18.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.425+0000 7f244d288640 1 -- 172.21.15.72:0/843105398 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24480689e0 con 0x7f24480d12b0 2024-04-15T15:16:18.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.425+0000 7f242f7fe640 1 -- 172.21.15.72:0/843105398 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2438031070 con 0x7f24480d12b0 2024-04-15T15:16:18.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.426+0000 7f242f7fe640 1 --2- 172.21.15.72:0/843105398 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f243406c8f0 0x7f243406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:18.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.426+0000 7f242f7fe640 1 -- 172.21.15.72:0/843105398 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f24380ad840 con 0x7f24480d12b0 2024-04-15T15:16:18.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.426+0000 7f24467fc640 1 --2- 172.21.15.72:0/843105398 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f243406c8f0 0x7f243406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:18.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.426+0000 7f24467fc640 1 --2- 172.21.15.72:0/843105398 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f243406c8f0 0x7f243406edb0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f243c007fd0 tx=0x7f243c007960 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:18.432 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.431+0000 7f242f7fe640 1 -- 172.21.15.72:0/843105398 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2438041070 con 0x7f24480d12b0 2024-04-15T15:16:18.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.644+0000 7f244d288640 1 -- 172.21.15.72:0/843105398 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f24480b6f00 con 0x7f24480d12b0 2024-04-15T15:16:18.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.644+0000 7f242f7fe640 1 -- 172.21.15.72:0/843105398 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+73 (secure 0 0 0) 0x7f243807bae0 con 0x7f24480d12b0 2024-04-15T15:16:18.647 INFO:teuthology.orchestra.run.smithi072.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-15T15:16:18.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.646+0000 7f244d288640 1 -- 172.21.15.72:0/843105398 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f243406c8f0 msgr2=0x7f243406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:18.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.646+0000 7f244d288640 1 --2- 172.21.15.72:0/843105398 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f243406c8f0 0x7f243406edb0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f243c007fd0 tx=0x7f243c007960 comp rx=0 tx=0).stop 2024-04-15T15:16:18.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.646+0000 7f244d288640 1 -- 172.21.15.72:0/843105398 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24480d12b0 msgr2=0x7f24481f41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:18.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.646+0000 7f244d288640 1 --2- 172.21.15.72:0/843105398 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24480d12b0 0x7f24481f41a0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f24380097d0 tx=0x7f24380046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:18.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.646+0000 7f244d288640 1 -- 172.21.15.72:0/843105398 shutdown_connections 2024-04-15T15:16:18.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.646+0000 7f244d288640 1 --2- 172.21.15.72:0/843105398 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f243406c8f0 0x7f243406edb0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:18.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.646+0000 7f244d288640 1 --2- 172.21.15.72:0/843105398 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2448161e40 0x7f24480b5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:18.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.646+0000 7f244d288640 1 --2- 172.21.15.72:0/843105398 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f24480d45b0 0x7f24480b3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:18.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.646+0000 7f244d288640 1 --2- 172.21.15.72:0/843105398 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f24480d12b0 0x7f24481f41a0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:18.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.646+0000 7f244d288640 1 -- 172.21.15.72:0/843105398 >> 172.21.15.72:0/843105398 conn(0x7f244809d970 msgr2=0x7f24480b7240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:18.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.646+0000 7f244d288640 1 -- 172.21.15.72:0/843105398 shutdown_connections 2024-04-15T15:16:18.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:18.646+0000 7f244d288640 1 -- 172.21.15.72:0/843105398 wait complete. 2024-04-15T15:16:18.659 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:16:18.659 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:16:18.660 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi175.front.sepia.ceph.com 2024-04-15T15:16:18.660 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-15T15:16:18.660 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:18.660 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:16:18.660 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-15T15:16:18.660 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:16:18.660 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-15T15:16:18.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:18.660 DEBUG:teuthology.orchestra.run.smithi175:> ip addr 2024-04-15T15:16:18.681 INFO:teuthology.orchestra.run.smithi175.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:16:18.682 INFO:teuthology.orchestra.run.smithi175.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:16:18.682 INFO:teuthology.orchestra.run.smithi175.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:16:18.682 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:18.682 INFO:teuthology.orchestra.run.smithi175.stdout: inet6 ::1/128 scope host 2024-04-15T15:16:18.682 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:18.682 INFO:teuthology.orchestra.run.smithi175.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:18.682 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether 0c:c4:7a:d9:39:ee brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:18.682 INFO:teuthology.orchestra.run.smithi175.stdout: altname enp7s0f0 2024-04-15T15:16:18.682 INFO:teuthology.orchestra.run.smithi175.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:18.682 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether 0c:c4:7a:d9:39:ef brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:18.682 INFO:teuthology.orchestra.run.smithi175.stdout: altname enp7s0f1 2024-04-15T15:16:18.683 INFO:teuthology.orchestra.run.smithi175.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:18.683 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether 0c:c4:7a:88:71:f0 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:18.683 INFO:teuthology.orchestra.run.smithi175.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:16:18.683 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether 0c:c4:7a:88:71:f1 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:18.683 INFO:teuthology.orchestra.run.smithi175.stdout: inet 172.21.15.175/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-15T15:16:18.683 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft 42619sec preferred_lft 42619sec 2024-04-15T15:16:18.683 INFO:teuthology.orchestra.run.smithi175.stdout: inet6 fe80::ec4:7aff:fe88:71f1/64 scope link 2024-04-15T15:16:18.683 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:18.683 INFO:teuthology.orchestra.run.smithi175.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:16:18.683 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether 6e:00:c4:f8:75:e4 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:18.683 INFO:teuthology.orchestra.run.smithi175.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-15T15:16:18.684 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:18.684 INFO:teuthology.orchestra.run.smithi175.stdout: inet6 fe80::d4be:d9ff:fe0d:518c/64 scope link 2024-04-15T15:16:18.684 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:18.684 INFO:teuthology.orchestra.run.smithi175.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:16:18.684 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether f2:c6:d9:f6:5e:3e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:18.684 INFO:teuthology.orchestra.run.smithi175.stdout: inet6 fe80::f0c6:d9ff:fef6:5e3e/64 scope link 2024-04-15T15:16:18.684 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:18.684 INFO:teuthology.orchestra.run.smithi175.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:16:18.684 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether 6e:00:c4:f8:75:e4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:16:18.684 INFO:teuthology.orchestra.run.smithi175.stdout: inet6 fe80::6c00:c4ff:fef8:75e4/64 scope link 2024-04-15T15:16:18.684 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:18.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:18.686 DEBUG:teuthology.orchestra.run.smithi175:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:16:18.753 INFO:teuthology.orchestra.run.smithi175.stdout:1 2024-04-15T15:16:18.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:18.755 DEBUG:teuthology.orchestra.run.smithi175:> route 2024-04-15T15:16:18.813 INFO:teuthology.orchestra.run.smithi175.stdout:Kernel IP routing table 2024-04-15T15:16:18.814 INFO:teuthology.orchestra.run.smithi175.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:16:18.814 INFO:teuthology.orchestra.run.smithi175.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-15T15:16:18.814 INFO:teuthology.orchestra.run.smithi175.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-15T15:16:18.814 INFO:teuthology.orchestra.run.smithi175.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:16:18.814 DEBUG:teuthology.orchestra.run.smithi175:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:16:18.814 DEBUG:teuthology.orchestra.run.smithi175:> set -e 2024-04-15T15:16:18.814 DEBUG:teuthology.orchestra.run.smithi175:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-15T15:16:18.814 DEBUG:teuthology.orchestra.run.smithi175:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-15T15:16:18.814 DEBUG:teuthology.orchestra.run.smithi175:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-15T15:16:18.814 DEBUG:teuthology.orchestra.run.smithi175:> ') 2024-04-15T15:16:18.907 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:16:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:16:18.953 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:16:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:18.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:18.959 DEBUG:teuthology.orchestra.run.smithi175:> ip netns list 2024-04-15T15:16:18.975 INFO:teuthology.orchestra.run.smithi175.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-15T15:16:18.975 INFO:teuthology.orchestra.run.smithi175.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-15T15:16:18.975 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-15T15:16:18.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:18.975 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip link set brx.0 up 2024-04-15T15:16:19.044 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-15T15:16:19.044 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:16:19.045 DEBUG:teuthology.orchestra.run.smithi175:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:19.099 INFO:teuthology.orchestra.run.smithi175.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-15T15:16:19.099 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:16:19.100 DEBUG:teuthology.orchestra.run.smithi175:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:19.155 DEBUG:teuthology.orchestra.run.smithi175:> sudo modprobe fuse 2024-04-15T15:16:19.225 DEBUG:teuthology.orchestra.run.smithi175:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:16:19.282 INFO:teuthology.orchestra.run.smithi175.stdout:/proc 2024-04-15T15:16:19.283 INFO:teuthology.orchestra.run.smithi175.stdout:/sys 2024-04-15T15:16:19.283 INFO:teuthology.orchestra.run.smithi175.stdout:/dev 2024-04-15T15:16:19.283 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/security 2024-04-15T15:16:19.283 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/shm 2024-04-15T15:16:19.283 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/pts 2024-04-15T15:16:19.283 INFO:teuthology.orchestra.run.smithi175.stdout:/run 2024-04-15T15:16:19.283 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/cgroup 2024-04-15T15:16:19.284 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/pstore 2024-04-15T15:16:19.284 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/bpf 2024-04-15T15:16:19.284 INFO:teuthology.orchestra.run.smithi175.stdout:/ 2024-04-15T15:16:19.284 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/selinux 2024-04-15T15:16:19.284 INFO:teuthology.orchestra.run.smithi175.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:16:19.284 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/mqueue 2024-04-15T15:16:19.284 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/hugepages 2024-04-15T15:16:19.284 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/debug 2024-04-15T15:16:19.285 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/tracing 2024-04-15T15:16:19.285 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/fuse/connections 2024-04-15T15:16:19.285 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/config 2024-04-15T15:16:19.285 INFO:teuthology.orchestra.run.smithi175.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:16:19.285 INFO:teuthology.orchestra.run.smithi175.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:16:19.285 INFO:teuthology.orchestra.run.smithi175.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:16:19.285 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:16:19.285 INFO:teuthology.orchestra.run.smithi175.stdout:/run/user/1000 2024-04-15T15:16:19.286 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph 2024-04-15T15:16:19.286 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:16:19.286 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:16:19.286 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:16:19.286 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:16:19.286 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns 2024-04-15T15:16:19.286 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:19.287 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:19.287 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:16:19.287 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:16:19.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:19.287 DEBUG:teuthology.orchestra.run.smithi175:> ls /sys/fs/fuse/connections 2024-04-15T15:16:19.338 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-15T15:16:19.338 DEBUG:teuthology.orchestra.run.smithi175:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-04-15T15:16:19.381 DEBUG:teuthology.orchestra.run.smithi175:> sudo modprobe fuse 2024-04-15T15:16:19.409 DEBUG:teuthology.orchestra.run.smithi175:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:16:19.466 INFO:teuthology.orchestra.run.smithi175.stdout:/proc 2024-04-15T15:16:19.466 INFO:teuthology.orchestra.run.smithi175.stdout:/sys 2024-04-15T15:16:19.466 INFO:teuthology.orchestra.run.smithi175.stdout:/dev 2024-04-15T15:16:19.467 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/security 2024-04-15T15:16:19.467 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/shm 2024-04-15T15:16:19.467 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/pts 2024-04-15T15:16:19.468 INFO:teuthology.orchestra.run.smithi175.stdout:/run 2024-04-15T15:16:19.468 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/cgroup 2024-04-15T15:16:19.468 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/pstore 2024-04-15T15:16:19.468 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/bpf 2024-04-15T15:16:19.468 INFO:teuthology.orchestra.run.smithi175.stdout:/ 2024-04-15T15:16:19.468 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/selinux 2024-04-15T15:16:19.468 INFO:teuthology.orchestra.run.smithi175.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:16:19.468 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/mqueue 2024-04-15T15:16:19.468 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/hugepages 2024-04-15T15:16:19.468 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/debug 2024-04-15T15:16:19.468 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/tracing 2024-04-15T15:16:19.469 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/fuse/connections 2024-04-15T15:16:19.469 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/config 2024-04-15T15:16:19.469 INFO:teuthology.orchestra.run.smithi175.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:16:19.469 INFO:teuthology.orchestra.run.smithi175.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:16:19.469 INFO:teuthology.orchestra.run.smithi175.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:16:19.469 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:16:19.469 INFO:teuthology.orchestra.run.smithi175.stdout:/run/user/1000 2024-04-15T15:16:19.470 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph 2024-04-15T15:16:19.470 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:16:19.470 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:16:19.470 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:16:19.470 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:16:19.470 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns 2024-04-15T15:16:19.470 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:19.470 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:19.470 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:16:19.471 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:16:19.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:19.472 DEBUG:teuthology.orchestra.run.smithi175:> ls /sys/fs/fuse/connections 2024-04-15T15:16:19.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi175.stderr:2024-04-15T15:16:19.468+0000 7f90a5844480 -1 init, newargv = 0x5638fb4e8f90 newargc=15 2024-04-15T15:16:19.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi175.stderr:ceph-fuse[45005]: starting ceph client 2024-04-15T15:16:19.475 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi175.stderr:ceph-fuse[45005]: starting fuse 2024-04-15T15:16:19.489 INFO:teuthology.orchestra.run.smithi175.stdout:43 2024-04-15T15:16:19.490 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-04-15T15:16:19.490 DEBUG:teuthology.orchestra.run.smithi175:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:16:19.490 DEBUG:teuthology.orchestra.run.smithi175:> import glob 2024-04-15T15:16:19.490 DEBUG:teuthology.orchestra.run.smithi175:> import re 2024-04-15T15:16:19.490 DEBUG:teuthology.orchestra.run.smithi175:> import os 2024-04-15T15:16:19.490 DEBUG:teuthology.orchestra.run.smithi175:> import subprocess 2024-04-15T15:16:19.490 DEBUG:teuthology.orchestra.run.smithi175:> 2024-04-15T15:16:19.490 DEBUG:teuthology.orchestra.run.smithi175:> def _find_admin_socket(client_name): 2024-04-15T15:16:19.491 DEBUG:teuthology.orchestra.run.smithi175:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-15T15:16:19.491 DEBUG:teuthology.orchestra.run.smithi175:> files = glob.glob(asok_path) 2024-04-15T15:16:19.491 DEBUG:teuthology.orchestra.run.smithi175:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-15T15:16:19.491 DEBUG:teuthology.orchestra.run.smithi175:> 2024-04-15T15:16:19.491 DEBUG:teuthology.orchestra.run.smithi175:> # Given a non-glob path, it better be there 2024-04-15T15:16:19.491 DEBUG:teuthology.orchestra.run.smithi175:> if "*" not in asok_path: 2024-04-15T15:16:19.491 DEBUG:teuthology.orchestra.run.smithi175:> assert(len(files) == 1) 2024-04-15T15:16:19.491 DEBUG:teuthology.orchestra.run.smithi175:> return files[0] 2024-04-15T15:16:19.491 DEBUG:teuthology.orchestra.run.smithi175:> 2024-04-15T15:16:19.491 DEBUG:teuthology.orchestra.run.smithi175:> for f in files: 2024-04-15T15:16:19.491 DEBUG:teuthology.orchestra.run.smithi175:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:16:19.491 DEBUG:teuthology.orchestra.run.smithi175:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:16:19.491 DEBUG:teuthology.orchestra.run.smithi175:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:16:19.491 DEBUG:teuthology.orchestra.run.smithi175:> contents = proc_f.read() 2024-04-15T15:16:19.491 DEBUG:teuthology.orchestra.run.smithi175:> if mountpoint in contents: 2024-04-15T15:16:19.491 DEBUG:teuthology.orchestra.run.smithi175:> return f 2024-04-15T15:16:19.492 DEBUG:teuthology.orchestra.run.smithi175:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:16:19.492 DEBUG:teuthology.orchestra.run.smithi175:> 2024-04-15T15:16:19.492 DEBUG:teuthology.orchestra.run.smithi175:> print(_find_admin_socket("client.1")) 2024-04-15T15:16:19.492 DEBUG:teuthology.orchestra.run.smithi175:> ' 2024-04-15T15:16:19.612 INFO:teuthology.orchestra.run.smithi175.stdout:/var/run/ceph/ceph-client.1.45005.asok 2024-04-15T15:16:19.615 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:16:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:19.622 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.45005.asok 2024-04-15T15:16:19.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:19.622 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.45005.asok status 2024-04-15T15:16:19.761 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-04-15T15:16:19.761 INFO:teuthology.orchestra.run.smithi175.stdout: "metadata": { 2024-04-15T15:16:19.761 INFO:teuthology.orchestra.run.smithi175.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:16:19.762 INFO:teuthology.orchestra.run.smithi175.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:16:19.762 INFO:teuthology.orchestra.run.smithi175.stdout: "entity_id": "1", 2024-04-15T15:16:19.762 INFO:teuthology.orchestra.run.smithi175.stdout: "hostname": "smithi175", 2024-04-15T15:16:19.762 INFO:teuthology.orchestra.run.smithi175.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-15T15:16:19.762 INFO:teuthology.orchestra.run.smithi175.stdout: "pid": "45005", 2024-04-15T15:16:19.762 INFO:teuthology.orchestra.run.smithi175.stdout: "root": "/" 2024-04-15T15:16:19.762 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-15T15:16:19.762 INFO:teuthology.orchestra.run.smithi175.stdout: "dentry_count": 0, 2024-04-15T15:16:19.762 INFO:teuthology.orchestra.run.smithi175.stdout: "dentry_pinned_count": 0, 2024-04-15T15:16:19.762 INFO:teuthology.orchestra.run.smithi175.stdout: "id": 5275, 2024-04-15T15:16:19.762 INFO:teuthology.orchestra.run.smithi175.stdout: "inst": { 2024-04-15T15:16:19.762 INFO:teuthology.orchestra.run.smithi175.stdout: "name": { 2024-04-15T15:16:19.763 INFO:teuthology.orchestra.run.smithi175.stdout: "type": "client", 2024-04-15T15:16:19.763 INFO:teuthology.orchestra.run.smithi175.stdout: "num": 5275 2024-04-15T15:16:19.763 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-15T15:16:19.763 INFO:teuthology.orchestra.run.smithi175.stdout: "addr": { 2024-04-15T15:16:19.763 INFO:teuthology.orchestra.run.smithi175.stdout: "type": "v1", 2024-04-15T15:16:19.763 INFO:teuthology.orchestra.run.smithi175.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:16:19.763 INFO:teuthology.orchestra.run.smithi175.stdout: "nonce": 1378074482 2024-04-15T15:16:19.763 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-15T15:16:19.763 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-15T15:16:19.763 INFO:teuthology.orchestra.run.smithi175.stdout: "addr": { 2024-04-15T15:16:19.763 INFO:teuthology.orchestra.run.smithi175.stdout: "type": "v1", 2024-04-15T15:16:19.763 INFO:teuthology.orchestra.run.smithi175.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:16:19.763 INFO:teuthology.orchestra.run.smithi175.stdout: "nonce": 1378074482 2024-04-15T15:16:19.763 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-15T15:16:19.763 INFO:teuthology.orchestra.run.smithi175.stdout: "inst_str": "client.5275 192.168.0.1:0/1378074482", 2024-04-15T15:16:19.764 INFO:teuthology.orchestra.run.smithi175.stdout: "addr_str": "192.168.0.1:0/1378074482", 2024-04-15T15:16:19.764 INFO:teuthology.orchestra.run.smithi175.stdout: "inode_count": 1, 2024-04-15T15:16:19.764 INFO:teuthology.orchestra.run.smithi175.stdout: "mds_epoch": 16, 2024-04-15T15:16:19.764 INFO:teuthology.orchestra.run.smithi175.stdout: "osd_epoch": 38, 2024-04-15T15:16:19.764 INFO:teuthology.orchestra.run.smithi175.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:16:19.764 INFO:teuthology.orchestra.run.smithi175.stdout: "blocklisted": false, 2024-04-15T15:16:19.764 INFO:teuthology.orchestra.run.smithi175.stdout: "fs_name": "cephfs" 2024-04-15T15:16:19.764 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-04-15T15:16:19.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:19.770 DEBUG:teuthology.orchestra.run.smithi175:> stat --file-system '--printf=%T 2024-04-15T15:16:19.770 DEBUG:teuthology.orchestra.run.smithi175:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:19.786 INFO:teuthology.orchestra.run.smithi175.stdout:fuseblk 2024-04-15T15:16:19.786 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:19.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:19.786 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:19.855 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds debug_mds 25 2024-04-15T15:16:19.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.972+0000 7f75d8700640 1 Processor -- start 2024-04-15T15:16:19.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.972+0000 7f75d8700640 1 -- start start 2024-04-15T15:16:19.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.973+0000 7f75d8700640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f75d015f9f0 0x7f75d00cca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:19.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.973+0000 7f75d8700640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f75d00d0010 0x7f75d00d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:19.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.973+0000 7f75d8700640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75d006df30 con 0x7f75d015c250 2024-04-15T15:16:19.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.973+0000 7f75d8700640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f75d006e210 con 0x7f75d00d0010 2024-04-15T15:16:19.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.973+0000 7f75d8700640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f75d006e4f0 con 0x7f75d015f9f0 2024-04-15T15:16:19.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.973+0000 7f75d5c9b640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f75d015f9f0 0x7f75d00cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:19.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.973+0000 7f75d6c9d640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f75d00d0010 0x7f75d00d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:19.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.973+0000 7f75d5c9b640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f75d015f9f0 0x7f75d00cca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3301/0 says I am v2:172.21.15.72:47714/0 (socket says 172.21.15.72:47714) 2024-04-15T15:16:19.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.973+0000 7f75d5c9b640 1 -- 172.21.15.72:0/4205001837 learned_addr learned my addr 172.21.15.72:0/4205001837 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:19.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.974+0000 7f75d6c9d640 1 -- 172.21.15.72:0/4205001837 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f75d015f9f0 msgr2=0x7f75d00cca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:19.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.974+0000 7f75d6c9d640 1 --2- 172.21.15.72:0/4205001837 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f75d015f9f0 0x7f75d00cca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:19.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.974+0000 7f75d6c9d640 1 -- 172.21.15.72:0/4205001837 >> v1:172.21.15.175:6789/0 conn(0x7f75d015c250 legacy=0x7f75d006d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:19.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.974+0000 7f75d6c9d640 1 -- 172.21.15.72:0/4205001837 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75d00b3610 con 0x7f75d00d0010 2024-04-15T15:16:19.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.974+0000 7f75d549a640 1 -- 172.21.15.72:0/4205001837 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 706603648 0 0) 0x7f75d006df30 con 0x7f75d015c250 2024-04-15T15:16:19.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.974+0000 7f75d5c9b640 1 --2- 172.21.15.72:0/4205001837 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f75d015f9f0 0x7f75d00cca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:19.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.974+0000 7f75d6c9d640 1 --2- 172.21.15.72:0/4205001837 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75d00d0010 0x7f75d00d04a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f75c8009d90 tx=0x7f75c802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c688d695e960257f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:19.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.975+0000 7f75d549a640 1 -- 172.21.15.72:0/4205001837 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c8039070 con 0x7f75d00d0010 2024-04-15T15:16:19.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.975+0000 7f75d549a640 1 -- 172.21.15.72:0/4205001837 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75c802cc50 con 0x7f75d00d0010 2024-04-15T15:16:19.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.975+0000 7f75d549a640 1 -- 172.21.15.72:0/4205001837 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75c8034440 con 0x7f75d00d0010 2024-04-15T15:16:19.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.975+0000 7f75d8700640 1 -- 172.21.15.72:0/4205001837 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75d00d0010 msgr2=0x7f75d00d04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:19.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.975+0000 7f75d8700640 1 --2- 172.21.15.72:0/4205001837 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75d00d0010 0x7f75d00d04a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f75c8009d90 tx=0x7f75c802bea0 comp rx=0 tx=0).stop 2024-04-15T15:16:19.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.975+0000 7f75d8700640 1 -- 172.21.15.72:0/4205001837 shutdown_connections 2024-04-15T15:16:19.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.975+0000 7f75d8700640 1 --2- 172.21.15.72:0/4205001837 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75d00d0010 0x7f75d00d04a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:19.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.975+0000 7f75d8700640 1 --2- 172.21.15.72:0/4205001837 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f75d015f9f0 0x7f75d00cca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:19.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.975+0000 7f75d8700640 1 -- 172.21.15.72:0/4205001837 >> 172.21.15.72:0/4205001837 conn(0x7f75d00ceec0 msgr2=0x7f75d00cf2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:19.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.975+0000 7f75d8700640 1 -- 172.21.15.72:0/4205001837 shutdown_connections 2024-04-15T15:16:19.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.975+0000 7f75d8700640 1 -- 172.21.15.72:0/4205001837 wait complete. 2024-04-15T15:16:19.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.976+0000 7f75d8700640 1 Processor -- start 2024-04-15T15:16:19.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.976+0000 7f75d8700640 1 -- start start 2024-04-15T15:16:19.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.977+0000 7f75d8700640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75d00d0010 0x7f75d0155270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:19.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.977+0000 7f75d8700640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75d015c250 0x7f75d00c9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:19.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.977+0000 7f75d8700640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f75d015f9f0 0x7f75d00cb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:19.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.977+0000 7f75d8700640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f75d015c7b0 con 0x7f75d00d0010 2024-04-15T15:16:19.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.977+0000 7f75d8700640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f75d015c630 con 0x7f75d015c250 2024-04-15T15:16:19.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.977+0000 7f75d8700640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f75d00ccfd0 con 0x7f75d015f9f0 2024-04-15T15:16:19.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.977+0000 7f75d649c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75d00d0010 0x7f75d0155270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:19.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.977+0000 7f75d649c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75d00d0010 0x7f75d0155270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56312/0 (socket says 172.21.15.72:56312) 2024-04-15T15:16:19.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.977+0000 7f75d649c640 1 -- 172.21.15.72:0/1166108875 learned_addr learned my addr 172.21.15.72:0/1166108875 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:19.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.978+0000 7f75d5c9b640 1 --2- 172.21.15.72:0/1166108875 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75d015c250 0x7f75d00c9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:19.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.978+0000 7f75d6c9d640 1 --2- 172.21.15.72:0/1166108875 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f75d015f9f0 0x7f75d00cb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:19.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.978+0000 7f75d649c640 1 -- 172.21.15.72:0/1166108875 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f75d015f9f0 msgr2=0x7f75d00cb990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:19.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.978+0000 7f75d649c640 1 --2- 172.21.15.72:0/1166108875 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f75d015f9f0 0x7f75d00cb990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:19.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.978+0000 7f75d649c640 1 -- 172.21.15.72:0/1166108875 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75d015c250 msgr2=0x7f75d00c9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:19.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.978+0000 7f75d649c640 1 --2- 172.21.15.72:0/1166108875 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75d015c250 0x7f75d00c9510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:19.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.978+0000 7f75d649c640 1 -- 172.21.15.72:0/1166108875 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75d00c73b0 con 0x7f75d00d0010 2024-04-15T15:16:19.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.978+0000 7f75d5c9b640 1 --2- 172.21.15.72:0/1166108875 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75d015c250 0x7f75d00c9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:19.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.978+0000 7f75d649c640 1 --2- 172.21.15.72:0/1166108875 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75d00d0010 0x7f75d0155270 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f75cc00c4e0 tx=0x7f75cc00c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:19.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.978+0000 7f75c6ffd640 1 -- 172.21.15.72:0/1166108875 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75cc017070 con 0x7f75d00d0010 2024-04-15T15:16:19.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.978+0000 7f75c6ffd640 1 -- 172.21.15.72:0/1166108875 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75cc0035f0 con 0x7f75d00d0010 2024-04-15T15:16:19.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.978+0000 7f75d8700640 1 -- 172.21.15.72:0/1166108875 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75d00c9fd0 con 0x7f75d00d0010 2024-04-15T15:16:19.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.978+0000 7f75c6ffd640 1 -- 172.21.15.72:0/1166108875 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75cc0039e0 con 0x7f75d00d0010 2024-04-15T15:16:19.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.978+0000 7f75d8700640 1 -- 172.21.15.72:0/1166108875 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75d00ca590 con 0x7f75d00d0010 2024-04-15T15:16:19.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.979+0000 7f75d8700640 1 -- 172.21.15.72:0/1166108875 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75d0075c50 con 0x7f75d00d0010 2024-04-15T15:16:19.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.980+0000 7f75c6ffd640 1 -- 172.21.15.72:0/1166108875 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f75cc01a070 con 0x7f75d00d0010 2024-04-15T15:16:19.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.980+0000 7f75c6ffd640 1 --2- 172.21.15.72:0/1166108875 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f75b406c8f0 0x7f75b406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:19.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.980+0000 7f75d5c9b640 1 --2- 172.21.15.72:0/1166108875 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f75b406c8f0 0x7f75b406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:19.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.980+0000 7f75c6ffd640 1 -- 172.21.15.72:0/1166108875 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f75cc08afa0 con 0x7f75d00d0010 2024-04-15T15:16:19.983 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.981+0000 7f75d5c9b640 1 --2- 172.21.15.72:0/1166108875 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f75b406c8f0 0x7f75b406edb0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f75d01557b0 tx=0x7f75c0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:19.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:19.985+0000 7f75c6ffd640 1 -- 172.21.15.72:0/1166108875 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f75d00c73b0 con 0x7f75d00d0010 2024-04-15T15:16:20.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.160+0000 7f75d8700640 1 -- 172.21.15.72:0/1166108875 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config set, name=debug_mds}] v 0) -- 0x7f75d00cf620 con 0x7f75d00d0010 2024-04-15T15:16:20.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.163+0000 7f75c6ffd640 1 -- 172.21.15.72:0/1166108875 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds}]=0 v11)=0 v11) ==== 108+0+0 (secure 0 0 0) 0x7f75cc058c00 con 0x7f75d00d0010 2024-04-15T15:16:20.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.166+0000 7f75d8700640 1 -- 172.21.15.72:0/1166108875 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f75b406c8f0 msgr2=0x7f75b406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.166+0000 7f75d8700640 1 --2- 172.21.15.72:0/1166108875 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f75b406c8f0 0x7f75b406edb0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f75d01557b0 tx=0x7f75c0008040 comp rx=0 tx=0).stop 2024-04-15T15:16:20.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.166+0000 7f75d8700640 1 -- 172.21.15.72:0/1166108875 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75d00d0010 msgr2=0x7f75d0155270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.166+0000 7f75d8700640 1 --2- 172.21.15.72:0/1166108875 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75d00d0010 0x7f75d0155270 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f75cc00c4e0 tx=0x7f75cc00c9a0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.166+0000 7f75d8700640 1 -- 172.21.15.72:0/1166108875 shutdown_connections 2024-04-15T15:16:20.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.166+0000 7f75d8700640 1 --2- 172.21.15.72:0/1166108875 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f75b406c8f0 0x7f75b406edb0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.166+0000 7f75d8700640 1 --2- 172.21.15.72:0/1166108875 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f75d015f9f0 0x7f75d00cb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.166+0000 7f75d8700640 1 --2- 172.21.15.72:0/1166108875 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f75d015c250 0x7f75d00c9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.166+0000 7f75d8700640 1 --2- 172.21.15.72:0/1166108875 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f75d00d0010 0x7f75d0155270 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.166+0000 7f75d8700640 1 -- 172.21.15.72:0/1166108875 >> 172.21.15.72:0/1166108875 conn(0x7f75d00ceec0 msgr2=0x7f75d006e730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:20.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.166+0000 7f75d8700640 1 -- 172.21.15.72:0/1166108875 shutdown_connections 2024-04-15T15:16:20.168 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.166+0000 7f75d8700640 1 -- 172.21.15.72:0/1166108875 wait complete. 2024-04-15T15:16:20.178 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_cache_quiesce_splitauth true 2024-04-15T15:16:20.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.282+0000 7f4b701b5640 1 Processor -- start 2024-04-15T15:16:20.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.283+0000 7f4b701b5640 1 -- start start 2024-04-15T15:16:20.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.283+0000 7f4b701b5640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f4b6815c5f0 0x7f4b680d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.283+0000 7f4b701b5640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4b680ce190 0x7f4b680d1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.283+0000 7f4b701b5640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b68160ff0 con 0x7f4b6806f510 2024-04-15T15:16:20.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.283+0000 7f4b701b5640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f4b68160da0 con 0x7f4b680ce190 2024-04-15T15:16:20.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.283+0000 7f4b701b5640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f4b681612d0 con 0x7f4b6815c5f0 2024-04-15T15:16:20.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.283+0000 7f4b6d750640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4b680ce190 0x7f4b680d1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.284+0000 7f4b6d750640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4b680ce190 0x7f4b680d1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56316/0 (socket says 172.21.15.72:56316) 2024-04-15T15:16:20.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.284+0000 7f4b6d750640 1 -- 172.21.15.72:0/932528845 learned_addr learned my addr 172.21.15.72:0/932528845 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:20.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.284+0000 7f4b6df51640 1 --2- 172.21.15.72:0/932528845 >> v2:172.21.15.175:3300/0 conn(0x7f4b6815c5f0 0x7f4b680d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.284+0000 7f4b6d750640 1 -- 172.21.15.72:0/932528845 >> v2:172.21.15.175:3300/0 conn(0x7f4b6815c5f0 msgr2=0x7f4b680d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:20.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.284+0000 7f4b6d750640 1 --2- 172.21.15.72:0/932528845 >> v2:172.21.15.175:3300/0 conn(0x7f4b6815c5f0 0x7f4b680d1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.284+0000 7f4b6d750640 1 -- 172.21.15.72:0/932528845 >> v1:172.21.15.175:6789/0 conn(0x7f4b6806f510 legacy=0x7f4b680ccb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.284+0000 7f4b6d750640 1 -- 172.21.15.72:0/932528845 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b680b5840 con 0x7f4b680ce190 2024-04-15T15:16:20.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.285+0000 7f4b6d750640 1 --2- 172.21.15.72:0/932528845 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b680ce190 0x7f4b680d1640 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f4b680696e0 tx=0x7f4b5802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8277591bc91b25d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:20.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.285+0000 7f4b6cf4f640 1 -- 172.21.15.72:0/932528845 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b58035020 con 0x7f4b680ce190 2024-04-15T15:16:20.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.285+0000 7f4b6cf4f640 1 -- 172.21.15.72:0/932528845 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b58031870 con 0x7f4b680ce190 2024-04-15T15:16:20.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.285+0000 7f4b6cf4f640 1 -- 172.21.15.72:0/932528845 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b58031c40 con 0x7f4b680ce190 2024-04-15T15:16:20.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.285+0000 7f4b701b5640 1 -- 172.21.15.72:0/932528845 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b680ce190 msgr2=0x7f4b680d1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.285+0000 7f4b701b5640 1 --2- 172.21.15.72:0/932528845 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b680ce190 0x7f4b680d1640 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f4b680696e0 tx=0x7f4b5802b910 comp rx=0 tx=0).stop 2024-04-15T15:16:20.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.285+0000 7f4b701b5640 1 -- 172.21.15.72:0/932528845 shutdown_connections 2024-04-15T15:16:20.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.285+0000 7f4b701b5640 1 --2- 172.21.15.72:0/932528845 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b680ce190 0x7f4b680d1640 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.285+0000 7f4b701b5640 1 --2- 172.21.15.72:0/932528845 >> v2:172.21.15.175:3300/0 conn(0x7f4b6815c5f0 0x7f4b680d1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.285+0000 7f4b701b5640 1 -- 172.21.15.72:0/932528845 >> 172.21.15.72:0/932528845 conn(0x7f4b680b4010 msgr2=0x7f4b680b4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:20.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.285+0000 7f4b701b5640 1 -- 172.21.15.72:0/932528845 shutdown_connections 2024-04-15T15:16:20.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.285+0000 7f4b701b5640 1 -- 172.21.15.72:0/932528845 wait complete. 2024-04-15T15:16:20.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.287+0000 7f4b701b5640 1 Processor -- start 2024-04-15T15:16:20.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.287+0000 7f4b701b5640 1 -- start start 2024-04-15T15:16:20.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.287+0000 7f4b701b5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b6806f510 0x7f4b681f1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.287+0000 7f4b701b5640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4b6815c5f0 0x7f4b68157710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.287+0000 7f4b701b5640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4b68157c50 0x7f4b681537a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.287+0000 7f4b701b5640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f4b680b5bb0 con 0x7f4b6806f510 2024-04-15T15:16:20.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.287+0000 7f4b701b5640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f4b680d23a0 con 0x7f4b6815c5f0 2024-04-15T15:16:20.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.287+0000 7f4b701b5640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f4b6815c9d0 con 0x7f4b68157c50 2024-04-15T15:16:20.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.287+0000 7f4b6df51640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b6806f510 0x7f4b681f1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.287+0000 7f4b6df51640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b6806f510 0x7f4b681f1aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56318/0 (socket says 172.21.15.72:56318) 2024-04-15T15:16:20.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.287+0000 7f4b6df51640 1 -- 172.21.15.72:0/200037985 learned_addr learned my addr 172.21.15.72:0/200037985 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:20.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.288+0000 7f4b6d750640 1 --2- 172.21.15.72:0/200037985 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4b6815c5f0 0x7f4b68157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.288+0000 7f4b6e752640 1 --2- 172.21.15.72:0/200037985 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4b68157c50 0x7f4b681537a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.288+0000 7f4b6df51640 1 -- 172.21.15.72:0/200037985 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4b68157c50 msgr2=0x7f4b681537a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.288+0000 7f4b6df51640 1 --2- 172.21.15.72:0/200037985 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4b68157c50 0x7f4b681537a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.288+0000 7f4b6df51640 1 -- 172.21.15.72:0/200037985 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4b6815c5f0 msgr2=0x7f4b68157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.288+0000 7f4b6df51640 1 --2- 172.21.15.72:0/200037985 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4b6815c5f0 0x7f4b68157710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.288+0000 7f4b6df51640 1 -- 172.21.15.72:0/200037985 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b6815fe30 con 0x7f4b6806f510 2024-04-15T15:16:20.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.288+0000 7f4b6e752640 1 --2- 172.21.15.72:0/200037985 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4b68157c50 0x7f4b681537a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:20.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.288+0000 7f4b6d750640 1 --2- 172.21.15.72:0/200037985 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4b6815c5f0 0x7f4b68157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:20.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.288+0000 7f4b6df51640 1 --2- 172.21.15.72:0/200037985 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b6806f510 0x7f4b681f1aa0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f4b64009d10 tx=0x7f4b6400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:20.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.288+0000 7f4b5e7fc640 1 -- 172.21.15.72:0/200037985 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b64018020 con 0x7f4b6806f510 2024-04-15T15:16:20.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.288+0000 7f4b5e7fc640 1 -- 172.21.15.72:0/200037985 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b64014480 con 0x7f4b6806f510 2024-04-15T15:16:20.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.288+0000 7f4b701b5640 1 -- 172.21.15.72:0/200037985 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b6815f590 con 0x7f4b6806f510 2024-04-15T15:16:20.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.288+0000 7f4b5e7fc640 1 -- 172.21.15.72:0/200037985 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b64014830 con 0x7f4b6806f510 2024-04-15T15:16:20.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.288+0000 7f4b701b5640 1 -- 172.21.15.72:0/200037985 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b6815fb50 con 0x7f4b6806f510 2024-04-15T15:16:20.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.289+0000 7f4b701b5640 1 -- 172.21.15.72:0/200037985 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b68075c50 con 0x7f4b6806f510 2024-04-15T15:16:20.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.290+0000 7f4b5e7fc640 1 -- 172.21.15.72:0/200037985 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4b64005d10 con 0x7f4b6806f510 2024-04-15T15:16:20.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.290+0000 7f4b5e7fc640 1 --2- 172.21.15.72:0/200037985 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4b4c06c8f0 0x7f4b4c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.290+0000 7f4b5e7fc640 1 -- 172.21.15.72:0/200037985 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f4b6408c750 con 0x7f4b6806f510 2024-04-15T15:16:20.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.290+0000 7f4b6d750640 1 --2- 172.21.15.72:0/200037985 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4b4c06c8f0 0x7f4b4c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.290+0000 7f4b6d750640 1 --2- 172.21.15.72:0/200037985 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4b4c06c8f0 0x7f4b4c06edb0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4b58036000 tx=0x7f4b5803c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:20.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.294+0000 7f4b5e7fc640 1 -- 172.21.15.72:0/200037985 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4b64022070 con 0x7f4b6806f510 2024-04-15T15:16:20.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.468+0000 7f4b701b5640 1 -- 172.21.15.72:0/200037985 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config set, name=mds_cache_quiesce_splitauth}] v 0) -- 0x7f4b6815bf20 con 0x7f4b6806f510 2024-04-15T15:16:20.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.474+0000 7f4b5e7fc640 1 -- 172.21.15.72:0/200037985 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_cache_quiesce_splitauth}]=0 v12)=0 v12) ==== 128+0+0 (secure 0 0 0) 0x7f4b64010040 con 0x7f4b6806f510 2024-04-15T15:16:20.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.476+0000 7f4b701b5640 1 -- 172.21.15.72:0/200037985 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4b4c06c8f0 msgr2=0x7f4b4c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.476+0000 7f4b701b5640 1 --2- 172.21.15.72:0/200037985 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4b4c06c8f0 0x7f4b4c06edb0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4b58036000 tx=0x7f4b5803c040 comp rx=0 tx=0).stop 2024-04-15T15:16:20.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.476+0000 7f4b701b5640 1 -- 172.21.15.72:0/200037985 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b6806f510 msgr2=0x7f4b681f1aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.476+0000 7f4b701b5640 1 --2- 172.21.15.72:0/200037985 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b6806f510 0x7f4b681f1aa0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f4b64009d10 tx=0x7f4b6400c490 comp rx=0 tx=0).stop 2024-04-15T15:16:20.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.476+0000 7f4b701b5640 1 -- 172.21.15.72:0/200037985 shutdown_connections 2024-04-15T15:16:20.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.476+0000 7f4b701b5640 1 --2- 172.21.15.72:0/200037985 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4b4c06c8f0 0x7f4b4c06edb0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.476+0000 7f4b701b5640 1 --2- 172.21.15.72:0/200037985 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4b68157c50 0x7f4b681537a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.476+0000 7f4b701b5640 1 --2- 172.21.15.72:0/200037985 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4b6815c5f0 0x7f4b68157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.476+0000 7f4b701b5640 1 --2- 172.21.15.72:0/200037985 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4b6806f510 0x7f4b681f1aa0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.476+0000 7f4b701b5640 1 -- 172.21.15.72:0/200037985 >> 172.21.15.72:0/200037985 conn(0x7f4b680b4010 msgr2=0x7f4b68070ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:20.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.476+0000 7f4b701b5640 1 -- 172.21.15.72:0/200037985 shutdown_connections 2024-04-15T15:16:20.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.476+0000 7f4b701b5640 1 -- 172.21.15.72:0/200037985 wait complete. 2024-04-15T15:16:20.489 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol_quiesce --mode=777 2024-04-15T15:16:20.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.594+0000 7fb9d7911640 1 Processor -- start 2024-04-15T15:16:20.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.594+0000 7fb9d7911640 1 -- start start 2024-04-15T15:16:20.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.595+0000 7fb9d7911640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb9d015fb70 0x7fb9d0160cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.595+0000 7fb9d7911640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb9d00d0650 0x7fb9d00703b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.595+0000 7fb9d7911640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb9d006d410 0x7fb9d006d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.595+0000 7fb9d7911640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fb9d006ddc0 con 0x7fb9d015fb70 2024-04-15T15:16:20.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.595+0000 7fb9d7911640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fb9d015ff50 con 0x7fb9d00d0650 2024-04-15T15:16:20.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.595+0000 7fb9d7911640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fb9d00708f0 con 0x7fb9d006d410 2024-04-15T15:16:20.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.595+0000 7fb9d56ad640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb9d015fb70 0x7fb9d0160cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.595+0000 7fb9d56ad640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb9d015fb70 0x7fb9d0160cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56334/0 (socket says 172.21.15.72:56334) 2024-04-15T15:16:20.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.595+0000 7fb9d56ad640 1 -- 172.21.15.72:0/2823277958 learned_addr learned my addr 172.21.15.72:0/2823277958 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:20.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.595+0000 7fb9d5eae640 1 --2- 172.21.15.72:0/2823277958 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb9d006d410 0x7fb9d006d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.595+0000 7fb9d4eac640 1 --2- 172.21.15.72:0/2823277958 >> v2:172.21.15.175:3300/0 conn(0x7fb9d00d0650 0x7fb9d00703b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.595+0000 7fb9d56ad640 1 -- 172.21.15.72:0/2823277958 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb9d006d410 msgr2=0x7fb9d006d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:20.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.595+0000 7fb9d56ad640 1 --2- 172.21.15.72:0/2823277958 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb9d006d410 0x7fb9d006d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.595+0000 7fb9d56ad640 1 -- 172.21.15.72:0/2823277958 >> v2:172.21.15.175:3300/0 conn(0x7fb9d00d0650 msgr2=0x7fb9d00703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:20.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.595+0000 7fb9d56ad640 1 --2- 172.21.15.72:0/2823277958 >> v2:172.21.15.175:3300/0 conn(0x7fb9d00d0650 0x7fb9d00703b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.595+0000 7fb9d56ad640 1 -- 172.21.15.72:0/2823277958 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9d0070b70 con 0x7fb9d015fb70 2024-04-15T15:16:20.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.595+0000 7fb9d5eae640 1 --2- 172.21.15.72:0/2823277958 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb9d006d410 0x7fb9d006d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:20.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.595+0000 7fb9d4eac640 1 --2- 172.21.15.72:0/2823277958 >> v2:172.21.15.175:3300/0 conn(0x7fb9d00d0650 0x7fb9d00703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:20.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.596+0000 7fb9d56ad640 1 --2- 172.21.15.72:0/2823277958 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9d015fb70 0x7fb9d0160cd0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fb9d00b27c0 tx=0x7fb9cc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=179bec911d20eec8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:20.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.596+0000 7fb9c7fff640 1 -- 172.21.15.72:0/2823277958 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9cc02c9e0 con 0x7fb9d015fb70 2024-04-15T15:16:20.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.596+0000 7fb9c7fff640 1 -- 172.21.15.72:0/2823277958 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9cc02cb80 con 0x7fb9d015fb70 2024-04-15T15:16:20.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.596+0000 7fb9c7fff640 1 -- 172.21.15.72:0/2823277958 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9cc0343f0 con 0x7fb9d015fb70 2024-04-15T15:16:20.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.597+0000 7fb9d7911640 1 -- 172.21.15.72:0/2823277958 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9d015fb70 msgr2=0x7fb9d0160cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.597+0000 7fb9d7911640 1 --2- 172.21.15.72:0/2823277958 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9d015fb70 0x7fb9d0160cd0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fb9d00b27c0 tx=0x7fb9cc02be40 comp rx=0 tx=0).stop 2024-04-15T15:16:20.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.597+0000 7fb9d7911640 1 -- 172.21.15.72:0/2823277958 shutdown_connections 2024-04-15T15:16:20.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.597+0000 7fb9d7911640 1 --2- 172.21.15.72:0/2823277958 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb9d006d410 0x7fb9d006d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.597+0000 7fb9d7911640 1 --2- 172.21.15.72:0/2823277958 >> v2:172.21.15.175:3300/0 conn(0x7fb9d00d0650 0x7fb9d00703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.597+0000 7fb9d7911640 1 --2- 172.21.15.72:0/2823277958 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9d015fb70 0x7fb9d0160cd0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.597+0000 7fb9d7911640 1 -- 172.21.15.72:0/2823277958 >> 172.21.15.72:0/2823277958 conn(0x7fb9d0072420 msgr2=0x7fb9d0072800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:20.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.597+0000 7fb9d7911640 1 -- 172.21.15.72:0/2823277958 shutdown_connections 2024-04-15T15:16:20.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.597+0000 7fb9d7911640 1 -- 172.21.15.72:0/2823277958 wait complete. 2024-04-15T15:16:20.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.598+0000 7fb9d7911640 1 Processor -- start 2024-04-15T15:16:20.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.598+0000 7fb9d7911640 1 -- start start 2024-04-15T15:16:20.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.599+0000 7fb9d7911640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb9d006d410 0x7fb9d01f2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.599+0000 7fb9d7911640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb9d00d0650 0x7fb9d015d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.599+0000 7fb9d7911640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9d015fb70 0x7fb9d015e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.599+0000 7fb9d7911640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d00d0130 con 0x7fb9d015fb70 2024-04-15T15:16:20.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.599+0000 7fb9d7911640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fb9d00cfe50 con 0x7fb9d00d0650 2024-04-15T15:16:20.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.599+0000 7fb9d7911640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fb9d00cf820 con 0x7fb9d006d410 2024-04-15T15:16:20.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.599+0000 7fb9d5eae640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9d015fb70 0x7fb9d015e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.599+0000 7fb9d5eae640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9d015fb70 0x7fb9d015e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56340/0 (socket says 172.21.15.72:56340) 2024-04-15T15:16:20.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.599+0000 7fb9d5eae640 1 -- 172.21.15.72:0/3938290710 learned_addr learned my addr 172.21.15.72:0/3938290710 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:20.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.599+0000 7fb9d4eac640 1 --2- 172.21.15.72:0/3938290710 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb9d00d0650 0x7fb9d015d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.599+0000 7fb9d56ad640 1 --2- 172.21.15.72:0/3938290710 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb9d006d410 0x7fb9d01f2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.599+0000 7fb9d5eae640 1 -- 172.21.15.72:0/3938290710 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb9d006d410 msgr2=0x7fb9d01f2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.599+0000 7fb9d5eae640 1 --2- 172.21.15.72:0/3938290710 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb9d006d410 0x7fb9d01f2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.599+0000 7fb9d5eae640 1 -- 172.21.15.72:0/3938290710 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb9d00d0650 msgr2=0x7fb9d015d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.599+0000 7fb9d5eae640 1 --2- 172.21.15.72:0/3938290710 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb9d00d0650 0x7fb9d015d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.599+0000 7fb9d5eae640 1 -- 172.21.15.72:0/3938290710 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9d01f3370 con 0x7fb9d015fb70 2024-04-15T15:16:20.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.599+0000 7fb9d56ad640 1 --2- 172.21.15.72:0/3938290710 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb9d006d410 0x7fb9d01f2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:20.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.599+0000 7fb9d4eac640 1 --2- 172.21.15.72:0/3938290710 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb9d00d0650 0x7fb9d015d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:20.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.599+0000 7fb9d5eae640 1 --2- 172.21.15.72:0/3938290710 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9d015fb70 0x7fb9d015e060 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fb9c800afd0 tx=0x7fb9c800ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:20.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.599+0000 7fb9c5ffb640 1 -- 172.21.15.72:0/3938290710 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c8015020 con 0x7fb9d015fb70 2024-04-15T15:16:20.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.599+0000 7fb9c5ffb640 1 -- 172.21.15.72:0/3938290710 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9c8004440 con 0x7fb9d015fb70 2024-04-15T15:16:20.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.599+0000 7fb9c5ffb640 1 -- 172.21.15.72:0/3938290710 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9c8003850 con 0x7fb9d015fb70 2024-04-15T15:16:20.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.599+0000 7fb9d7911640 1 -- 172.21.15.72:0/3938290710 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9d01506e0 con 0x7fb9d015fb70 2024-04-15T15:16:20.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.600+0000 7fb9d7911640 1 -- 172.21.15.72:0/3938290710 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9d0150f80 con 0x7fb9d015fb70 2024-04-15T15:16:20.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.601+0000 7fb9d7911640 1 -- 172.21.15.72:0/3938290710 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9d00680f0 con 0x7fb9d015fb70 2024-04-15T15:16:20.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.601+0000 7fb9c5ffb640 1 -- 172.21.15.72:0/3938290710 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb9c8018020 con 0x7fb9d015fb70 2024-04-15T15:16:20.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.602+0000 7fb9c5ffb640 1 --2- 172.21.15.72:0/3938290710 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb9b406c8f0 0x7fb9b406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.602+0000 7fb9d56ad640 1 --2- 172.21.15.72:0/3938290710 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb9b406c8f0 0x7fb9b406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.602+0000 7fb9c5ffb640 1 -- 172.21.15.72:0/3938290710 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fb9c808bc20 con 0x7fb9d015fb70 2024-04-15T15:16:20.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.602+0000 7fb9d56ad640 1 --2- 172.21.15.72:0/3938290710 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb9b406c8f0 0x7fb9b406edb0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc035000 tx=0x7fb9cc030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:20.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.606+0000 7fb9c5ffb640 1 -- 172.21.15.72:0/3938290710 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb9c801f020 con 0x7fb9d015fb70 2024-04-15T15:16:20.809 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.808+0000 7fb9d7911640 1 -- 172.21.15.72:0/3938290710 --> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol_quiesce", "mode": "777", "target": ["mon-mgr", ""]}) -- 0x7fb9d0073540 con 0x7fb9b406c8f0 2024-04-15T15:16:20.813 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-04-15T15:16:20.812+0000 7f8e5cbd8640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-15T15:16:20.813 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-04-15T15:16:20.812+0000 7f8e5cbd8640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-15T15:16:20.813 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-04-15T15:16:20.812+0000 7f8e5cbd8640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-15T15:16:20.813 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-04-15T15:16:20.812+0000 7f8e5cbd8640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-15T15:16:20.813 INFO:tasks.ceph.mgr.y.smithi072.stderr:2024-04-15T15:16:20.812+0000 7f8e5cbd8640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-15T15:16:20.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.840+0000 7fb9c5ffb640 1 -- 172.21.15.72:0/3938290710 <== mgr.4101 v2:172.21.15.72:6832/3503807805 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fb9d0073540 con 0x7fb9b406c8f0 2024-04-15T15:16:20.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.842+0000 7fb9d7911640 1 -- 172.21.15.72:0/3938290710 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb9b406c8f0 msgr2=0x7fb9b406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.842+0000 7fb9d7911640 1 --2- 172.21.15.72:0/3938290710 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb9b406c8f0 0x7fb9b406edb0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb9cc035000 tx=0x7fb9cc030000 comp rx=0 tx=0).stop 2024-04-15T15:16:20.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.842+0000 7fb9d7911640 1 -- 172.21.15.72:0/3938290710 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9d015fb70 msgr2=0x7fb9d015e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.842+0000 7fb9d7911640 1 --2- 172.21.15.72:0/3938290710 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9d015fb70 0x7fb9d015e060 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fb9c800afd0 tx=0x7fb9c800ca20 comp rx=0 tx=0).stop 2024-04-15T15:16:20.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.842+0000 7fb9d7911640 1 -- 172.21.15.72:0/3938290710 shutdown_connections 2024-04-15T15:16:20.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.842+0000 7fb9d7911640 1 --2- 172.21.15.72:0/3938290710 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb9b406c8f0 0x7fb9b406edb0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.842+0000 7fb9d7911640 1 --2- 172.21.15.72:0/3938290710 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9d015fb70 0x7fb9d015e060 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.842+0000 7fb9d7911640 1 --2- 172.21.15.72:0/3938290710 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb9d00d0650 0x7fb9d015d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.842+0000 7fb9d7911640 1 --2- 172.21.15.72:0/3938290710 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb9d006d410 0x7fb9d01f2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.842+0000 7fb9d7911640 1 -- 172.21.15.72:0/3938290710 >> 172.21.15.72:0/3938290710 conn(0x7fb9d0072420 msgr2=0x7fb9d006c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:20.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.842+0000 7fb9d7911640 1 -- 172.21.15.72:0/3938290710 shutdown_connections 2024-04-15T15:16:20.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.842+0000 7fb9d7911640 1 -- 172.21.15.72:0/3938290710 wait complete. 2024-04-15T15:16:20.855 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol_quiesce 2024-04-15T15:16:20.960 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.959+0000 7f114f688640 1 Processor -- start 2024-04-15T15:16:20.961 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.959+0000 7f114f688640 1 -- start start 2024-04-15T15:16:20.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.960+0000 7f114f688640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f11480b5410 0x7f11480b57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.960+0000 7f114f688640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f11480b0050 0x7f114806c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.960+0000 7f114f688640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11480d07c0 con 0x7f11480afae0 2024-04-15T15:16:20.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.960+0000 7f114f688640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f11480d0330 con 0x7f11480b0050 2024-04-15T15:16:20.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.960+0000 7f114f688640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f11480b5d30 con 0x7f11480b5410 2024-04-15T15:16:20.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.960+0000 7f114cc23640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f11480b0050 0x7f114806c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.960+0000 7f114cc23640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f11480b0050 0x7f114806c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56374/0 (socket says 172.21.15.72:56374) 2024-04-15T15:16:20.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.960+0000 7f114cc23640 1 -- 172.21.15.72:0/1382154256 learned_addr learned my addr 172.21.15.72:0/1382154256 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:20.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.960+0000 7f114dc25640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f11480afae0 0x7f11480ce950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:34270/0 (socket says 172.21.15.72:34270) 2024-04-15T15:16:20.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.960+0000 7f114d424640 1 --2- 172.21.15.72:0/1382154256 >> v2:172.21.15.175:3300/0 conn(0x7f11480b5410 0x7f11480b57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.960+0000 7f114cc23640 1 -- 172.21.15.72:0/1382154256 >> v2:172.21.15.175:3300/0 conn(0x7f11480b5410 msgr2=0x7f11480b57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:20.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.960+0000 7f114cc23640 1 --2- 172.21.15.72:0/1382154256 >> v2:172.21.15.175:3300/0 conn(0x7f11480b5410 0x7f11480b57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.960+0000 7f114cc23640 1 -- 172.21.15.72:0/1382154256 >> v1:172.21.15.72:6789/0 conn(0x7f11480afae0 legacy=0x7f11480ce950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.960+0000 7f114cc23640 1 -- 172.21.15.72:0/1382154256 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11480b3aa0 con 0x7f11480b0050 2024-04-15T15:16:20.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.960+0000 7f1147fff640 1 -- 172.21.15.72:0/1382154256 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3527186626 0 0) 0x7f11480d07c0 con 0x7f11480afae0 2024-04-15T15:16:20.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.960+0000 7f114d424640 1 --2- 172.21.15.72:0/1382154256 >> v2:172.21.15.175:3300/0 conn(0x7f11480b5410 0x7f11480b57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:20.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.961+0000 7f114cc23640 1 --2- 172.21.15.72:0/1382154256 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f11480b0050 0x7f114806c340 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f1148073da0 tx=0x7f113c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89e867b94a43ff7a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:20.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.961+0000 7f1147fff640 1 -- 172.21.15.72:0/1382154256 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f113c02ca80 con 0x7f11480b0050 2024-04-15T15:16:20.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.961+0000 7f1147fff640 1 -- 172.21.15.72:0/1382154256 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f113c02cc20 con 0x7f11480b0050 2024-04-15T15:16:20.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.961+0000 7f1147fff640 1 -- 172.21.15.72:0/1382154256 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f113c0343f0 con 0x7f11480b0050 2024-04-15T15:16:20.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.962+0000 7f114f688640 1 -- 172.21.15.72:0/1382154256 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f11480b0050 msgr2=0x7f114806c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.962+0000 7f114f688640 1 --2- 172.21.15.72:0/1382154256 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f11480b0050 0x7f114806c340 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f1148073da0 tx=0x7f113c02bee0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.962+0000 7f114f688640 1 -- 172.21.15.72:0/1382154256 shutdown_connections 2024-04-15T15:16:20.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.962+0000 7f114f688640 1 --2- 172.21.15.72:0/1382154256 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f11480b0050 0x7f114806c340 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.962+0000 7f114f688640 1 --2- 172.21.15.72:0/1382154256 >> v2:172.21.15.175:3300/0 conn(0x7f11480b5410 0x7f11480b57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.962+0000 7f114f688640 1 -- 172.21.15.72:0/1382154256 >> 172.21.15.72:0/1382154256 conn(0x7f11480d0a50 msgr2=0x7f11480d0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:20.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.962+0000 7f114f688640 1 -- 172.21.15.72:0/1382154256 shutdown_connections 2024-04-15T15:16:20.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.962+0000 7f114f688640 1 -- 172.21.15.72:0/1382154256 wait complete. 2024-04-15T15:16:20.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.963+0000 7f114f688640 1 Processor -- start 2024-04-15T15:16:20.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.963+0000 7f114f688640 1 -- start start 2024-04-15T15:16:20.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.964+0000 7f114f688640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f11480afae0 0x7f11481592a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.964+0000 7f114f688640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f11480b0050 0x7f114815bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.964+0000 7f114f688640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f11480b5410 0x7f114815f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.964+0000 7f114f688640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f11480d18f0 con 0x7f11480b5410 2024-04-15T15:16:20.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.964+0000 7f114f688640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f11480d1610 con 0x7f11480b0050 2024-04-15T15:16:20.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.964+0000 7f114f688640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f114806db40 con 0x7f11480afae0 2024-04-15T15:16:20.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.964+0000 7f114dc25640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f11480b5410 0x7f114815f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.964+0000 7f114dc25640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f11480b5410 0x7f114815f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56386/0 (socket says 172.21.15.72:56386) 2024-04-15T15:16:20.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.964+0000 7f114dc25640 1 -- 172.21.15.72:0/2898333981 learned_addr learned my addr 172.21.15.72:0/2898333981 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:20.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.964+0000 7f114cc23640 1 --2- 172.21.15.72:0/2898333981 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f11480b0050 0x7f114815bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.964+0000 7f114d424640 1 --2- 172.21.15.72:0/2898333981 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f11480afae0 0x7f11481592a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.964+0000 7f114dc25640 1 -- 172.21.15.72:0/2898333981 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f11480afae0 msgr2=0x7f11481592a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.964+0000 7f114dc25640 1 --2- 172.21.15.72:0/2898333981 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f11480afae0 0x7f11481592a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.964+0000 7f114dc25640 1 -- 172.21.15.72:0/2898333981 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f11480b0050 msgr2=0x7f114815bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.964+0000 7f114dc25640 1 --2- 172.21.15.72:0/2898333981 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f11480b0050 0x7f114815bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.964+0000 7f114dc25640 1 -- 172.21.15.72:0/2898333981 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f114815c0b0 con 0x7f11480b5410 2024-04-15T15:16:20.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.964+0000 7f114dc25640 1 --2- 172.21.15.72:0/2898333981 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f11480b5410 0x7f114815f480 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f113800c410 tx=0x7f113800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:20.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.964+0000 7f1145ffb640 1 -- 172.21.15.72:0/2898333981 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11380074e0 con 0x7f11480b5410 2024-04-15T15:16:20.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.965+0000 7f1145ffb640 1 -- 172.21.15.72:0/2898333981 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11380047e0 con 0x7f11480b5410 2024-04-15T15:16:20.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.965+0000 7f114f688640 1 -- 172.21.15.72:0/2898333981 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1148159880 con 0x7f11480b5410 2024-04-15T15:16:20.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.965+0000 7f1145ffb640 1 -- 172.21.15.72:0/2898333981 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1138004bd0 con 0x7f11480b5410 2024-04-15T15:16:20.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.965+0000 7f114f688640 1 -- 172.21.15.72:0/2898333981 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1148159e40 con 0x7f11480b5410 2024-04-15T15:16:20.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.966+0000 7f114f688640 1 -- 172.21.15.72:0/2898333981 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1148075c50 con 0x7f11480b5410 2024-04-15T15:16:20.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.966+0000 7f1145ffb640 1 -- 172.21.15.72:0/2898333981 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1138011070 con 0x7f11480b5410 2024-04-15T15:16:20.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.967+0000 7f1145ffb640 1 --2- 172.21.15.72:0/2898333981 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f113406c8f0 0x7f113406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.967+0000 7f114d424640 1 --2- 172.21.15.72:0/2898333981 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f113406c8f0 0x7f113406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.967+0000 7f1145ffb640 1 -- 172.21.15.72:0/2898333981 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f113808b800 con 0x7f11480b5410 2024-04-15T15:16:20.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.967+0000 7f114d424640 1 --2- 172.21.15.72:0/2898333981 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f113406c8f0 0x7f113406edb0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1140006fd0 tx=0x7f1140008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:20.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:20.971+0000 7f1145ffb640 1 -- 172.21.15.72:0/2898333981 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f114815c0b0 con 0x7f11480b5410 2024-04-15T15:16:21.002 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:16:21.005 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:16:21.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.179+0000 7f114f688640 1 -- 172.21.15.72:0/2898333981 --> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol_quiesce", "target": ["mon-mgr", ""]}) -- 0x7f114806d610 con 0x7f113406c8f0 2024-04-15T15:16:21.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.184+0000 7f1145ffb640 1 -- 172.21.15.72:0/2898333981 <== mgr.4101 v2:172.21.15.72:6832/3503807805 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+70 (secure 0 0 0) 0x7f114806d610 con 0x7f113406c8f0 2024-04-15T15:16:21.187 INFO:teuthology.orchestra.run.smithi072.stdout:/volumes/_nogroup/subvol_quiesce/6bdc4ba4-8793-4ca5-8487-64cfa9a984f4 2024-04-15T15:16:21.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.186+0000 7f114f688640 1 -- 172.21.15.72:0/2898333981 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f113406c8f0 msgr2=0x7f113406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.186+0000 7f114f688640 1 --2- 172.21.15.72:0/2898333981 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f113406c8f0 0x7f113406edb0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1140006fd0 tx=0x7f1140008040 comp rx=0 tx=0).stop 2024-04-15T15:16:21.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.186+0000 7f114f688640 1 -- 172.21.15.72:0/2898333981 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f11480b5410 msgr2=0x7f114815f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.186+0000 7f114f688640 1 --2- 172.21.15.72:0/2898333981 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f11480b5410 0x7f114815f480 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f113800c410 tx=0x7f113800c8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.186+0000 7f114f688640 1 -- 172.21.15.72:0/2898333981 shutdown_connections 2024-04-15T15:16:21.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.186+0000 7f114f688640 1 --2- 172.21.15.72:0/2898333981 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f113406c8f0 0x7f113406edb0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.186+0000 7f114f688640 1 --2- 172.21.15.72:0/2898333981 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f11480b5410 0x7f114815f480 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.186+0000 7f114f688640 1 --2- 172.21.15.72:0/2898333981 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f11480b0050 0x7f114815bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.186+0000 7f114f688640 1 --2- 172.21.15.72:0/2898333981 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f11480afae0 0x7f11481592a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.186+0000 7f114f688640 1 -- 172.21.15.72:0/2898333981 >> 172.21.15.72:0/2898333981 conn(0x7f11480d0a50 msgr2=0x7f114806f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:21.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.186+0000 7f114f688640 1 -- 172.21.15.72:0/2898333981 shutdown_connections 2024-04-15T15:16:21.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.186+0000 7f114f688640 1 -- 172.21.15.72:0/2898333981 wait complete. 2024-04-15T15:16:21.199 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:16:21.199 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:16:21.216 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:21.358 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.357+0000 7f17465a8640 1 Processor -- start 2024-04-15T15:16:21.359 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.357+0000 7f17465a8640 1 -- start start 2024-04-15T15:16:21.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.358+0000 7f17465a8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f17400af400 0x7f17400ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.358+0000 7f17465a8640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f17400af860 0x7f17400cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.358+0000 7f17465a8640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f174006f8f0 con 0x7f174015eba0 2024-04-15T15:16:21.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.358+0000 7f17465a8640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f174015ef80 con 0x7f17400af400 2024-04-15T15:16:21.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.358+0000 7f17465a8640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f17400ce2b0 con 0x7f17400af860 2024-04-15T15:16:21.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.358+0000 7f173f7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f17400af400 0x7f17400ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.360 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.358+0000 7f173f7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f17400af400 0x7f17400ccb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56396/0 (socket says 172.21.15.72:56396) 2024-04-15T15:16:21.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.358+0000 7f173f7fe640 1 -- 172.21.15.72:0/1646047325 learned_addr learned my addr 172.21.15.72:0/1646047325 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:21.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.358+0000 7f1744b45640 1 --2- 172.21.15.72:0/1646047325 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f17400af860 0x7f17400cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.358+0000 7f173f7fe640 1 -- 172.21.15.72:0/1646047325 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f17400af860 msgr2=0x7f17400cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:21.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.358+0000 7f173f7fe640 1 --2- 172.21.15.72:0/1646047325 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f17400af860 0x7f17400cd0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.358+0000 7f173f7fe640 1 -- 172.21.15.72:0/1646047325 >> v1:172.21.15.175:6789/0 conn(0x7f174015eba0 legacy=0x7f174006f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.358+0000 7f173f7fe640 1 -- 172.21.15.72:0/1646047325 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17400ce590 con 0x7f17400af400 2024-04-15T15:16:21.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.358+0000 7f1744b45640 1 --2- 172.21.15.72:0/1646047325 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f17400af860 0x7f17400cd0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:21.361 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.359+0000 7f173f7fe640 1 --2- 172.21.15.72:0/1646047325 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17400af400 0x7f17400ccb90 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f1734009870 tx=0x7f173402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21266f5a4017e351 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:21.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.359+0000 7f173effd640 1 -- 172.21.15.72:0/1646047325 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1734035020 con 0x7f17400af400 2024-04-15T15:16:21.362 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.359+0000 7f173effd640 1 -- 172.21.15.72:0/1646047325 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1734031640 con 0x7f17400af400 2024-04-15T15:16:21.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.359+0000 7f173effd640 1 -- 172.21.15.72:0/1646047325 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f173402d040 con 0x7f17400af400 2024-04-15T15:16:21.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.360+0000 7f17465a8640 1 -- 172.21.15.72:0/1646047325 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17400af400 msgr2=0x7f17400ccb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.360+0000 7f17465a8640 1 --2- 172.21.15.72:0/1646047325 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17400af400 0x7f17400ccb90 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f1734009870 tx=0x7f173402b8f0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.360+0000 7f17465a8640 1 -- 172.21.15.72:0/1646047325 shutdown_connections 2024-04-15T15:16:21.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.360+0000 7f17465a8640 1 --2- 172.21.15.72:0/1646047325 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f17400af860 0x7f17400cd0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.360+0000 7f17465a8640 1 --2- 172.21.15.72:0/1646047325 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f17400af400 0x7f17400ccb90 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.360+0000 7f17465a8640 1 -- 172.21.15.72:0/1646047325 >> 172.21.15.72:0/1646047325 conn(0x7f17400b1dd0 msgr2=0x7f17400b21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:21.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.360+0000 7f17465a8640 1 -- 172.21.15.72:0/1646047325 shutdown_connections 2024-04-15T15:16:21.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.360+0000 7f17465a8640 1 -- 172.21.15.72:0/1646047325 wait complete. 2024-04-15T15:16:21.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.361+0000 7f17465a8640 1 Processor -- start 2024-04-15T15:16:21.363 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.361+0000 7f17465a8640 1 -- start start 2024-04-15T15:16:21.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.362+0000 7f17465a8640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f17400af400 0x7f174014cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.362+0000 7f17465a8640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f17400af860 0x7f174015b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.362+0000 7f17465a8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f174015eba0 0x7f174014e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.362+0000 7f17465a8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f17400b2b80 con 0x7f174015eba0 2024-04-15T15:16:21.365 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.362+0000 7f17465a8640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f174006ecc0 con 0x7f17400af400 2024-04-15T15:16:21.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.362+0000 7f17465a8640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f17400ceb50 con 0x7f17400af860 2024-04-15T15:16:21.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.362+0000 7f1744b45640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f174015eba0 0x7f174014e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.362+0000 7f1744b45640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f174015eba0 0x7f174014e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:56406/0 (socket says 172.21.15.72:56406) 2024-04-15T15:16:21.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.362+0000 7f1744b45640 1 -- 172.21.15.72:0/2288488106 learned_addr learned my addr 172.21.15.72:0/2288488106 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:21.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.362+0000 7f173f7fe640 1 --2- 172.21.15.72:0/2288488106 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f17400af860 0x7f174015b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.362+0000 7f173ffff640 1 --2- 172.21.15.72:0/2288488106 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f17400af400 0x7f174014cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.362+0000 7f1744b45640 1 -- 172.21.15.72:0/2288488106 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f17400af860 msgr2=0x7f174015b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.362+0000 7f1744b45640 1 --2- 172.21.15.72:0/2288488106 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f17400af860 0x7f174015b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.362+0000 7f1744b45640 1 -- 172.21.15.72:0/2288488106 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f17400af400 msgr2=0x7f174014cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.362+0000 7f1744b45640 1 --2- 172.21.15.72:0/2288488106 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f17400af400 0x7f174014cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.362+0000 7f1744b45640 1 -- 172.21.15.72:0/2288488106 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1740155340 con 0x7f174015eba0 2024-04-15T15:16:21.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.362+0000 7f173ffff640 1 --2- 172.21.15.72:0/2288488106 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f17400af400 0x7f174014cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:21.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.362+0000 7f1744b45640 1 --2- 172.21.15.72:0/2288488106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f174015eba0 0x7f174014e1b0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f1738007ef0 tx=0x7f173800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:21.366 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.362+0000 7f173cff9640 1 -- 172.21.15.72:0/2288488106 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f173801a070 con 0x7f174015eba0 2024-04-15T15:16:21.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.363+0000 7f173cff9640 1 -- 172.21.15.72:0/2288488106 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1738003b90 con 0x7f174015eba0 2024-04-15T15:16:21.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.363+0000 7f17465a8640 1 -- 172.21.15.72:0/2288488106 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f174015be40 con 0x7f174015eba0 2024-04-15T15:16:21.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.363+0000 7f173cff9640 1 -- 172.21.15.72:0/2288488106 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1738004030 con 0x7f174015eba0 2024-04-15T15:16:21.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.363+0000 7f17465a8640 1 -- 172.21.15.72:0/2288488106 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f174014d6d0 con 0x7f174015eba0 2024-04-15T15:16:21.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.364+0000 7f17465a8640 1 -- 172.21.15.72:0/2288488106 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17400689e0 con 0x7f174015eba0 2024-04-15T15:16:21.367 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.364+0000 7f173cff9640 1 -- 172.21.15.72:0/2288488106 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f173801d070 con 0x7f174015eba0 2024-04-15T15:16:21.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.364+0000 7f173cff9640 1 --2- 172.21.15.72:0/2288488106 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f172c06c8f0 0x7f172c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.365+0000 7f173cff9640 1 -- 172.21.15.72:0/2288488106 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f173808be50 con 0x7f174015eba0 2024-04-15T15:16:21.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.365+0000 7f173ffff640 1 --2- 172.21.15.72:0/2288488106 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f172c06c8f0 0x7f172c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.368 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.365+0000 7f173ffff640 1 --2- 172.21.15.72:0/2288488106 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f172c06c8f0 0x7f172c06edb0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1730001700 tx=0x7f1730009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:21.370 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.369+0000 7f173cff9640 1 -- 172.21.15.72:0/2288488106 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1738010070 con 0x7f174015eba0 2024-04-15T15:16:21.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.545+0000 7f17465a8640 1 -- 172.21.15.72:0/2288488106 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f174006f310 con 0x7f174015eba0 2024-04-15T15:16:21.548 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.546+0000 7f173cff9640 1 -- 172.21.15.72:0/2288488106 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+16942 (secure 0 0 0) 0x7f1738059b30 con 0x7f174015eba0 2024-04-15T15:16:21.548 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:16:21.548 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":38,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:16:08.702307+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:05.551499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:06.003152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:21.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.548+0000 7f17465a8640 1 -- 172.21.15.72:0/2288488106 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f172c06c8f0 msgr2=0x7f172c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.548+0000 7f17465a8640 1 --2- 172.21.15.72:0/2288488106 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f172c06c8f0 0x7f172c06edb0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1730001700 tx=0x7f1730009040 comp rx=0 tx=0).stop 2024-04-15T15:16:21.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.548+0000 7f17465a8640 1 -- 172.21.15.72:0/2288488106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f174015eba0 msgr2=0x7f174014e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.548+0000 7f17465a8640 1 --2- 172.21.15.72:0/2288488106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f174015eba0 0x7f174014e1b0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f1738007ef0 tx=0x7f173800d4a0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.550 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.548+0000 7f17465a8640 1 -- 172.21.15.72:0/2288488106 shutdown_connections 2024-04-15T15:16:21.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.548+0000 7f17465a8640 1 --2- 172.21.15.72:0/2288488106 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f172c06c8f0 0x7f172c06edb0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.548+0000 7f17465a8640 1 --2- 172.21.15.72:0/2288488106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f174015eba0 0x7f174014e1b0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.548+0000 7f17465a8640 1 --2- 172.21.15.72:0/2288488106 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f17400af860 0x7f174015b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.548+0000 7f17465a8640 1 --2- 172.21.15.72:0/2288488106 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f17400af400 0x7f174014cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.548+0000 7f17465a8640 1 -- 172.21.15.72:0/2288488106 >> 172.21.15.72:0/2288488106 conn(0x7f17400b1dd0 msgr2=0x7f1740151b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:21.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.549+0000 7f17465a8640 1 -- 172.21.15.72:0/2288488106 shutdown_connections 2024-04-15T15:16:21.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.549+0000 7f17465a8640 1 -- 172.21.15.72:0/2288488106 wait complete. 2024-04-15T15:16:21.561 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:39.359102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:16:05.551499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:16:06.003152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:16:21.562 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:21.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.667+0000 7f662f58f640 1 Processor -- start 2024-04-15T15:16:21.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.667+0000 7f662f58f640 1 -- start start 2024-04-15T15:16:21.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.668+0000 7f662f58f640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f66280ccb50 0x7f662806dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.668+0000 7f662f58f640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f662815af20 0x7f66280ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.672 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.668+0000 7f662f58f640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66280cd470 con 0x7f662815e840 2024-04-15T15:16:21.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.668+0000 7f662f58f640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f662806ba90 con 0x7f662815af20 2024-04-15T15:16:21.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.668+0000 7f662f58f640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f662815c110 con 0x7f66280ccb50 2024-04-15T15:16:21.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.668+0000 7f662db2c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f662815e840 0x7f662815ecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60204/0 (socket says 172.21.15.72:60204) 2024-04-15T15:16:21.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.668+0000 7f662db2c640 1 -- 172.21.15.72:0/3867437644 learned_addr learned my addr 172.21.15.72:0/3867437644 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:21.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.668+0000 7f662cb2a640 1 --2- 172.21.15.72:0/3867437644 >> v2:172.21.15.175:3300/0 conn(0x7f662815af20 0x7f66280ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.668+0000 7f662d32b640 1 --2- 172.21.15.72:0/3867437644 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f66280ccb50 0x7f662806dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.668+0000 7f661ffff640 1 -- 172.21.15.72:0/3867437644 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1530708525 0 0) 0x7f66280cd470 con 0x7f662815e840 2024-04-15T15:16:21.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.669+0000 7f661ffff640 1 -- 172.21.15.72:0/3867437644 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6614003520 con 0x7f662815e840 2024-04-15T15:16:21.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.669+0000 7f661ffff640 1 -- 172.21.15.72:0/3867437644 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3618459185 0 0) 0x7f6614003520 con 0x7f662815e840 2024-04-15T15:16:21.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.669+0000 7f661ffff640 1 -- 172.21.15.72:0/3867437644 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f66280ccb50 msgr2=0x7f662806dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:21.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.669+0000 7f661ffff640 1 --2- 172.21.15.72:0/3867437644 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f66280ccb50 0x7f662806dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.669+0000 7f661ffff640 1 -- 172.21.15.72:0/3867437644 >> v2:172.21.15.175:3300/0 conn(0x7f662815af20 msgr2=0x7f66280ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:21.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.669+0000 7f661ffff640 1 --2- 172.21.15.72:0/3867437644 >> v2:172.21.15.175:3300/0 conn(0x7f662815af20 0x7f66280ccf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.669+0000 7f661ffff640 1 -- 172.21.15.72:0/3867437644 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66280cfae0 con 0x7f662815e840 2024-04-15T15:16:21.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.669+0000 7f661ffff640 1 -- 172.21.15.72:0/3867437644 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f66200019f0 con 0x7f662815e840 2024-04-15T15:16:21.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.669+0000 7f661ffff640 1 -- 172.21.15.72:0/3867437644 >> v1:172.21.15.72:6789/0 conn(0x7f662815e840 legacy=0x7f662815ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.669+0000 7f661ffff640 1 --2- 172.21.15.72:0/3867437644 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66140039b0 0x7f6614003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.669+0000 7f661ffff640 1 --2- 172.21.15.72:0/3867437644 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f66280ccb50 0x7f6614004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.669+0000 7f661ffff640 1 --2- 172.21.15.72:0/3867437644 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6614004d40 0x7f66140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.669+0000 7f661ffff640 1 -- 172.21.15.72:0/3867437644 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f662806ba90 con 0x7f66140039b0 2024-04-15T15:16:21.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.669+0000 7f661ffff640 1 -- 172.21.15.72:0/3867437644 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f662815c110 con 0x7f6614004d40 2024-04-15T15:16:21.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.669+0000 7f661ffff640 1 -- 172.21.15.72:0/3867437644 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f661400b9c0 con 0x7f66280ccb50 2024-04-15T15:16:21.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.669+0000 7f662d32b640 1 --2- 172.21.15.72:0/3867437644 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66140039b0 0x7f6614003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.669+0000 7f662db2c640 1 --2- 172.21.15.72:0/3867437644 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6614004d40 0x7f66140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.669+0000 7f662cb2a640 1 --2- 172.21.15.72:0/3867437644 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f66280ccb50 0x7f6614004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.669+0000 7f662d32b640 1 -- 172.21.15.72:0/3867437644 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f66280ccb50 msgr2=0x7f6614004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:21.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.669+0000 7f662d32b640 1 --2- 172.21.15.72:0/3867437644 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f66280ccb50 0x7f6614004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.669+0000 7f662d32b640 1 -- 172.21.15.72:0/3867437644 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6614004d40 msgr2=0x7f66140051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:21.674 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.669+0000 7f662d32b640 1 --2- 172.21.15.72:0/3867437644 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6614004d40 0x7f66140051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.669+0000 7f662d32b640 1 -- 172.21.15.72:0/3867437644 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6614003520 con 0x7f66140039b0 2024-04-15T15:16:21.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.670+0000 7f662d32b640 1 --2- 172.21.15.72:0/3867437644 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66140039b0 0x7f6614003dd0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f662400aac0 tx=0x7f6614005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bfdda577e1c4228 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:21.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.670+0000 7f661ffff640 1 -- 172.21.15.72:0/3867437644 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f662400b040 con 0x7f66140039b0 2024-04-15T15:16:21.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.670+0000 7f661ffff640 1 -- 172.21.15.72:0/3867437644 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6624034600 con 0x7f66140039b0 2024-04-15T15:16:21.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.670+0000 7f661ffff640 1 -- 172.21.15.72:0/3867437644 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66240349d0 con 0x7f66140039b0 2024-04-15T15:16:21.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.671+0000 7f662f58f640 1 -- 172.21.15.72:0/3867437644 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66140039b0 msgr2=0x7f6614003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.671+0000 7f662f58f640 1 --2- 172.21.15.72:0/3867437644 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66140039b0 0x7f6614003dd0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f662400aac0 tx=0x7f6614005830 comp rx=0 tx=0).stop 2024-04-15T15:16:21.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.671+0000 7f662db2c640 1 -- 172.21.15.72:0/3867437644 reap_dead start 2024-04-15T15:16:21.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.671+0000 7f662f58f640 1 -- 172.21.15.72:0/3867437644 shutdown_connections 2024-04-15T15:16:21.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.671+0000 7f662f58f640 1 -- 172.21.15.72:0/3867437644 >> 172.21.15.72:0/3867437644 conn(0x7f66280b31d0 msgr2=0x7f66280b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:21.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.671+0000 7f662f58f640 1 -- 172.21.15.72:0/3867437644 shutdown_connections 2024-04-15T15:16:21.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.671+0000 7f662f58f640 1 -- 172.21.15.72:0/3867437644 wait complete. 2024-04-15T15:16:21.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.672+0000 7f662f58f640 1 Processor -- start 2024-04-15T15:16:21.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.672+0000 7f662f58f640 1 -- start start 2024-04-15T15:16:21.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.673+0000 7f662f58f640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f66140039b0 0x7f66281574e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.673+0000 7f662f58f640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6628159dd0 0x7f662815a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.673+0000 7f662f58f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6628149de0 0x7f662814a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.679 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.673+0000 7f662f58f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f66280cf8c0 con 0x7f6628149de0 2024-04-15T15:16:21.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.673+0000 7f662f58f640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f66280cf740 con 0x7f6628159dd0 2024-04-15T15:16:21.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.673+0000 7f662f58f640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f66280cc770 con 0x7f66140039b0 2024-04-15T15:16:21.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.673+0000 7f662db2c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6628149de0 0x7f662814a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.673+0000 7f662db2c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6628149de0 0x7f662814a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42904/0 (socket says 172.21.15.72:42904) 2024-04-15T15:16:21.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.673+0000 7f662db2c640 1 -- 172.21.15.72:0/2145265440 learned_addr learned my addr 172.21.15.72:0/2145265440 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:21.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.673+0000 7f662cb2a640 1 --2- 172.21.15.72:0/2145265440 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6628159dd0 0x7f662815a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.673+0000 7f662d32b640 1 --2- 172.21.15.72:0/2145265440 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f66140039b0 0x7f66281574e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.673+0000 7f662db2c640 1 -- 172.21.15.72:0/2145265440 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f66140039b0 msgr2=0x7f66281574e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.673+0000 7f662db2c640 1 --2- 172.21.15.72:0/2145265440 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f66140039b0 0x7f66281574e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.673+0000 7f662db2c640 1 -- 172.21.15.72:0/2145265440 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6628159dd0 msgr2=0x7f662815a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.673+0000 7f662db2c640 1 --2- 172.21.15.72:0/2145265440 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6628159dd0 0x7f662815a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.680 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.673+0000 7f662db2c640 1 -- 172.21.15.72:0/2145265440 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66280cbfe0 con 0x7f6628149de0 2024-04-15T15:16:21.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.673+0000 7f662db2c640 1 --2- 172.21.15.72:0/2145265440 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6628149de0 0x7f662814a260 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f6628073f60 tx=0x7f6620007b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:21.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.674+0000 7f661dffb640 1 -- 172.21.15.72:0/2145265440 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f662000ab80 con 0x7f6628149de0 2024-04-15T15:16:21.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.674+0000 7f661dffb640 1 -- 172.21.15.72:0/2145265440 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f662000ad40 con 0x7f6628149de0 2024-04-15T15:16:21.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.674+0000 7f662f58f640 1 -- 172.21.15.72:0/2145265440 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66280af060 con 0x7f6628149de0 2024-04-15T15:16:21.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.674+0000 7f661dffb640 1 -- 172.21.15.72:0/2145265440 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66200133f0 con 0x7f6628149de0 2024-04-15T15:16:21.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.674+0000 7f662f58f640 1 -- 172.21.15.72:0/2145265440 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66281595d0 con 0x7f6628149de0 2024-04-15T15:16:21.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.675+0000 7f662f58f640 1 -- 172.21.15.72:0/2145265440 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6628075c50 con 0x7f6628149de0 2024-04-15T15:16:21.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.675+0000 7f661dffb640 1 -- 172.21.15.72:0/2145265440 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f662001b020 con 0x7f6628149de0 2024-04-15T15:16:21.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.676+0000 7f661dffb640 1 --2- 172.21.15.72:0/2145265440 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f660c06c8f0 0x7f660c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.676+0000 7f662d32b640 1 --2- 172.21.15.72:0/2145265440 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f660c06c8f0 0x7f660c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.676+0000 7f661dffb640 1 -- 172.21.15.72:0/2145265440 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f662008bb60 con 0x7f6628149de0 2024-04-15T15:16:21.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.676+0000 7f662d32b640 1 --2- 172.21.15.72:0/2145265440 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f660c06c8f0 0x7f660c06edb0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6624037000 tx=0x7f6624032000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:21.682 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.680+0000 7f661dffb640 1 -- 172.21.15.72:0/2145265440 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6620059e00 con 0x7f6628149de0 2024-04-15T15:16:21.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.740+0000 7f661dffb640 1 -- 172.21.15.72:0/2145265440 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f6620055d70 con 0x7f6628149de0 2024-04-15T15:16:21.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.863+0000 7f662f58f640 1 -- 172.21.15.72:0/2145265440 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f662806d1f0 con 0x7f6628149de0 2024-04-15T15:16:21.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.863+0000 7f661dffb640 1 -- 172.21.15.72:0/2145265440 <== mon.0 v2:172.21.15.72:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v38) ==== 93+0+10 (secure 0 0 0) 0x7f6620077990 con 0x7f6628149de0 2024-04-15T15:16:21.865 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-15T15:16:21.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.866+0000 7f662f58f640 1 -- 172.21.15.72:0/2145265440 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f660c06c8f0 msgr2=0x7f660c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.866+0000 7f662f58f640 1 --2- 172.21.15.72:0/2145265440 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f660c06c8f0 0x7f660c06edb0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6624037000 tx=0x7f6624032000 comp rx=0 tx=0).stop 2024-04-15T15:16:21.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.866+0000 7f662f58f640 1 -- 172.21.15.72:0/2145265440 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6628149de0 msgr2=0x7f662814a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.866+0000 7f662f58f640 1 --2- 172.21.15.72:0/2145265440 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6628149de0 0x7f662814a260 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f6628073f60 tx=0x7f6620007b10 comp rx=0 tx=0).stop 2024-04-15T15:16:21.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.866+0000 7f662f58f640 1 -- 172.21.15.72:0/2145265440 shutdown_connections 2024-04-15T15:16:21.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.866+0000 7f662f58f640 1 --2- 172.21.15.72:0/2145265440 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f660c06c8f0 0x7f660c06edb0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.866+0000 7f662f58f640 1 --2- 172.21.15.72:0/2145265440 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6628149de0 0x7f662814a260 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.866+0000 7f662f58f640 1 --2- 172.21.15.72:0/2145265440 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6628159dd0 0x7f662815a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.866+0000 7f662f58f640 1 --2- 172.21.15.72:0/2145265440 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f66140039b0 0x7f66281574e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.866+0000 7f662f58f640 1 -- 172.21.15.72:0/2145265440 >> 172.21.15.72:0/2145265440 conn(0x7f66280b31d0 msgr2=0x7f66280aff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:21.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.866+0000 7f662f58f640 1 -- 172.21.15.72:0/2145265440 shutdown_connections 2024-04-15T15:16:21.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.866+0000 7f662f58f640 1 -- 172.21.15.72:0/2145265440 wait complete. 2024-04-15T15:16:21.880 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:16:21.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.985+0000 7f126dffa640 1 Processor -- start 2024-04-15T15:16:21.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.985+0000 7f126dffa640 1 -- start start 2024-04-15T15:16:21.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.985+0000 7f126dffa640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f126815e610 0x7f12680d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.985+0000 7f126dffa640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f126806c400 0x7f126806c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.985+0000 7f126dffa640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12680d2320 con 0x7f126815e9f0 2024-04-15T15:16:21.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.985+0000 7f126dffa640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f12680d31d0 con 0x7f126806c400 2024-04-15T15:16:21.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.985+0000 7f126dffa640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f12680d25c0 con 0x7f126815e610 2024-04-15T15:16:21.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.986+0000 7f1266ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f126815e9f0 0x7f12680d3660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60214/0 (socket says 172.21.15.72:60214) 2024-04-15T15:16:21.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.986+0000 7f1266ffd640 1 -- 172.21.15.72:0/2153196880 learned_addr learned my addr 172.21.15.72:0/2153196880 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:21.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.986+0000 7f12677fe640 1 --2- 172.21.15.72:0/2153196880 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f126815e610 0x7f12680d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.986+0000 7f1267fff640 1 --2- 172.21.15.72:0/2153196880 >> v2:172.21.15.175:3300/0 conn(0x7f126806c400 0x7f126806c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.990 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.986+0000 7f12667fc640 1 -- 172.21.15.72:0/2153196880 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1431881324 0 0) 0x7f12680d2320 con 0x7f126815e9f0 2024-04-15T15:16:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.986+0000 7f12667fc640 1 -- 172.21.15.72:0/2153196880 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1254003520 con 0x7f126815e9f0 2024-04-15T15:16:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.986+0000 7f12667fc640 1 -- 172.21.15.72:0/2153196880 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2082997239 0 0) 0x7f1254003520 con 0x7f126815e9f0 2024-04-15T15:16:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.986+0000 7f12667fc640 1 -- 172.21.15.72:0/2153196880 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f126815e610 msgr2=0x7f12680d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.986+0000 7f12667fc640 1 --2- 172.21.15.72:0/2153196880 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f126815e610 0x7f12680d0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.986+0000 7f12667fc640 1 -- 172.21.15.72:0/2153196880 >> v2:172.21.15.175:3300/0 conn(0x7f126806c400 msgr2=0x7f126806c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.986+0000 7f12667fc640 1 --2- 172.21.15.72:0/2153196880 >> v2:172.21.15.175:3300/0 conn(0x7f126806c400 0x7f126806c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.986+0000 7f12667fc640 1 -- 172.21.15.72:0/2153196880 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12680d28a0 con 0x7f126815e9f0 2024-04-15T15:16:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.986+0000 7f12667fc640 1 -- 172.21.15.72:0/2153196880 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f1250001670 con 0x7f126815e9f0 2024-04-15T15:16:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.986+0000 7f12667fc640 1 -- 172.21.15.72:0/2153196880 >> v1:172.21.15.72:6789/0 conn(0x7f126815e9f0 legacy=0x7f12680d3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.986+0000 7f12667fc640 1 --2- 172.21.15.72:0/2153196880 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f12540039b0 0x7f1254003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.986+0000 7f12667fc640 1 --2- 172.21.15.72:0/2153196880 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f126815e610 0x7f1254004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.986+0000 7f12667fc640 1 --2- 172.21.15.72:0/2153196880 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1254004d40 0x7f12540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.986+0000 7f12667fc640 1 -- 172.21.15.72:0/2153196880 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f12680d31d0 con 0x7f12540039b0 2024-04-15T15:16:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.986+0000 7f12667fc640 1 -- 172.21.15.72:0/2153196880 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f12680d25c0 con 0x7f1254004d40 2024-04-15T15:16:21.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.986+0000 7f12667fc640 1 -- 172.21.15.72:0/2153196880 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f125400b9c0 con 0x7f126815e610 2024-04-15T15:16:21.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.986+0000 7f12677fe640 1 --2- 172.21.15.72:0/2153196880 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f12540039b0 0x7f1254003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.986+0000 7f1266ffd640 1 --2- 172.21.15.72:0/2153196880 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f126815e610 0x7f1254004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.986+0000 7f1267fff640 1 --2- 172.21.15.72:0/2153196880 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1254004d40 0x7f12540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.987+0000 7f12677fe640 1 -- 172.21.15.72:0/2153196880 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f126815e610 msgr2=0x7f1254004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:21.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.987+0000 7f12677fe640 1 --2- 172.21.15.72:0/2153196880 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f126815e610 0x7f1254004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.987+0000 7f12677fe640 1 -- 172.21.15.72:0/2153196880 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1254004d40 msgr2=0x7f12540051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:21.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.987+0000 7f12677fe640 1 --2- 172.21.15.72:0/2153196880 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1254004d40 0x7f12540051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.987+0000 7f12677fe640 1 -- 172.21.15.72:0/2153196880 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1254003520 con 0x7f12540039b0 2024-04-15T15:16:21.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.987+0000 7f12677fe640 1 --2- 172.21.15.72:0/2153196880 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f12540039b0 0x7f1254003dd0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f1258009ac0 tx=0x7f1254005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d50c0992be853b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:21.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.987+0000 7f12667fc640 1 -- 172.21.15.72:0/2153196880 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1258039070 con 0x7f12540039b0 2024-04-15T15:16:21.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.987+0000 7f12667fc640 1 -- 172.21.15.72:0/2153196880 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f125802c6b0 con 0x7f12540039b0 2024-04-15T15:16:21.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.987+0000 7f12667fc640 1 -- 172.21.15.72:0/2153196880 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1258009690 con 0x7f12540039b0 2024-04-15T15:16:21.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.988+0000 7f126dffa640 1 -- 172.21.15.72:0/2153196880 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f12540039b0 msgr2=0x7f1254003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.988+0000 7f126dffa640 1 --2- 172.21.15.72:0/2153196880 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f12540039b0 0x7f1254003dd0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f1258009ac0 tx=0x7f1254005830 comp rx=0 tx=0).stop 2024-04-15T15:16:21.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.988+0000 7f1267fff640 1 -- 172.21.15.72:0/2153196880 reap_dead start 2024-04-15T15:16:21.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.988+0000 7f126dffa640 1 -- 172.21.15.72:0/2153196880 shutdown_connections 2024-04-15T15:16:21.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.988+0000 7f126dffa640 1 -- 172.21.15.72:0/2153196880 >> 172.21.15.72:0/2153196880 conn(0x7f12680d4750 msgr2=0x7f12680d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:21.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.988+0000 7f126dffa640 1 -- 172.21.15.72:0/2153196880 shutdown_connections 2024-04-15T15:16:21.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.988+0000 7f126dffa640 1 -- 172.21.15.72:0/2153196880 wait complete. 2024-04-15T15:16:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.989+0000 7f126dffa640 1 Processor -- start 2024-04-15T15:16:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.989+0000 7f126dffa640 1 -- start start 2024-04-15T15:16:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.990+0000 7f126dffa640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f12540039b0 0x7f1268161930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.990+0000 7f126dffa640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12681590f0 0x7f126815f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.990+0000 7f126dffa640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f12680cd120 0x7f12680cd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.990+0000 7f126dffa640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f126806d130 con 0x7f12680cd120 2024-04-15T15:16:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.990+0000 7f126dffa640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f12680d2e60 con 0x7f12681590f0 2024-04-15T15:16:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.990+0000 7f126dffa640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f12680d0690 con 0x7f12540039b0 2024-04-15T15:16:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.990+0000 7f1267fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f12680cd120 0x7f12680cd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.990+0000 7f1267fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f12680cd120 0x7f12680cd5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42926/0 (socket says 172.21.15.72:42926) 2024-04-15T15:16:21.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.990+0000 7f1267fff640 1 -- 172.21.15.72:0/3614785758 learned_addr learned my addr 172.21.15.72:0/3614785758 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.990+0000 7f1266ffd640 1 --2- 172.21.15.72:0/3614785758 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12681590f0 0x7f126815f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.990+0000 7f12677fe640 1 --2- 172.21.15.72:0/3614785758 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f12540039b0 0x7f1268161930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.991+0000 7f1267fff640 1 -- 172.21.15.72:0/3614785758 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f12540039b0 msgr2=0x7f1268161930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.991+0000 7f1267fff640 1 --2- 172.21.15.72:0/3614785758 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f12540039b0 0x7f1268161930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.991+0000 7f1267fff640 1 -- 172.21.15.72:0/3614785758 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12681590f0 msgr2=0x7f126815f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.991+0000 7f1267fff640 1 --2- 172.21.15.72:0/3614785758 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12681590f0 0x7f126815f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.991+0000 7f1267fff640 1 -- 172.21.15.72:0/3614785758 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f126815fde0 con 0x7f12680cd120 2024-04-15T15:16:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.991+0000 7f1267fff640 1 --2- 172.21.15.72:0/3614785758 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f12680cd120 0x7f12680cd5a0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f12680d4700 tx=0x7f126000f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.991+0000 7f125ffff640 1 -- 172.21.15.72:0/3614785758 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f126000fd10 con 0x7f12680cd120 2024-04-15T15:16:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.991+0000 7f125ffff640 1 -- 172.21.15.72:0/3614785758 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f126001d070 con 0x7f12680cd120 2024-04-15T15:16:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.991+0000 7f126dffa640 1 -- 172.21.15.72:0/3614785758 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12681600b0 con 0x7f12680cd120 2024-04-15T15:16:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.991+0000 7f125ffff640 1 -- 172.21.15.72:0/3614785758 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1260015670 con 0x7f12680cd120 2024-04-15T15:16:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.991+0000 7f126dffa640 1 -- 172.21.15.72:0/3614785758 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1268160670 con 0x7f12680cd120 2024-04-15T15:16:21.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.992+0000 7f126dffa640 1 -- 172.21.15.72:0/3614785758 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1268075c50 con 0x7f12680cd120 2024-04-15T15:16:21.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.992+0000 7f125ffff640 1 -- 172.21.15.72:0/3614785758 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f1260015810 con 0x7f12680cd120 2024-04-15T15:16:21.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.993+0000 7f125ffff640 1 --2- 172.21.15.72:0/3614785758 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f124806c980 0x7f124806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.993+0000 7f125ffff640 1 -- 172.21.15.72:0/3614785758 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f126008c560 con 0x7f12680cd120 2024-04-15T15:16:21.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.993+0000 7f12677fe640 1 --2- 172.21.15.72:0/3614785758 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f124806c980 0x7f124806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.993+0000 7f12677fe640 1 --2- 172.21.15.72:0/3614785758 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f124806c980 0x7f124806ee40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1258036000 tx=0x7f1258030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:21.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:21.997+0000 7f125ffff640 1 -- 172.21.15.72:0/3614785758 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1260010040 con 0x7f12680cd120 2024-04-15T15:16:22.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.172+0000 7f126dffa640 1 -- 172.21.15.72:0/3614785758 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f12680cf740 con 0x7f12680cd120 2024-04-15T15:16:22.173 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.172+0000 7f125ffff640 1 -- 172.21.15.72:0/3614785758 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v38) ==== 104+0+11 (secure 0 0 0) 0x7f126005a700 con 0x7f12680cd120 2024-04-15T15:16:22.174 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:16:22.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.174+0000 7f126dffa640 1 -- 172.21.15.72:0/3614785758 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f124806c980 msgr2=0x7f124806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.174+0000 7f126dffa640 1 --2- 172.21.15.72:0/3614785758 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f124806c980 0x7f124806ee40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1258036000 tx=0x7f1258030000 comp rx=0 tx=0).stop 2024-04-15T15:16:22.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.174+0000 7f126dffa640 1 -- 172.21.15.72:0/3614785758 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f12680cd120 msgr2=0x7f12680cd5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.174+0000 7f126dffa640 1 --2- 172.21.15.72:0/3614785758 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f12680cd120 0x7f12680cd5a0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f12680d4700 tx=0x7f126000f1b0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.174+0000 7f126dffa640 1 -- 172.21.15.72:0/3614785758 shutdown_connections 2024-04-15T15:16:22.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.174+0000 7f126dffa640 1 --2- 172.21.15.72:0/3614785758 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f124806c980 0x7f124806ee40 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.174+0000 7f126dffa640 1 --2- 172.21.15.72:0/3614785758 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f12680cd120 0x7f12680cd5a0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.174+0000 7f126dffa640 1 --2- 172.21.15.72:0/3614785758 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f12681590f0 0x7f126815f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.174+0000 7f126dffa640 1 --2- 172.21.15.72:0/3614785758 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f12540039b0 0x7f1268161930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.174+0000 7f126dffa640 1 -- 172.21.15.72:0/3614785758 >> 172.21.15.72:0/3614785758 conn(0x7f12680d4750 msgr2=0x7f12680b4a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:22.176 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.174+0000 7f126dffa640 1 -- 172.21.15.72:0/3614785758 shutdown_connections 2024-04-15T15:16:22.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.174+0000 7f126dffa640 1 -- 172.21.15.72:0/3614785758 wait complete. 2024-04-15T15:16:22.187 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:16:22.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.292+0000 7f4576093640 1 Processor -- start 2024-04-15T15:16:22.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.292+0000 7f4576093640 1 -- start start 2024-04-15T15:16:22.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.292+0000 7f4576093640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f457015c630 0x7f457006cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.292+0000 7f4576093640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f457006bd50 con 0x7f457006ed30 2024-04-15T15:16:22.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.293+0000 7f4576093640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f457006bff0 con 0x7f45700ccb90 2024-04-15T15:16:22.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.293+0000 7f4576093640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f457006d2f0 con 0x7f457015c630 2024-04-15T15:16:22.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.293+0000 7f456ffff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f457006ed30 0x7f45700ad0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60220/0 (socket says 172.21.15.72:60220) 2024-04-15T15:16:22.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.293+0000 7f456ffff640 1 -- 172.21.15.72:0/2270027693 learned_addr learned my addr 172.21.15.72:0/2270027693 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:22.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.293+0000 7f456f7fe640 1 --2- 172.21.15.72:0/2270027693 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f457015c630 0x7f457006cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.293+0000 7f456effd640 1 -- 172.21.15.72:0/2270027693 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3306823281 0 0) 0x7f457006bd50 con 0x7f457006ed30 2024-04-15T15:16:22.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.293+0000 7f456effd640 1 -- 172.21.15.72:0/2270027693 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f455c003520 con 0x7f457006ed30 2024-04-15T15:16:22.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.293+0000 7f456f7fe640 1 -- 172.21.15.72:0/2270027693 >> v1:172.21.15.175:6789/0 conn(0x7f45700ccb90 legacy=0x7f457006e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.293+0000 7f456f7fe640 1 -- 172.21.15.72:0/2270027693 >> v1:172.21.15.72:6789/0 conn(0x7f457006ed30 legacy=0x7f45700ad0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.293+0000 7f456effd640 1 -- 172.21.15.72:0/2270027693 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4078146249 0 0) 0x7f457006bff0 con 0x7f45700ccb90 2024-04-15T15:16:22.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.293+0000 7f456f7fe640 1 -- 172.21.15.72:0/2270027693 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f457006c2d0 con 0x7f457015c630 2024-04-15T15:16:22.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.294+0000 7f456f7fe640 1 --2- 172.21.15.72:0/2270027693 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f457015c630 0x7f457006cdb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f45600097d0 tx=0x7f456002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25ca29ed5fe50f91 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:22.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.294+0000 7f456effd640 1 -- 172.21.15.72:0/2270027693 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4560035020 con 0x7f457015c630 2024-04-15T15:16:22.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.294+0000 7f456effd640 1 -- 172.21.15.72:0/2270027693 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4560031890 con 0x7f457015c630 2024-04-15T15:16:22.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.294+0000 7f456effd640 1 -- 172.21.15.72:0/2270027693 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4560031c40 con 0x7f457015c630 2024-04-15T15:16:22.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.295+0000 7f4576093640 1 -- 172.21.15.72:0/2270027693 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f457015c630 msgr2=0x7f457006cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.295+0000 7f4576093640 1 --2- 172.21.15.72:0/2270027693 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f457015c630 0x7f457006cdb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f45600097d0 tx=0x7f456002b930 comp rx=0 tx=0).stop 2024-04-15T15:16:22.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.295+0000 7f4576093640 1 -- 172.21.15.72:0/2270027693 shutdown_connections 2024-04-15T15:16:22.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.295+0000 7f4576093640 1 --2- 172.21.15.72:0/2270027693 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f457015c630 0x7f457006cdb0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.295+0000 7f4576093640 1 -- 172.21.15.72:0/2270027693 >> 172.21.15.72:0/2270027693 conn(0x7f45700b31d0 msgr2=0x7f45700b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:22.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.295+0000 7f4576093640 1 -- 172.21.15.72:0/2270027693 shutdown_connections 2024-04-15T15:16:22.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.295+0000 7f4576093640 1 -- 172.21.15.72:0/2270027693 wait complete. 2024-04-15T15:16:22.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.296+0000 7f4576093640 1 Processor -- start 2024-04-15T15:16:22.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.296+0000 7f4576093640 1 -- start start 2024-04-15T15:16:22.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.297+0000 7f4576093640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f457006ed30 0x7f45701f1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.297+0000 7f4576093640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f45700ccb90 0x7f4570154180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.297+0000 7f4576093640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f457015c630 0x7f45701546c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.297+0000 7f4576093640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f457015ca10 con 0x7f457006ed30 2024-04-15T15:16:22.300 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.297+0000 7f4576093640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f457006c920 con 0x7f457015c630 2024-04-15T15:16:22.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.297+0000 7f4576093640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f45700ccff0 con 0x7f45700ccb90 2024-04-15T15:16:22.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.297+0000 7f456f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f457006ed30 0x7f45701f1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.297+0000 7f456f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f457006ed30 0x7f45701f1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42936/0 (socket says 172.21.15.72:42936) 2024-04-15T15:16:22.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.297+0000 7f456f7fe640 1 -- 172.21.15.72:0/206050107 learned_addr learned my addr 172.21.15.72:0/206050107 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:22.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.297+0000 7f456ffff640 1 --2- 172.21.15.72:0/206050107 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f457015c630 0x7f45701546c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.297+0000 7f4567fff640 1 --2- 172.21.15.72:0/206050107 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f45700ccb90 0x7f4570154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.297+0000 7f456f7fe640 1 -- 172.21.15.72:0/206050107 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f45700ccb90 msgr2=0x7f4570154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.297+0000 7f456f7fe640 1 --2- 172.21.15.72:0/206050107 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f45700ccb90 0x7f4570154180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.297+0000 7f456f7fe640 1 -- 172.21.15.72:0/206050107 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f457015c630 msgr2=0x7f45701546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.297+0000 7f456f7fe640 1 --2- 172.21.15.72:0/206050107 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f457015c630 0x7f45701546c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.297+0000 7f456f7fe640 1 -- 172.21.15.72:0/206050107 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4570148f70 con 0x7f457006ed30 2024-04-15T15:16:22.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.297+0000 7f4567fff640 1 --2- 172.21.15.72:0/206050107 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f45700ccb90 0x7f4570154180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:22.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.297+0000 7f456ffff640 1 --2- 172.21.15.72:0/206050107 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f457015c630 0x7f45701546c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:22.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.297+0000 7f456f7fe640 1 --2- 172.21.15.72:0/206050107 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f457006ed30 0x7f45701f1c10 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f4560034040 tx=0x7f45600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:22.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.297+0000 7f456cff9640 1 -- 172.21.15.72:0/206050107 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4560035050 con 0x7f457006ed30 2024-04-15T15:16:22.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.297+0000 7f456cff9640 1 -- 172.21.15.72:0/206050107 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4560003a20 con 0x7f457006ed30 2024-04-15T15:16:22.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.297+0000 7f456cff9640 1 -- 172.21.15.72:0/206050107 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4560003df0 con 0x7f457006ed30 2024-04-15T15:16:22.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.297+0000 7f4576093640 1 -- 172.21.15.72:0/206050107 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4570156ef0 con 0x7f457006ed30 2024-04-15T15:16:22.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.297+0000 7f4576093640 1 -- 172.21.15.72:0/206050107 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45701574b0 con 0x7f457006ed30 2024-04-15T15:16:22.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.298+0000 7f4576093640 1 -- 172.21.15.72:0/206050107 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4570075c50 con 0x7f457006ed30 2024-04-15T15:16:22.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.299+0000 7f456cff9640 1 -- 172.21.15.72:0/206050107 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f4560038020 con 0x7f457006ed30 2024-04-15T15:16:22.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.299+0000 7f456cff9640 1 --2- 172.21.15.72:0/206050107 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f455006c980 0x7f455006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.300+0000 7f456cff9640 1 -- 172.21.15.72:0/206050107 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f45600ae050 con 0x7f457006ed30 2024-04-15T15:16:22.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.300+0000 7f4567fff640 1 --2- 172.21.15.72:0/206050107 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f455006c980 0x7f455006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.300+0000 7f4567fff640 1 --2- 172.21.15.72:0/206050107 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f455006c980 0x7f455006ee40 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f457014d2c0 tx=0x7f4558009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:22.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.304+0000 7f456cff9640 1 -- 172.21.15.72:0/206050107 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4570148f70 con 0x7f457006ed30 2024-04-15T15:16:22.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.479+0000 7f4576093640 1 -- 172.21.15.72:0/206050107 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f45700b20a0 con 0x7f457006ed30 2024-04-15T15:16:22.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.479+0000 7f456cff9640 1 -- 172.21.15.72:0/206050107 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v38) ==== 100+0+11 (secure 0 0 0) 0x7f456002d040 con 0x7f457006ed30 2024-04-15T15:16:22.481 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-04-15T15:16:22.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.481+0000 7f4576093640 1 -- 172.21.15.72:0/206050107 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f455006c980 msgr2=0x7f455006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.481+0000 7f4576093640 1 --2- 172.21.15.72:0/206050107 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f455006c980 0x7f455006ee40 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f457014d2c0 tx=0x7f4558009040 comp rx=0 tx=0).stop 2024-04-15T15:16:22.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.481+0000 7f4576093640 1 -- 172.21.15.72:0/206050107 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f457006ed30 msgr2=0x7f45701f1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.481+0000 7f4576093640 1 --2- 172.21.15.72:0/206050107 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f457006ed30 0x7f45701f1c10 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f4560034040 tx=0x7f45600046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.481+0000 7f4576093640 1 -- 172.21.15.72:0/206050107 shutdown_connections 2024-04-15T15:16:22.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.481+0000 7f4576093640 1 --2- 172.21.15.72:0/206050107 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f455006c980 0x7f455006ee40 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.481+0000 7f4576093640 1 --2- 172.21.15.72:0/206050107 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f457015c630 0x7f45701546c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.481+0000 7f4576093640 1 --2- 172.21.15.72:0/206050107 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f45700ccb90 0x7f4570154180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.481+0000 7f4576093640 1 --2- 172.21.15.72:0/206050107 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f457006ed30 0x7f45701f1c10 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.481+0000 7f4576093640 1 -- 172.21.15.72:0/206050107 >> 172.21.15.72:0/206050107 conn(0x7f45700b31d0 msgr2=0x7f457006e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:22.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.481+0000 7f4576093640 1 -- 172.21.15.72:0/206050107 shutdown_connections 2024-04-15T15:16:22.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.481+0000 7f4576093640 1 -- 172.21.15.72:0/206050107 wait complete. 2024-04-15T15:16:22.494 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:16:22.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.595+0000 7f4c9ea50640 1 Processor -- start 2024-04-15T15:16:22.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.595+0000 7f4c9ea50640 1 -- start start 2024-04-15T15:16:22.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.596+0000 7f4c9ea50640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4c9815c630 0x7f4c9806cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.596+0000 7f4c9ea50640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c9806bd50 con 0x7f4c980ccb90 2024-04-15T15:16:22.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.596+0000 7f4c9ea50640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c9806bff0 con 0x7f4c9806ed30 2024-04-15T15:16:22.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.596+0000 7f4c9ea50640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f4c9806d2f0 con 0x7f4c9815c630 2024-04-15T15:16:22.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.596+0000 7f4c977fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f4c980ccb90 0x7f4c9806e620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60230/0 (socket says 172.21.15.72:60230) 2024-04-15T15:16:22.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.596+0000 7f4c977fe640 1 -- 172.21.15.72:0/3385440338 learned_addr learned my addr 172.21.15.72:0/3385440338 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:22.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.596+0000 7f4c97fff640 1 --2- 172.21.15.72:0/3385440338 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4c9815c630 0x7f4c9806cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.597+0000 7f4c96ffd640 1 -- 172.21.15.72:0/3385440338 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 601216757 0 0) 0x7f4c9806bd50 con 0x7f4c980ccb90 2024-04-15T15:16:22.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.597+0000 7f4c96ffd640 1 -- 172.21.15.72:0/3385440338 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c80003520 con 0x7f4c980ccb90 2024-04-15T15:16:22.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.597+0000 7f4c96ffd640 1 -- 172.21.15.72:0/3385440338 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3837271782 0 0) 0x7f4c9806bff0 con 0x7f4c9806ed30 2024-04-15T15:16:22.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.597+0000 7f4c96ffd640 1 -- 172.21.15.72:0/3385440338 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4c9806bd50 con 0x7f4c9806ed30 2024-04-15T15:16:22.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.597+0000 7f4c97fff640 1 -- 172.21.15.72:0/3385440338 >> v1:172.21.15.175:6789/0 conn(0x7f4c9806ed30 legacy=0x7f4c980ad0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.597+0000 7f4c97fff640 1 -- 172.21.15.72:0/3385440338 >> v1:172.21.15.72:6789/0 conn(0x7f4c980ccb90 legacy=0x7f4c9806e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.597+0000 7f4c97fff640 1 -- 172.21.15.72:0/3385440338 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c9806c2d0 con 0x7f4c9815c630 2024-04-15T15:16:22.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.598+0000 7f4c97fff640 1 --2- 172.21.15.72:0/3385440338 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4c9815c630 0x7f4c9806cdb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4c880097d0 tx=0x7f4c8802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68ed5e2acbceaba8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:22.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.598+0000 7f4c96ffd640 1 -- 172.21.15.72:0/3385440338 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c88035020 con 0x7f4c9815c630 2024-04-15T15:16:22.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.598+0000 7f4c96ffd640 1 -- 172.21.15.72:0/3385440338 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c88031890 con 0x7f4c9815c630 2024-04-15T15:16:22.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.598+0000 7f4c96ffd640 1 -- 172.21.15.72:0/3385440338 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c88031c40 con 0x7f4c9815c630 2024-04-15T15:16:22.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.598+0000 7f4c9ea50640 1 -- 172.21.15.72:0/3385440338 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4c9815c630 msgr2=0x7f4c9806cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.598+0000 7f4c9ea50640 1 --2- 172.21.15.72:0/3385440338 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4c9815c630 0x7f4c9806cdb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4c880097d0 tx=0x7f4c8802b930 comp rx=0 tx=0).stop 2024-04-15T15:16:22.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.598+0000 7f4c9ea50640 1 -- 172.21.15.72:0/3385440338 shutdown_connections 2024-04-15T15:16:22.601 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.598+0000 7f4c9ea50640 1 --2- 172.21.15.72:0/3385440338 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4c9815c630 0x7f4c9806cdb0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.598+0000 7f4c9ea50640 1 -- 172.21.15.72:0/3385440338 >> 172.21.15.72:0/3385440338 conn(0x7f4c980b31d0 msgr2=0x7f4c980b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:22.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.599+0000 7f4c9ea50640 1 -- 172.21.15.72:0/3385440338 shutdown_connections 2024-04-15T15:16:22.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.599+0000 7f4c9ea50640 1 -- 172.21.15.72:0/3385440338 wait complete. 2024-04-15T15:16:22.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.600+0000 7f4c9ea50640 1 Processor -- start 2024-04-15T15:16:22.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.600+0000 7f4c9ea50640 1 -- start start 2024-04-15T15:16:22.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.600+0000 7f4c9ea50640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4c9806ed30 0x7f4c98151910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.600+0000 7f4c9ea50640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c980ccb90 0x7f4c98151e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.601+0000 7f4c9ea50640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4c9815c630 0x7f4c980ca1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.601+0000 7f4c9ea50640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f4c9815ca10 con 0x7f4c980ccb90 2024-04-15T15:16:22.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.601+0000 7f4c9ea50640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f4c9806c920 con 0x7f4c9806ed30 2024-04-15T15:16:22.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.601+0000 7f4c9ea50640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f4c980ccff0 con 0x7f4c9815c630 2024-04-15T15:16:22.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.601+0000 7f4c977fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c980ccb90 0x7f4c98151e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.601+0000 7f4c977fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c980ccb90 0x7f4c98151e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42946/0 (socket says 172.21.15.72:42946) 2024-04-15T15:16:22.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.601+0000 7f4c977fe640 1 -- 172.21.15.72:0/3813296629 learned_addr learned my addr 172.21.15.72:0/3813296629 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:22.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.601+0000 7f4c977fe640 1 -- 172.21.15.72:0/3813296629 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4c9815c630 msgr2=0x7f4c980ca1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.601+0000 7f4c977fe640 1 --2- 172.21.15.72:0/3813296629 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4c9815c630 0x7f4c980ca1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.601+0000 7f4c977fe640 1 -- 172.21.15.72:0/3813296629 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4c9806ed30 msgr2=0x7f4c98151910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.601+0000 7f4c977fe640 1 --2- 172.21.15.72:0/3813296629 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4c9806ed30 0x7f4c98151910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.601+0000 7f4c977fe640 1 -- 172.21.15.72:0/3813296629 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c980cafe0 con 0x7f4c980ccb90 2024-04-15T15:16:22.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.601+0000 7f4c977fe640 1 --2- 172.21.15.72:0/3813296629 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c980ccb90 0x7f4c98151e50 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c00c4d0 tx=0x7f4c8c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:22.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.601+0000 7f4c94ff9640 1 -- 172.21.15.72:0/3813296629 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c017070 con 0x7f4c980ccb90 2024-04-15T15:16:22.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.601+0000 7f4c94ff9640 1 -- 172.21.15.72:0/3813296629 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c8c005ce0 con 0x7f4c980ccb90 2024-04-15T15:16:22.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.601+0000 7f4c9ea50640 1 -- 172.21.15.72:0/3813296629 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c9814f1e0 con 0x7f4c980ccb90 2024-04-15T15:16:22.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.601+0000 7f4c94ff9640 1 -- 172.21.15.72:0/3813296629 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c8c007270 con 0x7f4c980ccb90 2024-04-15T15:16:22.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.601+0000 7f4c9ea50640 1 -- 172.21.15.72:0/3813296629 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c9814f7a0 con 0x7f4c980ccb90 2024-04-15T15:16:22.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.602+0000 7f4c9ea50640 1 -- 172.21.15.72:0/3813296629 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c9814fc90 con 0x7f4c980ccb90 2024-04-15T15:16:22.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.603+0000 7f4c94ff9640 1 -- 172.21.15.72:0/3813296629 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f4c8c01a070 con 0x7f4c980ccb90 2024-04-15T15:16:22.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.603+0000 7f4c94ff9640 1 --2- 172.21.15.72:0/3813296629 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4c8406c980 0x7f4c8406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.603+0000 7f4c94ff9640 1 -- 172.21.15.72:0/3813296629 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f4c8c08aff0 con 0x7f4c980ccb90 2024-04-15T15:16:22.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.603+0000 7f4c97fff640 1 --2- 172.21.15.72:0/3813296629 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4c8406c980 0x7f4c8406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.604+0000 7f4c97fff640 1 --2- 172.21.15.72:0/3813296629 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4c8406c980 0x7f4c8406ee40 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4c88036000 tx=0x7f4c88030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:22.609 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.607+0000 7f4c94ff9640 1 -- 172.21.15.72:0/3813296629 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4c8c058c60 con 0x7f4c980ccb90 2024-04-15T15:16:22.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.797+0000 7f4c9ea50640 1 -- 172.21.15.72:0/3813296629 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4c980d0f10 con 0x7f4c980ccb90 2024-04-15T15:16:22.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.797+0000 7f4c94ff9640 1 -- 172.21.15.72:0/3813296629 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v38) ==== 80+0+0 (secure 0 0 0) 0x7f4c8c05cf60 con 0x7f4c980ccb90 2024-04-15T15:16:22.799 INFO:teuthology.orchestra.run.smithi072.stderr:listed 0 entries 2024-04-15T15:16:22.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.799+0000 7f4c9ea50640 1 -- 172.21.15.72:0/3813296629 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4c8406c980 msgr2=0x7f4c8406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.799+0000 7f4c9ea50640 1 --2- 172.21.15.72:0/3813296629 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4c8406c980 0x7f4c8406ee40 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4c88036000 tx=0x7f4c88030000 comp rx=0 tx=0).stop 2024-04-15T15:16:22.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.799+0000 7f4c9ea50640 1 -- 172.21.15.72:0/3813296629 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c980ccb90 msgr2=0x7f4c98151e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.799+0000 7f4c9ea50640 1 --2- 172.21.15.72:0/3813296629 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c980ccb90 0x7f4c98151e50 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f4c8c00c4d0 tx=0x7f4c8c00c990 comp rx=0 tx=0).stop 2024-04-15T15:16:22.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.799+0000 7f4c9ea50640 1 -- 172.21.15.72:0/3813296629 shutdown_connections 2024-04-15T15:16:22.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.799+0000 7f4c9ea50640 1 --2- 172.21.15.72:0/3813296629 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4c8406c980 0x7f4c8406ee40 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.799+0000 7f4c9ea50640 1 --2- 172.21.15.72:0/3813296629 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4c9815c630 0x7f4c980ca1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.799+0000 7f4c9ea50640 1 --2- 172.21.15.72:0/3813296629 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4c980ccb90 0x7f4c98151e50 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.799+0000 7f4c9ea50640 1 --2- 172.21.15.72:0/3813296629 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4c9806ed30 0x7f4c98151910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.799+0000 7f4c9ea50640 1 -- 172.21.15.72:0/3813296629 >> 172.21.15.72:0/3813296629 conn(0x7f4c980b31d0 msgr2=0x7f4c98157570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:22.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.799+0000 7f4c9ea50640 1 -- 172.21.15.72:0/3813296629 shutdown_connections 2024-04-15T15:16:22.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.799+0000 7f4c9ea50640 1 -- 172.21.15.72:0/3813296629 wait complete. 2024-04-15T15:16:22.811 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:16:22.811 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:16:22.828 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:22.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.968+0000 7f51c4315640 1 Processor -- start 2024-04-15T15:16:22.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.968+0000 7f51c4315640 1 -- start start 2024-04-15T15:16:22.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.969+0000 7f51c4315640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f51bc161e40 0x7f51bc0b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.969+0000 7f51c4315640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f51bc0d12b0 0x7f51bc0d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.969+0000 7f51c4315640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51bc162220 con 0x7f51bc0d2ea0 2024-04-15T15:16:22.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.969+0000 7f51c4315640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f51bc0d21f0 con 0x7f51bc0d12b0 2024-04-15T15:16:22.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.969+0000 7f51c4315640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f51bc0d1820 con 0x7f51bc161e40 2024-04-15T15:16:22.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.970+0000 7f51c18b0640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f51bc0d12b0 0x7f51bc0d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.970+0000 7f51c28b2640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f51bc0d2ea0 0x7f51bc0d45b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60232/0 (socket says 172.21.15.72:60232) 2024-04-15T15:16:22.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.970+0000 7f51c18b0640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f51bc0d12b0 0x7f51bc0d2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42948/0 (socket says 172.21.15.72:42948) 2024-04-15T15:16:22.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.970+0000 7f51c28b2640 1 -- 172.21.15.72:0/1557678093 learned_addr learned my addr 172.21.15.72:0/1557678093 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:22.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.970+0000 7f51c18b0640 1 -- 172.21.15.72:0/1557678093 >> v2:172.21.15.175:3300/0 conn(0x7f51bc161e40 msgr2=0x7f51bc0b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:22.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.970+0000 7f51c20b1640 1 --2- 172.21.15.72:0/1557678093 >> v2:172.21.15.175:3300/0 conn(0x7f51bc161e40 0x7f51bc0b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.970+0000 7f51c18b0640 1 --2- 172.21.15.72:0/1557678093 >> v2:172.21.15.175:3300/0 conn(0x7f51bc161e40 0x7f51bc0b37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.970+0000 7f51c18b0640 1 -- 172.21.15.72:0/1557678093 >> v1:172.21.15.72:6789/0 conn(0x7f51bc0d2ea0 legacy=0x7f51bc0d45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.970+0000 7f51c10af640 1 -- 172.21.15.72:0/1557678093 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3279869380 0 0) 0x7f51bc162220 con 0x7f51bc0d2ea0 2024-04-15T15:16:22.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.970+0000 7f51c18b0640 1 -- 172.21.15.72:0/1557678093 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51bc0d1b00 con 0x7f51bc0d12b0 2024-04-15T15:16:22.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.971+0000 7f51c18b0640 1 --2- 172.21.15.72:0/1557678093 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51bc0d12b0 0x7f51bc0d2960 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f51b00097a0 tx=0x7f51b002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d2e01a4fa254edf server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:22.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.971+0000 7f51c10af640 1 -- 172.21.15.72:0/1557678093 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b00315a0 con 0x7f51bc0d12b0 2024-04-15T15:16:22.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.971+0000 7f51c10af640 1 -- 172.21.15.72:0/1557678093 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51b0037070 con 0x7f51bc0d12b0 2024-04-15T15:16:22.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.971+0000 7f51c10af640 1 -- 172.21.15.72:0/1557678093 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b0031960 con 0x7f51bc0d12b0 2024-04-15T15:16:22.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.971+0000 7f51c4315640 1 -- 172.21.15.72:0/1557678093 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51bc0d12b0 msgr2=0x7f51bc0d2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.971+0000 7f51c4315640 1 --2- 172.21.15.72:0/1557678093 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51bc0d12b0 0x7f51bc0d2960 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f51b00097a0 tx=0x7f51b002b840 comp rx=0 tx=0).stop 2024-04-15T15:16:22.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.971+0000 7f51c4315640 1 -- 172.21.15.72:0/1557678093 shutdown_connections 2024-04-15T15:16:22.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.971+0000 7f51c4315640 1 --2- 172.21.15.72:0/1557678093 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51bc0d12b0 0x7f51bc0d2960 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.971+0000 7f51c4315640 1 --2- 172.21.15.72:0/1557678093 >> v2:172.21.15.175:3300/0 conn(0x7f51bc161e40 0x7f51bc0b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.971+0000 7f51c4315640 1 -- 172.21.15.72:0/1557678093 >> 172.21.15.72:0/1557678093 conn(0x7f51bc09d970 msgr2=0x7f51bc0b95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:22.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.971+0000 7f51c4315640 1 -- 172.21.15.72:0/1557678093 shutdown_connections 2024-04-15T15:16:22.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.971+0000 7f51c4315640 1 -- 172.21.15.72:0/1557678093 wait complete. 2024-04-15T15:16:22.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.973+0000 7f51c4315640 1 Processor -- start 2024-04-15T15:16:22.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.973+0000 7f51c4315640 1 -- start start 2024-04-15T15:16:22.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.973+0000 7f51c4315640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51bc0d12b0 0x7f51bc159990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.977 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.973+0000 7f51c4315640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f51bc161e40 0x7f51bc0b2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.973+0000 7f51c4315640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f51bc1585f0 0x7f51bc157a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.973+0000 7f51c4315640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f51bc0b3d30 con 0x7f51bc0d12b0 2024-04-15T15:16:22.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.973+0000 7f51c4315640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f51bc0d2440 con 0x7f51bc1585f0 2024-04-15T15:16:22.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.973+0000 7f51c4315640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f51bc0b3eb0 con 0x7f51bc161e40 2024-04-15T15:16:22.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.973+0000 7f51c20b1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51bc0d12b0 0x7f51bc159990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.973+0000 7f51c20b1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51bc0d12b0 0x7f51bc159990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42958/0 (socket says 172.21.15.72:42958) 2024-04-15T15:16:22.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.973+0000 7f51c20b1640 1 -- 172.21.15.72:0/2202784336 learned_addr learned my addr 172.21.15.72:0/2202784336 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:22.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.974+0000 7f51c28b2640 1 --2- 172.21.15.72:0/2202784336 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f51bc1585f0 0x7f51bc157a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.974+0000 7f51c20b1640 1 -- 172.21.15.72:0/2202784336 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f51bc161e40 msgr2=0x7f51bc0b2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.974+0000 7f51c20b1640 1 --2- 172.21.15.72:0/2202784336 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f51bc161e40 0x7f51bc0b2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.974+0000 7f51c20b1640 1 -- 172.21.15.72:0/2202784336 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f51bc1585f0 msgr2=0x7f51bc157a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.978 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.974+0000 7f51c20b1640 1 --2- 172.21.15.72:0/2202784336 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f51bc1585f0 0x7f51bc157a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.974+0000 7f51c20b1640 1 -- 172.21.15.72:0/2202784336 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51bc0c8080 con 0x7f51bc0d12b0 2024-04-15T15:16:22.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.974+0000 7f51c28b2640 1 --2- 172.21.15.72:0/2202784336 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f51bc1585f0 0x7f51bc157a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:22.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.974+0000 7f51c20b1640 1 --2- 172.21.15.72:0/2202784336 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51bc0d12b0 0x7f51bc159990 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f51ac00cd00 tx=0x7f51ac00a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:22.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.974+0000 7f51baffd640 1 -- 172.21.15.72:0/2202784336 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ac003990 con 0x7f51bc0d12b0 2024-04-15T15:16:22.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.974+0000 7f51baffd640 1 -- 172.21.15.72:0/2202784336 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51ac003b30 con 0x7f51bc0d12b0 2024-04-15T15:16:22.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.974+0000 7f51c4315640 1 -- 172.21.15.72:0/2202784336 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51bc0c8360 con 0x7f51bc0d12b0 2024-04-15T15:16:22.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.974+0000 7f51baffd640 1 -- 172.21.15.72:0/2202784336 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51ac004020 con 0x7f51bc0d12b0 2024-04-15T15:16:22.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.974+0000 7f51c4315640 1 -- 172.21.15.72:0/2202784336 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51bc0c8920 con 0x7f51bc0d12b0 2024-04-15T15:16:22.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.975+0000 7f51c4315640 1 -- 172.21.15.72:0/2202784336 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51bc0689e0 con 0x7f51bc0d12b0 2024-04-15T15:16:22.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.976+0000 7f51baffd640 1 -- 172.21.15.72:0/2202784336 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f51ac0041c0 con 0x7f51bc0d12b0 2024-04-15T15:16:22.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.976+0000 7f51baffd640 1 --2- 172.21.15.72:0/2202784336 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f51a806c980 0x7f51a806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.976+0000 7f51c18b0640 1 --2- 172.21.15.72:0/2202784336 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f51a806c980 0x7f51a806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.976+0000 7f51baffd640 1 -- 172.21.15.72:0/2202784336 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f51ac08c380 con 0x7f51bc0d12b0 2024-04-15T15:16:22.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.977+0000 7f51c18b0640 1 --2- 172.21.15.72:0/2202784336 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f51a806c980 0x7f51a806ee40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f51b0036000 tx=0x7f51b0030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:22.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:22.981+0000 7f51baffd640 1 -- 172.21.15.72:0/2202784336 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f51ac014030 con 0x7f51bc0d12b0 2024-04-15T15:16:23.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.156+0000 7f51c4315640 1 -- 172.21.15.72:0/2202784336 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f51bc0d51b0 con 0x7f51bc0d12b0 2024-04-15T15:16:23.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.157+0000 7f51baffd640 1 -- 172.21.15.72:0/2202784336 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+16942 (secure 0 0 0) 0x7f51ac05a520 con 0x7f51bc0d12b0 2024-04-15T15:16:23.159 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:16:23.159 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":38,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:16:08.702307+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:05.551499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:06.003152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:23.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.159+0000 7f51c4315640 1 -- 172.21.15.72:0/2202784336 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f51a806c980 msgr2=0x7f51a806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.159+0000 7f51c4315640 1 --2- 172.21.15.72:0/2202784336 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f51a806c980 0x7f51a806ee40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f51b0036000 tx=0x7f51b0030000 comp rx=0 tx=0).stop 2024-04-15T15:16:23.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.159+0000 7f51c4315640 1 -- 172.21.15.72:0/2202784336 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51bc0d12b0 msgr2=0x7f51bc159990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.159+0000 7f51c4315640 1 --2- 172.21.15.72:0/2202784336 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51bc0d12b0 0x7f51bc159990 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f51ac00cd00 tx=0x7f51ac00a5c0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.159+0000 7f51c4315640 1 -- 172.21.15.72:0/2202784336 shutdown_connections 2024-04-15T15:16:23.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.159+0000 7f51c4315640 1 --2- 172.21.15.72:0/2202784336 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f51a806c980 0x7f51a806ee40 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.159+0000 7f51c4315640 1 --2- 172.21.15.72:0/2202784336 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f51bc1585f0 0x7f51bc157a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.159+0000 7f51c4315640 1 --2- 172.21.15.72:0/2202784336 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f51bc161e40 0x7f51bc0b2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.159+0000 7f51c4315640 1 --2- 172.21.15.72:0/2202784336 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51bc0d12b0 0x7f51bc159990 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.159+0000 7f51c4315640 1 -- 172.21.15.72:0/2202784336 >> 172.21.15.72:0/2202784336 conn(0x7f51bc09d970 msgr2=0x7f51bc1604c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:23.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.159+0000 7f51c4315640 1 -- 172.21.15.72:0/2202784336 shutdown_connections 2024-04-15T15:16:23.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.159+0000 7f51c4315640 1 -- 172.21.15.72:0/2202784336 wait complete. 2024-04-15T15:16:23.172 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:39.359102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:16:05.551499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:16:06.003152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:16:23.172 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:23.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.277+0000 7f1ebdf32640 1 Processor -- start 2024-04-15T15:16:23.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.277+0000 7f1ebdf32640 1 -- start start 2024-04-15T15:16:23.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.278+0000 7f1ebdf32640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1eb80b5410 0x7f1eb80b57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.278+0000 7f1ebdf32640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1eb80b0050 0x7f1eb806c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.278+0000 7f1ebdf32640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1eb80d07c0 con 0x7f1eb80afae0 2024-04-15T15:16:23.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.278+0000 7f1ebdf32640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f1eb80d0330 con 0x7f1eb80b0050 2024-04-15T15:16:23.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.278+0000 7f1ebdf32640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f1eb80b5d30 con 0x7f1eb80b5410 2024-04-15T15:16:23.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.278+0000 7f1eb6ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1eb80b0050 0x7f1eb806c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.278+0000 7f1eb6ffd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1eb80b0050 0x7f1eb806c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42968/0 (socket says 172.21.15.72:42968) 2024-04-15T15:16:23.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.278+0000 7f1eb6ffd640 1 -- 172.21.15.72:0/3744612438 learned_addr learned my addr 172.21.15.72:0/3744612438 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:23.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.278+0000 7f1eb7fff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f1eb80afae0 0x7f1eb80ce950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60236/0 (socket says 172.21.15.72:60236) 2024-04-15T15:16:23.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.278+0000 7f1eb77fe640 1 --2- 172.21.15.72:0/3744612438 >> v2:172.21.15.175:3300/0 conn(0x7f1eb80b5410 0x7f1eb80b57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.278+0000 7f1eb6ffd640 1 -- 172.21.15.72:0/3744612438 >> v2:172.21.15.175:3300/0 conn(0x7f1eb80b5410 msgr2=0x7f1eb80b57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:23.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.278+0000 7f1eb6ffd640 1 --2- 172.21.15.72:0/3744612438 >> v2:172.21.15.175:3300/0 conn(0x7f1eb80b5410 0x7f1eb80b57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.278+0000 7f1eb6ffd640 1 -- 172.21.15.72:0/3744612438 >> v1:172.21.15.72:6789/0 conn(0x7f1eb80afae0 legacy=0x7f1eb80ce950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.278+0000 7f1eb6ffd640 1 -- 172.21.15.72:0/3744612438 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eb80b3aa0 con 0x7f1eb80b0050 2024-04-15T15:16:23.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.278+0000 7f1eb67fc640 1 -- 172.21.15.72:0/3744612438 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3154804781 0 0) 0x7f1eb80d07c0 con 0x7f1eb80afae0 2024-04-15T15:16:23.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.279+0000 7f1eb6ffd640 1 --2- 172.21.15.72:0/3744612438 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1eb80b0050 0x7f1eb806c340 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8073da0 tx=0x7f1eac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8836212712c8ae4 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:23.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.279+0000 7f1eb67fc640 1 -- 172.21.15.72:0/3744612438 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eac035020 con 0x7f1eb80b0050 2024-04-15T15:16:23.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.279+0000 7f1eb67fc640 1 -- 172.21.15.72:0/3744612438 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1eac031870 con 0x7f1eb80b0050 2024-04-15T15:16:23.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.279+0000 7f1eb67fc640 1 -- 172.21.15.72:0/3744612438 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eac031c40 con 0x7f1eb80b0050 2024-04-15T15:16:23.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.280+0000 7f1ebdf32640 1 -- 172.21.15.72:0/3744612438 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1eb80b0050 msgr2=0x7f1eb806c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.280+0000 7f1ebdf32640 1 --2- 172.21.15.72:0/3744612438 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1eb80b0050 0x7f1eb806c340 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8073da0 tx=0x7f1eac02b910 comp rx=0 tx=0).stop 2024-04-15T15:16:23.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.280+0000 7f1ebdf32640 1 -- 172.21.15.72:0/3744612438 shutdown_connections 2024-04-15T15:16:23.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.280+0000 7f1ebdf32640 1 --2- 172.21.15.72:0/3744612438 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1eb80b0050 0x7f1eb806c340 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.280+0000 7f1ebdf32640 1 --2- 172.21.15.72:0/3744612438 >> v2:172.21.15.175:3300/0 conn(0x7f1eb80b5410 0x7f1eb80b57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.280+0000 7f1ebdf32640 1 -- 172.21.15.72:0/3744612438 >> 172.21.15.72:0/3744612438 conn(0x7f1eb80d0a50 msgr2=0x7f1eb80d0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:23.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.280+0000 7f1ebdf32640 1 -- 172.21.15.72:0/3744612438 shutdown_connections 2024-04-15T15:16:23.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.280+0000 7f1ebdf32640 1 -- 172.21.15.72:0/3744612438 wait complete. 2024-04-15T15:16:23.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.281+0000 7f1ebdf32640 1 Processor -- start 2024-04-15T15:16:23.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.281+0000 7f1ebdf32640 1 -- start start 2024-04-15T15:16:23.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.282+0000 7f1ebdf32640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1eb80afae0 0x7f1eb8149820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.282+0000 7f1ebdf32640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1eb80b0050 0x7f1eb814c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.282+0000 7f1ebdf32640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1eb80b5410 0x7f1eb80b0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.282+0000 7f1ebdf32640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb80d18f0 con 0x7f1eb80afae0 2024-04-15T15:16:23.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.282+0000 7f1ebdf32640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb80d1610 con 0x7f1eb80b5410 2024-04-15T15:16:23.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.282+0000 7f1ebdf32640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f1eb806db40 con 0x7f1eb80b0050 2024-04-15T15:16:23.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.282+0000 7f1eb77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1eb80afae0 0x7f1eb8149820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.282+0000 7f1eb77fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1eb80afae0 0x7f1eb8149820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42970/0 (socket says 172.21.15.72:42970) 2024-04-15T15:16:23.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.282+0000 7f1eb77fe640 1 -- 172.21.15.72:0/428011196 learned_addr learned my addr 172.21.15.72:0/428011196 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:23.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.282+0000 7f1eb77fe640 1 -- 172.21.15.72:0/428011196 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1eb80b0050 msgr2=0x7f1eb814c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.282+0000 7f1eb6ffd640 1 --2- 172.21.15.72:0/428011196 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1eb80b0050 0x7f1eb814c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.282+0000 7f1eb7fff640 1 --2- 172.21.15.72:0/428011196 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1eb80b5410 0x7f1eb80b0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.282+0000 7f1eb77fe640 1 --2- 172.21.15.72:0/428011196 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1eb80b0050 0x7f1eb814c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.282+0000 7f1eb77fe640 1 -- 172.21.15.72:0/428011196 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1eb80b5410 msgr2=0x7f1eb80b0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.282+0000 7f1eb77fe640 1 --2- 172.21.15.72:0/428011196 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1eb80b5410 0x7f1eb80b0530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.283+0000 7f1eb77fe640 1 -- 172.21.15.72:0/428011196 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eb8149d60 con 0x7f1eb80afae0 2024-04-15T15:16:23.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.283+0000 7f1eb6ffd640 1 --2- 172.21.15.72:0/428011196 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1eb80b0050 0x7f1eb814c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:23.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.283+0000 7f1eb7fff640 1 --2- 172.21.15.72:0/428011196 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1eb80b5410 0x7f1eb80b0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:23.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.283+0000 7f1eb77fe640 1 --2- 172.21.15.72:0/428011196 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1eb80afae0 0x7f1eb8149820 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f1ea800d800 tx=0x7f1ea800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:23.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.283+0000 7f1e9ffff640 1 -- 172.21.15.72:0/428011196 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea800b660 con 0x7f1eb80afae0 2024-04-15T15:16:23.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.283+0000 7f1e9ffff640 1 -- 172.21.15.72:0/428011196 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ea8019070 con 0x7f1eb80afae0 2024-04-15T15:16:23.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.283+0000 7f1ebdf32640 1 -- 172.21.15.72:0/428011196 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1eb814cdd0 con 0x7f1eb80afae0 2024-04-15T15:16:23.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.283+0000 7f1e9ffff640 1 -- 172.21.15.72:0/428011196 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea800ba20 con 0x7f1eb80afae0 2024-04-15T15:16:23.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.283+0000 7f1ebdf32640 1 -- 172.21.15.72:0/428011196 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1eb815bfb0 con 0x7f1eb80afae0 2024-04-15T15:16:23.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.284+0000 7f1ebdf32640 1 -- 172.21.15.72:0/428011196 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1eb814d2f0 con 0x7f1eb80afae0 2024-04-15T15:16:23.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.284+0000 7f1e9ffff640 1 -- 172.21.15.72:0/428011196 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f1ea800bbc0 con 0x7f1eb80afae0 2024-04-15T15:16:23.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.285+0000 7f1e9ffff640 1 --2- 172.21.15.72:0/428011196 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1ea406c980 0x7f1ea406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.285+0000 7f1e9ffff640 1 -- 172.21.15.72:0/428011196 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f1ea808c0c0 con 0x7f1eb80afae0 2024-04-15T15:16:23.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.285+0000 7f1eb6ffd640 1 --2- 172.21.15.72:0/428011196 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1ea406c980 0x7f1ea406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.285+0000 7f1eb6ffd640 1 --2- 172.21.15.72:0/428011196 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1ea406c980 0x7f1ea406ee40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1eac036000 tx=0x7f1eac008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:23.290 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.289+0000 7f1e9ffff640 1 -- 172.21.15.72:0/428011196 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1ea801c360 con 0x7f1eb80afae0 2024-04-15T15:16:23.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.463+0000 7f1ebdf32640 1 -- 172.21.15.72:0/428011196 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1eb8075c50 con 0x7f1eb80afae0 2024-04-15T15:16:23.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.463+0000 7f1e9ffff640 1 -- 172.21.15.72:0/428011196 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v38) ==== 93+0+10 (secure 0 0 0) 0x7f1ea805a400 con 0x7f1eb80afae0 2024-04-15T15:16:23.465 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-15T15:16:23.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.465+0000 7f1ebdf32640 1 -- 172.21.15.72:0/428011196 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1ea406c980 msgr2=0x7f1ea406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.465+0000 7f1ebdf32640 1 --2- 172.21.15.72:0/428011196 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1ea406c980 0x7f1ea406ee40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1eac036000 tx=0x7f1eac008030 comp rx=0 tx=0).stop 2024-04-15T15:16:23.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.465+0000 7f1ebdf32640 1 -- 172.21.15.72:0/428011196 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1eb80afae0 msgr2=0x7f1eb8149820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.465+0000 7f1ebdf32640 1 --2- 172.21.15.72:0/428011196 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1eb80afae0 0x7f1eb8149820 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f1ea800d800 tx=0x7f1ea800dcc0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.467 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.465+0000 7f1ebdf32640 1 -- 172.21.15.72:0/428011196 shutdown_connections 2024-04-15T15:16:23.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.465+0000 7f1ebdf32640 1 --2- 172.21.15.72:0/428011196 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1ea406c980 0x7f1ea406ee40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.465+0000 7f1ebdf32640 1 --2- 172.21.15.72:0/428011196 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1eb80b5410 0x7f1eb80b0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.465+0000 7f1ebdf32640 1 --2- 172.21.15.72:0/428011196 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1eb80b0050 0x7f1eb814c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.465+0000 7f1ebdf32640 1 --2- 172.21.15.72:0/428011196 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1eb80afae0 0x7f1eb8149820 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.465+0000 7f1ebdf32640 1 -- 172.21.15.72:0/428011196 >> 172.21.15.72:0/428011196 conn(0x7f1eb80d0a50 msgr2=0x7f1eb81610d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:23.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.466+0000 7f1ebdf32640 1 -- 172.21.15.72:0/428011196 shutdown_connections 2024-04-15T15:16:23.468 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.466+0000 7f1ebdf32640 1 -- 172.21.15.72:0/428011196 wait complete. 2024-04-15T15:16:23.478 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:16:23.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.584+0000 7f9af1fd4640 1 Processor -- start 2024-04-15T15:16:23.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.584+0000 7f9af1fd4640 1 -- start start 2024-04-15T15:16:23.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.585+0000 7f9af1fd4640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f9aec15e960 0x7f9aec0d1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.585+0000 7f9af1fd4640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9aec0d5010 0x7f9aec0cffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.585+0000 7f9af1fd4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9aec0b65b0 con 0x7f9aec0b11d0 2024-04-15T15:16:23.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.585+0000 7f9af1fd4640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f9aec0b6890 con 0x7f9aec15e960 2024-04-15T15:16:23.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.585+0000 7f9af1fd4640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f9aec0b6b70 con 0x7f9aec0d5010 2024-04-15T15:16:23.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.585+0000 7f9aeaffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f9aec0b11d0 0x7f9aec0d4900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60252/0 (socket says 172.21.15.72:60252) 2024-04-15T15:16:23.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.585+0000 7f9aeaffd640 1 -- 172.21.15.72:0/2420455307 learned_addr learned my addr 172.21.15.72:0/2420455307 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:23.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.585+0000 7f9aeb7fe640 1 --2- 172.21.15.72:0/2420455307 >> v2:172.21.15.175:3300/0 conn(0x7f9aec15e960 0x7f9aec0d1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.585+0000 7f9aebfff640 1 --2- 172.21.15.72:0/2420455307 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9aec0d5010 0x7f9aec0cffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.585+0000 7f9aea7fc640 1 -- 172.21.15.72:0/2420455307 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1994181898 0 0) 0x7f9aec0b65b0 con 0x7f9aec0b11d0 2024-04-15T15:16:23.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.585+0000 7f9aea7fc640 1 -- 172.21.15.72:0/2420455307 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ad4003520 con 0x7f9aec0b11d0 2024-04-15T15:16:23.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.585+0000 7f9aea7fc640 1 -- 172.21.15.72:0/2420455307 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3468185153 0 0) 0x7f9ad4003520 con 0x7f9aec0b11d0 2024-04-15T15:16:23.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.585+0000 7f9aea7fc640 1 -- 172.21.15.72:0/2420455307 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9aec0d5010 msgr2=0x7f9aec0cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:23.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.585+0000 7f9aea7fc640 1 --2- 172.21.15.72:0/2420455307 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9aec0d5010 0x7f9aec0cffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.585+0000 7f9aea7fc640 1 -- 172.21.15.72:0/2420455307 >> v2:172.21.15.175:3300/0 conn(0x7f9aec15e960 msgr2=0x7f9aec0d1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:23.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.585+0000 7f9aea7fc640 1 --2- 172.21.15.72:0/2420455307 >> v2:172.21.15.175:3300/0 conn(0x7f9aec15e960 0x7f9aec0d1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.585+0000 7f9aea7fc640 1 -- 172.21.15.72:0/2420455307 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aec0b6e50 con 0x7f9aec0b11d0 2024-04-15T15:16:23.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.585+0000 7f9aea7fc640 1 -- 172.21.15.72:0/2420455307 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f9ae0001670 con 0x7f9aec0b11d0 2024-04-15T15:16:23.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.586+0000 7f9aea7fc640 1 -- 172.21.15.72:0/2420455307 >> v1:172.21.15.72:6789/0 conn(0x7f9aec0b11d0 legacy=0x7f9aec0d4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.586+0000 7f9aea7fc640 1 --2- 172.21.15.72:0/2420455307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ad40039b0 0x7f9ad4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.586+0000 7f9aea7fc640 1 --2- 172.21.15.72:0/2420455307 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9ad4004800 0x7f9ad4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.586+0000 7f9aea7fc640 1 --2- 172.21.15.72:0/2420455307 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9aec0d5010 0x7f9ad40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.586+0000 7f9aea7fc640 1 -- 172.21.15.72:0/2420455307 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f9aec0b6890 con 0x7f9ad40039b0 2024-04-15T15:16:23.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.586+0000 7f9aea7fc640 1 -- 172.21.15.72:0/2420455307 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f9aec0b6b70 con 0x7f9ad4004800 2024-04-15T15:16:23.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.586+0000 7f9aea7fc640 1 -- 172.21.15.72:0/2420455307 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f9ad400b9c0 con 0x7f9aec0d5010 2024-04-15T15:16:23.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.586+0000 7f9aeb7fe640 1 --2- 172.21.15.72:0/2420455307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ad40039b0 0x7f9ad4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.586+0000 7f9aeaffd640 1 --2- 172.21.15.72:0/2420455307 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9ad4004800 0x7f9ad4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.586+0000 7f9aebfff640 1 --2- 172.21.15.72:0/2420455307 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9aec0d5010 0x7f9ad40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.586+0000 7f9aeb7fe640 1 -- 172.21.15.72:0/2420455307 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9aec0d5010 msgr2=0x7f9ad40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:23.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.586+0000 7f9aeb7fe640 1 --2- 172.21.15.72:0/2420455307 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9aec0d5010 0x7f9ad40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.586+0000 7f9aeb7fe640 1 -- 172.21.15.72:0/2420455307 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9ad4004800 msgr2=0x7f9ad4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:23.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.586+0000 7f9aeb7fe640 1 --2- 172.21.15.72:0/2420455307 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9ad4004800 0x7f9ad4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.586+0000 7f9aeb7fe640 1 -- 172.21.15.72:0/2420455307 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ad4003520 con 0x7f9ad40039b0 2024-04-15T15:16:23.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.586+0000 7f9aebfff640 1 --2- 172.21.15.72:0/2420455307 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9aec0d5010 0x7f9ad40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:23.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.587+0000 7f9aeb7fe640 1 --2- 172.21.15.72:0/2420455307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ad40039b0 0x7f9ad4003dd0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f9adc009ac0 tx=0x7f9ad4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=665036e81f4a4841 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:23.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.587+0000 7f9aea7fc640 1 -- 172.21.15.72:0/2420455307 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9adc039070 con 0x7f9ad40039b0 2024-04-15T15:16:23.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.587+0000 7f9aea7fc640 1 -- 172.21.15.72:0/2420455307 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9adc02c710 con 0x7f9ad40039b0 2024-04-15T15:16:23.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.587+0000 7f9aea7fc640 1 -- 172.21.15.72:0/2420455307 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9adc009690 con 0x7f9ad40039b0 2024-04-15T15:16:23.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.587+0000 7f9af1fd4640 1 -- 172.21.15.72:0/2420455307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ad40039b0 msgr2=0x7f9ad4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.587+0000 7f9af1fd4640 1 --2- 172.21.15.72:0/2420455307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ad40039b0 0x7f9ad4003dd0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f9adc009ac0 tx=0x7f9ad4005830 comp rx=0 tx=0).stop 2024-04-15T15:16:23.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.587+0000 7f9aebfff640 1 -- 172.21.15.72:0/2420455307 reap_dead start 2024-04-15T15:16:23.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.587+0000 7f9af1fd4640 1 -- 172.21.15.72:0/2420455307 shutdown_connections 2024-04-15T15:16:23.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.587+0000 7f9af1fd4640 1 -- 172.21.15.72:0/2420455307 >> 172.21.15.72:0/2420455307 conn(0x7f9aec0b9580 msgr2=0x7f9aec0b9990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:23.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.587+0000 7f9af1fd4640 1 -- 172.21.15.72:0/2420455307 shutdown_connections 2024-04-15T15:16:23.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.587+0000 7f9af1fd4640 1 -- 172.21.15.72:0/2420455307 wait complete. 2024-04-15T15:16:23.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.589+0000 7f9af1fd4640 1 Processor -- start 2024-04-15T15:16:23.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.589+0000 7f9af1fd4640 1 -- start start 2024-04-15T15:16:23.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.589+0000 7f9af1fd4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ad40039b0 0x7f9aec1f5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.590+0000 7f9af1fd4640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9aec1f7040 0x7f9aec0cd770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.590+0000 7f9af1fd4640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9aec0cdcb0 0x7f9aec15f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.590+0000 7f9af1fd4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f9aec0cfb40 con 0x7f9ad40039b0 2024-04-15T15:16:23.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.590+0000 7f9af1fd4640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f9aec0cf9c0 con 0x7f9aec1f7040 2024-04-15T15:16:23.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.590+0000 7f9af1fd4640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f9aec0b6040 con 0x7f9aec0cdcb0 2024-04-15T15:16:23.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.590+0000 7f9aeb7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ad40039b0 0x7f9aec1f5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.590+0000 7f9aeb7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ad40039b0 0x7f9aec1f5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:42988/0 (socket says 172.21.15.72:42988) 2024-04-15T15:16:23.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.590+0000 7f9aeb7fe640 1 -- 172.21.15.72:0/3801761679 learned_addr learned my addr 172.21.15.72:0/3801761679 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:23.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.590+0000 7f9aeaffd640 1 --2- 172.21.15.72:0/3801761679 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9aec1f7040 0x7f9aec0cd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.590+0000 7f9aebfff640 1 --2- 172.21.15.72:0/3801761679 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9aec0cdcb0 0x7f9aec15f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.590+0000 7f9aeb7fe640 1 -- 172.21.15.72:0/3801761679 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9aec0cdcb0 msgr2=0x7f9aec15f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.590+0000 7f9aeb7fe640 1 --2- 172.21.15.72:0/3801761679 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9aec0cdcb0 0x7f9aec15f9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.590+0000 7f9aeb7fe640 1 -- 172.21.15.72:0/3801761679 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9aec1f7040 msgr2=0x7f9aec0cd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.590+0000 7f9aeb7fe640 1 --2- 172.21.15.72:0/3801761679 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9aec1f7040 0x7f9aec0cd770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.590+0000 7f9aeb7fe640 1 -- 172.21.15.72:0/3801761679 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aec0cb830 con 0x7f9ad40039b0 2024-04-15T15:16:23.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.590+0000 7f9aeb7fe640 1 --2- 172.21.15.72:0/3801761679 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ad40039b0 0x7f9aec1f5310 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f9adc037040 tx=0x7f9adc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:23.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.590+0000 7f9ad3fff640 1 -- 172.21.15.72:0/3801761679 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9adc042020 con 0x7f9ad40039b0 2024-04-15T15:16:23.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.590+0000 7f9ad3fff640 1 -- 172.21.15.72:0/3801761679 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9adc03a040 con 0x7f9ad40039b0 2024-04-15T15:16:23.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.590+0000 7f9af1fd4640 1 -- 172.21.15.72:0/3801761679 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9aec0cbb10 con 0x7f9ad40039b0 2024-04-15T15:16:23.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.590+0000 7f9ad3fff640 1 -- 172.21.15.72:0/3801761679 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9adc039070 con 0x7f9ad40039b0 2024-04-15T15:16:23.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.590+0000 7f9af1fd4640 1 -- 172.21.15.72:0/3801761679 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9aec1f74c0 con 0x7f9ad40039b0 2024-04-15T15:16:23.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.591+0000 7f9af1fd4640 1 -- 172.21.15.72:0/3801761679 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9aec075c50 con 0x7f9ad40039b0 2024-04-15T15:16:23.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.592+0000 7f9ad3fff640 1 -- 172.21.15.72:0/3801761679 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f9adc045020 con 0x7f9ad40039b0 2024-04-15T15:16:23.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.592+0000 7f9ad3fff640 1 --2- 172.21.15.72:0/3801761679 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f9ad806c980 0x7f9ad806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.592+0000 7f9ad3fff640 1 -- 172.21.15.72:0/3801761679 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f9adc0ad480 con 0x7f9ad40039b0 2024-04-15T15:16:23.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.592+0000 7f9aeaffd640 1 --2- 172.21.15.72:0/3801761679 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f9ad806c980 0x7f9ad806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.593+0000 7f9aeaffd640 1 --2- 172.21.15.72:0/3801761679 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f9ad806c980 0x7f9ad806ee40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f9aec0a24d0 tx=0x7f9ae0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:23.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.596+0000 7f9ad3fff640 1 -- 172.21.15.72:0/3801761679 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9adc04c030 con 0x7f9ad40039b0 2024-04-15T15:16:23.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.775+0000 7f9af1fd4640 1 -- 172.21.15.72:0/3801761679 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9aec0ce090 con 0x7f9ad40039b0 2024-04-15T15:16:23.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.776+0000 7f9ad3fff640 1 -- 172.21.15.72:0/3801761679 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v38) ==== 104+0+11 (secure 0 0 0) 0x7f9adc07b730 con 0x7f9ad40039b0 2024-04-15T15:16:23.777 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:16:23.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.778+0000 7f9af1fd4640 1 -- 172.21.15.72:0/3801761679 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f9ad806c980 msgr2=0x7f9ad806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.778+0000 7f9af1fd4640 1 --2- 172.21.15.72:0/3801761679 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f9ad806c980 0x7f9ad806ee40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f9aec0a24d0 tx=0x7f9ae0009040 comp rx=0 tx=0).stop 2024-04-15T15:16:23.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.778+0000 7f9af1fd4640 1 -- 172.21.15.72:0/3801761679 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ad40039b0 msgr2=0x7f9aec1f5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.778+0000 7f9af1fd4640 1 --2- 172.21.15.72:0/3801761679 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ad40039b0 0x7f9aec1f5310 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f9adc037040 tx=0x7f9adc004730 comp rx=0 tx=0).stop 2024-04-15T15:16:23.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.778+0000 7f9af1fd4640 1 -- 172.21.15.72:0/3801761679 shutdown_connections 2024-04-15T15:16:23.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.778+0000 7f9af1fd4640 1 --2- 172.21.15.72:0/3801761679 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f9ad806c980 0x7f9ad806ee40 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.778+0000 7f9af1fd4640 1 --2- 172.21.15.72:0/3801761679 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9aec0cdcb0 0x7f9aec15f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.778+0000 7f9af1fd4640 1 --2- 172.21.15.72:0/3801761679 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9aec1f7040 0x7f9aec0cd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.778+0000 7f9af1fd4640 1 --2- 172.21.15.72:0/3801761679 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9ad40039b0 0x7f9aec1f5310 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.778+0000 7f9af1fd4640 1 -- 172.21.15.72:0/3801761679 >> 172.21.15.72:0/3801761679 conn(0x7f9aec0b9580 msgr2=0x7f9aec0b53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:23.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.778+0000 7f9af1fd4640 1 -- 172.21.15.72:0/3801761679 shutdown_connections 2024-04-15T15:16:23.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.778+0000 7f9af1fd4640 1 -- 172.21.15.72:0/3801761679 wait complete. 2024-04-15T15:16:23.790 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:16:23.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.895+0000 7f51653a1640 1 Processor -- start 2024-04-15T15:16:23.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.895+0000 7f51653a1640 1 -- start start 2024-04-15T15:16:23.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.896+0000 7f51653a1640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f51600cf890 0x7f51600d4370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.896+0000 7f51653a1640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f51600d48b0 0x7f51600d1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.896+0000 7f51653a1640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51600cfd80 con 0x7f516015e920 2024-04-15T15:16:23.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.896+0000 7f51653a1640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f51600d4d20 con 0x7f51600d48b0 2024-04-15T15:16:23.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.896+0000 7f51653a1640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f51600d0040 con 0x7f51600cf890 2024-04-15T15:16:23.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.896+0000 7f515ffff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f516015e920 0x7f51600b34d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60258/0 (socket says 172.21.15.72:60258) 2024-04-15T15:16:23.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.896+0000 7f515ffff640 1 -- 172.21.15.72:0/624378817 learned_addr learned my addr 172.21.15.72:0/624378817 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:23.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.896+0000 7f515effd640 1 -- 172.21.15.72:0/624378817 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2010924355 0 0) 0x7f51600cfd80 con 0x7f516015e920 2024-04-15T15:16:23.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.896+0000 7f515effd640 1 -- 172.21.15.72:0/624378817 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5148003520 con 0x7f516015e920 2024-04-15T15:16:23.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.896+0000 7f5164ba0640 1 --2- 172.21.15.72:0/624378817 >> v2:172.21.15.175:3300/0 conn(0x7f51600d48b0 0x7f51600d1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.896+0000 7f515f7fe640 1 --2- 172.21.15.72:0/624378817 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f51600cf890 0x7f51600d4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.896+0000 7f515effd640 1 -- 172.21.15.72:0/624378817 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4128073749 0 0) 0x7f5148003520 con 0x7f516015e920 2024-04-15T15:16:23.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.896+0000 7f515effd640 1 -- 172.21.15.72:0/624378817 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f51600cf890 msgr2=0x7f51600d4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:23.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.896+0000 7f515effd640 1 --2- 172.21.15.72:0/624378817 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f51600cf890 0x7f51600d4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.896+0000 7f515effd640 1 -- 172.21.15.72:0/624378817 >> v2:172.21.15.175:3300/0 conn(0x7f51600d48b0 msgr2=0x7f51600d1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:23.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.896+0000 7f515effd640 1 --2- 172.21.15.72:0/624378817 >> v2:172.21.15.175:3300/0 conn(0x7f51600d48b0 0x7f51600d1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.896+0000 7f515effd640 1 -- 172.21.15.72:0/624378817 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51600d0320 con 0x7f516015e920 2024-04-15T15:16:23.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.896+0000 7f515effd640 1 -- 172.21.15.72:0/624378817 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f5150001690 con 0x7f516015e920 2024-04-15T15:16:23.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.896+0000 7f515f7fe640 1 --2- 172.21.15.72:0/624378817 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f51600cf890 0x7f51600d4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:23.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.896+0000 7f515effd640 1 -- 172.21.15.72:0/624378817 >> v1:172.21.15.72:6789/0 conn(0x7f516015e920 legacy=0x7f51600b34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.896+0000 7f515effd640 1 --2- 172.21.15.72:0/624378817 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f51600cf890 0x7f51480039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.896+0000 7f515effd640 1 --2- 172.21.15.72:0/624378817 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5148003ef0 0x7f5148004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.896+0000 7f515effd640 1 --2- 172.21.15.72:0/624378817 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5148004d40 0x7f51480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.897+0000 7f515effd640 1 -- 172.21.15.72:0/624378817 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f51600d4d20 con 0x7f5148004d40 2024-04-15T15:16:23.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.897+0000 7f515effd640 1 -- 172.21.15.72:0/624378817 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f51600d0040 con 0x7f5148003ef0 2024-04-15T15:16:23.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.897+0000 7f515effd640 1 -- 172.21.15.72:0/624378817 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f514800ba70 con 0x7f51600cf890 2024-04-15T15:16:23.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.897+0000 7f5164ba0640 1 --2- 172.21.15.72:0/624378817 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5148004d40 0x7f51480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.897+0000 7f515ffff640 1 --2- 172.21.15.72:0/624378817 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f51600cf890 0x7f51480039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.897+0000 7f515f7fe640 1 --2- 172.21.15.72:0/624378817 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5148003ef0 0x7f5148004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.897+0000 7f5164ba0640 1 -- 172.21.15.72:0/624378817 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f51600cf890 msgr2=0x7f51480039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:23.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.897+0000 7f5164ba0640 1 --2- 172.21.15.72:0/624378817 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f51600cf890 0x7f51480039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.897+0000 7f5164ba0640 1 -- 172.21.15.72:0/624378817 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5148003ef0 msgr2=0x7f5148004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:23.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.897+0000 7f5164ba0640 1 --2- 172.21.15.72:0/624378817 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5148003ef0 0x7f5148004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.897+0000 7f5164ba0640 1 -- 172.21.15.72:0/624378817 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5148003520 con 0x7f5148004d40 2024-04-15T15:16:23.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.898+0000 7f5164ba0640 1 --2- 172.21.15.72:0/624378817 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5148004d40 0x7f51480051c0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f5158009fc0 tx=0x7f5148005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4a765f841951e5d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:23.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.898+0000 7f515effd640 1 -- 172.21.15.72:0/624378817 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5158039070 con 0x7f5148004d40 2024-04-15T15:16:23.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.898+0000 7f515effd640 1 -- 172.21.15.72:0/624378817 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f515802cc60 con 0x7f5148004d40 2024-04-15T15:16:23.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.898+0000 7f515effd640 1 -- 172.21.15.72:0/624378817 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5158009c10 con 0x7f5148004d40 2024-04-15T15:16:23.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.898+0000 7f51653a1640 1 -- 172.21.15.72:0/624378817 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5148004d40 msgr2=0x7f51480051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.898+0000 7f51653a1640 1 --2- 172.21.15.72:0/624378817 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5148004d40 0x7f51480051c0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f5158009fc0 tx=0x7f5148005830 comp rx=0 tx=0).stop 2024-04-15T15:16:23.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.898+0000 7f5164ba0640 1 -- 172.21.15.72:0/624378817 reap_dead start 2024-04-15T15:16:23.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.898+0000 7f51653a1640 1 -- 172.21.15.72:0/624378817 shutdown_connections 2024-04-15T15:16:23.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.898+0000 7f51653a1640 1 -- 172.21.15.72:0/624378817 >> 172.21.15.72:0/624378817 conn(0x7f51600b9c90 msgr2=0x7f516009d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:23.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.898+0000 7f51653a1640 1 -- 172.21.15.72:0/624378817 shutdown_connections 2024-04-15T15:16:23.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.898+0000 7f51653a1640 1 -- 172.21.15.72:0/624378817 wait complete. 2024-04-15T15:16:23.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.900+0000 7f51653a1640 1 Processor -- start 2024-04-15T15:16:23.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.900+0000 7f51653a1640 1 -- start start 2024-04-15T15:16:23.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.900+0000 7f51653a1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5148004d40 0x7f516006d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.901+0000 7f51653a1640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f51600b63c0 0x7f51600b6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.901+0000 7f51653a1640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f516014a3d0 0x7f516014a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.901+0000 7f51653a1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f51600b3be0 con 0x7f5148004d40 2024-04-15T15:16:23.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.901+0000 7f51653a1640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f51600d08e0 con 0x7f516014a3d0 2024-04-15T15:16:23.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.901+0000 7f51653a1640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f51600d4370 con 0x7f51600b63c0 2024-04-15T15:16:23.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.901+0000 7f515ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5148004d40 0x7f516006d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.901+0000 7f515ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5148004d40 0x7f516006d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43004/0 (socket says 172.21.15.72:43004) 2024-04-15T15:16:23.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.901+0000 7f515ffff640 1 -- 172.21.15.72:0/1459919620 learned_addr learned my addr 172.21.15.72:0/1459919620 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:23.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.901+0000 7f5164ba0640 1 --2- 172.21.15.72:0/1459919620 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f516014a3d0 0x7f516014a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.901+0000 7f515f7fe640 1 --2- 172.21.15.72:0/1459919620 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f51600b63c0 0x7f51600b6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.901+0000 7f515ffff640 1 -- 172.21.15.72:0/1459919620 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f51600b63c0 msgr2=0x7f51600b6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.901+0000 7f515ffff640 1 --2- 172.21.15.72:0/1459919620 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f51600b63c0 0x7f51600b6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.901+0000 7f515ffff640 1 -- 172.21.15.72:0/1459919620 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f516014a3d0 msgr2=0x7f516014a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.901+0000 7f515ffff640 1 --2- 172.21.15.72:0/1459919620 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f516014a3d0 0x7f516014a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.901+0000 7f515ffff640 1 -- 172.21.15.72:0/1459919620 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51600c9820 con 0x7f5148004d40 2024-04-15T15:16:23.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.901+0000 7f5164ba0640 1 --2- 172.21.15.72:0/1459919620 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f516014a3d0 0x7f516014a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:23.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.901+0000 7f515ffff640 1 --2- 172.21.15.72:0/1459919620 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5148004d40 0x7f516006d2e0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f5150001800 tx=0x7f51500077a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:23.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.901+0000 7f515cff9640 1 -- 172.21.15.72:0/1459919620 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f515000f9c0 con 0x7f5148004d40 2024-04-15T15:16:23.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.901+0000 7f515cff9640 1 -- 172.21.15.72:0/1459919620 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5150018070 con 0x7f5148004d40 2024-04-15T15:16:23.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.901+0000 7f51653a1640 1 -- 172.21.15.72:0/1459919620 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51600c8f80 con 0x7f5148004d40 2024-04-15T15:16:23.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.901+0000 7f515cff9640 1 -- 172.21.15.72:0/1459919620 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f515000fda0 con 0x7f5148004d40 2024-04-15T15:16:23.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.901+0000 7f51653a1640 1 -- 172.21.15.72:0/1459919620 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51600c9540 con 0x7f5148004d40 2024-04-15T15:16:23.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.902+0000 7f51653a1640 1 -- 172.21.15.72:0/1459919620 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f516009dc90 con 0x7f5148004d40 2024-04-15T15:16:23.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.903+0000 7f515cff9640 1 -- 172.21.15.72:0/1459919620 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f515001c410 con 0x7f5148004d40 2024-04-15T15:16:23.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.903+0000 7f515cff9640 1 --2- 172.21.15.72:0/1459919620 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f514c06c9b0 0x7f514c06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.903+0000 7f515f7fe640 1 --2- 172.21.15.72:0/1459919620 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f514c06c9b0 0x7f514c06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.903+0000 7f515cff9640 1 -- 172.21.15.72:0/1459919620 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f515008baf0 con 0x7f5148004d40 2024-04-15T15:16:23.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.904+0000 7f515f7fe640 1 --2- 172.21.15.72:0/1459919620 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f514c06c9b0 0x7f514c06ee70 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f516009d9e0 tx=0x7f5154008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:23.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:23.907+0000 7f515cff9640 1 -- 172.21.15.72:0/1459919620 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f516009dc90 con 0x7f5148004d40 2024-04-15T15:16:24.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.082+0000 7f51653a1640 1 -- 172.21.15.72:0/1459919620 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5160075aa0 con 0x7f5148004d40 2024-04-15T15:16:24.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.082+0000 7f515cff9640 1 -- 172.21.15.72:0/1459919620 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v38) ==== 100+0+11 (secure 0 0 0) 0x7f5150059e30 con 0x7f5148004d40 2024-04-15T15:16:24.084 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-04-15T15:16:24.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.084+0000 7f51653a1640 1 -- 172.21.15.72:0/1459919620 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f514c06c9b0 msgr2=0x7f514c06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.084+0000 7f51653a1640 1 --2- 172.21.15.72:0/1459919620 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f514c06c9b0 0x7f514c06ee70 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f516009d9e0 tx=0x7f5154008040 comp rx=0 tx=0).stop 2024-04-15T15:16:24.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.084+0000 7f51653a1640 1 -- 172.21.15.72:0/1459919620 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5148004d40 msgr2=0x7f516006d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.084+0000 7f51653a1640 1 --2- 172.21.15.72:0/1459919620 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5148004d40 0x7f516006d2e0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f5150001800 tx=0x7f51500077a0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.084+0000 7f51653a1640 1 -- 172.21.15.72:0/1459919620 shutdown_connections 2024-04-15T15:16:24.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.084+0000 7f51653a1640 1 --2- 172.21.15.72:0/1459919620 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f514c06c9b0 0x7f514c06ee70 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.084+0000 7f51653a1640 1 --2- 172.21.15.72:0/1459919620 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f516014a3d0 0x7f516014a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.084+0000 7f51653a1640 1 --2- 172.21.15.72:0/1459919620 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f51600b63c0 0x7f51600b6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.084+0000 7f51653a1640 1 --2- 172.21.15.72:0/1459919620 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5148004d40 0x7f516006d2e0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.084+0000 7f51653a1640 1 -- 172.21.15.72:0/1459919620 >> 172.21.15.72:0/1459919620 conn(0x7f51600b9c90 msgr2=0x7f51600b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:24.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.084+0000 7f51653a1640 1 -- 172.21.15.72:0/1459919620 shutdown_connections 2024-04-15T15:16:24.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.084+0000 7f51653a1640 1 -- 172.21.15.72:0/1459919620 wait complete. 2024-04-15T15:16:24.097 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:16:24.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.198+0000 7fe27b088640 1 Processor -- start 2024-04-15T15:16:24.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.199+0000 7fe27b088640 1 -- start start 2024-04-15T15:16:24.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.199+0000 7fe27b088640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe2740b0050 0x7fe2740cf960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.199+0000 7fe27b088640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe2740afae0 0x7fe27406c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.199+0000 7fe27b088640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2740d07c0 con 0x7fe2740b5410 2024-04-15T15:16:24.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.199+0000 7fe27b088640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fe2740d0330 con 0x7fe2740afae0 2024-04-15T15:16:24.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.199+0000 7fe27b088640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fe2740b3aa0 con 0x7fe2740b0050 2024-04-15T15:16:24.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.200+0000 7fe273fff640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe2740b0050 0x7fe2740cf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.200+0000 7fe279625640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe2740afae0 0x7fe27406c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.200+0000 7fe278e24640 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fe2740b5410 0x7fe2740b57f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.72:53860/0 (socket says 172.21.15.72:53860) 2024-04-15T15:16:24.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.200+0000 7fe278e24640 1 -- 172.21.15.72:0/3481245662 learned_addr learned my addr 172.21.15.72:0/3481245662 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:24.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.200+0000 7fe279625640 1 -- 172.21.15.72:0/3481245662 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe2740b0050 msgr2=0x7fe2740cf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:24.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.200+0000 7fe279625640 1 --2- 172.21.15.72:0/3481245662 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe2740b0050 0x7fe2740cf960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.200+0000 7fe279625640 1 -- 172.21.15.72:0/3481245662 >> v1:172.21.15.175:6789/0 conn(0x7fe2740b5410 legacy=0x7fe2740b57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.200+0000 7fe279625640 1 -- 172.21.15.72:0/3481245662 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2740b3d80 con 0x7fe2740afae0 2024-04-15T15:16:24.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.200+0000 7fe273fff640 1 --2- 172.21.15.72:0/3481245662 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe2740b0050 0x7fe2740cf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:24.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.201+0000 7fe279625640 1 --2- 172.21.15.72:0/3481245662 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe2740afae0 0x7fe27406c340 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe2740d04b0 tx=0x7fe26c02be00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5d81fda28c9d2f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:24.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.201+0000 7fe2737fe640 1 -- 172.21.15.72:0/3481245662 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe26c039070 con 0x7fe2740afae0 2024-04-15T15:16:24.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.201+0000 7fe2737fe640 1 -- 172.21.15.72:0/3481245662 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe26c02cbb0 con 0x7fe2740afae0 2024-04-15T15:16:24.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.201+0000 7fe2737fe640 1 -- 172.21.15.72:0/3481245662 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe26c034440 con 0x7fe2740afae0 2024-04-15T15:16:24.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.202+0000 7fe27b088640 1 -- 172.21.15.72:0/3481245662 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe2740afae0 msgr2=0x7fe27406c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.202+0000 7fe27b088640 1 --2- 172.21.15.72:0/3481245662 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe2740afae0 0x7fe27406c340 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe2740d04b0 tx=0x7fe26c02be00 comp rx=0 tx=0).stop 2024-04-15T15:16:24.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.202+0000 7fe27b088640 1 -- 172.21.15.72:0/3481245662 shutdown_connections 2024-04-15T15:16:24.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.202+0000 7fe27b088640 1 --2- 172.21.15.72:0/3481245662 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe2740afae0 0x7fe27406c340 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.202+0000 7fe27b088640 1 --2- 172.21.15.72:0/3481245662 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe2740b0050 0x7fe2740cf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.202+0000 7fe27b088640 1 -- 172.21.15.72:0/3481245662 >> 172.21.15.72:0/3481245662 conn(0x7fe2740d0a50 msgr2=0x7fe2740d0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:24.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.202+0000 7fe27b088640 1 -- 172.21.15.72:0/3481245662 shutdown_connections 2024-04-15T15:16:24.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.202+0000 7fe27b088640 1 -- 172.21.15.72:0/3481245662 wait complete. 2024-04-15T15:16:24.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.203+0000 7fe27b088640 1 Processor -- start 2024-04-15T15:16:24.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.203+0000 7fe27b088640 1 -- start start 2024-04-15T15:16:24.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.204+0000 7fe27b088640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe2740afae0 0x7fe27415bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.204+0000 7fe27b088640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe2740b0050 0x7fe2740b17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.204+0000 7fe27b088640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2740b5410 0x7fe274156d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.204+0000 7fe27b088640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe2740cfea0 con 0x7fe2740b5410 2024-04-15T15:16:24.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.204+0000 7fe27b088640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fe2740d18f0 con 0x7fe2740afae0 2024-04-15T15:16:24.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.204+0000 7fe27b088640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fe27406db40 con 0x7fe2740b0050 2024-04-15T15:16:24.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.204+0000 7fe279625640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2740b5410 0x7fe274156d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.204+0000 7fe279625640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2740b5410 0x7fe274156d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43010/0 (socket says 172.21.15.72:43010) 2024-04-15T15:16:24.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.204+0000 7fe279625640 1 -- 172.21.15.72:0/1688640462 learned_addr learned my addr 172.21.15.72:0/1688640462 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:24.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.204+0000 7fe273fff640 1 --2- 172.21.15.72:0/1688640462 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe2740b0050 0x7fe2740b17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.204+0000 7fe278e24640 1 --2- 172.21.15.72:0/1688640462 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe2740afae0 0x7fe27415bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.204+0000 7fe279625640 1 -- 172.21.15.72:0/1688640462 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe2740b0050 msgr2=0x7fe2740b17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.204+0000 7fe279625640 1 --2- 172.21.15.72:0/1688640462 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe2740b0050 0x7fe2740b17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.204+0000 7fe279625640 1 -- 172.21.15.72:0/1688640462 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe2740afae0 msgr2=0x7fe27415bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.204+0000 7fe279625640 1 --2- 172.21.15.72:0/1688640462 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe2740afae0 0x7fe27415bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.204+0000 7fe279625640 1 -- 172.21.15.72:0/1688640462 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2741f59e0 con 0x7fe2740b5410 2024-04-15T15:16:24.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.205+0000 7fe279625640 1 --2- 172.21.15.72:0/1688640462 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2740b5410 0x7fe274156d80 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fe26c009e80 tx=0x7fe26c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:24.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.205+0000 7fe2717fa640 1 -- 172.21.15.72:0/1688640462 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe26c039050 con 0x7fe2740b5410 2024-04-15T15:16:24.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.205+0000 7fe2717fa640 1 -- 172.21.15.72:0/1688640462 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe26c045070 con 0x7fe2740b5410 2024-04-15T15:16:24.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.205+0000 7fe2717fa640 1 -- 172.21.15.72:0/1688640462 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe26c03e830 con 0x7fe2740b5410 2024-04-15T15:16:24.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.205+0000 7fe27b088640 1 -- 172.21.15.72:0/1688640462 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2741f4d60 con 0x7fe2740b5410 2024-04-15T15:16:24.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.205+0000 7fe27b088640 1 -- 172.21.15.72:0/1688640462 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2741f5320 con 0x7fe2740b5410 2024-04-15T15:16:24.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.206+0000 7fe27b088640 1 -- 172.21.15.72:0/1688640462 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe274075c50 con 0x7fe2740b5410 2024-04-15T15:16:24.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.206+0000 7fe2717fa640 1 -- 172.21.15.72:0/1688640462 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fe26c03e9d0 con 0x7fe2740b5410 2024-04-15T15:16:24.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.207+0000 7fe2717fa640 1 --2- 172.21.15.72:0/1688640462 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe25806c980 0x7fe25806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.207+0000 7fe2717fa640 1 -- 172.21.15.72:0/1688640462 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fe26c0ae950 con 0x7fe2740b5410 2024-04-15T15:16:24.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.207+0000 7fe278e24640 1 --2- 172.21.15.72:0/1688640462 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe25806c980 0x7fe25806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.210 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.207+0000 7fe278e24640 1 --2- 172.21.15.72:0/1688640462 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe25806c980 0x7fe25806ee40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe268007fd0 tx=0x7fe268007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:24.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.211+0000 7fe2717fa640 1 -- 172.21.15.72:0/1688640462 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe26c07c760 con 0x7fe2740b5410 2024-04-15T15:16:24.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.397+0000 7fe27b088640 1 -- 172.21.15.72:0/1688640462 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe27406d610 con 0x7fe2740b5410 2024-04-15T15:16:24.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.398+0000 7fe2717fa640 1 -- 172.21.15.72:0/1688640462 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v38) ==== 80+0+0 (secure 0 0 0) 0x7fe26c0ace20 con 0x7fe2740b5410 2024-04-15T15:16:24.399 INFO:teuthology.orchestra.run.smithi072.stderr:listed 0 entries 2024-04-15T15:16:24.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.399+0000 7fe27b088640 1 -- 172.21.15.72:0/1688640462 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe25806c980 msgr2=0x7fe25806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.400+0000 7fe27b088640 1 --2- 172.21.15.72:0/1688640462 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe25806c980 0x7fe25806ee40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe268007fd0 tx=0x7fe268007e80 comp rx=0 tx=0).stop 2024-04-15T15:16:24.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.400+0000 7fe27b088640 1 -- 172.21.15.72:0/1688640462 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2740b5410 msgr2=0x7fe274156d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.400+0000 7fe27b088640 1 --2- 172.21.15.72:0/1688640462 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2740b5410 0x7fe274156d80 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fe26c009e80 tx=0x7fe26c004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:24.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.400+0000 7fe27b088640 1 -- 172.21.15.72:0/1688640462 shutdown_connections 2024-04-15T15:16:24.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.400+0000 7fe27b088640 1 --2- 172.21.15.72:0/1688640462 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe25806c980 0x7fe25806ee40 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.400+0000 7fe27b088640 1 --2- 172.21.15.72:0/1688640462 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe2740b5410 0x7fe274156d80 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.400+0000 7fe27b088640 1 --2- 172.21.15.72:0/1688640462 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe2740b0050 0x7fe2740b17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.400+0000 7fe27b088640 1 --2- 172.21.15.72:0/1688640462 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe2740afae0 0x7fe27415bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.400+0000 7fe27b088640 1 -- 172.21.15.72:0/1688640462 >> 172.21.15.72:0/1688640462 conn(0x7fe2740d0a50 msgr2=0x7fe2740ccc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:24.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.400+0000 7fe27b088640 1 -- 172.21.15.72:0/1688640462 shutdown_connections 2024-04-15T15:16:24.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.400+0000 7fe27b088640 1 -- 172.21.15.72:0/1688640462 wait complete. 2024-04-15T15:16:24.412 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi072.front.sepia.ceph.com... 2024-04-15T15:16:24.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:24.412 DEBUG:teuthology.orchestra.run.smithi072:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-15T15:16:24.440 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[54493]: fuse finished with error 0 and tester_r 0 2024-04-15T15:16:24.441 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:16:24.491 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi072.front.sepia.ceph.com 2024-04-15T15:16:24.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:24.491 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-15T15:16:24.507 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:16:24.507 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:16:24.527 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:24.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.648+0000 7f4e0a367640 1 Processor -- start 2024-04-15T15:16:24.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.648+0000 7f4e0a367640 1 -- start start 2024-04-15T15:16:24.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.649+0000 7f4e0a367640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4e04161fb0 0x7f4e04162450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.649+0000 7f4e0a367640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4e040c9570 0x7f4e04162990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.649+0000 7f4e0a367640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e040ccd40 con 0x7f4e040d12f0 2024-04-15T15:16:24.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.649+0000 7f4e0a367640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f4e040ccaf0 con 0x7f4e040c9570 2024-04-15T15:16:24.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.649+0000 7f4e0a367640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f4e040cd020 con 0x7f4e04161fb0 2024-04-15T15:16:24.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.649+0000 7f4e08904640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4e040c9570 0x7f4e04162990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.649+0000 7f4e08904640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f4e040c9570 0x7f4e04162990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43024/0 (socket says 172.21.15.72:43024) 2024-04-15T15:16:24.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.649+0000 7f4e08904640 1 -- 172.21.15.72:0/1390201054 learned_addr learned my addr 172.21.15.72:0/1390201054 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:24.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.649+0000 7f4e037fe640 1 --2- 172.21.15.72:0/1390201054 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4e04161fb0 0x7f4e04162450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.649+0000 7f4e08904640 1 -- 172.21.15.72:0/1390201054 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4e04161fb0 msgr2=0x7f4e04162450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:24.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.649+0000 7f4e08904640 1 --2- 172.21.15.72:0/1390201054 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4e04161fb0 0x7f4e04162450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.649+0000 7f4e08904640 1 -- 172.21.15.72:0/1390201054 >> v1:172.21.15.175:6789/0 conn(0x7f4e040d12f0 legacy=0x7f4e041618a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.650+0000 7f4e08904640 1 -- 172.21.15.72:0/1390201054 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e040cd300 con 0x7f4e040c9570 2024-04-15T15:16:24.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.650+0000 7f4e08904640 1 --2- 172.21.15.72:0/1390201054 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e040c9570 0x7f4e04162990 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc009e50 tx=0x7f4dfc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d01b8df3c19e4a7d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:24.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.650+0000 7f4e02ffd640 1 -- 172.21.15.72:0/1390201054 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc02cb00 con 0x7f4e040c9570 2024-04-15T15:16:24.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.650+0000 7f4e02ffd640 1 -- 172.21.15.72:0/1390201054 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dfc02cca0 con 0x7f4e040c9570 2024-04-15T15:16:24.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.650+0000 7f4e02ffd640 1 -- 172.21.15.72:0/1390201054 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dfc034610 con 0x7f4e040c9570 2024-04-15T15:16:24.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.651+0000 7f4e0a367640 1 -- 172.21.15.72:0/1390201054 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e040c9570 msgr2=0x7f4e04162990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.651+0000 7f4e0a367640 1 --2- 172.21.15.72:0/1390201054 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e040c9570 0x7f4e04162990 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc009e50 tx=0x7f4dfc02bf60 comp rx=0 tx=0).stop 2024-04-15T15:16:24.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.651+0000 7f4e0a367640 1 -- 172.21.15.72:0/1390201054 shutdown_connections 2024-04-15T15:16:24.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.651+0000 7f4e0a367640 1 --2- 172.21.15.72:0/1390201054 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e040c9570 0x7f4e04162990 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.651+0000 7f4e0a367640 1 --2- 172.21.15.72:0/1390201054 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4e04161fb0 0x7f4e04162450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.651+0000 7f4e0a367640 1 -- 172.21.15.72:0/1390201054 >> 172.21.15.72:0/1390201054 conn(0x7f4e040b3f90 msgr2=0x7f4e040b43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:24.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.651+0000 7f4e0a367640 1 -- 172.21.15.72:0/1390201054 shutdown_connections 2024-04-15T15:16:24.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.651+0000 7f4e0a367640 1 -- 172.21.15.72:0/1390201054 wait complete. 2024-04-15T15:16:24.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.652+0000 7f4e0a367640 1 Processor -- start 2024-04-15T15:16:24.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.652+0000 7f4e0a367640 1 -- start start 2024-04-15T15:16:24.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.653+0000 7f4e0a367640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e040c9570 0x7f4e04147050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.653+0000 7f4e0a367640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4e040d12f0 0x7f4e040c7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.653+0000 7f4e0a367640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4e04161fb0 0x7f4e040c83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.653+0000 7f4e0a367640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f4e040d2210 con 0x7f4e040c9570 2024-04-15T15:16:24.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.653+0000 7f4e0a367640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f4e040d2090 con 0x7f4e040d12f0 2024-04-15T15:16:24.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.653+0000 7f4e0a367640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f4e040b68e0 con 0x7f4e04161fb0 2024-04-15T15:16:24.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.653+0000 7f4e03fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e040c9570 0x7f4e04147050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.653+0000 7f4e03fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e040c9570 0x7f4e04147050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43026/0 (socket says 172.21.15.72:43026) 2024-04-15T15:16:24.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.653+0000 7f4e03fff640 1 -- 172.21.15.72:0/2278677288 learned_addr learned my addr 172.21.15.72:0/2278677288 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:24.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.653+0000 7f4e037fe640 1 --2- 172.21.15.72:0/2278677288 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4e040d12f0 0x7f4e040c7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.653+0000 7f4e08904640 1 --2- 172.21.15.72:0/2278677288 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4e04161fb0 0x7f4e040c83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.653+0000 7f4e03fff640 1 -- 172.21.15.72:0/2278677288 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4e04161fb0 msgr2=0x7f4e040c83e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.653+0000 7f4e03fff640 1 --2- 172.21.15.72:0/2278677288 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4e04161fb0 0x7f4e040c83e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.653+0000 7f4e03fff640 1 -- 172.21.15.72:0/2278677288 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4e040d12f0 msgr2=0x7f4e040c7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.653+0000 7f4e03fff640 1 --2- 172.21.15.72:0/2278677288 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4e040d12f0 0x7f4e040c7ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.653+0000 7f4e03fff640 1 -- 172.21.15.72:0/2278677288 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e040c9300 con 0x7f4e040c9570 2024-04-15T15:16:24.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.654+0000 7f4e08904640 1 --2- 172.21.15.72:0/2278677288 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4e04161fb0 0x7f4e040c83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:24.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.654+0000 7f4e03fff640 1 --2- 172.21.15.72:0/2278677288 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e040c9570 0x7f4e04147050 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f4df400c400 tx=0x7f4df400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:24.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.654+0000 7f4e00ff9640 1 -- 172.21.15.72:0/2278677288 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df40052d0 con 0x7f4e040c9570 2024-04-15T15:16:24.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.654+0000 7f4e00ff9640 1 -- 172.21.15.72:0/2278677288 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4df4005490 con 0x7f4e040c9570 2024-04-15T15:16:24.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.654+0000 7f4e0a367640 1 -- 172.21.15.72:0/2278677288 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e040c8a60 con 0x7f4e040c9570 2024-04-15T15:16:24.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.654+0000 7f4e00ff9640 1 -- 172.21.15.72:0/2278677288 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df4007260 con 0x7f4e040c9570 2024-04-15T15:16:24.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.654+0000 7f4e0a367640 1 -- 172.21.15.72:0/2278677288 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e040c9020 con 0x7f4e040c9570 2024-04-15T15:16:24.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.655+0000 7f4e0a367640 1 -- 172.21.15.72:0/2278677288 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e040689e0 con 0x7f4e040c9570 2024-04-15T15:16:24.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.655+0000 7f4e00ff9640 1 -- 172.21.15.72:0/2278677288 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f4df4007400 con 0x7f4e040c9570 2024-04-15T15:16:24.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.656+0000 7f4e00ff9640 1 --2- 172.21.15.72:0/2278677288 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4df006c980 0x7f4df006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.656+0000 7f4e00ff9640 1 -- 172.21.15.72:0/2278677288 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f4df408b2a0 con 0x7f4e040c9570 2024-04-15T15:16:24.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.656+0000 7f4e037fe640 1 --2- 172.21.15.72:0/2278677288 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4df006c980 0x7f4df006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.656+0000 7f4e037fe640 1 --2- 172.21.15.72:0/2278677288 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4df006c980 0x7f4df006ee40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f4df8006fd0 tx=0x7f4df8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:24.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.660+0000 7f4e00ff9640 1 -- 172.21.15.72:0/2278677288 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4df4059550 con 0x7f4e040c9570 2024-04-15T15:16:24.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.840+0000 7f4e0a367640 1 -- 172.21.15.72:0/2278677288 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4e040cb740 con 0x7f4e040c9570 2024-04-15T15:16:24.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.841+0000 7f4e00ff9640 1 -- 172.21.15.72:0/2278677288 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+16942 (secure 0 0 0) 0x7f4df4058e50 con 0x7f4e040c9570 2024-04-15T15:16:24.843 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:16:24.843 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":38,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:16:08.702307+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:05.551499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:06.003152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:24.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.843+0000 7f4e0a367640 1 -- 172.21.15.72:0/2278677288 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4df006c980 msgr2=0x7f4df006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.843+0000 7f4e0a367640 1 --2- 172.21.15.72:0/2278677288 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4df006c980 0x7f4df006ee40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f4df8006fd0 tx=0x7f4df8008040 comp rx=0 tx=0).stop 2024-04-15T15:16:24.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.843+0000 7f4e0a367640 1 -- 172.21.15.72:0/2278677288 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e040c9570 msgr2=0x7f4e04147050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.843+0000 7f4e0a367640 1 --2- 172.21.15.72:0/2278677288 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e040c9570 0x7f4e04147050 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f4df400c400 tx=0x7f4df400c8c0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.843+0000 7f4e0a367640 1 -- 172.21.15.72:0/2278677288 shutdown_connections 2024-04-15T15:16:24.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.843+0000 7f4e0a367640 1 --2- 172.21.15.72:0/2278677288 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f4df006c980 0x7f4df006ee40 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.843+0000 7f4e0a367640 1 --2- 172.21.15.72:0/2278677288 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4e04161fb0 0x7f4e040c83e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.843+0000 7f4e0a367640 1 --2- 172.21.15.72:0/2278677288 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f4e040d12f0 0x7f4e040c7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.843+0000 7f4e0a367640 1 --2- 172.21.15.72:0/2278677288 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4e040c9570 0x7f4e04147050 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.843+0000 7f4e0a367640 1 -- 172.21.15.72:0/2278677288 >> 172.21.15.72:0/2278677288 conn(0x7f4e040b3f90 msgr2=0x7f4e040cda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:24.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.843+0000 7f4e0a367640 1 -- 172.21.15.72:0/2278677288 shutdown_connections 2024-04-15T15:16:24.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.843+0000 7f4e0a367640 1 -- 172.21.15.72:0/2278677288 wait complete. 2024-04-15T15:16:24.856 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:39.359102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:16:05.551499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:16:06.003152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:16:24.857 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:24.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.961+0000 7f2875e88640 1 Processor -- start 2024-04-15T15:16:24.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.961+0000 7f2875e88640 1 -- start start 2024-04-15T15:16:24.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.962+0000 7f2875e88640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f287015c650 0x7f28700b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.962+0000 7f2875e88640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f28700b36f0 0x7f28700b5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.962+0000 7f2875e88640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f28700b57d0 0x7f28700d0a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.962+0000 7f2875e88640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f28700d02d0 con 0x7f28700b36f0 2024-04-15T15:16:24.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.962+0000 7f2875e88640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f287015ca30 con 0x7f287015c650 2024-04-15T15:16:24.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.962+0000 7f2875e88640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f28700b26b0 con 0x7f28700b57d0 2024-04-15T15:16:24.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.962+0000 7f286effd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f28700b36f0 0x7f28700b5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.962+0000 7f286effd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f28700b36f0 0x7f28700b5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43042/0 (socket says 172.21.15.72:43042) 2024-04-15T15:16:24.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.962+0000 7f286effd640 1 -- 172.21.15.72:0/1542583635 learned_addr learned my addr 172.21.15.72:0/1542583635 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:24.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.962+0000 7f286f7fe640 1 --2- 172.21.15.72:0/1542583635 >> v2:172.21.15.175:3300/0 conn(0x7f287015c650 0x7f28700b1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.962+0000 7f286ffff640 1 --2- 172.21.15.72:0/1542583635 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f28700b57d0 0x7f28700d0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.962+0000 7f286effd640 1 -- 172.21.15.72:0/1542583635 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f28700b57d0 msgr2=0x7f28700d0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:24.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.962+0000 7f286effd640 1 --2- 172.21.15.72:0/1542583635 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f28700b57d0 0x7f28700d0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.962+0000 7f286effd640 1 -- 172.21.15.72:0/1542583635 >> v2:172.21.15.175:3300/0 conn(0x7f287015c650 msgr2=0x7f28700b1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:24.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.962+0000 7f286effd640 1 --2- 172.21.15.72:0/1542583635 >> v2:172.21.15.175:3300/0 conn(0x7f287015c650 0x7f28700b1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.962+0000 7f286effd640 1 -- 172.21.15.72:0/1542583635 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28700b2990 con 0x7f28700b36f0 2024-04-15T15:16:24.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.962+0000 7f286ffff640 1 --2- 172.21.15.72:0/1542583635 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f28700b57d0 0x7f28700d0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:24.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.963+0000 7f286effd640 1 --2- 172.21.15.72:0/1542583635 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28700b36f0 0x7f28700b5290 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f28700d3540 tx=0x7f285802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14244ffb676d2783 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:24.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.963+0000 7f286e7fc640 1 -- 172.21.15.72:0/1542583635 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2858035020 con 0x7f28700b36f0 2024-04-15T15:16:24.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.963+0000 7f286e7fc640 1 -- 172.21.15.72:0/1542583635 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2858031870 con 0x7f28700b36f0 2024-04-15T15:16:24.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.963+0000 7f286e7fc640 1 -- 172.21.15.72:0/1542583635 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2858031c40 con 0x7f28700b36f0 2024-04-15T15:16:24.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.964+0000 7f2875e88640 1 -- 172.21.15.72:0/1542583635 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28700b36f0 msgr2=0x7f28700b5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.964+0000 7f2875e88640 1 --2- 172.21.15.72:0/1542583635 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28700b36f0 0x7f28700b5290 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f28700d3540 tx=0x7f285802b910 comp rx=0 tx=0).stop 2024-04-15T15:16:24.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.964+0000 7f2875e88640 1 -- 172.21.15.72:0/1542583635 shutdown_connections 2024-04-15T15:16:24.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.964+0000 7f2875e88640 1 --2- 172.21.15.72:0/1542583635 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f28700b57d0 0x7f28700d0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.964+0000 7f2875e88640 1 --2- 172.21.15.72:0/1542583635 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f28700b36f0 0x7f28700b5290 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.964+0000 7f2875e88640 1 --2- 172.21.15.72:0/1542583635 >> v2:172.21.15.175:3300/0 conn(0x7f287015c650 0x7f28700b1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.964+0000 7f2875e88640 1 -- 172.21.15.72:0/1542583635 >> 172.21.15.72:0/1542583635 conn(0x7f28700ceec0 msgr2=0x7f28700cf2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:24.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.964+0000 7f2875e88640 1 -- 172.21.15.72:0/1542583635 shutdown_connections 2024-04-15T15:16:24.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.964+0000 7f2875e88640 1 -- 172.21.15.72:0/1542583635 wait complete. 2024-04-15T15:16:24.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.965+0000 7f2875e88640 1 Processor -- start 2024-04-15T15:16:24.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.965+0000 7f2875e88640 1 -- start start 2024-04-15T15:16:24.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.966+0000 7f2875e88640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f28700b36f0 0x7f2870152280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.966+0000 7f2875e88640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f28700b57d0 0x7f28701519b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.966+0000 7f2875e88640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f287015c650 0x7f28700cb960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.966+0000 7f2875e88640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f28700b2280 con 0x7f287015c650 2024-04-15T15:16:24.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.966+0000 7f2875e88640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f28700b2c70 con 0x7f28700b57d0 2024-04-15T15:16:24.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.966+0000 7f2875e88640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f28700b2400 con 0x7f28700b36f0 2024-04-15T15:16:24.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.966+0000 7f286ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f287015c650 0x7f28700cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.966+0000 7f286ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f287015c650 0x7f28700cb960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43054/0 (socket says 172.21.15.72:43054) 2024-04-15T15:16:24.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.966+0000 7f286ffff640 1 -- 172.21.15.72:0/2583166639 learned_addr learned my addr 172.21.15.72:0/2583166639 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:24.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.966+0000 7f286f7fe640 1 --2- 172.21.15.72:0/2583166639 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f28700b36f0 0x7f2870152280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.966+0000 7f286effd640 1 --2- 172.21.15.72:0/2583166639 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f28700b57d0 0x7f28701519b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.966+0000 7f286ffff640 1 -- 172.21.15.72:0/2583166639 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f28700b36f0 msgr2=0x7f2870152280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.966+0000 7f286ffff640 1 --2- 172.21.15.72:0/2583166639 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f28700b36f0 0x7f2870152280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.966+0000 7f286ffff640 1 -- 172.21.15.72:0/2583166639 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f28700b57d0 msgr2=0x7f28701519b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.966+0000 7f286ffff640 1 --2- 172.21.15.72:0/2583166639 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f28700b57d0 0x7f28701519b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.966+0000 7f286ffff640 1 -- 172.21.15.72:0/2583166639 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28700c6c10 con 0x7f287015c650 2024-04-15T15:16:24.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.966+0000 7f286f7fe640 1 --2- 172.21.15.72:0/2583166639 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f28700b36f0 0x7f2870152280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:24.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.966+0000 7f286effd640 1 --2- 172.21.15.72:0/2583166639 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f28700b57d0 0x7f28701519b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:24.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.966+0000 7f286ffff640 1 --2- 172.21.15.72:0/2583166639 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f287015c650 0x7f28700cb960 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f2868007e10 tx=0x7f286800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:24.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.966+0000 7f2867fff640 1 -- 172.21.15.72:0/2583166639 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f286801a070 con 0x7f287015c650 2024-04-15T15:16:24.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.966+0000 7f2867fff640 1 -- 172.21.15.72:0/2583166639 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2868003b90 con 0x7f287015c650 2024-04-15T15:16:24.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.966+0000 7f2875e88640 1 -- 172.21.15.72:0/2583166639 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28700c6ef0 con 0x7f287015c650 2024-04-15T15:16:24.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.966+0000 7f2867fff640 1 -- 172.21.15.72:0/2583166639 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2868004030 con 0x7f287015c650 2024-04-15T15:16:24.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.966+0000 7f2875e88640 1 -- 172.21.15.72:0/2583166639 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28700c74b0 con 0x7f287015c650 2024-04-15T15:16:24.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.967+0000 7f2875e88640 1 -- 172.21.15.72:0/2583166639 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2870075c50 con 0x7f287015c650 2024-04-15T15:16:24.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.968+0000 7f2867fff640 1 -- 172.21.15.72:0/2583166639 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f286801d070 con 0x7f287015c650 2024-04-15T15:16:24.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.968+0000 7f2867fff640 1 --2- 172.21.15.72:0/2583166639 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f285006c980 0x7f285006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.968+0000 7f2867fff640 1 -- 172.21.15.72:0/2583166639 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f286808c0f0 con 0x7f287015c650 2024-04-15T15:16:24.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.968+0000 7f286f7fe640 1 --2- 172.21.15.72:0/2583166639 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f285006c980 0x7f285006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.969+0000 7f286f7fe640 1 --2- 172.21.15.72:0/2583166639 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f285006c980 0x7f285006ee40 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2860009a20 tx=0x7f2860008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:24.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:24.973+0000 7f2867fff640 1 -- 172.21.15.72:0/2583166639 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2868010070 con 0x7f287015c650 2024-04-15T15:16:25.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.148+0000 7f2875e88640 1 -- 172.21.15.72:0/2583166639 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f28700b1ec0 con 0x7f287015c650 2024-04-15T15:16:25.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.148+0000 7f2867fff640 1 -- 172.21.15.72:0/2583166639 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v38) ==== 93+0+10 (secure 0 0 0) 0x7f286805a320 con 0x7f287015c650 2024-04-15T15:16:25.150 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-15T15:16:25.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.150+0000 7f2875e88640 1 -- 172.21.15.72:0/2583166639 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f285006c980 msgr2=0x7f285006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.150+0000 7f2875e88640 1 --2- 172.21.15.72:0/2583166639 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f285006c980 0x7f285006ee40 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2860009a20 tx=0x7f2860008040 comp rx=0 tx=0).stop 2024-04-15T15:16:25.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.150+0000 7f2875e88640 1 -- 172.21.15.72:0/2583166639 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f287015c650 msgr2=0x7f28700cb960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.150+0000 7f2875e88640 1 --2- 172.21.15.72:0/2583166639 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f287015c650 0x7f28700cb960 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f2868007e10 tx=0x7f286800d4a0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.150+0000 7f2875e88640 1 -- 172.21.15.72:0/2583166639 shutdown_connections 2024-04-15T15:16:25.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.150+0000 7f2875e88640 1 --2- 172.21.15.72:0/2583166639 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f285006c980 0x7f285006ee40 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.150+0000 7f2875e88640 1 --2- 172.21.15.72:0/2583166639 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f287015c650 0x7f28700cb960 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.150+0000 7f2875e88640 1 --2- 172.21.15.72:0/2583166639 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f28700b57d0 0x7f28701519b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.150+0000 7f2875e88640 1 --2- 172.21.15.72:0/2583166639 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f28700b36f0 0x7f2870152280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.150+0000 7f2875e88640 1 -- 172.21.15.72:0/2583166639 >> 172.21.15.72:0/2583166639 conn(0x7f28700ceec0 msgr2=0x7f28700701a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:25.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.150+0000 7f2875e88640 1 -- 172.21.15.72:0/2583166639 shutdown_connections 2024-04-15T15:16:25.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.150+0000 7f2875e88640 1 -- 172.21.15.72:0/2583166639 wait complete. 2024-04-15T15:16:25.163 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:16:25.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.266+0000 7fd80fbbd640 1 Processor -- start 2024-04-15T15:16:25.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.266+0000 7fd80fbbd640 1 -- start start 2024-04-15T15:16:25.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.267+0000 7fd80fbbd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd8080ce2f0 0x7fd80815c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.267+0000 7fd80fbbd640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd8080d08f0 0x7fd8080cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.267+0000 7fd80fbbd640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd80806bbb0 con 0x7fd8080ce6d0 2024-04-15T15:16:25.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.267+0000 7fd80fbbd640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fd80806b960 con 0x7fd8080ce2f0 2024-04-15T15:16:25.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.267+0000 7fd80fbbd640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fd80806be90 con 0x7fd8080d08f0 2024-04-15T15:16:25.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.267+0000 7fd80d959640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd8080ce2f0 0x7fd80815c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.267+0000 7fd80e15a640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fd8080ce6d0 0x7fd808155600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60260/0 (socket says 172.21.15.72:60260) 2024-04-15T15:16:25.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.267+0000 7fd80e15a640 1 -- 172.21.15.72:0/1715238008 learned_addr learned my addr 172.21.15.72:0/1715238008 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:25.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.268+0000 7fd80d158640 1 --2- 172.21.15.72:0/1715238008 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd8080d08f0 0x7fd8080cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.268+0000 7fd80d959640 1 -- 172.21.15.72:0/1715238008 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd8080d08f0 msgr2=0x7fd8080cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:25.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.268+0000 7fd80d959640 1 --2- 172.21.15.72:0/1715238008 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd8080d08f0 0x7fd8080cc0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.268+0000 7fd80c957640 1 -- 172.21.15.72:0/1715238008 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 947229520 0 0) 0x7fd80806bbb0 con 0x7fd8080ce6d0 2024-04-15T15:16:25.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.268+0000 7fd80d959640 1 -- 172.21.15.72:0/1715238008 >> v1:172.21.15.72:6789/0 conn(0x7fd8080ce6d0 legacy=0x7fd808155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.268+0000 7fd80d959640 1 -- 172.21.15.72:0/1715238008 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd80806c170 con 0x7fd8080ce2f0 2024-04-15T15:16:25.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.268+0000 7fd80d158640 1 --2- 172.21.15.72:0/1715238008 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd8080d08f0 0x7fd8080cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:25.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.268+0000 7fd80d959640 1 --2- 172.21.15.72:0/1715238008 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8080ce2f0 0x7fd80815c530 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fd8040097a0 tx=0x7fd80402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe11e345987c5afc server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:25.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.268+0000 7fd80c957640 1 -- 172.21.15.72:0/1715238008 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd804035020 con 0x7fd8080ce2f0 2024-04-15T15:16:25.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.269+0000 7fd80c957640 1 -- 172.21.15.72:0/1715238008 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd804031830 con 0x7fd8080ce2f0 2024-04-15T15:16:25.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.269+0000 7fd80c957640 1 -- 172.21.15.72:0/1715238008 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd804031c00 con 0x7fd8080ce2f0 2024-04-15T15:16:25.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.269+0000 7fd80fbbd640 1 -- 172.21.15.72:0/1715238008 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8080ce2f0 msgr2=0x7fd80815c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.269+0000 7fd80fbbd640 1 --2- 172.21.15.72:0/1715238008 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8080ce2f0 0x7fd80815c530 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fd8040097a0 tx=0x7fd80402b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.269+0000 7fd80fbbd640 1 -- 172.21.15.72:0/1715238008 shutdown_connections 2024-04-15T15:16:25.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.269+0000 7fd80fbbd640 1 --2- 172.21.15.72:0/1715238008 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd8080d08f0 0x7fd8080cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.269+0000 7fd80fbbd640 1 --2- 172.21.15.72:0/1715238008 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8080ce2f0 0x7fd80815c530 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.269+0000 7fd80fbbd640 1 -- 172.21.15.72:0/1715238008 >> 172.21.15.72:0/1715238008 conn(0x7fd8080b31d0 msgr2=0x7fd8080b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:25.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.269+0000 7fd80fbbd640 1 -- 172.21.15.72:0/1715238008 shutdown_connections 2024-04-15T15:16:25.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.269+0000 7fd80fbbd640 1 -- 172.21.15.72:0/1715238008 wait complete. 2024-04-15T15:16:25.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.270+0000 7fd80fbbd640 1 Processor -- start 2024-04-15T15:16:25.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.270+0000 7fd80fbbd640 1 -- start start 2024-04-15T15:16:25.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.271+0000 7fd80fbbd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8080ce2f0 0x7fd80815d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.271+0000 7fd80fbbd640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd8080ce6d0 0x7fd808072c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.271+0000 7fd80fbbd640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd8080d08f0 0x7fd808073150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.271+0000 7fd80fbbd640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd808155d10 con 0x7fd8080ce2f0 2024-04-15T15:16:25.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.271+0000 7fd80fbbd640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fd80815ca70 con 0x7fd8080d08f0 2024-04-15T15:16:25.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.271+0000 7fd80fbbd640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fd808155e90 con 0x7fd8080ce6d0 2024-04-15T15:16:25.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.271+0000 7fd80d959640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8080ce2f0 0x7fd80815d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.271+0000 7fd80d959640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8080ce2f0 0x7fd80815d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43062/0 (socket says 172.21.15.72:43062) 2024-04-15T15:16:25.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.271+0000 7fd80d959640 1 -- 172.21.15.72:0/830581669 learned_addr learned my addr 172.21.15.72:0/830581669 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:25.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.271+0000 7fd80d959640 1 -- 172.21.15.72:0/830581669 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd8080ce6d0 msgr2=0x7fd808072c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.271+0000 7fd80d959640 1 --2- 172.21.15.72:0/830581669 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd8080ce6d0 0x7fd808072c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.271+0000 7fd80d959640 1 -- 172.21.15.72:0/830581669 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd8080d08f0 msgr2=0x7fd808073150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.271+0000 7fd80d959640 1 --2- 172.21.15.72:0/830581669 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd8080d08f0 0x7fd808073150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.271+0000 7fd80d959640 1 -- 172.21.15.72:0/830581669 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd80814a230 con 0x7fd8080ce2f0 2024-04-15T15:16:25.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.272+0000 7fd80d959640 1 --2- 172.21.15.72:0/830581669 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8080ce2f0 0x7fd80815d590 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fd80402bee0 tx=0x7fd8040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:25.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.272+0000 7fd7fe7fc640 1 -- 172.21.15.72:0/830581669 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd804041070 con 0x7fd8080ce2f0 2024-04-15T15:16:25.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.272+0000 7fd7fe7fc640 1 -- 172.21.15.72:0/830581669 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd80403a950 con 0x7fd8080ce2f0 2024-04-15T15:16:25.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.272+0000 7fd80fbbd640 1 -- 172.21.15.72:0/830581669 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd80814a510 con 0x7fd8080ce2f0 2024-04-15T15:16:25.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.272+0000 7fd7fe7fc640 1 -- 172.21.15.72:0/830581669 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd804035040 con 0x7fd8080ce2f0 2024-04-15T15:16:25.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.272+0000 7fd80fbbd640 1 -- 172.21.15.72:0/830581669 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd80814aad0 con 0x7fd8080ce2f0 2024-04-15T15:16:25.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.273+0000 7fd80fbbd640 1 -- 172.21.15.72:0/830581669 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd808075c50 con 0x7fd8080ce2f0 2024-04-15T15:16:25.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.273+0000 7fd7fe7fc640 1 -- 172.21.15.72:0/830581669 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fd80404c060 con 0x7fd8080ce2f0 2024-04-15T15:16:25.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.274+0000 7fd7fe7fc640 1 --2- 172.21.15.72:0/830581669 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd7ec06c980 0x7fd7ec06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.274+0000 7fd7fe7fc640 1 -- 172.21.15.72:0/830581669 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fd8040ad7a0 con 0x7fd8080ce2f0 2024-04-15T15:16:25.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.274+0000 7fd80d158640 1 --2- 172.21.15.72:0/830581669 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd7ec06c980 0x7fd7ec06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.274+0000 7fd80d158640 1 --2- 172.21.15.72:0/830581669 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd7ec06c980 0x7fd7ec06ee40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8006fd0 tx=0x7fd7f8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:25.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.278+0000 7fd7fe7fc640 1 -- 172.21.15.72:0/830581669 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd80407ba50 con 0x7fd8080ce2f0 2024-04-15T15:16:25.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.455+0000 7fd80fbbd640 1 -- 172.21.15.72:0/830581669 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd80806cab0 con 0x7fd8080ce2f0 2024-04-15T15:16:25.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.455+0000 7fd7fe7fc640 1 -- 172.21.15.72:0/830581669 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v38) ==== 104+0+11 (secure 0 0 0) 0x7fd80407b350 con 0x7fd8080ce2f0 2024-04-15T15:16:25.457 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:16:25.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.457+0000 7fd80fbbd640 1 -- 172.21.15.72:0/830581669 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd7ec06c980 msgr2=0x7fd7ec06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.457+0000 7fd80fbbd640 1 --2- 172.21.15.72:0/830581669 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd7ec06c980 0x7fd7ec06ee40 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8006fd0 tx=0x7fd7f8008040 comp rx=0 tx=0).stop 2024-04-15T15:16:25.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.457+0000 7fd80fbbd640 1 -- 172.21.15.72:0/830581669 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8080ce2f0 msgr2=0x7fd80815d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.457+0000 7fd80fbbd640 1 --2- 172.21.15.72:0/830581669 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8080ce2f0 0x7fd80815d590 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fd80402bee0 tx=0x7fd8040046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.457+0000 7fd80fbbd640 1 -- 172.21.15.72:0/830581669 shutdown_connections 2024-04-15T15:16:25.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.457+0000 7fd80fbbd640 1 --2- 172.21.15.72:0/830581669 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd7ec06c980 0x7fd7ec06ee40 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.457+0000 7fd80fbbd640 1 --2- 172.21.15.72:0/830581669 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd8080d08f0 0x7fd808073150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.457+0000 7fd80fbbd640 1 --2- 172.21.15.72:0/830581669 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd8080ce6d0 0x7fd808072c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.457+0000 7fd80fbbd640 1 --2- 172.21.15.72:0/830581669 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd8080ce2f0 0x7fd80815d590 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.457+0000 7fd80fbbd640 1 -- 172.21.15.72:0/830581669 >> 172.21.15.72:0/830581669 conn(0x7fd8080b31d0 msgr2=0x7fd808154b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:25.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.457+0000 7fd80fbbd640 1 -- 172.21.15.72:0/830581669 shutdown_connections 2024-04-15T15:16:25.460 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.457+0000 7fd80fbbd640 1 -- 172.21.15.72:0/830581669 wait complete. 2024-04-15T15:16:25.470 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:16:25.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.574+0000 7f6cc8e74640 1 Processor -- start 2024-04-15T15:16:25.576 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.574+0000 7f6cc8e74640 1 -- start start 2024-04-15T15:16:25.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.575+0000 7f6cc8e74640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6cc4145f10 0x7f6cc40d0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.577 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.575+0000 7f6cc8e74640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f6cc406f370 0x7f6cc40cdc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.575+0000 7f6cc8e74640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cc406d720 con 0x7f6cc406c1f0 2024-04-15T15:16:25.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.575+0000 7f6cc8e74640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f6cc41462f0 con 0x7f6cc406f370 2024-04-15T15:16:25.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.575+0000 7f6cc8e74640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f6cc406d290 con 0x7f6cc4145f10 2024-04-15T15:16:25.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.575+0000 7f6cc2d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f6cc406f370 0x7f6cc40cdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.575+0000 7f6cc2d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f6cc406f370 0x7f6cc40cdc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43072/0 (socket says 172.21.15.72:43072) 2024-04-15T15:16:25.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.575+0000 7f6cc2d9d640 1 -- 172.21.15.72:0/4232554467 learned_addr learned my addr 172.21.15.72:0/4232554467 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:25.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.576+0000 7f6cc259c640 1 --2- 172.21.15.72:0/4232554467 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6cc4145f10 0x7f6cc40d0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.576+0000 7f6cc2d9d640 1 -- 172.21.15.72:0/4232554467 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6cc4145f10 msgr2=0x7f6cc40d0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:25.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.576+0000 7f6cc2d9d640 1 --2- 172.21.15.72:0/4232554467 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6cc4145f10 0x7f6cc40d0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.576+0000 7f6cc2d9d640 1 -- 172.21.15.72:0/4232554467 >> v1:172.21.15.175:6789/0 conn(0x7f6cc406c1f0 legacy=0x7f6cc406ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:25.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.576+0000 7f6cc2d9d640 1 -- 172.21.15.72:0/4232554467 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cc406c680 con 0x7f6cc406f370 2024-04-15T15:16:25.578 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.576+0000 7f6cc2d9d640 1 --2- 172.21.15.72:0/4232554467 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6cc406f370 0x7f6cc40cdc10 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc009870 tx=0x7f6cbc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2989ca8f847ca9e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:25.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.576+0000 7f6cc159a640 1 -- 172.21.15.72:0/4232554467 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc035020 con 0x7f6cc406f370 2024-04-15T15:16:25.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.577+0000 7f6cc159a640 1 -- 172.21.15.72:0/4232554467 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cbc031640 con 0x7f6cc406f370 2024-04-15T15:16:25.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.577+0000 7f6cc159a640 1 -- 172.21.15.72:0/4232554467 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cbc02d040 con 0x7f6cc406f370 2024-04-15T15:16:25.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.577+0000 7f6cc8e74640 1 -- 172.21.15.72:0/4232554467 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6cc406f370 msgr2=0x7f6cc40cdc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.577+0000 7f6cc8e74640 1 --2- 172.21.15.72:0/4232554467 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6cc406f370 0x7f6cc40cdc10 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f6cbc009870 tx=0x7f6cbc02b8f0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.577+0000 7f6cc8e74640 1 -- 172.21.15.72:0/4232554467 shutdown_connections 2024-04-15T15:16:25.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.577+0000 7f6cc8e74640 1 --2- 172.21.15.72:0/4232554467 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6cc406f370 0x7f6cc40cdc10 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.577+0000 7f6cc8e74640 1 --2- 172.21.15.72:0/4232554467 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6cc4145f10 0x7f6cc40d0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.577+0000 7f6cc8e74640 1 -- 172.21.15.72:0/4232554467 >> 172.21.15.72:0/4232554467 conn(0x7f6cc40d13a0 msgr2=0x7f6cc40d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:25.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.577+0000 7f6cc8e74640 1 -- 172.21.15.72:0/4232554467 shutdown_connections 2024-04-15T15:16:25.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.577+0000 7f6cc8e74640 1 -- 172.21.15.72:0/4232554467 wait complete. 2024-04-15T15:16:25.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.578+0000 7f6cc8e74640 1 Processor -- start 2024-04-15T15:16:25.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.579+0000 7f6cc8e74640 1 -- start start 2024-04-15T15:16:25.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.579+0000 7f6cc8e74640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6cc406f370 0x7f6cc413b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.579+0000 7f6cc8e74640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6cc4145f10 0x7f6cc4142570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.579+0000 7f6cc8e74640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6cc4142ab0 0x7f6cc415e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.579+0000 7f6cc8e74640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc406cf20 con 0x7f6cc406f370 2024-04-15T15:16:25.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.579+0000 7f6cc8e74640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f6cc406cc40 con 0x7f6cc4142ab0 2024-04-15T15:16:25.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.579+0000 7f6cc8e74640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f6cc40d0ac0 con 0x7f6cc4145f10 2024-04-15T15:16:25.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.579+0000 7f6cc259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6cc406f370 0x7f6cc413b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.579+0000 7f6cc259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6cc406f370 0x7f6cc413b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43084/0 (socket says 172.21.15.72:43084) 2024-04-15T15:16:25.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.579+0000 7f6cc259c640 1 -- 172.21.15.72:0/1552933143 learned_addr learned my addr 172.21.15.72:0/1552933143 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:25.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.580+0000 7f6cc1d9b640 1 --2- 172.21.15.72:0/1552933143 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6cc4145f10 0x7f6cc4142570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.580+0000 7f6cc2d9d640 1 --2- 172.21.15.72:0/1552933143 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6cc4142ab0 0x7f6cc415e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.580+0000 7f6cc259c640 1 -- 172.21.15.72:0/1552933143 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6cc4145f10 msgr2=0x7f6cc4142570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.580+0000 7f6cc259c640 1 --2- 172.21.15.72:0/1552933143 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6cc4145f10 0x7f6cc4142570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.580+0000 7f6cc259c640 1 -- 172.21.15.72:0/1552933143 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6cc4142ab0 msgr2=0x7f6cc415e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.580+0000 7f6cc259c640 1 --2- 172.21.15.72:0/1552933143 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6cc4142ab0 0x7f6cc415e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.580+0000 7f6cc259c640 1 -- 172.21.15.72:0/1552933143 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cc41f31d0 con 0x7f6cc406f370 2024-04-15T15:16:25.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.580+0000 7f6cc259c640 1 --2- 172.21.15.72:0/1552933143 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6cc406f370 0x7f6cc413b240 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f6cb400ccf0 tx=0x7f6cb400a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:25.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.580+0000 7f6cbaffd640 1 -- 172.21.15.72:0/1552933143 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb4003990 con 0x7f6cc406f370 2024-04-15T15:16:25.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.580+0000 7f6cbaffd640 1 -- 172.21.15.72:0/1552933143 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cb4003b30 con 0x7f6cc406f370 2024-04-15T15:16:25.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.580+0000 7f6cc8e74640 1 -- 172.21.15.72:0/1552933143 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cc41f34b0 con 0x7f6cc406f370 2024-04-15T15:16:25.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.580+0000 7f6cbaffd640 1 -- 172.21.15.72:0/1552933143 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cb4004020 con 0x7f6cc406f370 2024-04-15T15:16:25.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.580+0000 7f6cc8e74640 1 -- 172.21.15.72:0/1552933143 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cc41f3a70 con 0x7f6cc406f370 2024-04-15T15:16:25.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.581+0000 7f6cc8e74640 1 -- 172.21.15.72:0/1552933143 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cc4075c50 con 0x7f6cc406f370 2024-04-15T15:16:25.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.581+0000 7f6cbaffd640 1 -- 172.21.15.72:0/1552933143 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f6cb40041c0 con 0x7f6cc406f370 2024-04-15T15:16:25.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.582+0000 7f6cbaffd640 1 --2- 172.21.15.72:0/1552933143 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6ca406c9b0 0x7f6ca406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.582+0000 7f6cbaffd640 1 -- 172.21.15.72:0/1552933143 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f6cb408c2e0 con 0x7f6cc406f370 2024-04-15T15:16:25.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.582+0000 7f6cc1d9b640 1 --2- 172.21.15.72:0/1552933143 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6ca406c9b0 0x7f6ca406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.582+0000 7f6cc1d9b640 1 --2- 172.21.15.72:0/1552933143 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6ca406c9b0 0x7f6ca406ee70 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6cc406ed90 tx=0x7f6cac009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:25.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.586+0000 7f6cbaffd640 1 -- 172.21.15.72:0/1552933143 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6cb4014030 con 0x7f6cc406f370 2024-04-15T15:16:25.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.762+0000 7f6cc8e74640 1 -- 172.21.15.72:0/1552933143 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6cc40d2130 con 0x7f6cc406f370 2024-04-15T15:16:25.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.763+0000 7f6cbaffd640 1 -- 172.21.15.72:0/1552933143 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v38) ==== 100+0+11 (secure 0 0 0) 0x7f6cb405a480 con 0x7f6cc406f370 2024-04-15T15:16:25.764 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-04-15T15:16:25.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.765+0000 7f6cc8e74640 1 -- 172.21.15.72:0/1552933143 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6ca406c9b0 msgr2=0x7f6ca406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.765+0000 7f6cc8e74640 1 --2- 172.21.15.72:0/1552933143 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6ca406c9b0 0x7f6ca406ee70 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6cc406ed90 tx=0x7f6cac009040 comp rx=0 tx=0).stop 2024-04-15T15:16:25.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.765+0000 7f6cc8e74640 1 -- 172.21.15.72:0/1552933143 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6cc406f370 msgr2=0x7f6cc413b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.765+0000 7f6cc8e74640 1 --2- 172.21.15.72:0/1552933143 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6cc406f370 0x7f6cc413b240 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f6cb400ccf0 tx=0x7f6cb400a5c0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.765+0000 7f6cc8e74640 1 -- 172.21.15.72:0/1552933143 shutdown_connections 2024-04-15T15:16:25.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.765+0000 7f6cc8e74640 1 --2- 172.21.15.72:0/1552933143 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f6ca406c9b0 0x7f6ca406ee70 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.765+0000 7f6cc8e74640 1 --2- 172.21.15.72:0/1552933143 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f6cc4142ab0 0x7f6cc415e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.765+0000 7f6cc8e74640 1 --2- 172.21.15.72:0/1552933143 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f6cc4145f10 0x7f6cc4142570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.765+0000 7f6cc8e74640 1 --2- 172.21.15.72:0/1552933143 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6cc406f370 0x7f6cc413b240 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.765+0000 7f6cc8e74640 1 -- 172.21.15.72:0/1552933143 >> 172.21.15.72:0/1552933143 conn(0x7f6cc40d13a0 msgr2=0x7f6cc40d0050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:25.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.765+0000 7f6cc8e74640 1 -- 172.21.15.72:0/1552933143 shutdown_connections 2024-04-15T15:16:25.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.765+0000 7f6cc8e74640 1 -- 172.21.15.72:0/1552933143 wait complete. 2024-04-15T15:16:25.777 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:16:25.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.880+0000 7fa5dc515640 1 Processor -- start 2024-04-15T15:16:25.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.880+0000 7fa5dc515640 1 -- start start 2024-04-15T15:16:25.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.881+0000 7fa5dc515640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa5d4159140 0x7fa5d406b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.881+0000 7fa5dc515640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5d41600e0 con 0x7fa5d40b4cc0 2024-04-15T15:16:25.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.881+0000 7fa5dc515640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5d415bcd0 con 0x7fa5d406d300 2024-04-15T15:16:25.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.881+0000 7fa5dc515640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fa5d406bea0 con 0x7fa5d4159140 2024-04-15T15:16:25.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.881+0000 7fa5daab2640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fa5d40b4cc0 0x7fa5d415f9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60274/0 (socket says 172.21.15.72:60274) 2024-04-15T15:16:25.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.881+0000 7fa5daab2640 1 -- 172.21.15.72:0/998954929 learned_addr learned my addr 172.21.15.72:0/998954929 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:25.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.881+0000 7fa5da2b1640 1 --2- 172.21.15.72:0/998954929 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa5d4159140 0x7fa5d406b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.881+0000 7fa5d92af640 1 -- 172.21.15.72:0/998954929 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3713902521 0 0) 0x7fa5d41600e0 con 0x7fa5d40b4cc0 2024-04-15T15:16:25.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.881+0000 7fa5d92af640 1 -- 172.21.15.72:0/998954929 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5c0003520 con 0x7fa5d40b4cc0 2024-04-15T15:16:25.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.881+0000 7fa5d92af640 1 -- 172.21.15.72:0/998954929 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4118370451 0 0) 0x7fa5c0003520 con 0x7fa5d40b4cc0 2024-04-15T15:16:25.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.881+0000 7fa5d92af640 1 -- 172.21.15.72:0/998954929 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa5d4159140 msgr2=0x7fa5d406b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:25.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.881+0000 7fa5d92af640 1 --2- 172.21.15.72:0/998954929 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa5d4159140 0x7fa5d406b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.881+0000 7fa5d92af640 1 -- 172.21.15.72:0/998954929 >> v1:172.21.15.175:6789/0 conn(0x7fa5d406d300 legacy=0x7fa5d415f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.881+0000 7fa5d92af640 1 -- 172.21.15.72:0/998954929 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5d415bf90 con 0x7fa5d40b4cc0 2024-04-15T15:16:25.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.881+0000 7fa5d92af640 1 -- 172.21.15.72:0/998954929 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7fa5cc0019f0 con 0x7fa5d40b4cc0 2024-04-15T15:16:25.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.881+0000 7fa5d92af640 1 -- 172.21.15.72:0/998954929 >> v1:172.21.15.72:6789/0 conn(0x7fa5d40b4cc0 legacy=0x7fa5d415f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.881+0000 7fa5d92af640 1 --2- 172.21.15.72:0/998954929 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5c00039e0 0x7fa5c0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.881+0000 7fa5d92af640 1 --2- 172.21.15.72:0/998954929 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa5d4159140 0x7fa5c0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.882+0000 7fa5d92af640 1 --2- 172.21.15.72:0/998954929 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa5c0004d40 0x7fa5c00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.882+0000 7fa5d92af640 1 -- 172.21.15.72:0/998954929 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c000bb00 con 0x7fa5c00039e0 2024-04-15T15:16:25.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.882+0000 7fa5d92af640 1 -- 172.21.15.72:0/998954929 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c000bc80 con 0x7fa5c0004d40 2024-04-15T15:16:25.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.882+0000 7fa5d92af640 1 -- 172.21.15.72:0/998954929 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fa5c000be80 con 0x7fa5d4159140 2024-04-15T15:16:25.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.882+0000 7fa5da2b1640 1 --2- 172.21.15.72:0/998954929 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5c00039e0 0x7fa5c0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.882+0000 7fa5daab2640 1 --2- 172.21.15.72:0/998954929 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa5c0004d40 0x7fa5c00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.882+0000 7fa5d9ab0640 1 --2- 172.21.15.72:0/998954929 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa5d4159140 0x7fa5c0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.882+0000 7fa5da2b1640 1 -- 172.21.15.72:0/998954929 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa5d4159140 msgr2=0x7fa5c0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:25.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.882+0000 7fa5da2b1640 1 --2- 172.21.15.72:0/998954929 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa5d4159140 0x7fa5c0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.882+0000 7fa5da2b1640 1 -- 172.21.15.72:0/998954929 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa5c0004d40 msgr2=0x7fa5c00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:25.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.882+0000 7fa5da2b1640 1 --2- 172.21.15.72:0/998954929 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa5c0004d40 0x7fa5c00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.882+0000 7fa5da2b1640 1 -- 172.21.15.72:0/998954929 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5c0003520 con 0x7fa5c00039e0 2024-04-15T15:16:25.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.883+0000 7fa5da2b1640 1 --2- 172.21.15.72:0/998954929 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5c00039e0 0x7fa5c0003e00 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0009a90 tx=0x7fa5c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfecb5ca1b75fe18 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:25.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.883+0000 7fa5d92af640 1 -- 172.21.15.72:0/998954929 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5d0039070 con 0x7fa5c00039e0 2024-04-15T15:16:25.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.883+0000 7fa5d92af640 1 -- 172.21.15.72:0/998954929 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5d002c680 con 0x7fa5c00039e0 2024-04-15T15:16:25.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.883+0000 7fa5d92af640 1 -- 172.21.15.72:0/998954929 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5d0009690 con 0x7fa5c00039e0 2024-04-15T15:16:25.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.883+0000 7fa5dc515640 1 -- 172.21.15.72:0/998954929 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5c00039e0 msgr2=0x7fa5c0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.883+0000 7fa5dc515640 1 --2- 172.21.15.72:0/998954929 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5c00039e0 0x7fa5c0003e00 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0009a90 tx=0x7fa5c0005830 comp rx=0 tx=0).stop 2024-04-15T15:16:25.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.883+0000 7fa5daab2640 1 -- 172.21.15.72:0/998954929 reap_dead start 2024-04-15T15:16:25.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.883+0000 7fa5dc515640 1 -- 172.21.15.72:0/998954929 shutdown_connections 2024-04-15T15:16:25.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.883+0000 7fa5dc515640 1 -- 172.21.15.72:0/998954929 >> 172.21.15.72:0/998954929 conn(0x7fa5d40d0a50 msgr2=0x7fa5d40d0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:25.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.883+0000 7fa5dc515640 1 -- 172.21.15.72:0/998954929 shutdown_connections 2024-04-15T15:16:25.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.883+0000 7fa5dc515640 1 -- 172.21.15.72:0/998954929 wait complete. 2024-04-15T15:16:25.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.885+0000 7fa5dc515640 1 Processor -- start 2024-04-15T15:16:25.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.885+0000 7fa5dc515640 1 -- start start 2024-04-15T15:16:25.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.885+0000 7fa5dc515640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5c00039e0 0x7fa5d41f61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.885+0000 7fa5dc515640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa5d413b1a0 0x7fa5d40b1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.885+0000 7fa5dc515640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa5d40b1cd0 0x7fa5d413caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.885+0000 7fa5dc515640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d415c830 con 0x7fa5c00039e0 2024-04-15T15:16:25.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.885+0000 7fa5dc515640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d415c550 con 0x7fa5d40b1cd0 2024-04-15T15:16:25.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.885+0000 7fa5dc515640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fa5d40d06c0 con 0x7fa5d413b1a0 2024-04-15T15:16:25.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.885+0000 7fa5da2b1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5c00039e0 0x7fa5d41f61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.885+0000 7fa5da2b1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5c00039e0 0x7fa5d41f61c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43106/0 (socket says 172.21.15.72:43106) 2024-04-15T15:16:25.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.885+0000 7fa5da2b1640 1 -- 172.21.15.72:0/2640881377 learned_addr learned my addr 172.21.15.72:0/2640881377 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:25.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.886+0000 7fa5da2b1640 1 -- 172.21.15.72:0/2640881377 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa5d413b1a0 msgr2=0x7fa5d40b1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.886+0000 7fa5da2b1640 1 --2- 172.21.15.72:0/2640881377 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa5d413b1a0 0x7fa5d40b1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.886+0000 7fa5da2b1640 1 -- 172.21.15.72:0/2640881377 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa5d40b1cd0 msgr2=0x7fa5d413caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.886+0000 7fa5da2b1640 1 --2- 172.21.15.72:0/2640881377 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa5d40b1cd0 0x7fa5d413caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.886+0000 7fa5da2b1640 1 -- 172.21.15.72:0/2640881377 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5d4150bd0 con 0x7fa5c00039e0 2024-04-15T15:16:25.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.886+0000 7fa5da2b1640 1 --2- 172.21.15.72:0/2640881377 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5c00039e0 0x7fa5d41f61c0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0000c00 tx=0x7fa5d0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:25.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.886+0000 7fa5caffd640 1 -- 172.21.15.72:0/2640881377 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5d0042020 con 0x7fa5c00039e0 2024-04-15T15:16:25.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.886+0000 7fa5caffd640 1 -- 172.21.15.72:0/2640881377 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5d003a040 con 0x7fa5c00039e0 2024-04-15T15:16:25.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.886+0000 7fa5dc515640 1 -- 172.21.15.72:0/2640881377 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5d413d860 con 0x7fa5c00039e0 2024-04-15T15:16:25.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.886+0000 7fa5caffd640 1 -- 172.21.15.72:0/2640881377 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5d0039070 con 0x7fa5c00039e0 2024-04-15T15:16:25.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.886+0000 7fa5dc515640 1 -- 172.21.15.72:0/2640881377 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5d413de20 con 0x7fa5c00039e0 2024-04-15T15:16:25.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.887+0000 7fa5dc515640 1 -- 172.21.15.72:0/2640881377 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5d40b20f0 con 0x7fa5c00039e0 2024-04-15T15:16:25.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.887+0000 7fa5caffd640 1 -- 172.21.15.72:0/2640881377 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fa5d0045020 con 0x7fa5c00039e0 2024-04-15T15:16:25.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.888+0000 7fa5caffd640 1 --2- 172.21.15.72:0/2640881377 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa5b806c9b0 0x7fa5b806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.888+0000 7fa5d9ab0640 1 --2- 172.21.15.72:0/2640881377 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa5b806c9b0 0x7fa5b806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.888+0000 7fa5caffd640 1 -- 172.21.15.72:0/2640881377 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fa5d00ad100 con 0x7fa5c00039e0 2024-04-15T15:16:25.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.888+0000 7fa5d9ab0640 1 --2- 172.21.15.72:0/2640881377 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa5b806c9b0 0x7fa5b806ee70 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4007fd0 tx=0x7fa5c4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:25.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:25.892+0000 7fa5caffd640 1 -- 172.21.15.72:0/2640881377 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa5d004c070 con 0x7fa5c00039e0 2024-04-15T15:16:26.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.079+0000 7fa5dc515640 1 -- 172.21.15.72:0/2640881377 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fa5d4068010 con 0x7fa5c00039e0 2024-04-15T15:16:26.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.079+0000 7fa5caffd640 1 -- 172.21.15.72:0/2640881377 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v38) ==== 80+0+0 (secure 0 0 0) 0x7fa5d007b3b0 con 0x7fa5c00039e0 2024-04-15T15:16:26.081 INFO:teuthology.orchestra.run.smithi072.stderr:listed 0 entries 2024-04-15T15:16:26.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.081+0000 7fa5dc515640 1 -- 172.21.15.72:0/2640881377 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa5b806c9b0 msgr2=0x7fa5b806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.081+0000 7fa5dc515640 1 --2- 172.21.15.72:0/2640881377 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa5b806c9b0 0x7fa5b806ee70 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4007fd0 tx=0x7fa5c4009040 comp rx=0 tx=0).stop 2024-04-15T15:16:26.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.081+0000 7fa5dc515640 1 -- 172.21.15.72:0/2640881377 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5c00039e0 msgr2=0x7fa5d41f61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.081+0000 7fa5dc515640 1 --2- 172.21.15.72:0/2640881377 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5c00039e0 0x7fa5d41f61c0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0000c00 tx=0x7fa5d0004730 comp rx=0 tx=0).stop 2024-04-15T15:16:26.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.081+0000 7fa5dc515640 1 -- 172.21.15.72:0/2640881377 shutdown_connections 2024-04-15T15:16:26.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.081+0000 7fa5dc515640 1 --2- 172.21.15.72:0/2640881377 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa5b806c9b0 0x7fa5b806ee70 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.081+0000 7fa5dc515640 1 --2- 172.21.15.72:0/2640881377 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa5d40b1cd0 0x7fa5d413caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.081+0000 7fa5dc515640 1 --2- 172.21.15.72:0/2640881377 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa5d413b1a0 0x7fa5d40b1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.081+0000 7fa5dc515640 1 --2- 172.21.15.72:0/2640881377 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5c00039e0 0x7fa5d41f61c0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.081+0000 7fa5dc515640 1 -- 172.21.15.72:0/2640881377 >> 172.21.15.72:0/2640881377 conn(0x7fa5d40d0a50 msgr2=0x7fa5d406fa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:26.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.081+0000 7fa5dc515640 1 -- 172.21.15.72:0/2640881377 shutdown_connections 2024-04-15T15:16:26.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.081+0000 7fa5dc515640 1 -- 172.21.15.72:0/2640881377 wait complete. 2024-04-15T15:16:26.094 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:16:26.094 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:16:26.114 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:26.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.215+0000 7f8b35159640 1 Processor -- start 2024-04-15T15:16:26.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.216+0000 7f8b35159640 1 -- start start 2024-04-15T15:16:26.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.216+0000 7f8b35159640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8b30161e40 0x7f8b300b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.216+0000 7f8b35159640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b30162220 con 0x7f8b300d12b0 2024-04-15T15:16:26.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.216+0000 7f8b35159640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b300d3070 con 0x7f8b300d45b0 2024-04-15T15:16:26.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.216+0000 7f8b35159640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f8b300d21f0 con 0x7f8b30161e40 2024-04-15T15:16:26.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.216+0000 7f8b2ed9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8b30161e40 0x7f8b300b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.216+0000 7f8b2e59c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f8b300d12b0 0x7f8b300d2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60290/0 (socket says 172.21.15.72:60290) 2024-04-15T15:16:26.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.216+0000 7f8b2ed9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8b30161e40 0x7f8b300b37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43112/0 (socket says 172.21.15.72:43112) 2024-04-15T15:16:26.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.216+0000 7f8b2e59c640 1 -- 172.21.15.72:0/2010466106 learned_addr learned my addr 172.21.15.72:0/2010466106 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:26.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.217+0000 7f8b2ed9d640 1 -- 172.21.15.72:0/2010466106 >> v1:172.21.15.175:6789/0 conn(0x7f8b300d45b0 legacy=0x7f8b30162490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:26.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.217+0000 7f8b2ed9d640 1 -- 172.21.15.72:0/2010466106 >> v1:172.21.15.72:6789/0 conn(0x7f8b300d12b0 legacy=0x7f8b300d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.217+0000 7f8b2ed9d640 1 -- 172.21.15.72:0/2010466106 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b300d1820 con 0x7f8b30161e40 2024-04-15T15:16:26.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.217+0000 7f8b2ed9d640 1 --2- 172.21.15.72:0/2010466106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b30161e40 0x7f8b300b37f0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f8b280097d0 tx=0x7f8b2802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf6d0df823af04e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:26.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.218+0000 7f8b2dd9b640 1 -- 172.21.15.72:0/2010466106 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b280315a0 con 0x7f8b30161e40 2024-04-15T15:16:26.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.218+0000 7f8b2dd9b640 1 -- 172.21.15.72:0/2010466106 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b28037070 con 0x7f8b30161e40 2024-04-15T15:16:26.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.218+0000 7f8b2dd9b640 1 -- 172.21.15.72:0/2010466106 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b28031960 con 0x7f8b30161e40 2024-04-15T15:16:26.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.218+0000 7f8b35159640 1 -- 172.21.15.72:0/2010466106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b30161e40 msgr2=0x7f8b300b37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.218+0000 7f8b35159640 1 --2- 172.21.15.72:0/2010466106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b30161e40 0x7f8b300b37f0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f8b280097d0 tx=0x7f8b2802b840 comp rx=0 tx=0).stop 2024-04-15T15:16:26.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.218+0000 7f8b35159640 1 -- 172.21.15.72:0/2010466106 shutdown_connections 2024-04-15T15:16:26.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.218+0000 7f8b35159640 1 --2- 172.21.15.72:0/2010466106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b30161e40 0x7f8b300b37f0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.218+0000 7f8b35159640 1 -- 172.21.15.72:0/2010466106 >> 172.21.15.72:0/2010466106 conn(0x7f8b3009d970 msgr2=0x7f8b300b95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:26.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.218+0000 7f8b35159640 1 -- 172.21.15.72:0/2010466106 shutdown_connections 2024-04-15T15:16:26.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.218+0000 7f8b35159640 1 -- 172.21.15.72:0/2010466106 wait complete. 2024-04-15T15:16:26.223 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.220+0000 7f8b35159640 1 Processor -- start 2024-04-15T15:16:26.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.220+0000 7f8b35159640 1 -- start start 2024-04-15T15:16:26.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.220+0000 7f8b35159640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b300d12b0 0x7f8b30153860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.220+0000 7f8b35159640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8b300d45b0 0x7f8b3015fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.220+0000 7f8b35159640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b30161e40 0x7f8b3015e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.220+0000 7f8b35159640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8b300d2440 con 0x7f8b30161e40 2024-04-15T15:16:26.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.220+0000 7f8b35159640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f8b300d1de0 con 0x7f8b300d12b0 2024-04-15T15:16:26.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.220+0000 7f8b35159640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f8b300b3d30 con 0x7f8b300d45b0 2024-04-15T15:16:26.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.220+0000 7f8b2f59e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b30161e40 0x7f8b3015e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.221+0000 7f8b2f59e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b30161e40 0x7f8b3015e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43120/0 (socket says 172.21.15.72:43120) 2024-04-15T15:16:26.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.221+0000 7f8b2f59e640 1 -- 172.21.15.72:0/2593836657 learned_addr learned my addr 172.21.15.72:0/2593836657 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:26.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.221+0000 7f8b2ed9d640 1 --2- 172.21.15.72:0/2593836657 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b300d12b0 0x7f8b30153860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.221+0000 7f8b2e59c640 1 --2- 172.21.15.72:0/2593836657 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8b300d45b0 0x7f8b3015fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.221+0000 7f8b2f59e640 1 -- 172.21.15.72:0/2593836657 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8b300d45b0 msgr2=0x7f8b3015fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.221+0000 7f8b2f59e640 1 --2- 172.21.15.72:0/2593836657 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8b300d45b0 0x7f8b3015fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.221+0000 7f8b2f59e640 1 -- 172.21.15.72:0/2593836657 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b300d12b0 msgr2=0x7f8b30153860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.221+0000 7f8b2f59e640 1 --2- 172.21.15.72:0/2593836657 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b300d12b0 0x7f8b30153860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.221+0000 7f8b2f59e640 1 -- 172.21.15.72:0/2593836657 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b3015ec80 con 0x7f8b30161e40 2024-04-15T15:16:26.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.221+0000 7f8b2f59e640 1 --2- 172.21.15.72:0/2593836657 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b30161e40 0x7f8b3015e230 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f8b2000c440 tx=0x7f8b2000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:26.225 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.221+0000 7f8b277fe640 1 -- 172.21.15.72:0/2593836657 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b20011790 con 0x7f8b30161e40 2024-04-15T15:16:26.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.221+0000 7f8b277fe640 1 -- 172.21.15.72:0/2593836657 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b20017070 con 0x7f8b30161e40 2024-04-15T15:16:26.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.221+0000 7f8b35159640 1 -- 172.21.15.72:0/2593836657 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b3015ef60 con 0x7f8b30161e40 2024-04-15T15:16:26.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.221+0000 7f8b277fe640 1 -- 172.21.15.72:0/2593836657 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b20011b70 con 0x7f8b30161e40 2024-04-15T15:16:26.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.221+0000 7f8b35159640 1 -- 172.21.15.72:0/2593836657 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b3015f800 con 0x7f8b30161e40 2024-04-15T15:16:26.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.222+0000 7f8b35159640 1 -- 172.21.15.72:0/2593836657 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b3015f350 con 0x7f8b30161e40 2024-04-15T15:16:26.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.223+0000 7f8b277fe640 1 -- 172.21.15.72:0/2593836657 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f8b20011d10 con 0x7f8b30161e40 2024-04-15T15:16:26.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.223+0000 7f8b277fe640 1 --2- 172.21.15.72:0/2593836657 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8b1006c980 0x7f8b1006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.223+0000 7f8b277fe640 1 -- 172.21.15.72:0/2593836657 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f8b2008ba90 con 0x7f8b30161e40 2024-04-15T15:16:26.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.223+0000 7f8b2ed9d640 1 --2- 172.21.15.72:0/2593836657 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8b1006c980 0x7f8b1006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.224+0000 7f8b2ed9d640 1 --2- 172.21.15.72:0/2593836657 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8b1006c980 0x7f8b1006ee40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8b28036000 tx=0x7f8b28008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:26.229 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.227+0000 7f8b277fe640 1 -- 172.21.15.72:0/2593836657 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8b20059dd0 con 0x7f8b30161e40 2024-04-15T15:16:26.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.403+0000 7f8b35159640 1 -- 172.21.15.72:0/2593836657 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8b300689e0 con 0x7f8b30161e40 2024-04-15T15:16:26.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.404+0000 7f8b277fe640 1 -- 172.21.15.72:0/2593836657 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+16942 (secure 0 0 0) 0x7f8b200596d0 con 0x7f8b30161e40 2024-04-15T15:16:26.406 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:16:26.406 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":38,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:16:08.702307+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:05.551499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:06.003152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:26.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.406+0000 7f8b35159640 1 -- 172.21.15.72:0/2593836657 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8b1006c980 msgr2=0x7f8b1006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.406+0000 7f8b35159640 1 --2- 172.21.15.72:0/2593836657 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8b1006c980 0x7f8b1006ee40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8b28036000 tx=0x7f8b28008030 comp rx=0 tx=0).stop 2024-04-15T15:16:26.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.406+0000 7f8b35159640 1 -- 172.21.15.72:0/2593836657 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b30161e40 msgr2=0x7f8b3015e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.406+0000 7f8b35159640 1 --2- 172.21.15.72:0/2593836657 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b30161e40 0x7f8b3015e230 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f8b2000c440 tx=0x7f8b2000c900 comp rx=0 tx=0).stop 2024-04-15T15:16:26.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.406+0000 7f8b35159640 1 -- 172.21.15.72:0/2593836657 shutdown_connections 2024-04-15T15:16:26.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.406+0000 7f8b35159640 1 --2- 172.21.15.72:0/2593836657 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8b1006c980 0x7f8b1006ee40 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.406+0000 7f8b35159640 1 --2- 172.21.15.72:0/2593836657 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b30161e40 0x7f8b3015e230 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.406+0000 7f8b35159640 1 --2- 172.21.15.72:0/2593836657 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8b300d45b0 0x7f8b3015fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.406+0000 7f8b35159640 1 --2- 172.21.15.72:0/2593836657 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b300d12b0 0x7f8b30153860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.406+0000 7f8b35159640 1 -- 172.21.15.72:0/2593836657 >> 172.21.15.72:0/2593836657 conn(0x7f8b3009d970 msgr2=0x7f8b300b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:26.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.406+0000 7f8b35159640 1 -- 172.21.15.72:0/2593836657 shutdown_connections 2024-04-15T15:16:26.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.406+0000 7f8b35159640 1 -- 172.21.15.72:0/2593836657 wait complete. 2024-04-15T15:16:26.419 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:39.359102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:16:05.551499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:16:06.003152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:16:26.420 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:26.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.524+0000 7f53f57ba640 1 Processor -- start 2024-04-15T15:16:26.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.524+0000 7f53f57ba640 1 -- start start 2024-04-15T15:16:26.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.525+0000 7f53f57ba640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f53f01622f0 0x7f53f0162760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.525+0000 7f53f57ba640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f53f015ff30 0x7f53f0160360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.525+0000 7f53f57ba640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53f0155b70 con 0x7f53f0155790 2024-04-15T15:16:26.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.525+0000 7f53f57ba640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f53f00b6580 con 0x7f53f01622f0 2024-04-15T15:16:26.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.525+0000 7f53f57ba640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f53f0162ca0 con 0x7f53f015ff30 2024-04-15T15:16:26.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.525+0000 7f53eeffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f53f0155790 0x7f53f00b5e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60298/0 (socket says 172.21.15.72:60298) 2024-04-15T15:16:26.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.525+0000 7f53eeffd640 1 -- 172.21.15.72:0/3669271931 learned_addr learned my addr 172.21.15.72:0/3669271931 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:26.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.525+0000 7f53ef7fe640 1 --2- 172.21.15.72:0/3669271931 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f53f015ff30 0x7f53f0160360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.525+0000 7f53ee7fc640 1 --2- 172.21.15.72:0/3669271931 >> v2:172.21.15.175:3300/0 conn(0x7f53f01622f0 0x7f53f0162760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.525+0000 7f53edffb640 1 -- 172.21.15.72:0/3669271931 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2430889400 0 0) 0x7f53f0155b70 con 0x7f53f0155790 2024-04-15T15:16:26.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.526+0000 7f53edffb640 1 -- 172.21.15.72:0/3669271931 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53dc003520 con 0x7f53f0155790 2024-04-15T15:16:26.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.526+0000 7f53edffb640 1 -- 172.21.15.72:0/3669271931 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2525434302 0 0) 0x7f53dc003520 con 0x7f53f0155790 2024-04-15T15:16:26.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.526+0000 7f53edffb640 1 -- 172.21.15.72:0/3669271931 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f53f015ff30 msgr2=0x7f53f0160360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:26.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.526+0000 7f53edffb640 1 --2- 172.21.15.72:0/3669271931 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f53f015ff30 0x7f53f0160360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.526+0000 7f53edffb640 1 -- 172.21.15.72:0/3669271931 >> v2:172.21.15.175:3300/0 conn(0x7f53f01622f0 msgr2=0x7f53f0162760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:26.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.526+0000 7f53edffb640 1 --2- 172.21.15.72:0/3669271931 >> v2:172.21.15.175:3300/0 conn(0x7f53f01622f0 0x7f53f0162760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.526+0000 7f53edffb640 1 -- 172.21.15.72:0/3669271931 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53f006ef00 con 0x7f53f0155790 2024-04-15T15:16:26.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.526+0000 7f53edffb640 1 -- 172.21.15.72:0/3669271931 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f53e00019f0 con 0x7f53f0155790 2024-04-15T15:16:26.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.526+0000 7f53edffb640 1 -- 172.21.15.72:0/3669271931 >> v1:172.21.15.72:6789/0 conn(0x7f53f0155790 legacy=0x7f53f00b5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.526+0000 7f53edffb640 1 --2- 172.21.15.72:0/3669271931 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f53dc0039b0 0x7f53dc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.526+0000 7f53edffb640 1 --2- 172.21.15.72:0/3669271931 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f53f015ff30 0x7f53dc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.526+0000 7f53edffb640 1 --2- 172.21.15.72:0/3669271931 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53dc004d40 0x7f53dc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.526+0000 7f53edffb640 1 -- 172.21.15.72:0/3669271931 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f53f00b6580 con 0x7f53dc004d40 2024-04-15T15:16:26.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.526+0000 7f53edffb640 1 -- 172.21.15.72:0/3669271931 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f53f0162ca0 con 0x7f53dc0039b0 2024-04-15T15:16:26.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.526+0000 7f53edffb640 1 -- 172.21.15.72:0/3669271931 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f53dc00b9c0 con 0x7f53f015ff30 2024-04-15T15:16:26.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.526+0000 7f53ef7fe640 1 --2- 172.21.15.72:0/3669271931 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53dc004d40 0x7f53dc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.526+0000 7f53eeffd640 1 --2- 172.21.15.72:0/3669271931 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f53dc0039b0 0x7f53dc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.526+0000 7f53ee7fc640 1 --2- 172.21.15.72:0/3669271931 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f53f015ff30 0x7f53dc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.526+0000 7f53ef7fe640 1 -- 172.21.15.72:0/3669271931 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f53f015ff30 msgr2=0x7f53dc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:26.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.526+0000 7f53ef7fe640 1 --2- 172.21.15.72:0/3669271931 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f53f015ff30 0x7f53dc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.526+0000 7f53ef7fe640 1 -- 172.21.15.72:0/3669271931 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f53dc0039b0 msgr2=0x7f53dc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:26.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.526+0000 7f53ef7fe640 1 --2- 172.21.15.72:0/3669271931 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f53dc0039b0 0x7f53dc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.526+0000 7f53ef7fe640 1 -- 172.21.15.72:0/3669271931 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53dc003520 con 0x7f53dc004d40 2024-04-15T15:16:26.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.527+0000 7f53ef7fe640 1 --2- 172.21.15.72:0/3669271931 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53dc004d40 0x7f53dc0051c0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f53e8009ac0 tx=0x7f53dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6305d76c9cee78ae server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:26.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.527+0000 7f53edffb640 1 -- 172.21.15.72:0/3669271931 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53e8039070 con 0x7f53dc004d40 2024-04-15T15:16:26.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.527+0000 7f53edffb640 1 -- 172.21.15.72:0/3669271931 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53e802c6b0 con 0x7f53dc004d40 2024-04-15T15:16:26.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.527+0000 7f53edffb640 1 -- 172.21.15.72:0/3669271931 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53e8009690 con 0x7f53dc004d40 2024-04-15T15:16:26.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.528+0000 7f53f57ba640 1 -- 172.21.15.72:0/3669271931 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53dc004d40 msgr2=0x7f53dc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.528+0000 7f53f57ba640 1 --2- 172.21.15.72:0/3669271931 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53dc004d40 0x7f53dc0051c0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f53e8009ac0 tx=0x7f53dc005830 comp rx=0 tx=0).stop 2024-04-15T15:16:26.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.528+0000 7f53ef7fe640 1 -- 172.21.15.72:0/3669271931 reap_dead start 2024-04-15T15:16:26.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.528+0000 7f53f57ba640 1 -- 172.21.15.72:0/3669271931 shutdown_connections 2024-04-15T15:16:26.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.528+0000 7f53f57ba640 1 -- 172.21.15.72:0/3669271931 >> 172.21.15.72:0/3669271931 conn(0x7f53f00d24d0 msgr2=0x7f53f00d28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:26.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.528+0000 7f53f57ba640 1 -- 172.21.15.72:0/3669271931 shutdown_connections 2024-04-15T15:16:26.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.528+0000 7f53f57ba640 1 -- 172.21.15.72:0/3669271931 wait complete. 2024-04-15T15:16:26.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.529+0000 7f53f57ba640 1 Processor -- start 2024-04-15T15:16:26.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.529+0000 7f53f57ba640 1 -- start start 2024-04-15T15:16:26.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.530+0000 7f53f57ba640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f53dc004d40 0x7f53f01f8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.530+0000 7f53f57ba640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f53f0143e80 0x7f53f0152010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.530+0000 7f53f57ba640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53f0152550 0x7f53f014fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.530+0000 7f53f57ba640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f53f00b59d0 con 0x7f53f0152550 2024-04-15T15:16:26.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.530+0000 7f53f57ba640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f53f006ecb0 con 0x7f53dc004d40 2024-04-15T15:16:26.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.530+0000 7f53f57ba640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f53f00b5380 con 0x7f53f0143e80 2024-04-15T15:16:26.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.530+0000 7f53ef7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53f0152550 0x7f53f014fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.530+0000 7f53ef7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53f0152550 0x7f53f014fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43130/0 (socket says 172.21.15.72:43130) 2024-04-15T15:16:26.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.530+0000 7f53ef7fe640 1 -- 172.21.15.72:0/3435123757 learned_addr learned my addr 172.21.15.72:0/3435123757 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:26.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.530+0000 7f53ef7fe640 1 -- 172.21.15.72:0/3435123757 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f53f0143e80 msgr2=0x7f53f0152010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.530+0000 7f53eeffd640 1 --2- 172.21.15.72:0/3435123757 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f53dc004d40 0x7f53f01f8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.530+0000 7f53ee7fc640 1 --2- 172.21.15.72:0/3435123757 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f53f0143e80 0x7f53f0152010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.530+0000 7f53ef7fe640 1 --2- 172.21.15.72:0/3435123757 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f53f0143e80 0x7f53f0152010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.530+0000 7f53ef7fe640 1 -- 172.21.15.72:0/3435123757 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f53dc004d40 msgr2=0x7f53f01f8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.530+0000 7f53ef7fe640 1 --2- 172.21.15.72:0/3435123757 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f53dc004d40 0x7f53f01f8380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.530+0000 7f53ef7fe640 1 -- 172.21.15.72:0/3435123757 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53f0152a90 con 0x7f53f0152550 2024-04-15T15:16:26.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.530+0000 7f53ef7fe640 1 --2- 172.21.15.72:0/3435123757 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53f0152550 0x7f53f014fa40 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f53e8009ac0 tx=0x7f53e8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:26.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.530+0000 7f53e77fe640 1 -- 172.21.15.72:0/3435123757 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53e8039070 con 0x7f53f0152550 2024-04-15T15:16:26.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.531+0000 7f53e77fe640 1 -- 172.21.15.72:0/3435123757 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53e8043070 con 0x7f53f0152550 2024-04-15T15:16:26.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.531+0000 7f53f57ba640 1 -- 172.21.15.72:0/3435123757 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53f0152d70 con 0x7f53f0152550 2024-04-15T15:16:26.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.531+0000 7f53e77fe640 1 -- 172.21.15.72:0/3435123757 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53e8003850 con 0x7f53f0152550 2024-04-15T15:16:26.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.531+0000 7f53f57ba640 1 -- 172.21.15.72:0/3435123757 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53f01f7b80 con 0x7f53f0152550 2024-04-15T15:16:26.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.532+0000 7f53f57ba640 1 -- 172.21.15.72:0/3435123757 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53f00b6700 con 0x7f53f0152550 2024-04-15T15:16:26.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.532+0000 7f53e77fe640 1 -- 172.21.15.72:0/3435123757 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f53e80039f0 con 0x7f53f0152550 2024-04-15T15:16:26.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.532+0000 7f53e77fe640 1 --2- 172.21.15.72:0/3435123757 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f53d006c9b0 0x7f53d006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.533+0000 7f53e77fe640 1 -- 172.21.15.72:0/3435123757 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f53e80acbe0 con 0x7f53f0152550 2024-04-15T15:16:26.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.533+0000 7f53eeffd640 1 --2- 172.21.15.72:0/3435123757 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f53d006c9b0 0x7f53d006ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.533+0000 7f53eeffd640 1 --2- 172.21.15.72:0/3435123757 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f53d006c9b0 0x7f53d006ee70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f53e00013d0 tx=0x7f53e0006040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:26.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.537+0000 7f53e77fe640 1 -- 172.21.15.72:0/3435123757 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f53f00b6700 con 0x7f53f0152550 2024-04-15T15:16:26.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.712+0000 7f53f57ba640 1 -- 172.21.15.72:0/3435123757 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f53f006e650 con 0x7f53f0152550 2024-04-15T15:16:26.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.712+0000 7f53e77fe640 1 -- 172.21.15.72:0/3435123757 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v38) ==== 93+0+10 (secure 0 0 0) 0x7f53e807af20 con 0x7f53f0152550 2024-04-15T15:16:26.714 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-15T15:16:26.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.714+0000 7f53f57ba640 1 -- 172.21.15.72:0/3435123757 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f53d006c9b0 msgr2=0x7f53d006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.714+0000 7f53f57ba640 1 --2- 172.21.15.72:0/3435123757 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f53d006c9b0 0x7f53d006ee70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f53e00013d0 tx=0x7f53e0006040 comp rx=0 tx=0).stop 2024-04-15T15:16:26.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.714+0000 7f53f57ba640 1 -- 172.21.15.72:0/3435123757 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53f0152550 msgr2=0x7f53f014fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.714+0000 7f53f57ba640 1 --2- 172.21.15.72:0/3435123757 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53f0152550 0x7f53f014fa40 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f53e8009ac0 tx=0x7f53e8004730 comp rx=0 tx=0).stop 2024-04-15T15:16:26.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.714+0000 7f53f57ba640 1 -- 172.21.15.72:0/3435123757 shutdown_connections 2024-04-15T15:16:26.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.714+0000 7f53f57ba640 1 --2- 172.21.15.72:0/3435123757 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f53d006c9b0 0x7f53d006ee70 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.714+0000 7f53f57ba640 1 --2- 172.21.15.72:0/3435123757 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f53f0152550 0x7f53f014fa40 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.714+0000 7f53f57ba640 1 --2- 172.21.15.72:0/3435123757 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f53f0143e80 0x7f53f0152010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.714+0000 7f53f57ba640 1 --2- 172.21.15.72:0/3435123757 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f53dc004d40 0x7f53f01f8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.715+0000 7f53f57ba640 1 -- 172.21.15.72:0/3435123757 >> 172.21.15.72:0/3435123757 conn(0x7f53f00d24d0 msgr2=0x7f53f006f730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:26.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.715+0000 7f53f57ba640 1 -- 172.21.15.72:0/3435123757 shutdown_connections 2024-04-15T15:16:26.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.715+0000 7f53f57ba640 1 -- 172.21.15.72:0/3435123757 wait complete. 2024-04-15T15:16:26.727 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:16:26.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.833+0000 7f35fcb96640 1 Processor -- start 2024-04-15T15:16:26.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.833+0000 7f35fcb96640 1 -- start start 2024-04-15T15:16:26.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.834+0000 7f35fcb96640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f35f8145f10 0x7f35f80d0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.834+0000 7f35fcb96640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f35f806c1f0 0x7f35f806ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.834+0000 7f35fcb96640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f35f806f1a0 0x7f35f8160240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.834+0000 7f35fcb96640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f35f81462f0 con 0x7f35f806f1a0 2024-04-15T15:16:26.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.834+0000 7f35fcb96640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f35f806d290 con 0x7f35f8145f10 2024-04-15T15:16:26.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.834+0000 7f35fcb96640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f35f806d720 con 0x7f35f806c1f0 2024-04-15T15:16:26.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.834+0000 7f35f6d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f35f806f1a0 0x7f35f8160240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.834+0000 7f35f6d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f35f806f1a0 0x7f35f8160240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43136/0 (socket says 172.21.15.72:43136) 2024-04-15T15:16:26.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.834+0000 7f35f6d9d640 1 -- 172.21.15.72:0/3689967535 learned_addr learned my addr 172.21.15.72:0/3689967535 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:26.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.834+0000 7f35f659c640 1 --2- 172.21.15.72:0/3689967535 >> v2:172.21.15.175:3300/0 conn(0x7f35f8145f10 0x7f35f80d0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.834+0000 7f35edd9b640 1 --2- 172.21.15.72:0/3689967535 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f35f806c1f0 0x7f35f806ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.834+0000 7f35f6d9d640 1 -- 172.21.15.72:0/3689967535 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f35f806c1f0 msgr2=0x7f35f806ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:26.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.834+0000 7f35f6d9d640 1 --2- 172.21.15.72:0/3689967535 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f35f806c1f0 0x7f35f806ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.834+0000 7f35f6d9d640 1 -- 172.21.15.72:0/3689967535 >> v2:172.21.15.175:3300/0 conn(0x7f35f8145f10 msgr2=0x7f35f80d0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:26.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.834+0000 7f35f6d9d640 1 --2- 172.21.15.72:0/3689967535 >> v2:172.21.15.175:3300/0 conn(0x7f35f8145f10 0x7f35f80d0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.834+0000 7f35f6d9d640 1 -- 172.21.15.72:0/3689967535 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35f806c680 con 0x7f35f806f1a0 2024-04-15T15:16:26.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.835+0000 7f35f6d9d640 1 --2- 172.21.15.72:0/3689967535 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35f806f1a0 0x7f35f8160240 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f35f80727c0 tx=0x7f35f002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3641560e1decc4ff server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:26.837 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.835+0000 7f35f5d9b640 1 -- 172.21.15.72:0/3689967535 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f002ca70 con 0x7f35f806f1a0 2024-04-15T15:16:26.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.835+0000 7f35f5d9b640 1 -- 172.21.15.72:0/3689967535 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35f002cc10 con 0x7f35f806f1a0 2024-04-15T15:16:26.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.835+0000 7f35f5d9b640 1 -- 172.21.15.72:0/3689967535 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35f00343f0 con 0x7f35f806f1a0 2024-04-15T15:16:26.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.835+0000 7f35fcb96640 1 -- 172.21.15.72:0/3689967535 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35f806f1a0 msgr2=0x7f35f8160240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.835+0000 7f35fcb96640 1 --2- 172.21.15.72:0/3689967535 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35f806f1a0 0x7f35f8160240 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f35f80727c0 tx=0x7f35f002bed0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.835+0000 7f35fcb96640 1 -- 172.21.15.72:0/3689967535 shutdown_connections 2024-04-15T15:16:26.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.835+0000 7f35fcb96640 1 --2- 172.21.15.72:0/3689967535 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35f806f1a0 0x7f35f8160240 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.835+0000 7f35fcb96640 1 --2- 172.21.15.72:0/3689967535 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f35f806c1f0 0x7f35f806ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.835+0000 7f35fcb96640 1 --2- 172.21.15.72:0/3689967535 >> v2:172.21.15.175:3300/0 conn(0x7f35f8145f10 0x7f35f80d0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.835+0000 7f35fcb96640 1 -- 172.21.15.72:0/3689967535 >> 172.21.15.72:0/3689967535 conn(0x7f35f80d13a0 msgr2=0x7f35f80d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:26.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.836+0000 7f35fcb96640 1 -- 172.21.15.72:0/3689967535 shutdown_connections 2024-04-15T15:16:26.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.836+0000 7f35fcb96640 1 -- 172.21.15.72:0/3689967535 wait complete. 2024-04-15T15:16:26.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.837+0000 7f35fcb96640 1 Processor -- start 2024-04-15T15:16:26.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.837+0000 7f35fcb96640 1 -- start start 2024-04-15T15:16:26.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.838+0000 7f35fcb96640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35f806f1a0 0x7f35f813d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.838+0000 7f35fcb96640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35f8145f10 0x7f35f81416c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.838+0000 7f35fcb96640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f35f8141c00 0x7f35f81f2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.838+0000 7f35fcb96640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f35f806cf20 con 0x7f35f806f1a0 2024-04-15T15:16:26.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.838+0000 7f35fcb96640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f35f806cc40 con 0x7f35f8145f10 2024-04-15T15:16:26.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.838+0000 7f35fcb96640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f35f806f600 con 0x7f35f8141c00 2024-04-15T15:16:26.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.838+0000 7f35f659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35f806f1a0 0x7f35f813d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.838+0000 7f35f659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35f806f1a0 0x7f35f813d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43152/0 (socket says 172.21.15.72:43152) 2024-04-15T15:16:26.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.838+0000 7f35f659c640 1 -- 172.21.15.72:0/178675780 learned_addr learned my addr 172.21.15.72:0/178675780 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:26.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.838+0000 7f35f659c640 1 -- 172.21.15.72:0/178675780 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f35f8141c00 msgr2=0x7f35f81f2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.838+0000 7f35f659c640 1 --2- 172.21.15.72:0/178675780 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f35f8141c00 0x7f35f81f2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.838+0000 7f35f659c640 1 -- 172.21.15.72:0/178675780 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35f8145f10 msgr2=0x7f35f81416c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.838+0000 7f35f659c640 1 --2- 172.21.15.72:0/178675780 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35f8145f10 0x7f35f81416c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.838+0000 7f35f659c640 1 -- 172.21.15.72:0/178675780 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35f813c730 con 0x7f35f806f1a0 2024-04-15T15:16:26.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.838+0000 7f35f659c640 1 --2- 172.21.15.72:0/178675780 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35f806f1a0 0x7f35f813d7c0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f35e800d800 tx=0x7f35e800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:26.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.838+0000 7f35ef7fe640 1 -- 172.21.15.72:0/178675780 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e800b6c0 con 0x7f35f806f1a0 2024-04-15T15:16:26.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.838+0000 7f35ef7fe640 1 -- 172.21.15.72:0/178675780 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35e8019070 con 0x7f35f806f1a0 2024-04-15T15:16:26.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.838+0000 7f35ef7fe640 1 -- 172.21.15.72:0/178675780 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e800ba80 con 0x7f35f806f1a0 2024-04-15T15:16:26.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.838+0000 7f35fcb96640 1 -- 172.21.15.72:0/178675780 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35f813b6b0 con 0x7f35f806f1a0 2024-04-15T15:16:26.842 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.839+0000 7f35fcb96640 1 -- 172.21.15.72:0/178675780 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35f813bf50 con 0x7f35f806f1a0 2024-04-15T15:16:26.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.839+0000 7f35fcb96640 1 -- 172.21.15.72:0/178675780 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35f8075c50 con 0x7f35f806f1a0 2024-04-15T15:16:26.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.840+0000 7f35ef7fe640 1 -- 172.21.15.72:0/178675780 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f35e800bc20 con 0x7f35f806f1a0 2024-04-15T15:16:26.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.840+0000 7f35ef7fe640 1 --2- 172.21.15.72:0/178675780 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f35d806c980 0x7f35d806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.840+0000 7f35ef7fe640 1 -- 172.21.15.72:0/178675780 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f35e808c210 con 0x7f35f806f1a0 2024-04-15T15:16:26.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.840+0000 7f35edd9b640 1 --2- 172.21.15.72:0/178675780 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f35d806c980 0x7f35d806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.841+0000 7f35edd9b640 1 --2- 172.21.15.72:0/178675780 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f35d806c980 0x7f35d806ee40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f35f81408d0 tx=0x7f35e0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:26.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:26.845+0000 7f35ef7fe640 1 -- 172.21.15.72:0/178675780 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f35e805a4c0 con 0x7f35f806f1a0 2024-04-15T15:16:27.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.018+0000 7f35fcb96640 1 -- 172.21.15.72:0/178675780 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f35f80d2130 con 0x7f35f806f1a0 2024-04-15T15:16:27.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.018+0000 7f35ef7fe640 1 -- 172.21.15.72:0/178675780 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v38) ==== 104+0+11 (secure 0 0 0) 0x7f35e8059dc0 con 0x7f35f806f1a0 2024-04-15T15:16:27.020 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:16:27.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.020+0000 7f35fcb96640 1 -- 172.21.15.72:0/178675780 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f35d806c980 msgr2=0x7f35d806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.020+0000 7f35fcb96640 1 --2- 172.21.15.72:0/178675780 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f35d806c980 0x7f35d806ee40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f35f81408d0 tx=0x7f35e0008040 comp rx=0 tx=0).stop 2024-04-15T15:16:27.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.020+0000 7f35fcb96640 1 -- 172.21.15.72:0/178675780 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35f806f1a0 msgr2=0x7f35f813d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.020+0000 7f35fcb96640 1 --2- 172.21.15.72:0/178675780 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35f806f1a0 0x7f35f813d7c0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f35e800d800 tx=0x7f35e800dcc0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.020+0000 7f35fcb96640 1 -- 172.21.15.72:0/178675780 shutdown_connections 2024-04-15T15:16:27.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.020+0000 7f35fcb96640 1 --2- 172.21.15.72:0/178675780 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f35d806c980 0x7f35d806ee40 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.020+0000 7f35fcb96640 1 --2- 172.21.15.72:0/178675780 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f35f8141c00 0x7f35f81f2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.020+0000 7f35fcb96640 1 --2- 172.21.15.72:0/178675780 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f35f8145f10 0x7f35f81416c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.020+0000 7f35fcb96640 1 --2- 172.21.15.72:0/178675780 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35f806f1a0 0x7f35f813d7c0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.020+0000 7f35fcb96640 1 -- 172.21.15.72:0/178675780 >> 172.21.15.72:0/178675780 conn(0x7f35f80d13a0 msgr2=0x7f35f80b0f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:27.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.021+0000 7f35fcb96640 1 -- 172.21.15.72:0/178675780 shutdown_connections 2024-04-15T15:16:27.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.021+0000 7f35fcb96640 1 -- 172.21.15.72:0/178675780 wait complete. 2024-04-15T15:16:27.033 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:16:27.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.137+0000 7fe21feeb640 1 Processor -- start 2024-04-15T15:16:27.139 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.138+0000 7fe21feeb640 1 -- start start 2024-04-15T15:16:27.140 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.138+0000 7fe21feeb640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe21815e610 0x7fe2180d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.138+0000 7fe21feeb640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe21815e9f0 0x7fe2180d3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.138+0000 7fe21feeb640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe21806c400 0x7fe21806c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.138+0000 7fe21feeb640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fe2180d31d0 con 0x7fe21815e9f0 2024-04-15T15:16:27.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.138+0000 7fe21feeb640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fe2180d2320 con 0x7fe21806c400 2024-04-15T15:16:27.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.138+0000 7fe21feeb640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fe2180d25c0 con 0x7fe21815e610 2024-04-15T15:16:27.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.139+0000 7fe21d486640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe21815e9f0 0x7fe2180d3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.139+0000 7fe21d486640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe21815e9f0 0x7fe2180d3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43154/0 (socket says 172.21.15.72:43154) 2024-04-15T15:16:27.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.139+0000 7fe21d486640 1 -- 172.21.15.72:0/3340016955 learned_addr learned my addr 172.21.15.72:0/3340016955 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:27.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.139+0000 7fe21e488640 1 --2- 172.21.15.72:0/3340016955 >> v2:172.21.15.175:3300/0 conn(0x7fe21806c400 0x7fe21806c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.139+0000 7fe21dc87640 1 --2- 172.21.15.72:0/3340016955 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe21815e610 0x7fe2180d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.139+0000 7fe21d486640 1 -- 172.21.15.72:0/3340016955 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe21815e610 msgr2=0x7fe2180d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:27.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.139+0000 7fe21d486640 1 --2- 172.21.15.72:0/3340016955 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe21815e610 0x7fe2180d0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.139+0000 7fe21d486640 1 -- 172.21.15.72:0/3340016955 >> v2:172.21.15.175:3300/0 conn(0x7fe21806c400 msgr2=0x7fe21806c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:27.141 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.139+0000 7fe21d486640 1 --2- 172.21.15.72:0/3340016955 >> v2:172.21.15.175:3300/0 conn(0x7fe21806c400 0x7fe21806c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.142 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.139+0000 7fe21d486640 1 -- 172.21.15.72:0/3340016955 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2180d28a0 con 0x7fe21815e9f0 2024-04-15T15:16:27.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.140+0000 7fe21d486640 1 --2- 172.21.15.72:0/3340016955 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe21815e9f0 0x7fe2180d3660 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fe21815ee30 tx=0x7fe20802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77f598eb5a4079de server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:27.143 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.140+0000 7fe21cc85640 1 -- 172.21.15.72:0/3340016955 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe208034070 con 0x7fe21815e9f0 2024-04-15T15:16:27.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.140+0000 7fe21cc85640 1 -- 172.21.15.72:0/3340016955 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe208030860 con 0x7fe21815e9f0 2024-04-15T15:16:27.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.140+0000 7fe21cc85640 1 -- 172.21.15.72:0/3340016955 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe208030c30 con 0x7fe21815e9f0 2024-04-15T15:16:27.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.140+0000 7fe21feeb640 1 -- 172.21.15.72:0/3340016955 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe21815e9f0 msgr2=0x7fe2180d3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.140+0000 7fe21feeb640 1 --2- 172.21.15.72:0/3340016955 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe21815e9f0 0x7fe2180d3660 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fe21815ee30 tx=0x7fe20802b9a0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.140+0000 7fe21feeb640 1 -- 172.21.15.72:0/3340016955 shutdown_connections 2024-04-15T15:16:27.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.140+0000 7fe21feeb640 1 --2- 172.21.15.72:0/3340016955 >> v2:172.21.15.175:3300/0 conn(0x7fe21806c400 0x7fe21806c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.140+0000 7fe21feeb640 1 --2- 172.21.15.72:0/3340016955 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe21815e9f0 0x7fe2180d3660 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.140+0000 7fe21feeb640 1 --2- 172.21.15.72:0/3340016955 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe21815e610 0x7fe2180d0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.144 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.140+0000 7fe21feeb640 1 -- 172.21.15.72:0/3340016955 >> 172.21.15.72:0/3340016955 conn(0x7fe2180d4750 msgr2=0x7fe2180d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:27.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.141+0000 7fe21feeb640 1 -- 172.21.15.72:0/3340016955 shutdown_connections 2024-04-15T15:16:27.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.141+0000 7fe21feeb640 1 -- 172.21.15.72:0/3340016955 wait complete. 2024-04-15T15:16:27.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.142+0000 7fe21feeb640 1 Processor -- start 2024-04-15T15:16:27.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.142+0000 7fe21feeb640 1 -- start start 2024-04-15T15:16:27.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.142+0000 7fe21feeb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe21806c400 0x7fe2180ca830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.142+0000 7fe21feeb640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe21815e610 0x7fe218160c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.142+0000 7fe21feeb640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe21815e9f0 0x7fe218161180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.142+0000 7fe21feeb640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe21806d130 con 0x7fe21806c400 2024-04-15T15:16:27.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.142+0000 7fe21feeb640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fe2180d2e60 con 0x7fe21815e9f0 2024-04-15T15:16:27.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.142+0000 7fe21feeb640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fe2180d0bd0 con 0x7fe21815e610 2024-04-15T15:16:27.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.142+0000 7fe21dc87640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe21806c400 0x7fe2180ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.142+0000 7fe21dc87640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe21806c400 0x7fe2180ca830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43172/0 (socket says 172.21.15.72:43172) 2024-04-15T15:16:27.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.142+0000 7fe21dc87640 1 -- 172.21.15.72:0/2427540303 learned_addr learned my addr 172.21.15.72:0/2427540303 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:27.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.143+0000 7fe21e488640 1 --2- 172.21.15.72:0/2427540303 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe21815e9f0 0x7fe218161180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.143+0000 7fe21d486640 1 --2- 172.21.15.72:0/2427540303 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe21815e610 0x7fe218160c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.143+0000 7fe21dc87640 1 -- 172.21.15.72:0/2427540303 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe21815e610 msgr2=0x7fe218160c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.143+0000 7fe21dc87640 1 --2- 172.21.15.72:0/2427540303 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe21815e610 0x7fe218160c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.143+0000 7fe21dc87640 1 -- 172.21.15.72:0/2427540303 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe21815e9f0 msgr2=0x7fe218161180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.143+0000 7fe21dc87640 1 --2- 172.21.15.72:0/2427540303 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe21815e9f0 0x7fe218161180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.143+0000 7fe21dc87640 1 -- 172.21.15.72:0/2427540303 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2181ea000 con 0x7fe21806c400 2024-04-15T15:16:27.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.143+0000 7fe21e488640 1 --2- 172.21.15.72:0/2427540303 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe21815e9f0 0x7fe218161180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:27.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.143+0000 7fe21d486640 1 --2- 172.21.15.72:0/2427540303 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe21815e610 0x7fe218160c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:27.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.143+0000 7fe21dc87640 1 --2- 172.21.15.72:0/2427540303 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe21806c400 0x7fe2180ca830 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fe21400d830 tx=0x7fe21400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:27.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.143+0000 7fe20e7fc640 1 -- 172.21.15.72:0/2427540303 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe21400b6c0 con 0x7fe21806c400 2024-04-15T15:16:27.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.143+0000 7fe20e7fc640 1 -- 172.21.15.72:0/2427540303 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe214019070 con 0x7fe21806c400 2024-04-15T15:16:27.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.143+0000 7fe20e7fc640 1 -- 172.21.15.72:0/2427540303 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe21400ba80 con 0x7fe21806c400 2024-04-15T15:16:27.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.143+0000 7fe21feeb640 1 -- 172.21.15.72:0/2427540303 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2180cc000 con 0x7fe21806c400 2024-04-15T15:16:27.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.143+0000 7fe21feeb640 1 -- 172.21.15.72:0/2427540303 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2180cc5c0 con 0x7fe21806c400 2024-04-15T15:16:27.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.144+0000 7fe21feeb640 1 -- 172.21.15.72:0/2427540303 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe218075c50 con 0x7fe21806c400 2024-04-15T15:16:27.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.145+0000 7fe20e7fc640 1 -- 172.21.15.72:0/2427540303 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fe21401e460 con 0x7fe21806c400 2024-04-15T15:16:27.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.145+0000 7fe20e7fc640 1 --2- 172.21.15.72:0/2427540303 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe1fc06c9b0 0x7fe1fc06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.145+0000 7fe21d486640 1 --2- 172.21.15.72:0/2427540303 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe1fc06c9b0 0x7fe1fc06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.145+0000 7fe20e7fc640 1 -- 172.21.15.72:0/2427540303 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fe21408c330 con 0x7fe21806c400 2024-04-15T15:16:27.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.146+0000 7fe21d486640 1 --2- 172.21.15.72:0/2427540303 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe1fc06c9b0 0x7fe1fc06ee70 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe208036000 tx=0x7fe20803b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:27.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.150+0000 7fe20e7fc640 1 -- 172.21.15.72:0/2427540303 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe21405a670 con 0x7fe21806c400 2024-04-15T15:16:27.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.325+0000 7fe21feeb640 1 -- 172.21.15.72:0/2427540303 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe2180cf740 con 0x7fe21806c400 2024-04-15T15:16:27.327 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.326+0000 7fe20e7fc640 1 -- 172.21.15.72:0/2427540303 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v38) ==== 100+0+11 (secure 0 0 0) 0x7fe214059f70 con 0x7fe21806c400 2024-04-15T15:16:27.328 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-04-15T15:16:27.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.328+0000 7fe21feeb640 1 -- 172.21.15.72:0/2427540303 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe1fc06c9b0 msgr2=0x7fe1fc06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.328+0000 7fe21feeb640 1 --2- 172.21.15.72:0/2427540303 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe1fc06c9b0 0x7fe1fc06ee70 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe208036000 tx=0x7fe20803b040 comp rx=0 tx=0).stop 2024-04-15T15:16:27.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.328+0000 7fe21feeb640 1 -- 172.21.15.72:0/2427540303 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe21806c400 msgr2=0x7fe2180ca830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.328+0000 7fe21feeb640 1 --2- 172.21.15.72:0/2427540303 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe21806c400 0x7fe2180ca830 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fe21400d830 tx=0x7fe21400dcf0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.328+0000 7fe21feeb640 1 -- 172.21.15.72:0/2427540303 shutdown_connections 2024-04-15T15:16:27.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.328+0000 7fe21feeb640 1 --2- 172.21.15.72:0/2427540303 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe1fc06c9b0 0x7fe1fc06ee70 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.328+0000 7fe21feeb640 1 --2- 172.21.15.72:0/2427540303 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe21815e9f0 0x7fe218161180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.328+0000 7fe21feeb640 1 --2- 172.21.15.72:0/2427540303 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe21815e610 0x7fe218160c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.328+0000 7fe21feeb640 1 --2- 172.21.15.72:0/2427540303 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe21806c400 0x7fe2180ca830 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.328+0000 7fe21feeb640 1 -- 172.21.15.72:0/2427540303 >> 172.21.15.72:0/2427540303 conn(0x7fe2180d4750 msgr2=0x7fe2180b4490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:27.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.328+0000 7fe21feeb640 1 -- 172.21.15.72:0/2427540303 shutdown_connections 2024-04-15T15:16:27.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.328+0000 7fe21feeb640 1 -- 172.21.15.72:0/2427540303 wait complete. 2024-04-15T15:16:27.341 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:16:27.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.443+0000 7f88b1f00640 1 Processor -- start 2024-04-15T15:16:27.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.443+0000 7f88b1f00640 1 -- start start 2024-04-15T15:16:27.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.444+0000 7f88b1f00640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f88ac0ce2f0 0x7f88ac15c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.444+0000 7f88b1f00640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88ac06bbb0 con 0x7f88ac0d08f0 2024-04-15T15:16:27.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.444+0000 7f88b1f00640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88ac06be90 con 0x7f88ac0cc7c0 2024-04-15T15:16:27.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.444+0000 7f88b1f00640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f88ac06b960 con 0x7f88ac0ce2f0 2024-04-15T15:16:27.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.444+0000 7f88aaffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f88ac0d08f0 0x7f88ac0cc0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60304/0 (socket says 172.21.15.72:60304) 2024-04-15T15:16:27.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.444+0000 7f88aaffd640 1 -- 172.21.15.72:0/2228417307 learned_addr learned my addr 172.21.15.72:0/2228417307 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:27.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.444+0000 7f88ab7fe640 1 --2- 172.21.15.72:0/2228417307 >> v2:172.21.15.175:3300/0 conn(0x7f88ac0ce2f0 0x7f88ac15c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.444+0000 7f88aa7fc640 1 -- 172.21.15.72:0/2228417307 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2486488108 0 0) 0x7f88ac06bbb0 con 0x7f88ac0d08f0 2024-04-15T15:16:27.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.444+0000 7f88aa7fc640 1 -- 172.21.15.72:0/2228417307 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8894003520 con 0x7f88ac0d08f0 2024-04-15T15:16:27.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.445+0000 7f88aa7fc640 1 -- 172.21.15.72:0/2228417307 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1743448132 0 0) 0x7f8894003520 con 0x7f88ac0d08f0 2024-04-15T15:16:27.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.445+0000 7f88aa7fc640 1 -- 172.21.15.72:0/2228417307 >> v2:172.21.15.175:3300/0 conn(0x7f88ac0ce2f0 msgr2=0x7f88ac15c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:27.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.445+0000 7f88aa7fc640 1 --2- 172.21.15.72:0/2228417307 >> v2:172.21.15.175:3300/0 conn(0x7f88ac0ce2f0 0x7f88ac15c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.445+0000 7f88aa7fc640 1 -- 172.21.15.72:0/2228417307 >> v1:172.21.15.175:6789/0 conn(0x7f88ac0cc7c0 legacy=0x7f88ac155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.445+0000 7f88aa7fc640 1 -- 172.21.15.72:0/2228417307 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88ac06c170 con 0x7f88ac0d08f0 2024-04-15T15:16:27.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.445+0000 7f88aa7fc640 1 -- 172.21.15.72:0/2228417307 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f88a0001670 con 0x7f88ac0d08f0 2024-04-15T15:16:27.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.445+0000 7f88aa7fc640 1 -- 172.21.15.72:0/2228417307 >> v1:172.21.15.72:6789/0 conn(0x7f88ac0d08f0 legacy=0x7f88ac0cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.445+0000 7f88aa7fc640 1 --2- 172.21.15.72:0/2228417307 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f88940039b0 0x7f8894003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.445+0000 7f88aa7fc640 1 --2- 172.21.15.72:0/2228417307 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8894004800 0x7f8894004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.445+0000 7f88aa7fc640 1 --2- 172.21.15.72:0/2228417307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88940051c0 0x7f889400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.445+0000 7f88aa7fc640 1 -- 172.21.15.72:0/2228417307 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac06b960 con 0x7f88940051c0 2024-04-15T15:16:27.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.445+0000 7f88aa7fc640 1 -- 172.21.15.72:0/2228417307 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f889400c090 con 0x7f8894004800 2024-04-15T15:16:27.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.445+0000 7f88aa7fc640 1 -- 172.21.15.72:0/2228417307 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f889400c290 con 0x7f88940039b0 2024-04-15T15:16:27.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.445+0000 7f88abfff640 1 --2- 172.21.15.72:0/2228417307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88940051c0 0x7f889400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.445+0000 7f88aaffd640 1 --2- 172.21.15.72:0/2228417307 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8894004800 0x7f8894004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.445+0000 7f88ab7fe640 1 --2- 172.21.15.72:0/2228417307 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f88940039b0 0x7f8894003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.445+0000 7f88abfff640 1 -- 172.21.15.72:0/2228417307 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f88940039b0 msgr2=0x7f8894003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:27.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.445+0000 7f88abfff640 1 --2- 172.21.15.72:0/2228417307 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f88940039b0 0x7f8894003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.445+0000 7f88abfff640 1 -- 172.21.15.72:0/2228417307 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8894004800 msgr2=0x7f8894004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:27.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.445+0000 7f88abfff640 1 --2- 172.21.15.72:0/2228417307 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8894004800 0x7f8894004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.445+0000 7f88abfff640 1 -- 172.21.15.72:0/2228417307 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8894003520 con 0x7f88940051c0 2024-04-15T15:16:27.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.446+0000 7f88abfff640 1 -- 172.21.15.72:0/2228417307 reap_dead start 2024-04-15T15:16:27.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.446+0000 7f88abfff640 1 --2- 172.21.15.72:0/2228417307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88940051c0 0x7f889400b9c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f88a4001720 tx=0x7f8894005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41c7e1304c03f1f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:27.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.446+0000 7f88aa7fc640 1 -- 172.21.15.72:0/2228417307 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a4001370 con 0x7f88940051c0 2024-04-15T15:16:27.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.446+0000 7f88aa7fc640 1 -- 172.21.15.72:0/2228417307 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88a40052d0 con 0x7f88940051c0 2024-04-15T15:16:27.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.446+0000 7f88aa7fc640 1 -- 172.21.15.72:0/2228417307 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88a4003830 con 0x7f88940051c0 2024-04-15T15:16:27.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.446+0000 7f88b1f00640 1 -- 172.21.15.72:0/2228417307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88940051c0 msgr2=0x7f889400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.446+0000 7f88b1f00640 1 --2- 172.21.15.72:0/2228417307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88940051c0 0x7f889400b9c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f88a4001720 tx=0x7f8894005830 comp rx=0 tx=0).stop 2024-04-15T15:16:27.450 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.446+0000 7f88b1f00640 1 -- 172.21.15.72:0/2228417307 shutdown_connections 2024-04-15T15:16:27.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.446+0000 7f88b1f00640 1 --2- 172.21.15.72:0/2228417307 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88940051c0 0x7f889400b9c0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.447+0000 7f88b1f00640 1 -- 172.21.15.72:0/2228417307 >> 172.21.15.72:0/2228417307 conn(0x7f88ac0b31d0 msgr2=0x7f88ac0b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:27.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.447+0000 7f88b1f00640 1 -- 172.21.15.72:0/2228417307 shutdown_connections 2024-04-15T15:16:27.451 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.447+0000 7f88b1f00640 1 -- 172.21.15.72:0/2228417307 wait complete. 2024-04-15T15:16:27.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.448+0000 7f88b1f00640 1 Processor -- start 2024-04-15T15:16:27.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.448+0000 7f88b1f00640 1 -- start start 2024-04-15T15:16:27.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.449+0000 7f88b1f00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88940051c0 0x7f88ac153b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.449+0000 7f88b1f00640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f88ac155260 0x7f88ac0b2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.449+0000 7f88b1f00640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f88ac148720 0x7f88ac0ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.449+0000 7f88b1f00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac0ce850 con 0x7f88940051c0 2024-04-15T15:16:27.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.449+0000 7f88b1f00640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f88ac0ce6d0 con 0x7f88ac155260 2024-04-15T15:16:27.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.449+0000 7f88b1f00640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f88ac0ce9d0 con 0x7f88ac148720 2024-04-15T15:16:27.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.449+0000 7f88ab7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88940051c0 0x7f88ac153b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.449+0000 7f88ab7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88940051c0 0x7f88ac153b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43188/0 (socket says 172.21.15.72:43188) 2024-04-15T15:16:27.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.449+0000 7f88ab7fe640 1 -- 172.21.15.72:0/1518467095 learned_addr learned my addr 172.21.15.72:0/1518467095 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:27.454 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.449+0000 7f88ab7fe640 1 -- 172.21.15.72:0/1518467095 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f88ac148720 msgr2=0x7f88ac0ccc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.449+0000 7f88ab7fe640 1 --2- 172.21.15.72:0/1518467095 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f88ac148720 0x7f88ac0ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.449+0000 7f88ab7fe640 1 -- 172.21.15.72:0/1518467095 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f88ac155260 msgr2=0x7f88ac0b2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.449+0000 7f88ab7fe640 1 --2- 172.21.15.72:0/1518467095 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f88ac155260 0x7f88ac0b2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.449+0000 7f88ab7fe640 1 -- 172.21.15.72:0/1518467095 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88ac0b2bc0 con 0x7f88940051c0 2024-04-15T15:16:27.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.449+0000 7f88ab7fe640 1 --2- 172.21.15.72:0/1518467095 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88940051c0 0x7f88ac153b00 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f889c007f00 tx=0x7f889c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:27.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.449+0000 7f8893fff640 1 -- 172.21.15.72:0/1518467095 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f889c01a020 con 0x7f88940051c0 2024-04-15T15:16:27.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.449+0000 7f8893fff640 1 -- 172.21.15.72:0/1518467095 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f889c016470 con 0x7f88940051c0 2024-04-15T15:16:27.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.449+0000 7f88b1f00640 1 -- 172.21.15.72:0/1518467095 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88ac1541c0 con 0x7f88940051c0 2024-04-15T15:16:27.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.449+0000 7f8893fff640 1 -- 172.21.15.72:0/1518467095 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f889c016860 con 0x7f88940051c0 2024-04-15T15:16:27.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.449+0000 7f88b1f00640 1 -- 172.21.15.72:0/1518467095 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88ac154780 con 0x7f88940051c0 2024-04-15T15:16:27.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.450+0000 7f88b1f00640 1 -- 172.21.15.72:0/1518467095 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88ac075c50 con 0x7f88940051c0 2024-04-15T15:16:27.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.451+0000 7f8893fff640 1 -- 172.21.15.72:0/1518467095 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f889c005d10 con 0x7f88940051c0 2024-04-15T15:16:27.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.451+0000 7f8893fff640 1 --2- 172.21.15.72:0/1518467095 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f889806c980 0x7f889806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.451+0000 7f8893fff640 1 -- 172.21.15.72:0/1518467095 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f889c08cca0 con 0x7f88940051c0 2024-04-15T15:16:27.455 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.451+0000 7f88aaffd640 1 --2- 172.21.15.72:0/1518467095 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f889806c980 0x7f889806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.456 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.452+0000 7f88aaffd640 1 --2- 172.21.15.72:0/1518467095 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f889806c980 0x7f889806ee40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f88a0001eb0 tx=0x7f88a0000f30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:27.457 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.455+0000 7f8893fff640 1 -- 172.21.15.72:0/1518467095 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f889c091070 con 0x7f88940051c0 2024-04-15T15:16:27.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.643+0000 7f88b1f00640 1 -- 172.21.15.72:0/1518467095 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f88ac06cab0 con 0x7f88940051c0 2024-04-15T15:16:27.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.643+0000 7f8893fff640 1 -- 172.21.15.72:0/1518467095 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v38) ==== 80+0+0 (secure 0 0 0) 0x7f889c024030 con 0x7f88940051c0 2024-04-15T15:16:27.645 INFO:teuthology.orchestra.run.smithi072.stderr:listed 0 entries 2024-04-15T15:16:27.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.645+0000 7f88b1f00640 1 -- 172.21.15.72:0/1518467095 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f889806c980 msgr2=0x7f889806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.645+0000 7f88b1f00640 1 --2- 172.21.15.72:0/1518467095 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f889806c980 0x7f889806ee40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f88a0001eb0 tx=0x7f88a0000f30 comp rx=0 tx=0).stop 2024-04-15T15:16:27.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.645+0000 7f88b1f00640 1 -- 172.21.15.72:0/1518467095 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88940051c0 msgr2=0x7f88ac153b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.645+0000 7f88b1f00640 1 --2- 172.21.15.72:0/1518467095 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88940051c0 0x7f88ac153b00 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f889c007f00 tx=0x7f889c0115a0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.645+0000 7f88b1f00640 1 -- 172.21.15.72:0/1518467095 shutdown_connections 2024-04-15T15:16:27.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.645+0000 7f88b1f00640 1 --2- 172.21.15.72:0/1518467095 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f889806c980 0x7f889806ee40 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.645+0000 7f88b1f00640 1 --2- 172.21.15.72:0/1518467095 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f88ac148720 0x7f88ac0ccc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.645+0000 7f88b1f00640 1 --2- 172.21.15.72:0/1518467095 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f88ac155260 0x7f88ac0b2680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.645+0000 7f88b1f00640 1 --2- 172.21.15.72:0/1518467095 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f88940051c0 0x7f88ac153b00 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.645+0000 7f88b1f00640 1 -- 172.21.15.72:0/1518467095 >> 172.21.15.72:0/1518467095 conn(0x7f88ac0b31d0 msgr2=0x7f88ac072170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:27.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.645+0000 7f88b1f00640 1 -- 172.21.15.72:0/1518467095 shutdown_connections 2024-04-15T15:16:27.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:27.645+0000 7f88b1f00640 1 -- 172.21.15.72:0/1518467095 wait complete. 2024-04-15T15:16:27.658 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi175.front.sepia.ceph.com... 2024-04-15T15:16:27.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:27.658 DEBUG:teuthology.orchestra.run.smithi175:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:27.691 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi175.stderr:ceph-fuse[45005]: fuse finished with error 0 and tester_r 0 2024-04-15T15:16:27.693 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:16:33.794 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi175.front.sepia.ceph.com 2024-04-15T15:16:33.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:33.795 DEBUG:teuthology.orchestra.run.smithi175:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-15T15:16:33.815 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:16:33.815 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:16:33.834 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16951 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248808k,nr_inodes=812202,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-15T15:16:33.835 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.2 is not mounted at ubuntu@smithi072.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.2 2024-04-15T15:16:33.835 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi072.front.sepia.ceph.com 2024-04-15T15:16:33.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:33.836 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-15T15:16:33.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:16:33.894 INFO:teuthology.orchestra.run.smithi072.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-04-15T15:16:33.894 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:16:33.894 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:16:33.913 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20056 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-15T15:16:33.913 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.3 is not mounted at ubuntu@smithi175.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.3 2024-04-15T15:16:33.914 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi175.front.sepia.ceph.com 2024-04-15T15:16:33.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:33.914 DEBUG:teuthology.orchestra.run.smithi175:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-15T15:16:33.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:16:33.972 INFO:teuthology.orchestra.run.smithi175.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-04-15T15:16:33.973 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:16:34.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.084+0000 7fe7dc11c640 1 Processor -- start 2024-04-15T15:16:34.085 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.084+0000 7fe7dc11c640 1 -- start start 2024-04-15T15:16:34.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.085+0000 7fe7dc11c640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe7d415c7c0 0x7fe7d406e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.085+0000 7fe7dc11c640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe7d40af700 0x7fe7d406e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.085+0000 7fe7dc11c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7d40cffd0 con 0x7fe7d40ccb70 2024-04-15T15:16:34.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.085+0000 7fe7dc11c640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fe7d415cba0 con 0x7fe7d40af700 2024-04-15T15:16:34.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.085+0000 7fe7dc11c640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fe7d406dc10 con 0x7fe7d415c7c0 2024-04-15T15:16:34.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.085+0000 7fe7da6b9640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fe7d40ccb70 0x7fe7d40cd000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:55960/0 (socket says 172.21.15.72:55960) 2024-04-15T15:16:34.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.085+0000 7fe7da6b9640 1 -- 172.21.15.72:0/607393688 learned_addr learned my addr 172.21.15.72:0/607393688 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:34.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.085+0000 7fe7d9eb8640 1 --2- 172.21.15.72:0/607393688 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe7d415c7c0 0x7fe7d406e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.085+0000 7fe7d8eb6640 1 -- 172.21.15.72:0/607393688 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 298524481 0 0) 0x7fe7d40cffd0 con 0x7fe7d40ccb70 2024-04-15T15:16:34.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.085+0000 7fe7d96b7640 1 --2- 172.21.15.72:0/607393688 >> v2:172.21.15.175:3300/0 conn(0x7fe7d40af700 0x7fe7d406e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.085+0000 7fe7d8eb6640 1 -- 172.21.15.72:0/607393688 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7c0003520 con 0x7fe7d40ccb70 2024-04-15T15:16:34.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.085+0000 7fe7d8eb6640 1 -- 172.21.15.72:0/607393688 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1647584167 0 0) 0x7fe7c0003520 con 0x7fe7d40ccb70 2024-04-15T15:16:34.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.085+0000 7fe7d8eb6640 1 -- 172.21.15.72:0/607393688 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe7d415c7c0 msgr2=0x7fe7d406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:34.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.085+0000 7fe7d8eb6640 1 --2- 172.21.15.72:0/607393688 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe7d415c7c0 0x7fe7d406e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.085+0000 7fe7d8eb6640 1 -- 172.21.15.72:0/607393688 >> v2:172.21.15.175:3300/0 conn(0x7fe7d40af700 msgr2=0x7fe7d406e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:34.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.085+0000 7fe7d8eb6640 1 --2- 172.21.15.72:0/607393688 >> v2:172.21.15.175:3300/0 conn(0x7fe7d40af700 0x7fe7d406e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.085+0000 7fe7d8eb6640 1 -- 172.21.15.72:0/607393688 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7d406de60 con 0x7fe7d40ccb70 2024-04-15T15:16:34.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.085+0000 7fe7d8eb6640 1 -- 172.21.15.72:0/607393688 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7fe7cc0019f0 con 0x7fe7d40ccb70 2024-04-15T15:16:34.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.085+0000 7fe7d8eb6640 1 -- 172.21.15.72:0/607393688 >> v1:172.21.15.72:6789/0 conn(0x7fe7d40ccb70 legacy=0x7fe7d40cd000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.085+0000 7fe7d8eb6640 1 --2- 172.21.15.72:0/607393688 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe7d415c7c0 0x7fe7c00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.085+0000 7fe7d8eb6640 1 --2- 172.21.15.72:0/607393688 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe7c0003f30 0x7fe7c0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.085+0000 7fe7d8eb6640 1 --2- 172.21.15.72:0/607393688 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7c0004d40 0x7fe7c00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.086+0000 7fe7d8eb6640 1 -- 172.21.15.72:0/607393688 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d415cba0 con 0x7fe7c0004d40 2024-04-15T15:16:34.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.086+0000 7fe7d8eb6640 1 -- 172.21.15.72:0/607393688 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d406dc10 con 0x7fe7c0003f30 2024-04-15T15:16:34.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.086+0000 7fe7d8eb6640 1 -- 172.21.15.72:0/607393688 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fe7c000b9c0 con 0x7fe7d415c7c0 2024-04-15T15:16:34.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.086+0000 7fe7da6b9640 1 --2- 172.21.15.72:0/607393688 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7c0004d40 0x7fe7c00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.086+0000 7fe7d96b7640 1 --2- 172.21.15.72:0/607393688 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe7c0003f30 0x7fe7c0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.086+0000 7fe7d9eb8640 1 --2- 172.21.15.72:0/607393688 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe7d415c7c0 0x7fe7c00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.086+0000 7fe7da6b9640 1 -- 172.21.15.72:0/607393688 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe7d415c7c0 msgr2=0x7fe7c00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:34.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.086+0000 7fe7da6b9640 1 --2- 172.21.15.72:0/607393688 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe7d415c7c0 0x7fe7c00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.086+0000 7fe7da6b9640 1 -- 172.21.15.72:0/607393688 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe7c0003f30 msgr2=0x7fe7c0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:34.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.086+0000 7fe7da6b9640 1 --2- 172.21.15.72:0/607393688 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe7c0003f30 0x7fe7c0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.086+0000 7fe7da6b9640 1 -- 172.21.15.72:0/607393688 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7c0003520 con 0x7fe7c0004d40 2024-04-15T15:16:34.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.087+0000 7fe7da6b9640 1 --2- 172.21.15.72:0/607393688 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7c0004d40 0x7fe7c00051c0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc00a870 tx=0x7fe7c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb3f530b67d0a744 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:34.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.087+0000 7fe7d8eb6640 1 -- 172.21.15.72:0/607393688 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc00a4c0 con 0x7fe7c0004d40 2024-04-15T15:16:34.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.087+0000 7fe7d8eb6640 1 -- 172.21.15.72:0/607393688 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7cc034630 con 0x7fe7c0004d40 2024-04-15T15:16:34.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.087+0000 7fe7d8eb6640 1 -- 172.21.15.72:0/607393688 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc034a20 con 0x7fe7c0004d40 2024-04-15T15:16:34.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.087+0000 7fe7dc11c640 1 -- 172.21.15.72:0/607393688 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7c0004d40 msgr2=0x7fe7c00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.087+0000 7fe7dc11c640 1 --2- 172.21.15.72:0/607393688 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7c0004d40 0x7fe7c00051c0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc00a870 tx=0x7fe7c0005830 comp rx=0 tx=0).stop 2024-04-15T15:16:34.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.087+0000 7fe7da6b9640 1 -- 172.21.15.72:0/607393688 reap_dead start 2024-04-15T15:16:34.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.087+0000 7fe7dc11c640 1 -- 172.21.15.72:0/607393688 shutdown_connections 2024-04-15T15:16:34.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.087+0000 7fe7dc11c640 1 -- 172.21.15.72:0/607393688 >> 172.21.15.72:0/607393688 conn(0x7fe7d40b1dd0 msgr2=0x7fe7d40b21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:34.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.087+0000 7fe7dc11c640 1 -- 172.21.15.72:0/607393688 shutdown_connections 2024-04-15T15:16:34.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.087+0000 7fe7dc11c640 1 -- 172.21.15.72:0/607393688 wait complete. 2024-04-15T15:16:34.094 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.089+0000 7fe7dc11c640 1 Processor -- start 2024-04-15T15:16:34.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.089+0000 7fe7dc11c640 1 -- start start 2024-04-15T15:16:34.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.089+0000 7fe7dc11c640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe7c0004d40 0x7fe7d4071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.089+0000 7fe7dc11c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7d4157450 0x7fe7d4153960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.089+0000 7fe7dc11c640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe7d4153ea0 0x7fe7d414bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.089+0000 7fe7dc11c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d40cfab0 con 0x7fe7d4157450 2024-04-15T15:16:34.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.089+0000 7fe7dc11c640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fe7d40cf7d0 con 0x7fe7c0004d40 2024-04-15T15:16:34.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.089+0000 7fe7dc11c640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fe7d40cdc60 con 0x7fe7d4153ea0 2024-04-15T15:16:34.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.090+0000 7fe7d96b7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7d4157450 0x7fe7d4153960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.090+0000 7fe7d96b7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7d4157450 0x7fe7d4153960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60448/0 (socket says 172.21.15.72:60448) 2024-04-15T15:16:34.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.090+0000 7fe7d96b7640 1 -- 172.21.15.72:0/3789055079 learned_addr learned my addr 172.21.15.72:0/3789055079 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:34.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.090+0000 7fe7d9eb8640 1 --2- 172.21.15.72:0/3789055079 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe7c0004d40 0x7fe7d4071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.090+0000 7fe7da6b9640 1 --2- 172.21.15.72:0/3789055079 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe7d4153ea0 0x7fe7d414bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.090+0000 7fe7d96b7640 1 -- 172.21.15.72:0/3789055079 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe7d4153ea0 msgr2=0x7fe7d414bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.090+0000 7fe7d96b7640 1 --2- 172.21.15.72:0/3789055079 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe7d4153ea0 0x7fe7d414bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.095 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.090+0000 7fe7d96b7640 1 -- 172.21.15.72:0/3789055079 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe7c0004d40 msgr2=0x7fe7d4071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.090+0000 7fe7d96b7640 1 --2- 172.21.15.72:0/3789055079 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe7c0004d40 0x7fe7d4071bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.090+0000 7fe7d96b7640 1 -- 172.21.15.72:0/3789055079 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7d41f41d0 con 0x7fe7d4157450 2024-04-15T15:16:34.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.090+0000 7fe7d96b7640 1 --2- 172.21.15.72:0/3789055079 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7d4157450 0x7fe7d4153960 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4011960 tx=0x7fe7c4011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:34.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.090+0000 7fe7ca7fc640 1 -- 172.21.15.72:0/3789055079 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c400faf0 con 0x7fe7d4157450 2024-04-15T15:16:34.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.090+0000 7fe7ca7fc640 1 -- 172.21.15.72:0/3789055079 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7c400fc90 con 0x7fe7d4157450 2024-04-15T15:16:34.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.090+0000 7fe7ca7fc640 1 -- 172.21.15.72:0/3789055079 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7c4014440 con 0x7fe7d4157450 2024-04-15T15:16:34.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.090+0000 7fe7dc11c640 1 -- 172.21.15.72:0/3789055079 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7d40723e0 con 0x7fe7d4157450 2024-04-15T15:16:34.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.090+0000 7fe7dc11c640 1 -- 172.21.15.72:0/3789055079 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7d40729a0 con 0x7fe7d4157450 2024-04-15T15:16:34.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.091+0000 7fe7dc11c640 1 -- 172.21.15.72:0/3789055079 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7d40689e0 con 0x7fe7d4157450 2024-04-15T15:16:34.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.092+0000 7fe7ca7fc640 1 -- 172.21.15.72:0/3789055079 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fe7c4012040 con 0x7fe7d4157450 2024-04-15T15:16:34.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.092+0000 7fe7ca7fc640 1 --2- 172.21.15.72:0/3789055079 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe7b806c9b0 0x7fe7b806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.092+0000 7fe7ca7fc640 1 -- 172.21.15.72:0/3789055079 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6130+0+0 (secure 0 0 0) 0x7fe7c408c3b0 con 0x7fe7d4157450 2024-04-15T15:16:34.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.092+0000 7fe7d9eb8640 1 --2- 172.21.15.72:0/3789055079 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe7b806c9b0 0x7fe7b806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.096 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.093+0000 7fe7d9eb8640 1 --2- 172.21.15.72:0/3789055079 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe7b806c9b0 0x7fe7b806ee70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fe7d00099e0 tx=0x7fe7d0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:34.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.096+0000 7fe7ca7fc640 1 -- 172.21.15.72:0/3789055079 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe7c4019030 con 0x7fe7d4157450 2024-04-15T15:16:34.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.307+0000 7fe7dc11c640 1 -- 172.21.15.72:0/3789055079 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fe7d40cc600 con 0x7fe7d4157450 2024-04-15T15:16:34.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.308+0000 7fe7ca7fc640 1 -- 172.21.15.72:0/3789055079 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+73 (secure 0 0 0) 0x7fe7c405a640 con 0x7fe7d4157450 2024-04-15T15:16:34.309 INFO:teuthology.orchestra.run.smithi072.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-15T15:16:34.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.310+0000 7fe7dc11c640 1 -- 172.21.15.72:0/3789055079 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe7b806c9b0 msgr2=0x7fe7b806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.310+0000 7fe7dc11c640 1 --2- 172.21.15.72:0/3789055079 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe7b806c9b0 0x7fe7b806ee70 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fe7d00099e0 tx=0x7fe7d0008040 comp rx=0 tx=0).stop 2024-04-15T15:16:34.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.310+0000 7fe7dc11c640 1 -- 172.21.15.72:0/3789055079 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7d4157450 msgr2=0x7fe7d4153960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.310+0000 7fe7dc11c640 1 --2- 172.21.15.72:0/3789055079 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7d4157450 0x7fe7d4153960 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fe7c4011960 tx=0x7fe7c4011e20 comp rx=0 tx=0).stop 2024-04-15T15:16:34.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.310+0000 7fe7dc11c640 1 -- 172.21.15.72:0/3789055079 shutdown_connections 2024-04-15T15:16:34.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.310+0000 7fe7dc11c640 1 --2- 172.21.15.72:0/3789055079 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe7b806c9b0 0x7fe7b806ee70 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.310+0000 7fe7dc11c640 1 --2- 172.21.15.72:0/3789055079 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe7d4153ea0 0x7fe7d414bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.310+0000 7fe7dc11c640 1 --2- 172.21.15.72:0/3789055079 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe7d4157450 0x7fe7d4153960 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.310+0000 7fe7dc11c640 1 --2- 172.21.15.72:0/3789055079 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe7c0004d40 0x7fe7d4071bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.310+0000 7fe7dc11c640 1 -- 172.21.15.72:0/3789055079 >> 172.21.15.72:0/3789055079 conn(0x7fe7d40b1dd0 msgr2=0x7fe7d40ad0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:34.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.310+0000 7fe7dc11c640 1 -- 172.21.15.72:0/3789055079 shutdown_connections 2024-04-15T15:16:34.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34.310+0000 7fe7dc11c640 1 -- 172.21.15.72:0/3789055079 wait complete. 2024-04-15T15:16:34.322 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:16:34.322 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:16:34.322 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi072.front.sepia.ceph.com 2024-04-15T15:16:34.322 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-15T15:16:34.323 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-15T15:16:34.323 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:16:34.323 INFO:tasks.cephfs.mount:self.cephfs_mntpt = /volumes/_nogroup/subvol_quiesce/6bdc4ba4-8793-4ca5-8487-64cfa9a984f4 2024-04-15T15:16:34.323 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:16:34.323 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-15T15:16:34.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:34.323 DEBUG:teuthology.orchestra.run.smithi072:> ip addr 2024-04-15T15:16:34.339 INFO:teuthology.orchestra.run.smithi072.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:16:34.339 INFO:teuthology.orchestra.run.smithi072.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:16:34.339 INFO:teuthology.orchestra.run.smithi072.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:16:34.339 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:34.339 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 ::1/128 scope host 2024-04-15T15:16:34.339 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:34.340 INFO:teuthology.orchestra.run.smithi072.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:34.340 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f4 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:34.340 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f0 2024-04-15T15:16:34.340 INFO:teuthology.orchestra.run.smithi072.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:34.340 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f5 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:34.340 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f1 2024-04-15T15:16:34.340 INFO:teuthology.orchestra.run.smithi072.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:34.340 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:64 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:34.340 INFO:teuthology.orchestra.run.smithi072.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:16:34.340 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:65 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:34.340 INFO:teuthology.orchestra.run.smithi072.stdout: inet 172.21.15.72/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-15T15:16:34.340 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft 42609sec preferred_lft 42609sec 2024-04-15T15:16:34.340 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::ec4:7aff:fe8f:b765/64 scope link 2024-04-15T15:16:34.340 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:34.340 INFO:teuthology.orchestra.run.smithi072.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:16:34.341 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 62:2f:60:9d:8a:ab brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:34.341 INFO:teuthology.orchestra.run.smithi072.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-15T15:16:34.341 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:34.341 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::6871:2cff:fe2a:4823/64 scope link 2024-04-15T15:16:34.341 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:34.341 INFO:teuthology.orchestra.run.smithi072.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:16:34.341 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether b6:ae:7a:2e:38:96 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:34.341 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::b4ae:7aff:fe2e:3896/64 scope link 2024-04-15T15:16:34.341 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:34.341 INFO:teuthology.orchestra.run.smithi072.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:16:34.342 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 62:2f:60:9d:8a:ab brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:16:34.342 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::602f:60ff:fe9d:8aab/64 scope link 2024-04-15T15:16:34.342 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:34.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:34.342 DEBUG:teuthology.orchestra.run.smithi072:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:16:34.406 INFO:teuthology.orchestra.run.smithi072.stdout:1 2024-04-15T15:16:34.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:34.409 DEBUG:teuthology.orchestra.run.smithi072:> route 2024-04-15T15:16:34.469 INFO:teuthology.orchestra.run.smithi072.stdout:Kernel IP routing table 2024-04-15T15:16:34.469 INFO:teuthology.orchestra.run.smithi072.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:16:34.469 INFO:teuthology.orchestra.run.smithi072.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-15T15:16:34.469 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-15T15:16:34.469 INFO:teuthology.orchestra.run.smithi072.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:16:34.470 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:16:34.470 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-04-15T15:16:34.470 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-15T15:16:34.470 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-15T15:16:34.470 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-15T15:16:34.470 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-04-15T15:16:34.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:16:34.602 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:34.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:34.608 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-04-15T15:16:34.624 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-15T15:16:34.624 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-15T15:16:34.624 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-15T15:16:34.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:34.624 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set brx.0 up 2024-04-15T15:16:34.693 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-15T15:16:34.693 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:16:34.693 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-15T15:16:34.749 INFO:teuthology.orchestra.run.smithi072.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-15T15:16:34.749 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:16:34.749 DEBUG:teuthology.orchestra.run.smithi072:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-15T15:16:34.805 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-04-15T15:16:34.875 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:16:34.932 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-04-15T15:16:34.932 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-04-15T15:16:34.932 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-04-15T15:16:34.932 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-04-15T15:16:34.932 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-04-15T15:16:34.932 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-04-15T15:16:34.932 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-04-15T15:16:34.933 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-04-15T15:16:34.933 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-04-15T15:16:34.933 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-04-15T15:16:34.933 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-04-15T15:16:34.933 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-04-15T15:16:34.933 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:16:34.933 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-04-15T15:16:34.933 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-04-15T15:16:34.933 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-04-15T15:16:34.933 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-04-15T15:16:34.933 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:16:34.933 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-04-15T15:16:34.933 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-04-15T15:16:34.933 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:16:34.934 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:16:34.934 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:16:34.934 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-04-15T15:16:34.934 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-04-15T15:16:34.934 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:16:34.934 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:16:34.934 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:16:34.934 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:16:34.934 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-04-15T15:16:34.934 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:34.934 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:34.934 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:16:34.934 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:16:34.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:34.935 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-04-15T15:16:34.988 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-15T15:16:34.988 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/volumes/_nogroup/subvol_quiesce/6bdc4ba4-8793-4ca5-8487-64cfa9a984f4) 2024-04-15T15:16:35.031 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-04-15T15:16:35.058 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:16:35.116 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-04-15T15:16:35.116 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-04-15T15:16:35.116 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-04-15T15:16:35.117 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-04-15T15:16:35.117 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-04-15T15:16:35.117 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-04-15T15:16:35.117 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-04-15T15:16:35.117 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-04-15T15:16:35.117 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-04-15T15:16:35.117 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-04-15T15:16:35.117 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-04-15T15:16:35.117 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-04-15T15:16:35.117 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:16:35.117 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-04-15T15:16:35.117 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-04-15T15:16:35.117 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-04-15T15:16:35.117 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-04-15T15:16:35.117 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:16:35.118 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-04-15T15:16:35.118 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-04-15T15:16:35.118 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:16:35.118 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:16:35.118 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:16:35.118 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-04-15T15:16:35.118 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-04-15T15:16:35.118 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:16:35.118 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:16:35.118 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:16:35.118 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:16:35.118 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-04-15T15:16:35.118 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:35.119 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:35.119 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:16:35.119 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:16:35.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:35.119 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-04-15T15:16:35.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:2024-04-15T15:16:35.118+0000 7f6b11941480 -1 init, newargv = 0x55fc80e1bbc0 newargc=15 2024-04-15T15:16:35.120 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[56093]: starting ceph client 2024-04-15T15:16:35.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[56093]: starting fuse 2024-04-15T15:16:35.137 INFO:teuthology.orchestra.run.smithi072.stdout:43 2024-04-15T15:16:35.137 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-04-15T15:16:35.137 DEBUG:teuthology.orchestra.run.smithi072:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:16:35.137 DEBUG:teuthology.orchestra.run.smithi072:> import glob 2024-04-15T15:16:35.138 DEBUG:teuthology.orchestra.run.smithi072:> import re 2024-04-15T15:16:35.138 DEBUG:teuthology.orchestra.run.smithi072:> import os 2024-04-15T15:16:35.138 DEBUG:teuthology.orchestra.run.smithi072:> import subprocess 2024-04-15T15:16:35.138 DEBUG:teuthology.orchestra.run.smithi072:> 2024-04-15T15:16:35.138 DEBUG:teuthology.orchestra.run.smithi072:> def _find_admin_socket(client_name): 2024-04-15T15:16:35.138 DEBUG:teuthology.orchestra.run.smithi072:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-15T15:16:35.138 DEBUG:teuthology.orchestra.run.smithi072:> files = glob.glob(asok_path) 2024-04-15T15:16:35.138 DEBUG:teuthology.orchestra.run.smithi072:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-15T15:16:35.138 DEBUG:teuthology.orchestra.run.smithi072:> 2024-04-15T15:16:35.138 DEBUG:teuthology.orchestra.run.smithi072:> # Given a non-glob path, it better be there 2024-04-15T15:16:35.138 DEBUG:teuthology.orchestra.run.smithi072:> if "*" not in asok_path: 2024-04-15T15:16:35.138 DEBUG:teuthology.orchestra.run.smithi072:> assert(len(files) == 1) 2024-04-15T15:16:35.138 DEBUG:teuthology.orchestra.run.smithi072:> return files[0] 2024-04-15T15:16:35.138 DEBUG:teuthology.orchestra.run.smithi072:> 2024-04-15T15:16:35.138 DEBUG:teuthology.orchestra.run.smithi072:> for f in files: 2024-04-15T15:16:35.139 DEBUG:teuthology.orchestra.run.smithi072:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:16:35.139 DEBUG:teuthology.orchestra.run.smithi072:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:16:35.139 DEBUG:teuthology.orchestra.run.smithi072:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:16:35.139 DEBUG:teuthology.orchestra.run.smithi072:> contents = proc_f.read() 2024-04-15T15:16:35.139 DEBUG:teuthology.orchestra.run.smithi072:> if mountpoint in contents: 2024-04-15T15:16:35.139 DEBUG:teuthology.orchestra.run.smithi072:> return f 2024-04-15T15:16:35.139 DEBUG:teuthology.orchestra.run.smithi072:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:16:35.139 DEBUG:teuthology.orchestra.run.smithi072:> 2024-04-15T15:16:35.139 DEBUG:teuthology.orchestra.run.smithi072:> print(_find_admin_socket("client.0")) 2024-04-15T15:16:35.139 DEBUG:teuthology.orchestra.run.smithi072:> ' 2024-04-15T15:16:35.259 INFO:teuthology.orchestra.run.smithi072.stdout:/var/run/ceph/ceph-client.0.56093.asok 2024-04-15T15:16:35.262 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:35.270 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.56093.asok 2024-04-15T15:16:35.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:35.270 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.56093.asok status 2024-04-15T15:16:35.409 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-04-15T15:16:35.409 INFO:teuthology.orchestra.run.smithi072.stdout: "metadata": { 2024-04-15T15:16:35.409 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:16:35.409 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:16:35.409 INFO:teuthology.orchestra.run.smithi072.stdout: "entity_id": "0", 2024-04-15T15:16:35.409 INFO:teuthology.orchestra.run.smithi072.stdout: "hostname": "smithi072", 2024-04-15T15:16:35.409 INFO:teuthology.orchestra.run.smithi072.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-15T15:16:35.409 INFO:teuthology.orchestra.run.smithi072.stdout: "pid": "56093", 2024-04-15T15:16:35.410 INFO:teuthology.orchestra.run.smithi072.stdout: "root": "/volumes/_nogroup/subvol_quiesce/6bdc4ba4-8793-4ca5-8487-64cfa9a984f4" 2024-04-15T15:16:35.410 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:35.410 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_count": 0, 2024-04-15T15:16:35.410 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_pinned_count": 0, 2024-04-15T15:16:35.410 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 5970, 2024-04-15T15:16:35.410 INFO:teuthology.orchestra.run.smithi072.stdout: "inst": { 2024-04-15T15:16:35.410 INFO:teuthology.orchestra.run.smithi072.stdout: "name": { 2024-04-15T15:16:35.410 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "client", 2024-04-15T15:16:35.410 INFO:teuthology.orchestra.run.smithi072.stdout: "num": 5970 2024-04-15T15:16:35.410 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:35.410 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-04-15T15:16:35.410 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-04-15T15:16:35.410 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:16:35.410 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 3459987399 2024-04-15T15:16:35.411 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:16:35.411 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:35.411 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-04-15T15:16:35.411 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-04-15T15:16:35.411 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:16:35.411 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 3459987399 2024-04-15T15:16:35.411 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:35.411 INFO:teuthology.orchestra.run.smithi072.stdout: "inst_str": "client.5970 192.168.0.1:0/3459987399", 2024-04-15T15:16:35.411 INFO:teuthology.orchestra.run.smithi072.stdout: "addr_str": "192.168.0.1:0/3459987399", 2024-04-15T15:16:35.411 INFO:teuthology.orchestra.run.smithi072.stdout: "inode_count": 5, 2024-04-15T15:16:35.411 INFO:teuthology.orchestra.run.smithi072.stdout: "mds_epoch": 16, 2024-04-15T15:16:35.411 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch": 40, 2024-04-15T15:16:35.411 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:16:35.411 INFO:teuthology.orchestra.run.smithi072.stdout: "blocklisted": false, 2024-04-15T15:16:35.411 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs" 2024-04-15T15:16:35.412 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-04-15T15:16:35.419 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:16:35.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.519+0000 7f77332a5640 1 Processor -- start 2024-04-15T15:16:35.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.519+0000 7f77332a5640 1 -- start start 2024-04-15T15:16:35.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.520+0000 7f77332a5640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f772c0cc5e0 0x7f772c15b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:35.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.520+0000 7f77332a5640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f772c0af740 0x7f772c1648a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:35.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.520+0000 7f77332a5640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f772c0cf990 con 0x7f772c15ad10 2024-04-15T15:16:35.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.520+0000 7f77332a5640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f772c0cd4b0 con 0x7f772c0cc5e0 2024-04-15T15:16:35.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.520+0000 7f77332a5640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f772c0cd790 con 0x7f772c0af740 2024-04-15T15:16:35.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.520+0000 7f7731041640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f772c15ad10 0x7f772c15b0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:55966/0 (socket says 172.21.15.72:55966) 2024-04-15T15:16:35.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.520+0000 7f7731041640 1 -- 172.21.15.72:0/819144014 learned_addr learned my addr 172.21.15.72:0/819144014 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:35.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.520+0000 7f7730840640 1 --2- 172.21.15.72:0/819144014 >> v2:172.21.15.175:3300/0 conn(0x7f772c0cc5e0 0x7f772c15b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:35.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.520+0000 7f7731842640 1 --2- 172.21.15.72:0/819144014 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f772c0af740 0x7f772c1648a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:35.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.520+0000 7f7723fff640 1 -- 172.21.15.72:0/819144014 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1091615600 0 0) 0x7f772c0cf990 con 0x7f772c15ad10 2024-04-15T15:16:35.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.520+0000 7f7723fff640 1 -- 172.21.15.72:0/819144014 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7718004630 con 0x7f772c15ad10 2024-04-15T15:16:35.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.520+0000 7f7723fff640 1 -- 172.21.15.72:0/819144014 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3180836088 0 0) 0x7f7718004630 con 0x7f772c15ad10 2024-04-15T15:16:35.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.520+0000 7f7723fff640 1 -- 172.21.15.72:0/819144014 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f772c0af740 msgr2=0x7f772c1648a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:35.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.520+0000 7f7723fff640 1 --2- 172.21.15.72:0/819144014 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f772c0af740 0x7f772c1648a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.520+0000 7f7723fff640 1 -- 172.21.15.72:0/819144014 >> v2:172.21.15.175:3300/0 conn(0x7f772c0cc5e0 msgr2=0x7f772c15b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:35.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.520+0000 7f7723fff640 1 --2- 172.21.15.72:0/819144014 >> v2:172.21.15.175:3300/0 conn(0x7f772c0cc5e0 0x7f772c15b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.520+0000 7f7723fff640 1 -- 172.21.15.72:0/819144014 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f772c15bf10 con 0x7f772c15ad10 2024-04-15T15:16:35.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.520+0000 7f7723fff640 1 -- 172.21.15.72:0/819144014 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f7728001690 con 0x7f772c15ad10 2024-04-15T15:16:35.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.520+0000 7f7723fff640 1 -- 172.21.15.72:0/819144014 >> v1:172.21.15.72:6789/0 conn(0x7f772c15ad10 legacy=0x7f772c15b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:35.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.520+0000 7f7723fff640 1 --2- 172.21.15.72:0/819144014 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7718004ac0 0x7f7718004ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:35.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.520+0000 7f7723fff640 1 --2- 172.21.15.72:0/819144014 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f772c0af740 0x7f7718005910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:35.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.520+0000 7f7723fff640 1 --2- 172.21.15.72:0/819144014 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7718005e50 0x7f77180062d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:35.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.521+0000 7f7723fff640 1 -- 172.21.15.72:0/819144014 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f772c0cd4b0 con 0x7f7718005e50 2024-04-15T15:16:35.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.521+0000 7f7723fff640 1 -- 172.21.15.72:0/819144014 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f772c0cd790 con 0x7f7718004ac0 2024-04-15T15:16:35.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.521+0000 7f7723fff640 1 -- 172.21.15.72:0/819144014 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f771800cad0 con 0x7f772c0af740 2024-04-15T15:16:35.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.521+0000 7f7731842640 1 --2- 172.21.15.72:0/819144014 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7718005e50 0x7f77180062d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:35.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.521+0000 7f7731041640 1 --2- 172.21.15.72:0/819144014 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7718004ac0 0x7f7718004ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:35.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.521+0000 7f7730840640 1 --2- 172.21.15.72:0/819144014 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f772c0af740 0x7f7718005910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:35.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.521+0000 7f7731842640 1 -- 172.21.15.72:0/819144014 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f772c0af740 msgr2=0x7f7718005910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:35.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.521+0000 7f7731842640 1 --2- 172.21.15.72:0/819144014 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f772c0af740 0x7f7718005910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.521+0000 7f7731842640 1 -- 172.21.15.72:0/819144014 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7718004ac0 msgr2=0x7f7718004ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:35.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.521+0000 7f7731842640 1 --2- 172.21.15.72:0/819144014 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7718004ac0 0x7f7718004ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.521+0000 7f7731842640 1 -- 172.21.15.72:0/819144014 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7718004630 con 0x7f7718005e50 2024-04-15T15:16:35.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.521+0000 7f7730840640 1 --2- 172.21.15.72:0/819144014 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f772c0af740 0x7f7718005910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:35.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.522+0000 7f7731842640 1 --2- 172.21.15.72:0/819144014 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7718005e50 0x7f77180062d0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f7724009fc0 tx=0x7f7718006940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c35971813d1a25c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:35.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.522+0000 7f7723fff640 1 -- 172.21.15.72:0/819144014 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7724039070 con 0x7f7718005e50 2024-04-15T15:16:35.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.522+0000 7f7723fff640 1 -- 172.21.15.72:0/819144014 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f772402cc20 con 0x7f7718005e50 2024-04-15T15:16:35.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.522+0000 7f7723fff640 1 -- 172.21.15.72:0/819144014 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7724009c10 con 0x7f7718005e50 2024-04-15T15:16:35.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.522+0000 7f77332a5640 1 -- 172.21.15.72:0/819144014 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7718005e50 msgr2=0x7f77180062d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:35.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.522+0000 7f77332a5640 1 --2- 172.21.15.72:0/819144014 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7718005e50 0x7f77180062d0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f7724009fc0 tx=0x7f7718006940 comp rx=0 tx=0).stop 2024-04-15T15:16:35.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.522+0000 7f7731842640 1 -- 172.21.15.72:0/819144014 reap_dead start 2024-04-15T15:16:35.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.522+0000 7f77332a5640 1 -- 172.21.15.72:0/819144014 shutdown_connections 2024-04-15T15:16:35.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.522+0000 7f77332a5640 1 -- 172.21.15.72:0/819144014 >> 172.21.15.72:0/819144014 conn(0x7f772c0b1dd0 msgr2=0x7f772c0b21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:35.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.522+0000 7f77332a5640 1 -- 172.21.15.72:0/819144014 shutdown_connections 2024-04-15T15:16:35.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.522+0000 7f77332a5640 1 -- 172.21.15.72:0/819144014 wait complete. 2024-04-15T15:16:35.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.524+0000 7f77332a5640 1 Processor -- start 2024-04-15T15:16:35.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.524+0000 7f77332a5640 1 -- start start 2024-04-15T15:16:35.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.524+0000 7f77332a5640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7718005e50 0x7f772c1f02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:35.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.524+0000 7f77332a5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f772c073940 0x7f772c15d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:35.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.524+0000 7f77332a5640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f772c14f720 0x7f772c15d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:35.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.524+0000 7f77332a5640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f772c0ce2d0 con 0x7f772c073940 2024-04-15T15:16:35.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.524+0000 7f77332a5640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f772c15c1f0 con 0x7f772c14f720 2024-04-15T15:16:35.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.524+0000 7f77332a5640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f772c06dc10 con 0x7f7718005e50 2024-04-15T15:16:35.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.524+0000 7f7730840640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f772c073940 0x7f772c15d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:35.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.524+0000 7f7730840640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f772c073940 0x7f772c15d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60464/0 (socket says 172.21.15.72:60464) 2024-04-15T15:16:35.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.524+0000 7f7730840640 1 -- 172.21.15.72:0/678520637 learned_addr learned my addr 172.21.15.72:0/678520637 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:35.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.525+0000 7f7731842640 1 --2- 172.21.15.72:0/678520637 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f772c14f720 0x7f772c15d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:35.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.525+0000 7f7731041640 1 --2- 172.21.15.72:0/678520637 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7718005e50 0x7f772c1f02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:35.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.525+0000 7f7730840640 1 -- 172.21.15.72:0/678520637 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7718005e50 msgr2=0x7f772c1f02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:35.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.525+0000 7f7730840640 1 --2- 172.21.15.72:0/678520637 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7718005e50 0x7f772c1f02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.525+0000 7f7730840640 1 -- 172.21.15.72:0/678520637 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f772c14f720 msgr2=0x7f772c15d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:35.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.525+0000 7f7730840640 1 --2- 172.21.15.72:0/678520637 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f772c14f720 0x7f772c15d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.525+0000 7f7730840640 1 -- 172.21.15.72:0/678520637 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f772c073200 con 0x7f772c073940 2024-04-15T15:16:35.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.525+0000 7f7730840640 1 --2- 172.21.15.72:0/678520637 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f772c073940 0x7f772c15d3c0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f771c011960 tx=0x7f771c011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:35.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.525+0000 7f7721ffb640 1 -- 172.21.15.72:0/678520637 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f771c01f070 con 0x7f772c073940 2024-04-15T15:16:35.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.525+0000 7f7721ffb640 1 -- 172.21.15.72:0/678520637 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f771c00ca00 con 0x7f772c073940 2024-04-15T15:16:35.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.525+0000 7f77332a5640 1 -- 172.21.15.72:0/678520637 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f772c0734e0 con 0x7f772c073940 2024-04-15T15:16:35.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.525+0000 7f7721ffb640 1 -- 172.21.15.72:0/678520637 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f771c00cdf0 con 0x7f772c073940 2024-04-15T15:16:35.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.525+0000 7f77332a5640 1 -- 172.21.15.72:0/678520637 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f772c1554a0 con 0x7f772c073940 2024-04-15T15:16:35.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.526+0000 7f77332a5640 1 -- 172.21.15.72:0/678520637 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f772c155880 con 0x7f772c073940 2024-04-15T15:16:35.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.527+0000 7f7721ffb640 1 -- 172.21.15.72:0/678520637 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f771c022070 con 0x7f772c073940 2024-04-15T15:16:35.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.527+0000 7f7721ffb640 1 --2- 172.21.15.72:0/678520637 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f771006c980 0x7f771006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:35.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.527+0000 7f7731041640 1 --2- 172.21.15.72:0/678520637 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f771006c980 0x7f771006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:35.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.527+0000 7f7721ffb640 1 -- 172.21.15.72:0/678520637 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6130+0+0 (secure 0 0 0) 0x7f771c08d990 con 0x7f772c073940 2024-04-15T15:16:35.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.528+0000 7f7731041640 1 --2- 172.21.15.72:0/678520637 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f771006c980 0x7f771006ee40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f772c0cfb60 tx=0x7f7728009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:35.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.532+0000 7f7721ffb640 1 -- 172.21.15.72:0/678520637 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f772c155880 con 0x7f772c073940 2024-04-15T15:16:35.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.746+0000 7f77332a5640 1 -- 172.21.15.72:0/678520637 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f772c0689e0 con 0x7f772c073940 2024-04-15T15:16:35.748 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.746+0000 7f7721ffb640 1 -- 172.21.15.72:0/678520637 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+73 (secure 0 0 0) 0x7f771c05b5e0 con 0x7f772c073940 2024-04-15T15:16:35.748 INFO:teuthology.orchestra.run.smithi072.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-15T15:16:35.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.748+0000 7f77332a5640 1 -- 172.21.15.72:0/678520637 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f771006c980 msgr2=0x7f771006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:35.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.748+0000 7f77332a5640 1 --2- 172.21.15.72:0/678520637 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f771006c980 0x7f771006ee40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f772c0cfb60 tx=0x7f7728009040 comp rx=0 tx=0).stop 2024-04-15T15:16:35.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.748+0000 7f77332a5640 1 -- 172.21.15.72:0/678520637 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f772c073940 msgr2=0x7f772c15d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:35.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.748+0000 7f77332a5640 1 --2- 172.21.15.72:0/678520637 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f772c073940 0x7f772c15d3c0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f771c011960 tx=0x7f771c011e20 comp rx=0 tx=0).stop 2024-04-15T15:16:35.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.748+0000 7f77332a5640 1 -- 172.21.15.72:0/678520637 shutdown_connections 2024-04-15T15:16:35.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.748+0000 7f77332a5640 1 --2- 172.21.15.72:0/678520637 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f771006c980 0x7f771006ee40 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.748+0000 7f77332a5640 1 --2- 172.21.15.72:0/678520637 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f772c14f720 0x7f772c15d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.748+0000 7f77332a5640 1 --2- 172.21.15.72:0/678520637 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f772c073940 0x7f772c15d3c0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.748+0000 7f77332a5640 1 --2- 172.21.15.72:0/678520637 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7718005e50 0x7f772c1f02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.748+0000 7f77332a5640 1 -- 172.21.15.72:0/678520637 >> 172.21.15.72:0/678520637 conn(0x7f772c0b1dd0 msgr2=0x7f772c06c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:35.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.749+0000 7f77332a5640 1 -- 172.21.15.72:0/678520637 shutdown_connections 2024-04-15T15:16:35.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:35.749+0000 7f77332a5640 1 -- 172.21.15.72:0/678520637 wait complete. 2024-04-15T15:16:35.761 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:16:35.761 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:16:35.761 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi175.front.sepia.ceph.com 2024-04-15T15:16:35.761 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-15T15:16:35.761 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:35.761 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:16:35.761 INFO:tasks.cephfs.mount:self.cephfs_mntpt = /volumes/_nogroup/subvol_quiesce/6bdc4ba4-8793-4ca5-8487-64cfa9a984f4 2024-04-15T15:16:35.761 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:16:35.761 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-15T15:16:35.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:35.761 DEBUG:teuthology.orchestra.run.smithi175:> ip addr 2024-04-15T15:16:35.781 INFO:teuthology.orchestra.run.smithi175.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:16:35.781 INFO:teuthology.orchestra.run.smithi175.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:16:35.781 INFO:teuthology.orchestra.run.smithi175.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:16:35.781 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:35.781 INFO:teuthology.orchestra.run.smithi175.stdout: inet6 ::1/128 scope host 2024-04-15T15:16:35.781 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:35.782 INFO:teuthology.orchestra.run.smithi175.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:35.782 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether 0c:c4:7a:d9:39:ee brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:35.782 INFO:teuthology.orchestra.run.smithi175.stdout: altname enp7s0f0 2024-04-15T15:16:35.782 INFO:teuthology.orchestra.run.smithi175.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:35.782 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether 0c:c4:7a:d9:39:ef brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:35.782 INFO:teuthology.orchestra.run.smithi175.stdout: altname enp7s0f1 2024-04-15T15:16:35.782 INFO:teuthology.orchestra.run.smithi175.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:35.782 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether 0c:c4:7a:88:71:f0 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:35.782 INFO:teuthology.orchestra.run.smithi175.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:16:35.782 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether 0c:c4:7a:88:71:f1 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:35.782 INFO:teuthology.orchestra.run.smithi175.stdout: inet 172.21.15.175/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-15T15:16:35.782 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft 42602sec preferred_lft 42602sec 2024-04-15T15:16:35.782 INFO:teuthology.orchestra.run.smithi175.stdout: inet6 fe80::ec4:7aff:fe88:71f1/64 scope link 2024-04-15T15:16:35.782 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:35.783 INFO:teuthology.orchestra.run.smithi175.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:16:35.783 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether 6e:00:c4:f8:75:e4 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:35.783 INFO:teuthology.orchestra.run.smithi175.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-15T15:16:35.783 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:35.783 INFO:teuthology.orchestra.run.smithi175.stdout: inet6 fe80::d4be:d9ff:fe0d:518c/64 scope link 2024-04-15T15:16:35.783 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:35.783 INFO:teuthology.orchestra.run.smithi175.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:16:35.783 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether f2:c6:d9:f6:5e:3e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:35.783 INFO:teuthology.orchestra.run.smithi175.stdout: inet6 fe80::f0c6:d9ff:fef6:5e3e/64 scope link 2024-04-15T15:16:35.783 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:35.783 INFO:teuthology.orchestra.run.smithi175.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:16:35.783 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether 6e:00:c4:f8:75:e4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:16:35.783 INFO:teuthology.orchestra.run.smithi175.stdout: inet6 fe80::6c00:c4ff:fef8:75e4/64 scope link 2024-04-15T15:16:35.783 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:35.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:35.784 DEBUG:teuthology.orchestra.run.smithi175:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:16:35.853 INFO:teuthology.orchestra.run.smithi175.stdout:1 2024-04-15T15:16:35.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:35.855 DEBUG:teuthology.orchestra.run.smithi175:> route 2024-04-15T15:16:35.912 INFO:teuthology.orchestra.run.smithi175.stdout:Kernel IP routing table 2024-04-15T15:16:35.912 INFO:teuthology.orchestra.run.smithi175.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:16:35.912 INFO:teuthology.orchestra.run.smithi175.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-15T15:16:35.912 INFO:teuthology.orchestra.run.smithi175.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-15T15:16:35.912 INFO:teuthology.orchestra.run.smithi175.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:16:35.913 DEBUG:teuthology.orchestra.run.smithi175:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:16:35.913 DEBUG:teuthology.orchestra.run.smithi175:> set -e 2024-04-15T15:16:35.913 DEBUG:teuthology.orchestra.run.smithi175:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-15T15:16:35.913 DEBUG:teuthology.orchestra.run.smithi175:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-15T15:16:35.913 DEBUG:teuthology.orchestra.run.smithi175:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-15T15:16:35.914 DEBUG:teuthology.orchestra.run.smithi175:> ') 2024-04-15T15:16:36.004 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:16:36 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:16:36.048 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:16:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:36.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:36.055 DEBUG:teuthology.orchestra.run.smithi175:> ip netns list 2024-04-15T15:16:36.069 INFO:teuthology.orchestra.run.smithi175.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-15T15:16:36.070 INFO:teuthology.orchestra.run.smithi175.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-15T15:16:36.070 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-15T15:16:36.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:36.070 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip link set brx.0 up 2024-04-15T15:16:36.138 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-15T15:16:36.138 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:16:36.138 DEBUG:teuthology.orchestra.run.smithi175:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:36.195 INFO:teuthology.orchestra.run.smithi175.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-15T15:16:36.195 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:16:36.195 DEBUG:teuthology.orchestra.run.smithi175:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:36.254 DEBUG:teuthology.orchestra.run.smithi175:> sudo modprobe fuse 2024-04-15T15:16:36.323 DEBUG:teuthology.orchestra.run.smithi175:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:16:36.380 INFO:teuthology.orchestra.run.smithi175.stdout:/proc 2024-04-15T15:16:36.380 INFO:teuthology.orchestra.run.smithi175.stdout:/sys 2024-04-15T15:16:36.380 INFO:teuthology.orchestra.run.smithi175.stdout:/dev 2024-04-15T15:16:36.380 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/security 2024-04-15T15:16:36.380 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/shm 2024-04-15T15:16:36.380 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/pts 2024-04-15T15:16:36.380 INFO:teuthology.orchestra.run.smithi175.stdout:/run 2024-04-15T15:16:36.380 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/cgroup 2024-04-15T15:16:36.380 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/pstore 2024-04-15T15:16:36.380 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/bpf 2024-04-15T15:16:36.381 INFO:teuthology.orchestra.run.smithi175.stdout:/ 2024-04-15T15:16:36.381 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/selinux 2024-04-15T15:16:36.381 INFO:teuthology.orchestra.run.smithi175.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:16:36.381 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/mqueue 2024-04-15T15:16:36.381 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/hugepages 2024-04-15T15:16:36.381 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/debug 2024-04-15T15:16:36.381 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/tracing 2024-04-15T15:16:36.381 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/fuse/connections 2024-04-15T15:16:36.381 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/config 2024-04-15T15:16:36.381 INFO:teuthology.orchestra.run.smithi175.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:16:36.381 INFO:teuthology.orchestra.run.smithi175.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:16:36.381 INFO:teuthology.orchestra.run.smithi175.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:16:36.381 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:16:36.381 INFO:teuthology.orchestra.run.smithi175.stdout:/run/user/1000 2024-04-15T15:16:36.382 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph 2024-04-15T15:16:36.382 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:16:36.382 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:16:36.382 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:16:36.382 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:16:36.382 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns 2024-04-15T15:16:36.382 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:36.382 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:36.382 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:16:36.382 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:16:36.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:36.383 DEBUG:teuthology.orchestra.run.smithi175:> ls /sys/fs/fuse/connections 2024-04-15T15:16:36.436 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-15T15:16:36.436 DEBUG:teuthology.orchestra.run.smithi175:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1 --client_mountpoint=/volumes/_nogroup/subvol_quiesce/6bdc4ba4-8793-4ca5-8487-64cfa9a984f4) 2024-04-15T15:16:36.479 DEBUG:teuthology.orchestra.run.smithi175:> sudo modprobe fuse 2024-04-15T15:16:36.506 DEBUG:teuthology.orchestra.run.smithi175:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:16:36.563 INFO:teuthology.orchestra.run.smithi175.stdout:/proc 2024-04-15T15:16:36.563 INFO:teuthology.orchestra.run.smithi175.stdout:/sys 2024-04-15T15:16:36.563 INFO:teuthology.orchestra.run.smithi175.stdout:/dev 2024-04-15T15:16:36.563 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/security 2024-04-15T15:16:36.563 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/shm 2024-04-15T15:16:36.563 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/pts 2024-04-15T15:16:36.563 INFO:teuthology.orchestra.run.smithi175.stdout:/run 2024-04-15T15:16:36.564 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/cgroup 2024-04-15T15:16:36.564 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/pstore 2024-04-15T15:16:36.564 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/bpf 2024-04-15T15:16:36.564 INFO:teuthology.orchestra.run.smithi175.stdout:/ 2024-04-15T15:16:36.564 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/selinux 2024-04-15T15:16:36.564 INFO:teuthology.orchestra.run.smithi175.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:16:36.564 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/mqueue 2024-04-15T15:16:36.564 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/hugepages 2024-04-15T15:16:36.564 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/debug 2024-04-15T15:16:36.564 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/tracing 2024-04-15T15:16:36.564 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/fuse/connections 2024-04-15T15:16:36.564 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/config 2024-04-15T15:16:36.564 INFO:teuthology.orchestra.run.smithi175.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:16:36.564 INFO:teuthology.orchestra.run.smithi175.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:16:36.565 INFO:teuthology.orchestra.run.smithi175.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:16:36.565 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:16:36.565 INFO:teuthology.orchestra.run.smithi175.stdout:/run/user/1000 2024-04-15T15:16:36.565 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph 2024-04-15T15:16:36.565 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:16:36.565 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:16:36.565 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:16:36.565 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:16:36.565 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns 2024-04-15T15:16:36.565 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:36.565 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:36.565 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:16:36.565 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:16:36.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:36.566 DEBUG:teuthology.orchestra.run.smithi175:> ls /sys/fs/fuse/connections 2024-04-15T15:16:36.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi175.stderr:2024-04-15T15:16:36.565+0000 7f019343c480 -1 init, newargv = 0x55c17637ff50 newargc=15 2024-04-15T15:16:36.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi175.stderr:ceph-fuse[45499]: starting ceph client 2024-04-15T15:16:36.573 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi175.stderr:ceph-fuse[45499]: starting fuse 2024-04-15T15:16:36.582 INFO:teuthology.orchestra.run.smithi175.stdout:43 2024-04-15T15:16:36.582 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-04-15T15:16:36.582 DEBUG:teuthology.orchestra.run.smithi175:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:16:36.582 DEBUG:teuthology.orchestra.run.smithi175:> import glob 2024-04-15T15:16:36.582 DEBUG:teuthology.orchestra.run.smithi175:> import re 2024-04-15T15:16:36.583 DEBUG:teuthology.orchestra.run.smithi175:> import os 2024-04-15T15:16:36.583 DEBUG:teuthology.orchestra.run.smithi175:> import subprocess 2024-04-15T15:16:36.583 DEBUG:teuthology.orchestra.run.smithi175:> 2024-04-15T15:16:36.583 DEBUG:teuthology.orchestra.run.smithi175:> def _find_admin_socket(client_name): 2024-04-15T15:16:36.583 DEBUG:teuthology.orchestra.run.smithi175:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-15T15:16:36.583 DEBUG:teuthology.orchestra.run.smithi175:> files = glob.glob(asok_path) 2024-04-15T15:16:36.583 DEBUG:teuthology.orchestra.run.smithi175:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-15T15:16:36.583 DEBUG:teuthology.orchestra.run.smithi175:> 2024-04-15T15:16:36.583 DEBUG:teuthology.orchestra.run.smithi175:> # Given a non-glob path, it better be there 2024-04-15T15:16:36.583 DEBUG:teuthology.orchestra.run.smithi175:> if "*" not in asok_path: 2024-04-15T15:16:36.583 DEBUG:teuthology.orchestra.run.smithi175:> assert(len(files) == 1) 2024-04-15T15:16:36.583 DEBUG:teuthology.orchestra.run.smithi175:> return files[0] 2024-04-15T15:16:36.583 DEBUG:teuthology.orchestra.run.smithi175:> 2024-04-15T15:16:36.583 DEBUG:teuthology.orchestra.run.smithi175:> for f in files: 2024-04-15T15:16:36.583 DEBUG:teuthology.orchestra.run.smithi175:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:16:36.584 DEBUG:teuthology.orchestra.run.smithi175:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:16:36.584 DEBUG:teuthology.orchestra.run.smithi175:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:16:36.584 DEBUG:teuthology.orchestra.run.smithi175:> contents = proc_f.read() 2024-04-15T15:16:36.584 DEBUG:teuthology.orchestra.run.smithi175:> if mountpoint in contents: 2024-04-15T15:16:36.584 DEBUG:teuthology.orchestra.run.smithi175:> return f 2024-04-15T15:16:36.584 DEBUG:teuthology.orchestra.run.smithi175:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:16:36.584 DEBUG:teuthology.orchestra.run.smithi175:> 2024-04-15T15:16:36.584 DEBUG:teuthology.orchestra.run.smithi175:> print(_find_admin_socket("client.1")) 2024-04-15T15:16:36.584 DEBUG:teuthology.orchestra.run.smithi175:> ' 2024-04-15T15:16:36.704 INFO:teuthology.orchestra.run.smithi175.stdout:/var/run/ceph/ceph-client.1.45499.asok 2024-04-15T15:16:36.707 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:16:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:36.715 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.45499.asok 2024-04-15T15:16:36.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:36.715 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.45499.asok status 2024-04-15T15:16:36.854 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-04-15T15:16:36.855 INFO:teuthology.orchestra.run.smithi175.stdout: "metadata": { 2024-04-15T15:16:36.855 INFO:teuthology.orchestra.run.smithi175.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:16:36.855 INFO:teuthology.orchestra.run.smithi175.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:16:36.855 INFO:teuthology.orchestra.run.smithi175.stdout: "entity_id": "1", 2024-04-15T15:16:36.855 INFO:teuthology.orchestra.run.smithi175.stdout: "hostname": "smithi175", 2024-04-15T15:16:36.855 INFO:teuthology.orchestra.run.smithi175.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-15T15:16:36.855 INFO:teuthology.orchestra.run.smithi175.stdout: "pid": "45499", 2024-04-15T15:16:36.855 INFO:teuthology.orchestra.run.smithi175.stdout: "root": "/volumes/_nogroup/subvol_quiesce/6bdc4ba4-8793-4ca5-8487-64cfa9a984f4" 2024-04-15T15:16:36.855 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-15T15:16:36.855 INFO:teuthology.orchestra.run.smithi175.stdout: "dentry_count": 0, 2024-04-15T15:16:36.855 INFO:teuthology.orchestra.run.smithi175.stdout: "dentry_pinned_count": 0, 2024-04-15T15:16:36.855 INFO:teuthology.orchestra.run.smithi175.stdout: "id": 5075, 2024-04-15T15:16:36.855 INFO:teuthology.orchestra.run.smithi175.stdout: "inst": { 2024-04-15T15:16:36.856 INFO:teuthology.orchestra.run.smithi175.stdout: "name": { 2024-04-15T15:16:36.856 INFO:teuthology.orchestra.run.smithi175.stdout: "type": "client", 2024-04-15T15:16:36.856 INFO:teuthology.orchestra.run.smithi175.stdout: "num": 5075 2024-04-15T15:16:36.856 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-15T15:16:36.856 INFO:teuthology.orchestra.run.smithi175.stdout: "addr": { 2024-04-15T15:16:36.856 INFO:teuthology.orchestra.run.smithi175.stdout: "type": "v1", 2024-04-15T15:16:36.856 INFO:teuthology.orchestra.run.smithi175.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:16:36.856 INFO:teuthology.orchestra.run.smithi175.stdout: "nonce": 534343064 2024-04-15T15:16:36.856 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-15T15:16:36.856 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-15T15:16:36.856 INFO:teuthology.orchestra.run.smithi175.stdout: "addr": { 2024-04-15T15:16:36.856 INFO:teuthology.orchestra.run.smithi175.stdout: "type": "v1", 2024-04-15T15:16:36.856 INFO:teuthology.orchestra.run.smithi175.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:16:36.856 INFO:teuthology.orchestra.run.smithi175.stdout: "nonce": 534343064 2024-04-15T15:16:36.856 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-15T15:16:36.856 INFO:teuthology.orchestra.run.smithi175.stdout: "inst_str": "client.5075 192.168.0.1:0/534343064", 2024-04-15T15:16:36.857 INFO:teuthology.orchestra.run.smithi175.stdout: "addr_str": "192.168.0.1:0/534343064", 2024-04-15T15:16:36.857 INFO:teuthology.orchestra.run.smithi175.stdout: "inode_count": 5, 2024-04-15T15:16:36.857 INFO:teuthology.orchestra.run.smithi175.stdout: "mds_epoch": 16, 2024-04-15T15:16:36.857 INFO:teuthology.orchestra.run.smithi175.stdout: "osd_epoch": 40, 2024-04-15T15:16:36.857 INFO:teuthology.orchestra.run.smithi175.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:16:36.857 INFO:teuthology.orchestra.run.smithi175.stdout: "blocklisted": false, 2024-04-15T15:16:36.857 INFO:teuthology.orchestra.run.smithi175.stdout: "fs_name": "cephfs" 2024-04-15T15:16:36.857 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-04-15T15:16:36.863 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:16:36.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.974+0000 7f2249dd4640 1 Processor -- start 2024-04-15T15:16:36.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.974+0000 7f2249dd4640 1 -- start start 2024-04-15T15:16:36.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.974+0000 7f2249dd4640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f22440cff80 0x7f22440d0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:36.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.974+0000 7f2249dd4640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f22440d0950 0x7f22440ce630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:36.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.974+0000 7f2249dd4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22440d1ab0 con 0x7f224415c6b0 2024-04-15T15:16:36.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.974+0000 7f2249dd4640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f22440d0da0 con 0x7f22440cff80 2024-04-15T15:16:36.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.974+0000 7f2249dd4640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f22440ccaa0 con 0x7f22440d0950 2024-04-15T15:16:36.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.975+0000 7f22437fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f224415c6b0 0x7f224406ff20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:55968/0 (socket says 172.21.15.72:55968) 2024-04-15T15:16:36.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.975+0000 7f22437fe640 1 -- 172.21.15.72:0/4160558329 learned_addr learned my addr 172.21.15.72:0/4160558329 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:36.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.975+0000 7f2242ffd640 1 -- 172.21.15.72:0/4160558329 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3894476882 0 0) 0x7f22440d1ab0 con 0x7f224415c6b0 2024-04-15T15:16:36.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.975+0000 7f2242ffd640 1 -- 172.21.15.72:0/4160558329 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2230003520 con 0x7f224415c6b0 2024-04-15T15:16:36.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.975+0000 7f223affd640 1 --2- 172.21.15.72:0/4160558329 >> v2:172.21.15.175:3300/0 conn(0x7f22440cff80 0x7f22440d0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:36.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.975+0000 7f2243fff640 1 --2- 172.21.15.72:0/4160558329 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f22440d0950 0x7f22440ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:36.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.975+0000 7f2242ffd640 1 -- 172.21.15.72:0/4160558329 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3518402122 0 0) 0x7f2230003520 con 0x7f224415c6b0 2024-04-15T15:16:36.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.975+0000 7f2242ffd640 1 -- 172.21.15.72:0/4160558329 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f22440d0950 msgr2=0x7f22440ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:36.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.975+0000 7f2242ffd640 1 --2- 172.21.15.72:0/4160558329 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f22440d0950 0x7f22440ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.975+0000 7f2242ffd640 1 -- 172.21.15.72:0/4160558329 >> v2:172.21.15.175:3300/0 conn(0x7f22440cff80 msgr2=0x7f22440d0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:36.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.975+0000 7f2242ffd640 1 --2- 172.21.15.72:0/4160558329 >> v2:172.21.15.175:3300/0 conn(0x7f22440cff80 0x7f22440d0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.975+0000 7f2242ffd640 1 -- 172.21.15.72:0/4160558329 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22440ccd80 con 0x7f224415c6b0 2024-04-15T15:16:36.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.975+0000 7f2242ffd640 1 -- 172.21.15.72:0/4160558329 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f2234001670 con 0x7f224415c6b0 2024-04-15T15:16:36.980 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.975+0000 7f2242ffd640 1 -- 172.21.15.72:0/4160558329 >> v1:172.21.15.72:6789/0 conn(0x7f224415c6b0 legacy=0x7f224406ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:36.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.975+0000 7f2242ffd640 1 --2- 172.21.15.72:0/4160558329 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f22300039b0 0x7f2230003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:36.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.975+0000 7f2242ffd640 1 --2- 172.21.15.72:0/4160558329 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f22440d0950 0x7f2230004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:36.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.975+0000 7f2242ffd640 1 --2- 172.21.15.72:0/4160558329 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2230004d40 0x7f22300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:36.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.975+0000 7f2242ffd640 1 -- 172.21.15.72:0/4160558329 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f22440d0da0 con 0x7f2230004d40 2024-04-15T15:16:36.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.975+0000 7f2242ffd640 1 -- 172.21.15.72:0/4160558329 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f22440ccaa0 con 0x7f22300039b0 2024-04-15T15:16:36.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.975+0000 7f2242ffd640 1 -- 172.21.15.72:0/4160558329 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f223000b9c0 con 0x7f22440d0950 2024-04-15T15:16:36.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.975+0000 7f22437fe640 1 --2- 172.21.15.72:0/4160558329 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f22300039b0 0x7f2230003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:36.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.975+0000 7f2243fff640 1 --2- 172.21.15.72:0/4160558329 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2230004d40 0x7f22300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:36.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.975+0000 7f223affd640 1 --2- 172.21.15.72:0/4160558329 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f22440d0950 0x7f2230004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:36.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.976+0000 7f22437fe640 1 -- 172.21.15.72:0/4160558329 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f22440d0950 msgr2=0x7f2230004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:36.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.976+0000 7f22437fe640 1 --2- 172.21.15.72:0/4160558329 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f22440d0950 0x7f2230004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.976+0000 7f22437fe640 1 -- 172.21.15.72:0/4160558329 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2230004d40 msgr2=0x7f22300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:36.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.976+0000 7f22437fe640 1 --2- 172.21.15.72:0/4160558329 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2230004d40 0x7f22300051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.976+0000 7f22437fe640 1 -- 172.21.15.72:0/4160558329 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2230003520 con 0x7f22300039b0 2024-04-15T15:16:36.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.976+0000 7f22437fe640 1 --2- 172.21.15.72:0/4160558329 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f22300039b0 0x7f2230003dd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f223400c160 tx=0x7f2230005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3b6c50879ac1a15 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:36.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.976+0000 7f2242ffd640 1 -- 172.21.15.72:0/4160558329 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2234007c80 con 0x7f22300039b0 2024-04-15T15:16:36.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.977+0000 7f2242ffd640 1 -- 172.21.15.72:0/4160558329 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2234030070 con 0x7f22300039b0 2024-04-15T15:16:36.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.977+0000 7f2242ffd640 1 -- 172.21.15.72:0/4160558329 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2234005d30 con 0x7f22300039b0 2024-04-15T15:16:36.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.977+0000 7f2249dd4640 1 -- 172.21.15.72:0/4160558329 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f22300039b0 msgr2=0x7f2230003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:36.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.977+0000 7f2249dd4640 1 --2- 172.21.15.72:0/4160558329 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f22300039b0 0x7f2230003dd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f223400c160 tx=0x7f2230005830 comp rx=0 tx=0).stop 2024-04-15T15:16:36.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.977+0000 7f2243fff640 1 -- 172.21.15.72:0/4160558329 reap_dead start 2024-04-15T15:16:36.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.977+0000 7f2249dd4640 1 -- 172.21.15.72:0/4160558329 shutdown_connections 2024-04-15T15:16:36.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.977+0000 7f2249dd4640 1 -- 172.21.15.72:0/4160558329 >> 172.21.15.72:0/4160558329 conn(0x7f22440b3fa0 msgr2=0x7f22440b43b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:36.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.977+0000 7f2249dd4640 1 -- 172.21.15.72:0/4160558329 shutdown_connections 2024-04-15T15:16:36.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.977+0000 7f2249dd4640 1 -- 172.21.15.72:0/4160558329 wait complete. 2024-04-15T15:16:36.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.978+0000 7f2249dd4640 1 Processor -- start 2024-04-15T15:16:36.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.979+0000 7f2249dd4640 1 -- start start 2024-04-15T15:16:36.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.979+0000 7f2249dd4640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f22300039b0 0x7f22441f4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:36.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.979+0000 7f2249dd4640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2244158e20 0x7f224406e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:36.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.979+0000 7f2249dd4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2244156ce0 0x7f2244157160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:36.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.979+0000 7f2249dd4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f22440cd1e0 con 0x7f2244156ce0 2024-04-15T15:16:36.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.979+0000 7f2249dd4640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f22440cd060 con 0x7f2244158e20 2024-04-15T15:16:36.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.979+0000 7f2249dd4640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f22440ce630 con 0x7f22300039b0 2024-04-15T15:16:36.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.979+0000 7f2243fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2244156ce0 0x7f2244157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:36.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.979+0000 7f2243fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2244156ce0 0x7f2244157160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60494/0 (socket says 172.21.15.72:60494) 2024-04-15T15:16:36.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.979+0000 7f2243fff640 1 -- 172.21.15.72:0/775658693 learned_addr learned my addr 172.21.15.72:0/775658693 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:36.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.979+0000 7f223affd640 1 --2- 172.21.15.72:0/775658693 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2244158e20 0x7f224406e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:36.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.979+0000 7f22437fe640 1 --2- 172.21.15.72:0/775658693 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f22300039b0 0x7f22441f4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:36.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.980+0000 7f2243fff640 1 -- 172.21.15.72:0/775658693 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f22300039b0 msgr2=0x7f22441f4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:36.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.980+0000 7f2243fff640 1 --2- 172.21.15.72:0/775658693 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f22300039b0 0x7f22441f4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.980+0000 7f2243fff640 1 -- 172.21.15.72:0/775658693 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2244158e20 msgr2=0x7f224406e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:36.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.980+0000 7f2243fff640 1 --2- 172.21.15.72:0/775658693 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2244158e20 0x7f224406e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.986 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.980+0000 7f2243fff640 1 -- 172.21.15.72:0/775658693 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22441f5140 con 0x7f2244156ce0 2024-04-15T15:16:36.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.980+0000 7f2243fff640 1 --2- 172.21.15.72:0/775658693 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2244156ce0 0x7f2244157160 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f22440707b0 tx=0x7f223c00f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:36.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.980+0000 7f2240ff9640 1 -- 172.21.15.72:0/775658693 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f223c00fcb0 con 0x7f2244156ce0 2024-04-15T15:16:36.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.980+0000 7f2240ff9640 1 -- 172.21.15.72:0/775658693 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f223c01d070 con 0x7f2244156ce0 2024-04-15T15:16:36.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.980+0000 7f2249dd4640 1 -- 172.21.15.72:0/775658693 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22441593e0 con 0x7f2244156ce0 2024-04-15T15:16:36.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.980+0000 7f2240ff9640 1 -- 172.21.15.72:0/775658693 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f223c015670 con 0x7f2244156ce0 2024-04-15T15:16:36.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.980+0000 7f2249dd4640 1 -- 172.21.15.72:0/775658693 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22441599a0 con 0x7f2244156ce0 2024-04-15T15:16:36.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.981+0000 7f2249dd4640 1 -- 172.21.15.72:0/775658693 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22440690f0 con 0x7f2244156ce0 2024-04-15T15:16:36.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.981+0000 7f2240ff9640 1 -- 172.21.15.72:0/775658693 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f223c015810 con 0x7f2244156ce0 2024-04-15T15:16:36.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.982+0000 7f2240ff9640 1 --2- 172.21.15.72:0/775658693 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f222406c9b0 0x7f222406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:36.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.982+0000 7f22437fe640 1 --2- 172.21.15.72:0/775658693 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f222406c9b0 0x7f222406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:36.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.982+0000 7f2240ff9640 1 -- 172.21.15.72:0/775658693 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6130+0+0 (secure 0 0 0) 0x7f223c08c5a0 con 0x7f2244156ce0 2024-04-15T15:16:36.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.982+0000 7f22437fe640 1 --2- 172.21.15.72:0/775658693 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f222406c9b0 0x7f222406ee70 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f2234034000 tx=0x7f2234007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:36.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:36.987+0000 7f2240ff9640 1 -- 172.21.15.72:0/775658693 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f223c010040 con 0x7f2244156ce0 2024-04-15T15:16:37.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:37.198+0000 7f2249dd4640 1 -- 172.21.15.72:0/775658693 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f22440b5950 con 0x7f2244156ce0 2024-04-15T15:16:37.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:37.198+0000 7f2240ff9640 1 -- 172.21.15.72:0/775658693 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+73 (secure 0 0 0) 0x7f223c05a720 con 0x7f2244156ce0 2024-04-15T15:16:37.200 INFO:teuthology.orchestra.run.smithi072.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-15T15:16:37.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:37.200+0000 7f2249dd4640 1 -- 172.21.15.72:0/775658693 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f222406c9b0 msgr2=0x7f222406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:37.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:37.200+0000 7f2249dd4640 1 --2- 172.21.15.72:0/775658693 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f222406c9b0 0x7f222406ee70 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f2234034000 tx=0x7f2234007000 comp rx=0 tx=0).stop 2024-04-15T15:16:37.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:37.200+0000 7f2249dd4640 1 -- 172.21.15.72:0/775658693 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2244156ce0 msgr2=0x7f2244157160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:37.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:37.200+0000 7f2249dd4640 1 --2- 172.21.15.72:0/775658693 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2244156ce0 0x7f2244157160 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f22440707b0 tx=0x7f223c00f1b0 comp rx=0 tx=0).stop 2024-04-15T15:16:37.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:37.200+0000 7f2249dd4640 1 -- 172.21.15.72:0/775658693 shutdown_connections 2024-04-15T15:16:37.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:37.200+0000 7f2249dd4640 1 --2- 172.21.15.72:0/775658693 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f222406c9b0 0x7f222406ee70 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:37.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:37.200+0000 7f2249dd4640 1 --2- 172.21.15.72:0/775658693 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2244156ce0 0x7f2244157160 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:37.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:37.200+0000 7f2249dd4640 1 --2- 172.21.15.72:0/775658693 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2244158e20 0x7f224406e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:37.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:37.200+0000 7f2249dd4640 1 --2- 172.21.15.72:0/775658693 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f22300039b0 0x7f22441f4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:37.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:37.200+0000 7f2249dd4640 1 -- 172.21.15.72:0/775658693 >> 172.21.15.72:0/775658693 conn(0x7f22440b3fa0 msgr2=0x7f224406fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:37.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:37.200+0000 7f2249dd4640 1 -- 172.21.15.72:0/775658693 shutdown_connections 2024-04-15T15:16:37.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:37.200+0000 7f2249dd4640 1 -- 172.21.15.72:0/775658693 wait complete. 2024-04-15T15:16:37.212 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:16:37.212 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:16:37.212 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi072.front.sepia.ceph.com 2024-04-15T15:16:37.212 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-04-15T15:16:37.212 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-15T15:16:37.212 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:16:37.212 INFO:tasks.cephfs.mount:self.cephfs_mntpt = /volumes/_nogroup/subvol_quiesce/6bdc4ba4-8793-4ca5-8487-64cfa9a984f4 2024-04-15T15:16:37.213 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:16:37.213 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.2' netns for '/home/ubuntu/cephtest/mnt.2' 2024-04-15T15:16:37.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:37.213 DEBUG:teuthology.orchestra.run.smithi072:> ip addr 2024-04-15T15:16:37.229 INFO:teuthology.orchestra.run.smithi072.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:16:37.229 INFO:teuthology.orchestra.run.smithi072.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:16:37.229 INFO:teuthology.orchestra.run.smithi072.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:16:37.229 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:37.229 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 ::1/128 scope host 2024-04-15T15:16:37.229 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:37.229 INFO:teuthology.orchestra.run.smithi072.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:37.229 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f4 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:37.230 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f0 2024-04-15T15:16:37.230 INFO:teuthology.orchestra.run.smithi072.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:37.230 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:d8:da:f5 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:37.230 INFO:teuthology.orchestra.run.smithi072.stdout: altname enp7s0f1 2024-04-15T15:16:37.230 INFO:teuthology.orchestra.run.smithi072.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:37.230 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:64 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:37.230 INFO:teuthology.orchestra.run.smithi072.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:16:37.230 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 0c:c4:7a:8f:b7:65 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:37.230 INFO:teuthology.orchestra.run.smithi072.stdout: inet 172.21.15.72/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-15T15:16:37.230 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft 42606sec preferred_lft 42606sec 2024-04-15T15:16:37.230 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::ec4:7aff:fe8f:b765/64 scope link 2024-04-15T15:16:37.230 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:37.230 INFO:teuthology.orchestra.run.smithi072.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:16:37.230 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 62:2f:60:9d:8a:ab brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:37.231 INFO:teuthology.orchestra.run.smithi072.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-15T15:16:37.231 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:37.231 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::6871:2cff:fe2a:4823/64 scope link 2024-04-15T15:16:37.231 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:37.231 INFO:teuthology.orchestra.run.smithi072.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:16:37.231 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether b6:ae:7a:2e:38:96 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:37.231 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::b4ae:7aff:fe2e:3896/64 scope link 2024-04-15T15:16:37.231 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:37.231 INFO:teuthology.orchestra.run.smithi072.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:16:37.231 INFO:teuthology.orchestra.run.smithi072.stdout: link/ether 62:2f:60:9d:8a:ab brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:16:37.231 INFO:teuthology.orchestra.run.smithi072.stdout: inet6 fe80::602f:60ff:fe9d:8aab/64 scope link 2024-04-15T15:16:37.231 INFO:teuthology.orchestra.run.smithi072.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:37.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:37.232 DEBUG:teuthology.orchestra.run.smithi072:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:16:37.296 INFO:teuthology.orchestra.run.smithi072.stdout:1 2024-04-15T15:16:37.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:37.298 DEBUG:teuthology.orchestra.run.smithi072:> route 2024-04-15T15:16:37.355 INFO:teuthology.orchestra.run.smithi072.stdout:Kernel IP routing table 2024-04-15T15:16:37.355 INFO:teuthology.orchestra.run.smithi072.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:16:37.355 INFO:teuthology.orchestra.run.smithi072.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-15T15:16:37.355 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-15T15:16:37.355 INFO:teuthology.orchestra.run.smithi072.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:16:37.355 DEBUG:teuthology.orchestra.run.smithi072:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:16:37.355 DEBUG:teuthology.orchestra.run.smithi072:> set -e 2024-04-15T15:16:37.355 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-15T15:16:37.356 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-15T15:16:37.356 DEBUG:teuthology.orchestra.run.smithi072:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-15T15:16:37.356 DEBUG:teuthology.orchestra.run.smithi072:> ') 2024-04-15T15:16:37.444 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:16:37.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:37.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:37.494 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-04-15T15:16:37.510 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-15T15:16:37.510 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-15T15:16:37.510 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.2' netns for '/home/ubuntu/cephtest/mnt.2' 2024-04-15T15:16:37.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:37.510 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link set brx.1 up 2024-04-15T15:16:37.580 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-04-15T15:16:37.580 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:16:37.580 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-04-15T15:16:37.635 INFO:teuthology.orchestra.run.smithi072.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-04-15T15:16:37.635 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:16:37.635 DEBUG:teuthology.orchestra.run.smithi072:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-04-15T15:16:37.692 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-04-15T15:16:37.761 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:16:37.818 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-04-15T15:16:37.818 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-04-15T15:16:37.818 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-04-15T15:16:37.818 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-04-15T15:16:37.818 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-04-15T15:16:37.818 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-04-15T15:16:37.818 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-04-15T15:16:37.818 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-04-15T15:16:37.819 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-04-15T15:16:37.819 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-04-15T15:16:37.819 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-04-15T15:16:37.819 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-04-15T15:16:37.819 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:16:37.819 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-04-15T15:16:37.819 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-04-15T15:16:37.819 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-04-15T15:16:37.819 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-04-15T15:16:37.819 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:16:37.819 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-04-15T15:16:37.819 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-04-15T15:16:37.819 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:16:37.819 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:16:37.819 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:16:37.820 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-04-15T15:16:37.820 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-04-15T15:16:37.820 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:16:37.820 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:16:37.820 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:16:37.820 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:16:37.820 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-04-15T15:16:37.820 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:37.820 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:37.820 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:16:37.820 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:16:37.820 INFO:teuthology.orchestra.run.smithi072.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-15T15:16:37.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:37.821 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-04-15T15:16:37.874 INFO:teuthology.orchestra.run.smithi072.stdout:43 2024-04-15T15:16:37.874 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2024-04-15T15:16:37.874 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2 --client_mountpoint=/volumes/_nogroup/subvol_quiesce/6bdc4ba4-8793-4ca5-8487-64cfa9a984f4) 2024-04-15T15:16:37.919 DEBUG:teuthology.orchestra.run.smithi072:> sudo modprobe fuse 2024-04-15T15:16:37.947 DEBUG:teuthology.orchestra.run.smithi072:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:16:38.003 INFO:teuthology.orchestra.run.smithi072.stdout:/proc 2024-04-15T15:16:38.004 INFO:teuthology.orchestra.run.smithi072.stdout:/sys 2024-04-15T15:16:38.004 INFO:teuthology.orchestra.run.smithi072.stdout:/dev 2024-04-15T15:16:38.004 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/security 2024-04-15T15:16:38.004 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/shm 2024-04-15T15:16:38.004 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/pts 2024-04-15T15:16:38.004 INFO:teuthology.orchestra.run.smithi072.stdout:/run 2024-04-15T15:16:38.004 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/cgroup 2024-04-15T15:16:38.004 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/pstore 2024-04-15T15:16:38.004 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/bpf 2024-04-15T15:16:38.004 INFO:teuthology.orchestra.run.smithi072.stdout:/ 2024-04-15T15:16:38.004 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/selinux 2024-04-15T15:16:38.004 INFO:teuthology.orchestra.run.smithi072.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:16:38.004 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/mqueue 2024-04-15T15:16:38.005 INFO:teuthology.orchestra.run.smithi072.stdout:/dev/hugepages 2024-04-15T15:16:38.005 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/debug 2024-04-15T15:16:38.005 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/tracing 2024-04-15T15:16:38.005 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:16:38.005 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/fs/fuse/connections 2024-04-15T15:16:38.005 INFO:teuthology.orchestra.run.smithi072.stdout:/sys/kernel/config 2024-04-15T15:16:38.005 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:16:38.005 INFO:teuthology.orchestra.run.smithi072.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:16:38.005 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:16:38.005 INFO:teuthology.orchestra.run.smithi072.stdout:/run/user/1000 2024-04-15T15:16:38.005 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph 2024-04-15T15:16:38.005 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:16:38.005 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:16:38.005 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:16:38.005 INFO:teuthology.orchestra.run.smithi072.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:16:38.005 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns 2024-04-15T15:16:38.006 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:38.006 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:38.006 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:16:38.006 INFO:teuthology.orchestra.run.smithi072.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:16:38.006 INFO:teuthology.orchestra.run.smithi072.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-15T15:16:38.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:38.006 DEBUG:teuthology.orchestra.run.smithi072:> ls /sys/fs/fuse/connections 2024-04-15T15:16:38.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi072.stderr:2024-04-15T15:16:38.006+0000 7fd24b640480 -1 init, newargv = 0x5575e300b7a0 newargc=15 2024-04-15T15:16:38.008 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi072.stderr:ceph-fuse[56518]: starting ceph client 2024-04-15T15:16:38.014 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi072.stderr:ceph-fuse[56518]: starting fuse 2024-04-15T15:16:38.023 INFO:teuthology.orchestra.run.smithi072.stdout:43 2024-04-15T15:16:38.023 INFO:teuthology.orchestra.run.smithi072.stdout:44 2024-04-15T15:16:38.023 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2024-04-15T15:16:38.024 DEBUG:teuthology.orchestra.run.smithi072:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:16:38.024 DEBUG:teuthology.orchestra.run.smithi072:> import glob 2024-04-15T15:16:38.024 DEBUG:teuthology.orchestra.run.smithi072:> import re 2024-04-15T15:16:38.024 DEBUG:teuthology.orchestra.run.smithi072:> import os 2024-04-15T15:16:38.024 DEBUG:teuthology.orchestra.run.smithi072:> import subprocess 2024-04-15T15:16:38.024 DEBUG:teuthology.orchestra.run.smithi072:> 2024-04-15T15:16:38.024 DEBUG:teuthology.orchestra.run.smithi072:> def _find_admin_socket(client_name): 2024-04-15T15:16:38.024 DEBUG:teuthology.orchestra.run.smithi072:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-04-15T15:16:38.024 DEBUG:teuthology.orchestra.run.smithi072:> files = glob.glob(asok_path) 2024-04-15T15:16:38.024 DEBUG:teuthology.orchestra.run.smithi072:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-04-15T15:16:38.024 DEBUG:teuthology.orchestra.run.smithi072:> 2024-04-15T15:16:38.024 DEBUG:teuthology.orchestra.run.smithi072:> # Given a non-glob path, it better be there 2024-04-15T15:16:38.024 DEBUG:teuthology.orchestra.run.smithi072:> if "*" not in asok_path: 2024-04-15T15:16:38.024 DEBUG:teuthology.orchestra.run.smithi072:> assert(len(files) == 1) 2024-04-15T15:16:38.025 DEBUG:teuthology.orchestra.run.smithi072:> return files[0] 2024-04-15T15:16:38.025 DEBUG:teuthology.orchestra.run.smithi072:> 2024-04-15T15:16:38.025 DEBUG:teuthology.orchestra.run.smithi072:> for f in files: 2024-04-15T15:16:38.025 DEBUG:teuthology.orchestra.run.smithi072:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:16:38.025 DEBUG:teuthology.orchestra.run.smithi072:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:16:38.025 DEBUG:teuthology.orchestra.run.smithi072:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:16:38.025 DEBUG:teuthology.orchestra.run.smithi072:> contents = proc_f.read() 2024-04-15T15:16:38.025 DEBUG:teuthology.orchestra.run.smithi072:> if mountpoint in contents: 2024-04-15T15:16:38.025 DEBUG:teuthology.orchestra.run.smithi072:> return f 2024-04-15T15:16:38.025 DEBUG:teuthology.orchestra.run.smithi072:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:16:38.025 DEBUG:teuthology.orchestra.run.smithi072:> 2024-04-15T15:16:38.025 DEBUG:teuthology.orchestra.run.smithi072:> print(_find_admin_socket("client.2")) 2024-04-15T15:16:38.025 DEBUG:teuthology.orchestra.run.smithi072:> ' 2024-04-15T15:16:38.145 INFO:teuthology.orchestra.run.smithi072.stdout:/var/run/ceph/ceph-client.2.56518.asok 2024-04-15T15:16:38.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:38.156 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.56518.asok 2024-04-15T15:16:38.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:38.156 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.56518.asok status 2024-04-15T15:16:38.294 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-04-15T15:16:38.294 INFO:teuthology.orchestra.run.smithi072.stdout: "metadata": { 2024-04-15T15:16:38.294 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:16:38.295 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:16:38.295 INFO:teuthology.orchestra.run.smithi072.stdout: "entity_id": "2", 2024-04-15T15:16:38.295 INFO:teuthology.orchestra.run.smithi072.stdout: "hostname": "smithi072", 2024-04-15T15:16:38.295 INFO:teuthology.orchestra.run.smithi072.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-04-15T15:16:38.295 INFO:teuthology.orchestra.run.smithi072.stdout: "pid": "56518", 2024-04-15T15:16:38.295 INFO:teuthology.orchestra.run.smithi072.stdout: "root": "/volumes/_nogroup/subvol_quiesce/6bdc4ba4-8793-4ca5-8487-64cfa9a984f4" 2024-04-15T15:16:38.295 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:38.295 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_count": 0, 2024-04-15T15:16:38.295 INFO:teuthology.orchestra.run.smithi072.stdout: "dentry_pinned_count": 0, 2024-04-15T15:16:38.295 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 5991, 2024-04-15T15:16:38.295 INFO:teuthology.orchestra.run.smithi072.stdout: "inst": { 2024-04-15T15:16:38.295 INFO:teuthology.orchestra.run.smithi072.stdout: "name": { 2024-04-15T15:16:38.295 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "client", 2024-04-15T15:16:38.295 INFO:teuthology.orchestra.run.smithi072.stdout: "num": 5991 2024-04-15T15:16:38.295 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:38.296 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-04-15T15:16:38.296 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-04-15T15:16:38.296 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.2:0", 2024-04-15T15:16:38.296 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 1017049788 2024-04-15T15:16:38.296 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:16:38.296 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:38.296 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": { 2024-04-15T15:16:38.296 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "v1", 2024-04-15T15:16:38.296 INFO:teuthology.orchestra.run.smithi072.stdout: "addr": "192.168.0.2:0", 2024-04-15T15:16:38.296 INFO:teuthology.orchestra.run.smithi072.stdout: "nonce": 1017049788 2024-04-15T15:16:38.296 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:38.296 INFO:teuthology.orchestra.run.smithi072.stdout: "inst_str": "client.5991 192.168.0.2:0/1017049788", 2024-04-15T15:16:38.297 INFO:teuthology.orchestra.run.smithi072.stdout: "addr_str": "192.168.0.2:0/1017049788", 2024-04-15T15:16:38.297 INFO:teuthology.orchestra.run.smithi072.stdout: "inode_count": 5, 2024-04-15T15:16:38.297 INFO:teuthology.orchestra.run.smithi072.stdout: "mds_epoch": 16, 2024-04-15T15:16:38.297 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch": 40, 2024-04-15T15:16:38.297 INFO:teuthology.orchestra.run.smithi072.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:16:38.297 INFO:teuthology.orchestra.run.smithi072.stdout: "blocklisted": false, 2024-04-15T15:16:38.297 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs" 2024-04-15T15:16:38.297 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-04-15T15:16:38.303 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:16:38.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.404+0000 7f2f815ba640 1 Processor -- start 2024-04-15T15:16:38.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.404+0000 7f2f815ba640 1 -- start start 2024-04-15T15:16:38.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.405+0000 7f2f815ba640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2f7c161fb0 0x7f2f7c162450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.405+0000 7f2f815ba640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2f7c0c9570 0x7f2f7c162990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.405+0000 7f2f815ba640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f7c0ccd40 con 0x7f2f7c0d12f0 2024-04-15T15:16:38.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.405+0000 7f2f815ba640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f2f7c0ccaf0 con 0x7f2f7c0c9570 2024-04-15T15:16:38.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.405+0000 7f2f815ba640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f2f7c0cd020 con 0x7f2f7c161fb0 2024-04-15T15:16:38.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.405+0000 7f2f7affd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f2f7c0d12f0 0x7f2f7c1618a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:55984/0 (socket says 172.21.15.72:55984) 2024-04-15T15:16:38.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.405+0000 7f2f7b7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f2f7c0c9570 0x7f2f7c162990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.405+0000 7f2f7affd640 1 -- 172.21.15.72:0/4045044359 learned_addr learned my addr 172.21.15.72:0/4045044359 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:38.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.405+0000 7f2f7b7fe640 1 -- 172.21.15.72:0/4045044359 >> v2:172.21.15.175:3300/0 conn(0x7f2f7c161fb0 msgr2=0x7f2f7c162450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:38.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.405+0000 7f2f7a7fc640 1 --2- 172.21.15.72:0/4045044359 >> v2:172.21.15.175:3300/0 conn(0x7f2f7c161fb0 0x7f2f7c162450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.405+0000 7f2f7b7fe640 1 --2- 172.21.15.72:0/4045044359 >> v2:172.21.15.175:3300/0 conn(0x7f2f7c161fb0 0x7f2f7c162450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.405+0000 7f2f79ffb640 1 -- 172.21.15.72:0/4045044359 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2977426585 0 0) 0x7f2f7c0ccd40 con 0x7f2f7c0d12f0 2024-04-15T15:16:38.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.405+0000 7f2f7b7fe640 1 -- 172.21.15.72:0/4045044359 >> v1:172.21.15.72:6789/0 conn(0x7f2f7c0d12f0 legacy=0x7f2f7c1618a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.405+0000 7f2f7b7fe640 1 -- 172.21.15.72:0/4045044359 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f7c0cd300 con 0x7f2f7c0c9570 2024-04-15T15:16:38.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.406+0000 7f2f7a7fc640 1 --2- 172.21.15.72:0/4045044359 >> v2:172.21.15.175:3300/0 conn(0x7f2f7c161fb0 0x7f2f7c162450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:38.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.406+0000 7f2f7b7fe640 1 --2- 172.21.15.72:0/4045044359 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f7c0c9570 0x7f2f7c162990 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f2f74009d90 tx=0x7f2f7402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bce6f3e34992b938 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:38.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.406+0000 7f2f79ffb640 1 -- 172.21.15.72:0/4045044359 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f74039070 con 0x7f2f7c0c9570 2024-04-15T15:16:38.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.406+0000 7f2f79ffb640 1 -- 172.21.15.72:0/4045044359 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f7402cc60 con 0x7f2f7c0c9570 2024-04-15T15:16:38.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.406+0000 7f2f79ffb640 1 -- 172.21.15.72:0/4045044359 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f74034660 con 0x7f2f7c0c9570 2024-04-15T15:16:38.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.407+0000 7f2f815ba640 1 -- 172.21.15.72:0/4045044359 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f7c0c9570 msgr2=0x7f2f7c162990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.407+0000 7f2f815ba640 1 --2- 172.21.15.72:0/4045044359 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f7c0c9570 0x7f2f7c162990 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f2f74009d90 tx=0x7f2f7402beb0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.407+0000 7f2f815ba640 1 -- 172.21.15.72:0/4045044359 shutdown_connections 2024-04-15T15:16:38.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.407+0000 7f2f815ba640 1 --2- 172.21.15.72:0/4045044359 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f7c0c9570 0x7f2f7c162990 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.407+0000 7f2f815ba640 1 --2- 172.21.15.72:0/4045044359 >> v2:172.21.15.175:3300/0 conn(0x7f2f7c161fb0 0x7f2f7c162450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.407+0000 7f2f815ba640 1 -- 172.21.15.72:0/4045044359 >> 172.21.15.72:0/4045044359 conn(0x7f2f7c0b3f90 msgr2=0x7f2f7c0b43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:38.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.407+0000 7f2f815ba640 1 -- 172.21.15.72:0/4045044359 shutdown_connections 2024-04-15T15:16:38.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.407+0000 7f2f815ba640 1 -- 172.21.15.72:0/4045044359 wait complete. 2024-04-15T15:16:38.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.408+0000 7f2f815ba640 1 Processor -- start 2024-04-15T15:16:38.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.408+0000 7f2f815ba640 1 -- start start 2024-04-15T15:16:38.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.409+0000 7f2f815ba640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f7c0c9570 0x7f2f7c0ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.409+0000 7f2f815ba640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f7c0d12f0 0x7f2f7c15aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.409+0000 7f2f815ba640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2f7c161fb0 0x7f2f7c147a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.409+0000 7f2f815ba640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2f7c0d2210 con 0x7f2f7c0d12f0 2024-04-15T15:16:38.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.409+0000 7f2f815ba640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f2f7c0d2090 con 0x7f2f7c0c9570 2024-04-15T15:16:38.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.409+0000 7f2f815ba640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f2f7c0b68e0 con 0x7f2f7c161fb0 2024-04-15T15:16:38.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.409+0000 7f2f7a7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f7c0d12f0 0x7f2f7c15aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.409+0000 7f2f7a7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f7c0d12f0 0x7f2f7c15aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60522/0 (socket says 172.21.15.72:60522) 2024-04-15T15:16:38.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.409+0000 7f2f7a7fc640 1 -- 172.21.15.72:0/2788707652 learned_addr learned my addr 172.21.15.72:0/2788707652 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:38.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.409+0000 7f2f7affd640 1 --2- 172.21.15.72:0/2788707652 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f7c0c9570 0x7f2f7c0ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.409+0000 7f2f7b7fe640 1 --2- 172.21.15.72:0/2788707652 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2f7c161fb0 0x7f2f7c147a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.409+0000 7f2f7a7fc640 1 -- 172.21.15.72:0/2788707652 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2f7c161fb0 msgr2=0x7f2f7c147a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.409+0000 7f2f7a7fc640 1 --2- 172.21.15.72:0/2788707652 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2f7c161fb0 0x7f2f7c147a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.409+0000 7f2f7a7fc640 1 -- 172.21.15.72:0/2788707652 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f7c0c9570 msgr2=0x7f2f7c0ca6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.409+0000 7f2f7a7fc640 1 --2- 172.21.15.72:0/2788707652 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f7c0c9570 0x7f2f7c0ca6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.409+0000 7f2f7a7fc640 1 -- 172.21.15.72:0/2788707652 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f7c147f50 con 0x7f2f7c0d12f0 2024-04-15T15:16:38.413 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.410+0000 7f2f7a7fc640 1 --2- 172.21.15.72:0/2788707652 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f7c0d12f0 0x7f2f7c15aab0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f2f7000c7f0 tx=0x7f2f7000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:38.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.410+0000 7f2f637fe640 1 -- 172.21.15.72:0/2788707652 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f70015070 con 0x7f2f7c0d12f0 2024-04-15T15:16:38.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.410+0000 7f2f637fe640 1 -- 172.21.15.72:0/2788707652 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f70007820 con 0x7f2f7c0d12f0 2024-04-15T15:16:38.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.410+0000 7f2f815ba640 1 -- 172.21.15.72:0/2788707652 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f7c0cad90 con 0x7f2f7c0d12f0 2024-04-15T15:16:38.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.410+0000 7f2f637fe640 1 -- 172.21.15.72:0/2788707652 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f70007bf0 con 0x7f2f7c0d12f0 2024-04-15T15:16:38.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.410+0000 7f2f815ba640 1 -- 172.21.15.72:0/2788707652 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f7c1484a0 con 0x7f2f7c0d12f0 2024-04-15T15:16:38.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.411+0000 7f2f815ba640 1 -- 172.21.15.72:0/2788707652 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f7c0689e0 con 0x7f2f7c0d12f0 2024-04-15T15:16:38.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.411+0000 7f2f637fe640 1 -- 172.21.15.72:0/2788707652 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f2f70018070 con 0x7f2f7c0d12f0 2024-04-15T15:16:38.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.412+0000 7f2f637fe640 1 --2- 172.21.15.72:0/2788707652 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2f6806c980 0x7f2f6806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.412+0000 7f2f637fe640 1 -- 172.21.15.72:0/2788707652 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6130+0+0 (secure 0 0 0) 0x7f2f7008bc30 con 0x7f2f7c0d12f0 2024-04-15T15:16:38.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.412+0000 7f2f7affd640 1 --2- 172.21.15.72:0/2788707652 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2f6806c980 0x7f2f6806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.412+0000 7f2f7affd640 1 --2- 172.21.15.72:0/2788707652 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2f6806c980 0x7f2f6806ee40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c000f60 tx=0x7f2f6c007d50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:38.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.416+0000 7f2f637fe640 1 -- 172.21.15.72:0/2788707652 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2f700597f0 con 0x7f2f7c0d12f0 2024-04-15T15:16:38.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.629+0000 7f2f815ba640 1 -- 172.21.15.72:0/2788707652 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f2f7c0cdce0 con 0x7f2f7c0d12f0 2024-04-15T15:16:38.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.629+0000 7f2f637fe640 1 -- 172.21.15.72:0/2788707652 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+73 (secure 0 0 0) 0x7f2f7005daf0 con 0x7f2f7c0d12f0 2024-04-15T15:16:38.631 INFO:teuthology.orchestra.run.smithi072.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-15T15:16:38.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.631+0000 7f2f815ba640 1 -- 172.21.15.72:0/2788707652 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2f6806c980 msgr2=0x7f2f6806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.631+0000 7f2f815ba640 1 --2- 172.21.15.72:0/2788707652 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2f6806c980 0x7f2f6806ee40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c000f60 tx=0x7f2f6c007d50 comp rx=0 tx=0).stop 2024-04-15T15:16:38.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.631+0000 7f2f815ba640 1 -- 172.21.15.72:0/2788707652 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f7c0d12f0 msgr2=0x7f2f7c15aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.631+0000 7f2f815ba640 1 --2- 172.21.15.72:0/2788707652 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f7c0d12f0 0x7f2f7c15aab0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f2f7000c7f0 tx=0x7f2f7000ccb0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.631+0000 7f2f815ba640 1 -- 172.21.15.72:0/2788707652 shutdown_connections 2024-04-15T15:16:38.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.631+0000 7f2f815ba640 1 --2- 172.21.15.72:0/2788707652 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2f6806c980 0x7f2f6806ee40 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.631+0000 7f2f815ba640 1 --2- 172.21.15.72:0/2788707652 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2f7c161fb0 0x7f2f7c147a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.631+0000 7f2f815ba640 1 --2- 172.21.15.72:0/2788707652 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2f7c0d12f0 0x7f2f7c15aab0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.631+0000 7f2f815ba640 1 --2- 172.21.15.72:0/2788707652 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2f7c0c9570 0x7f2f7c0ca6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.631+0000 7f2f815ba640 1 -- 172.21.15.72:0/2788707652 >> 172.21.15.72:0/2788707652 conn(0x7f2f7c0b3f90 msgr2=0x7f2f7c0b4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:38.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.631+0000 7f2f815ba640 1 -- 172.21.15.72:0/2788707652 shutdown_connections 2024-04-15T15:16:38.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:38.631+0000 7f2f815ba640 1 -- 172.21.15.72:0/2788707652 wait complete. 2024-04-15T15:16:38.643 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:16:38.643 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:16:38.643 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi175.front.sepia.ceph.com 2024-04-15T15:16:38.643 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-04-15T15:16:38.644 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-15T15:16:38.644 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:16:38.644 INFO:tasks.cephfs.mount:self.cephfs_mntpt = /volumes/_nogroup/subvol_quiesce/6bdc4ba4-8793-4ca5-8487-64cfa9a984f4 2024-04-15T15:16:38.644 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:16:38.644 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.3' netns for '/home/ubuntu/cephtest/mnt.3' 2024-04-15T15:16:38.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:38.644 DEBUG:teuthology.orchestra.run.smithi175:> ip addr 2024-04-15T15:16:38.664 INFO:teuthology.orchestra.run.smithi175.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:16:38.664 INFO:teuthology.orchestra.run.smithi175.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:16:38.664 INFO:teuthology.orchestra.run.smithi175.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:16:38.664 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:38.664 INFO:teuthology.orchestra.run.smithi175.stdout: inet6 ::1/128 scope host 2024-04-15T15:16:38.665 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:38.665 INFO:teuthology.orchestra.run.smithi175.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:38.665 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether 0c:c4:7a:d9:39:ee brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:38.665 INFO:teuthology.orchestra.run.smithi175.stdout: altname enp7s0f0 2024-04-15T15:16:38.665 INFO:teuthology.orchestra.run.smithi175.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:38.665 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether 0c:c4:7a:d9:39:ef brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:38.665 INFO:teuthology.orchestra.run.smithi175.stdout: altname enp7s0f1 2024-04-15T15:16:38.665 INFO:teuthology.orchestra.run.smithi175.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:38.665 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether 0c:c4:7a:88:71:f0 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:38.665 INFO:teuthology.orchestra.run.smithi175.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:16:38.665 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether 0c:c4:7a:88:71:f1 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:38.665 INFO:teuthology.orchestra.run.smithi175.stdout: inet 172.21.15.175/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-15T15:16:38.665 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft 42599sec preferred_lft 42599sec 2024-04-15T15:16:38.665 INFO:teuthology.orchestra.run.smithi175.stdout: inet6 fe80::ec4:7aff:fe88:71f1/64 scope link 2024-04-15T15:16:38.665 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:38.666 INFO:teuthology.orchestra.run.smithi175.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:16:38.666 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether 6e:00:c4:f8:75:e4 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:38.666 INFO:teuthology.orchestra.run.smithi175.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-15T15:16:38.666 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:38.666 INFO:teuthology.orchestra.run.smithi175.stdout: inet6 fe80::d4be:d9ff:fe0d:518c/64 scope link 2024-04-15T15:16:38.666 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:38.666 INFO:teuthology.orchestra.run.smithi175.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:16:38.666 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether f2:c6:d9:f6:5e:3e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:38.666 INFO:teuthology.orchestra.run.smithi175.stdout: inet6 fe80::f0c6:d9ff:fef6:5e3e/64 scope link 2024-04-15T15:16:38.666 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:38.666 INFO:teuthology.orchestra.run.smithi175.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:16:38.666 INFO:teuthology.orchestra.run.smithi175.stdout: link/ether 6e:00:c4:f8:75:e4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:16:38.666 INFO:teuthology.orchestra.run.smithi175.stdout: inet6 fe80::6c00:c4ff:fef8:75e4/64 scope link 2024-04-15T15:16:38.666 INFO:teuthology.orchestra.run.smithi175.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:38.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:38.667 DEBUG:teuthology.orchestra.run.smithi175:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:16:38.735 INFO:teuthology.orchestra.run.smithi175.stdout:1 2024-04-15T15:16:38.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:38.737 DEBUG:teuthology.orchestra.run.smithi175:> route 2024-04-15T15:16:38.796 INFO:teuthology.orchestra.run.smithi175.stdout:Kernel IP routing table 2024-04-15T15:16:38.797 INFO:teuthology.orchestra.run.smithi175.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:16:38.797 INFO:teuthology.orchestra.run.smithi175.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-15T15:16:38.797 INFO:teuthology.orchestra.run.smithi175.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-15T15:16:38.797 INFO:teuthology.orchestra.run.smithi175.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:16:38.797 DEBUG:teuthology.orchestra.run.smithi175:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:16:38.797 DEBUG:teuthology.orchestra.run.smithi175:> set -e 2024-04-15T15:16:38.797 DEBUG:teuthology.orchestra.run.smithi175:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-15T15:16:38.797 DEBUG:teuthology.orchestra.run.smithi175:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-15T15:16:38.798 DEBUG:teuthology.orchestra.run.smithi175:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-15T15:16:38.798 DEBUG:teuthology.orchestra.run.smithi175:> ') 2024-04-15T15:16:38.893 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:16:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:16:38.939 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:16:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:38.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:38.946 DEBUG:teuthology.orchestra.run.smithi175:> ip netns list 2024-04-15T15:16:38.962 INFO:teuthology.orchestra.run.smithi175.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-15T15:16:38.963 INFO:teuthology.orchestra.run.smithi175.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-15T15:16:38.963 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.3' netns for '/home/ubuntu/cephtest/mnt.3' 2024-04-15T15:16:38.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:38.963 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip link set brx.1 up 2024-04-15T15:16:39.032 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-04-15T15:16:39.032 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:16:39.032 DEBUG:teuthology.orchestra.run.smithi175:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-04-15T15:16:39.087 INFO:teuthology.orchestra.run.smithi175.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-04-15T15:16:39.088 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:16:39.088 DEBUG:teuthology.orchestra.run.smithi175:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-04-15T15:16:39.147 DEBUG:teuthology.orchestra.run.smithi175:> sudo modprobe fuse 2024-04-15T15:16:39.216 DEBUG:teuthology.orchestra.run.smithi175:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:16:39.273 INFO:teuthology.orchestra.run.smithi175.stdout:/proc 2024-04-15T15:16:39.273 INFO:teuthology.orchestra.run.smithi175.stdout:/sys 2024-04-15T15:16:39.273 INFO:teuthology.orchestra.run.smithi175.stdout:/dev 2024-04-15T15:16:39.273 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/security 2024-04-15T15:16:39.273 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/shm 2024-04-15T15:16:39.273 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/pts 2024-04-15T15:16:39.274 INFO:teuthology.orchestra.run.smithi175.stdout:/run 2024-04-15T15:16:39.274 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/cgroup 2024-04-15T15:16:39.274 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/pstore 2024-04-15T15:16:39.274 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/bpf 2024-04-15T15:16:39.274 INFO:teuthology.orchestra.run.smithi175.stdout:/ 2024-04-15T15:16:39.274 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/selinux 2024-04-15T15:16:39.274 INFO:teuthology.orchestra.run.smithi175.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:16:39.274 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/mqueue 2024-04-15T15:16:39.274 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/hugepages 2024-04-15T15:16:39.274 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/debug 2024-04-15T15:16:39.274 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/tracing 2024-04-15T15:16:39.274 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/fuse/connections 2024-04-15T15:16:39.274 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/config 2024-04-15T15:16:39.274 INFO:teuthology.orchestra.run.smithi175.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:16:39.275 INFO:teuthology.orchestra.run.smithi175.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:16:39.275 INFO:teuthology.orchestra.run.smithi175.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:16:39.275 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:16:39.275 INFO:teuthology.orchestra.run.smithi175.stdout:/run/user/1000 2024-04-15T15:16:39.275 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph 2024-04-15T15:16:39.275 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:16:39.275 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:16:39.275 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:16:39.275 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:16:39.275 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns 2024-04-15T15:16:39.275 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:39.275 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:39.275 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:16:39.275 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:16:39.275 INFO:teuthology.orchestra.run.smithi175.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:39.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:39.276 DEBUG:teuthology.orchestra.run.smithi175:> ls /sys/fs/fuse/connections 2024-04-15T15:16:39.329 INFO:teuthology.orchestra.run.smithi175.stdout:43 2024-04-15T15:16:39.329 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2024-04-15T15:16:39.329 DEBUG:teuthology.orchestra.run.smithi175:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3 --client_mountpoint=/volumes/_nogroup/subvol_quiesce/6bdc4ba4-8793-4ca5-8487-64cfa9a984f4) 2024-04-15T15:16:39.372 DEBUG:teuthology.orchestra.run.smithi175:> sudo modprobe fuse 2024-04-15T15:16:39.400 DEBUG:teuthology.orchestra.run.smithi175:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:16:39.457 INFO:teuthology.orchestra.run.smithi175.stdout:/proc 2024-04-15T15:16:39.457 INFO:teuthology.orchestra.run.smithi175.stdout:/sys 2024-04-15T15:16:39.457 INFO:teuthology.orchestra.run.smithi175.stdout:/dev 2024-04-15T15:16:39.457 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/security 2024-04-15T15:16:39.457 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/shm 2024-04-15T15:16:39.457 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/pts 2024-04-15T15:16:39.457 INFO:teuthology.orchestra.run.smithi175.stdout:/run 2024-04-15T15:16:39.457 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/cgroup 2024-04-15T15:16:39.457 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/pstore 2024-04-15T15:16:39.458 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/bpf 2024-04-15T15:16:39.458 INFO:teuthology.orchestra.run.smithi175.stdout:/ 2024-04-15T15:16:39.458 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/selinux 2024-04-15T15:16:39.458 INFO:teuthology.orchestra.run.smithi175.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:16:39.458 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/mqueue 2024-04-15T15:16:39.458 INFO:teuthology.orchestra.run.smithi175.stdout:/dev/hugepages 2024-04-15T15:16:39.458 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/debug 2024-04-15T15:16:39.458 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/tracing 2024-04-15T15:16:39.458 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/fs/fuse/connections 2024-04-15T15:16:39.458 INFO:teuthology.orchestra.run.smithi175.stdout:/sys/kernel/config 2024-04-15T15:16:39.458 INFO:teuthology.orchestra.run.smithi175.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:16:39.458 INFO:teuthology.orchestra.run.smithi175.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:16:39.458 INFO:teuthology.orchestra.run.smithi175.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:16:39.458 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:16:39.459 INFO:teuthology.orchestra.run.smithi175.stdout:/run/user/1000 2024-04-15T15:16:39.459 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph 2024-04-15T15:16:39.459 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:16:39.459 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:16:39.459 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:16:39.459 INFO:teuthology.orchestra.run.smithi175.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:16:39.459 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns 2024-04-15T15:16:39.459 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:39.459 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:39.459 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:16:39.459 INFO:teuthology.orchestra.run.smithi175.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:16:39.459 INFO:teuthology.orchestra.run.smithi175.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:39.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:39.460 DEBUG:teuthology.orchestra.run.smithi175:> ls /sys/fs/fuse/connections 2024-04-15T15:16:39.460 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi175.stderr:2024-04-15T15:16:39.459+0000 7fabad40c480 -1 init, newargv = 0x56416bb59120 newargc=15 2024-04-15T15:16:39.461 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi175.stderr:ceph-fuse[45840]: starting ceph client 2024-04-15T15:16:39.467 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi175.stderr:ceph-fuse[45840]: starting fuse 2024-04-15T15:16:39.476 INFO:teuthology.orchestra.run.smithi175.stdout:43 2024-04-15T15:16:39.476 INFO:teuthology.orchestra.run.smithi175.stdout:44 2024-04-15T15:16:39.476 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2024-04-15T15:16:39.477 DEBUG:teuthology.orchestra.run.smithi175:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:16:39.477 DEBUG:teuthology.orchestra.run.smithi175:> import glob 2024-04-15T15:16:39.477 DEBUG:teuthology.orchestra.run.smithi175:> import re 2024-04-15T15:16:39.477 DEBUG:teuthology.orchestra.run.smithi175:> import os 2024-04-15T15:16:39.477 DEBUG:teuthology.orchestra.run.smithi175:> import subprocess 2024-04-15T15:16:39.477 DEBUG:teuthology.orchestra.run.smithi175:> 2024-04-15T15:16:39.477 DEBUG:teuthology.orchestra.run.smithi175:> def _find_admin_socket(client_name): 2024-04-15T15:16:39.477 DEBUG:teuthology.orchestra.run.smithi175:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-04-15T15:16:39.477 DEBUG:teuthology.orchestra.run.smithi175:> files = glob.glob(asok_path) 2024-04-15T15:16:39.477 DEBUG:teuthology.orchestra.run.smithi175:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-04-15T15:16:39.477 DEBUG:teuthology.orchestra.run.smithi175:> 2024-04-15T15:16:39.477 DEBUG:teuthology.orchestra.run.smithi175:> # Given a non-glob path, it better be there 2024-04-15T15:16:39.477 DEBUG:teuthology.orchestra.run.smithi175:> if "*" not in asok_path: 2024-04-15T15:16:39.477 DEBUG:teuthology.orchestra.run.smithi175:> assert(len(files) == 1) 2024-04-15T15:16:39.478 DEBUG:teuthology.orchestra.run.smithi175:> return files[0] 2024-04-15T15:16:39.478 DEBUG:teuthology.orchestra.run.smithi175:> 2024-04-15T15:16:39.478 DEBUG:teuthology.orchestra.run.smithi175:> for f in files: 2024-04-15T15:16:39.478 DEBUG:teuthology.orchestra.run.smithi175:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:16:39.478 DEBUG:teuthology.orchestra.run.smithi175:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:16:39.478 DEBUG:teuthology.orchestra.run.smithi175:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:16:39.478 DEBUG:teuthology.orchestra.run.smithi175:> contents = proc_f.read() 2024-04-15T15:16:39.478 DEBUG:teuthology.orchestra.run.smithi175:> if mountpoint in contents: 2024-04-15T15:16:39.478 DEBUG:teuthology.orchestra.run.smithi175:> return f 2024-04-15T15:16:39.478 DEBUG:teuthology.orchestra.run.smithi175:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:16:39.478 DEBUG:teuthology.orchestra.run.smithi175:> 2024-04-15T15:16:39.478 DEBUG:teuthology.orchestra.run.smithi175:> print(_find_admin_socket("client.3")) 2024-04-15T15:16:39.478 DEBUG:teuthology.orchestra.run.smithi175:> ' 2024-04-15T15:16:39.598 INFO:teuthology.orchestra.run.smithi175.stdout:/var/run/ceph/ceph-client.3.45840.asok 2024-04-15T15:16:39.601 INFO:teuthology.orchestra.run.smithi175.stderr:2024-04-15T15:16:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:39.608 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.45840.asok 2024-04-15T15:16:39.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:39.608 DEBUG:teuthology.orchestra.run.smithi175:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.45840.asok status 2024-04-15T15:16:39.748 INFO:teuthology.orchestra.run.smithi175.stdout:{ 2024-04-15T15:16:39.749 INFO:teuthology.orchestra.run.smithi175.stdout: "metadata": { 2024-04-15T15:16:39.749 INFO:teuthology.orchestra.run.smithi175.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:16:39.749 INFO:teuthology.orchestra.run.smithi175.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:16:39.749 INFO:teuthology.orchestra.run.smithi175.stdout: "entity_id": "3", 2024-04-15T15:16:39.749 INFO:teuthology.orchestra.run.smithi175.stdout: "hostname": "smithi175", 2024-04-15T15:16:39.749 INFO:teuthology.orchestra.run.smithi175.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-04-15T15:16:39.749 INFO:teuthology.orchestra.run.smithi175.stdout: "pid": "45840", 2024-04-15T15:16:39.749 INFO:teuthology.orchestra.run.smithi175.stdout: "root": "/volumes/_nogroup/subvol_quiesce/6bdc4ba4-8793-4ca5-8487-64cfa9a984f4" 2024-04-15T15:16:39.749 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-15T15:16:39.749 INFO:teuthology.orchestra.run.smithi175.stdout: "dentry_count": 0, 2024-04-15T15:16:39.749 INFO:teuthology.orchestra.run.smithi175.stdout: "dentry_pinned_count": 0, 2024-04-15T15:16:39.749 INFO:teuthology.orchestra.run.smithi175.stdout: "id": 5090, 2024-04-15T15:16:39.749 INFO:teuthology.orchestra.run.smithi175.stdout: "inst": { 2024-04-15T15:16:39.750 INFO:teuthology.orchestra.run.smithi175.stdout: "name": { 2024-04-15T15:16:39.750 INFO:teuthology.orchestra.run.smithi175.stdout: "type": "client", 2024-04-15T15:16:39.750 INFO:teuthology.orchestra.run.smithi175.stdout: "num": 5090 2024-04-15T15:16:39.750 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-15T15:16:39.750 INFO:teuthology.orchestra.run.smithi175.stdout: "addr": { 2024-04-15T15:16:39.750 INFO:teuthology.orchestra.run.smithi175.stdout: "type": "v1", 2024-04-15T15:16:39.750 INFO:teuthology.orchestra.run.smithi175.stdout: "addr": "192.168.0.2:0", 2024-04-15T15:16:39.750 INFO:teuthology.orchestra.run.smithi175.stdout: "nonce": 3869342304 2024-04-15T15:16:39.750 INFO:teuthology.orchestra.run.smithi175.stdout: } 2024-04-15T15:16:39.750 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-15T15:16:39.750 INFO:teuthology.orchestra.run.smithi175.stdout: "addr": { 2024-04-15T15:16:39.750 INFO:teuthology.orchestra.run.smithi175.stdout: "type": "v1", 2024-04-15T15:16:39.750 INFO:teuthology.orchestra.run.smithi175.stdout: "addr": "192.168.0.2:0", 2024-04-15T15:16:39.750 INFO:teuthology.orchestra.run.smithi175.stdout: "nonce": 3869342304 2024-04-15T15:16:39.751 INFO:teuthology.orchestra.run.smithi175.stdout: }, 2024-04-15T15:16:39.751 INFO:teuthology.orchestra.run.smithi175.stdout: "inst_str": "client.5090 192.168.0.2:0/3869342304", 2024-04-15T15:16:39.751 INFO:teuthology.orchestra.run.smithi175.stdout: "addr_str": "192.168.0.2:0/3869342304", 2024-04-15T15:16:39.751 INFO:teuthology.orchestra.run.smithi175.stdout: "inode_count": 5, 2024-04-15T15:16:39.751 INFO:teuthology.orchestra.run.smithi175.stdout: "mds_epoch": 16, 2024-04-15T15:16:39.751 INFO:teuthology.orchestra.run.smithi175.stdout: "osd_epoch": 40, 2024-04-15T15:16:39.751 INFO:teuthology.orchestra.run.smithi175.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:16:39.751 INFO:teuthology.orchestra.run.smithi175.stdout: "blocklisted": false, 2024-04-15T15:16:39.751 INFO:teuthology.orchestra.run.smithi175.stdout: "fs_name": "cephfs" 2024-04-15T15:16:39.751 INFO:teuthology.orchestra.run.smithi175.stdout:} 2024-04-15T15:16:39.757 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'touch file') 2024-04-15T15:16:39.810 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-15T15:16:39.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:39.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:39.823 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quiesce.block -v 1 file) 2024-04-15T15:16:39.841 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.2:0 quiesce path /volumes/_nogroup/subvol_quiesce/6bdc4ba4-8793-4ca5-8487-64cfa9a984f4 --wait 2024-04-15T15:16:39.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.988+0000 7f69c6f1b640 1 Processor -- start 2024-04-15T15:16:39.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.988+0000 7f69c6f1b640 1 -- start start 2024-04-15T15:16:39.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.988+0000 7f69c6f1b640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f69c015ca60 0x7f69c00b5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.988+0000 7f69c6f1b640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69c015a4b0 con 0x7f69c006e210 2024-04-15T15:16:39.991 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.988+0000 7f69c6f1b640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69c006c680 con 0x7f69c015a0d0 2024-04-15T15:16:39.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.988+0000 7f69c6f1b640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f69c006c960 con 0x7f69c015ca60 2024-04-15T15:16:39.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.989+0000 7f69c54b8640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f69c006e210 0x7f69c006e640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:55990/0 (socket says 172.21.15.72:55990) 2024-04-15T15:16:39.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.989+0000 7f69c54b8640 1 -- 172.21.15.72:0/2801929398 learned_addr learned my addr 172.21.15.72:0/2801929398 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:39.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.989+0000 7f69bffff640 1 --2- 172.21.15.72:0/2801929398 >> v2:172.21.15.175:3300/0 conn(0x7f69c015ca60 0x7f69c00b5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.989+0000 7f69bf7fe640 1 -- 172.21.15.72:0/2801929398 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2461651313 0 0) 0x7f69c015a4b0 con 0x7f69c006e210 2024-04-15T15:16:39.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.989+0000 7f69bf7fe640 1 -- 172.21.15.72:0/2801929398 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69b0003520 con 0x7f69c006e210 2024-04-15T15:16:39.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.989+0000 7f69bf7fe640 1 -- 172.21.15.72:0/2801929398 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2968576937 0 0) 0x7f69c006c680 con 0x7f69c015a0d0 2024-04-15T15:16:39.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.989+0000 7f69bf7fe640 1 -- 172.21.15.72:0/2801929398 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69c015a4b0 con 0x7f69c015a0d0 2024-04-15T15:16:39.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.989+0000 7f69bf7fe640 1 -- 172.21.15.72:0/2801929398 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1056989999 0 0) 0x7f69b0003520 con 0x7f69c006e210 2024-04-15T15:16:39.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.989+0000 7f69bffff640 1 -- 172.21.15.72:0/2801929398 >> v1:172.21.15.175:6789/0 conn(0x7f69c015a0d0 legacy=0x7f69c00b5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.989+0000 7f69bffff640 1 -- 172.21.15.72:0/2801929398 >> v1:172.21.15.72:6789/0 conn(0x7f69c006e210 legacy=0x7f69c006e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.989+0000 7f69bffff640 1 -- 172.21.15.72:0/2801929398 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69c006cc40 con 0x7f69c015ca60 2024-04-15T15:16:39.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.990+0000 7f69bffff640 1 --2- 172.21.15.72:0/2801929398 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69c015ca60 0x7f69c00b5a80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f69ac0097a0 tx=0x7f69ac02b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=291844dacc40641c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:39.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.990+0000 7f69bf7fe640 1 -- 172.21.15.72:0/2801929398 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ac038070 con 0x7f69c015ca60 2024-04-15T15:16:39.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.990+0000 7f69bf7fe640 1 -- 172.21.15.72:0/2801929398 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69ac031500 con 0x7f69c015ca60 2024-04-15T15:16:39.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.990+0000 7f69bf7fe640 1 -- 172.21.15.72:0/2801929398 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b80019f0 con 0x7f69c015ca60 2024-04-15T15:16:39.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.991+0000 7f69c6f1b640 1 -- 172.21.15.72:0/2801929398 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69c015ca60 msgr2=0x7f69c00b5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.991+0000 7f69c6f1b640 1 --2- 172.21.15.72:0/2801929398 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69c015ca60 0x7f69c00b5a80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f69ac0097a0 tx=0x7f69ac02b820 comp rx=0 tx=0).stop 2024-04-15T15:16:39.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.991+0000 7f69c6f1b640 1 -- 172.21.15.72:0/2801929398 shutdown_connections 2024-04-15T15:16:39.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.991+0000 7f69c6f1b640 1 --2- 172.21.15.72:0/2801929398 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69c015ca60 0x7f69c00b5a80 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.991+0000 7f69c6f1b640 1 -- 172.21.15.72:0/2801929398 >> 172.21.15.72:0/2801929398 conn(0x7f69c00b1ee0 msgr2=0x7f69c00b22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:39.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.991+0000 7f69c6f1b640 1 -- 172.21.15.72:0/2801929398 shutdown_connections 2024-04-15T15:16:39.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.991+0000 7f69c6f1b640 1 -- 172.21.15.72:0/2801929398 wait complete. 2024-04-15T15:16:39.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.992+0000 7f69c6f1b640 1 Processor -- start 2024-04-15T15:16:39.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.992+0000 7f69c6f1b640 1 -- start start 2024-04-15T15:16:39.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.993+0000 7f69c6f1b640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69c006e210 0x7f69c009d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.993+0000 7f69c6f1b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69c015a0d0 0x7f69c009de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.993+0000 7f69c6f1b640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f69c015ca60 0x7f69c01f0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.993+0000 7f69c6f1b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f69c006c310 con 0x7f69c015a0d0 2024-04-15T15:16:39.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.993+0000 7f69c6f1b640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f69c006d200 con 0x7f69c006e210 2024-04-15T15:16:39.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.993+0000 7f69c6f1b640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f69c00b5fc0 con 0x7f69c015ca60 2024-04-15T15:16:39.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.993+0000 7f69bffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69c015a0d0 0x7f69c009de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.993+0000 7f69bffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69c015a0d0 0x7f69c009de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60534/0 (socket says 172.21.15.72:60534) 2024-04-15T15:16:39.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.993+0000 7f69bffff640 1 -- 172.21.15.72:0/546079836 learned_addr learned my addr 172.21.15.72:0/546079836 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:39.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.993+0000 7f69c4cb7640 1 --2- 172.21.15.72:0/546079836 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69c006e210 0x7f69c009d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.993+0000 7f69c54b8640 1 --2- 172.21.15.72:0/546079836 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f69c015ca60 0x7f69c01f0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.993+0000 7f69bffff640 1 -- 172.21.15.72:0/546079836 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f69c015ca60 msgr2=0x7f69c01f0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.993+0000 7f69bffff640 1 --2- 172.21.15.72:0/546079836 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f69c015ca60 0x7f69c01f0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.993+0000 7f69bffff640 1 -- 172.21.15.72:0/546079836 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69c006e210 msgr2=0x7f69c009d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.993+0000 7f69bffff640 1 --2- 172.21.15.72:0/546079836 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69c006e210 0x7f69c009d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.993+0000 7f69bffff640 1 -- 172.21.15.72:0/546079836 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69c009e340 con 0x7f69c015a0d0 2024-04-15T15:16:39.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.993+0000 7f69bffff640 1 --2- 172.21.15.72:0/546079836 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69c015a0d0 0x7f69c009de00 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f69ac000c00 tx=0x7f69ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:39.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.993+0000 7f69bd7fa640 1 -- 172.21.15.72:0/546079836 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ac042020 con 0x7f69c015a0d0 2024-04-15T15:16:39.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.993+0000 7f69bd7fa640 1 -- 172.21.15.72:0/546079836 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69ac03d870 con 0x7f69c015a0d0 2024-04-15T15:16:39.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.993+0000 7f69c6f1b640 1 -- 172.21.15.72:0/546079836 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69c00cc050 con 0x7f69c015a0d0 2024-04-15T15:16:39.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.993+0000 7f69bd7fa640 1 -- 172.21.15.72:0/546079836 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69ac038040 con 0x7f69c015a0d0 2024-04-15T15:16:39.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.994+0000 7f69c6f1b640 1 -- 172.21.15.72:0/546079836 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69c00cc610 con 0x7f69c015a0d0 2024-04-15T15:16:40.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.995+0000 7f69bd7fa640 1 -- 172.21.15.72:0/546079836 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f69ac045020 con 0x7f69c015a0d0 2024-04-15T15:16:40.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.995+0000 7f69bd7fa640 1 --2- 172.21.15.72:0/546079836 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f69a406c980 0x7f69a406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.996+0000 7f69bd7fa640 1 -- 172.21.15.72:0/546079836 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6130+0+0 (secure 0 0 0) 0x7f69ac0ad720 con 0x7f69c015a0d0 2024-04-15T15:16:40.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.996+0000 7f69c4cb7640 1 --2- 172.21.15.72:0/546079836 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f69a406c980 0x7f69a406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.996+0000 7f69c4cb7640 1 --2- 172.21.15.72:0/546079836 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f69a406c980 0x7f69a406ee40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f69b40011e0 tx=0x7f69b4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.996+0000 7f69c9bd7740 1 Processor -- start 2024-04-15T15:16:40.001 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.996+0000 7f69c9bd7740 1 -- start start 2024-04-15T15:16:40.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.997+0000 7f69c9bd7740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b56923e80 0x560b568f6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.997+0000 7f69c9bd7740 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x560b56924260 0x560b56926c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.997+0000 7f69c9bd7740 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560b568f7360 0x560b56927150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.997+0000 7f69c9bd7740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x560b568bead0 con 0x560b56923e80 2024-04-15T15:16:40.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.997+0000 7f69c9bd7740 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x560b56862560 con 0x560b568f7360 2024-04-15T15:16:40.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.997+0000 7f69c9bd7740 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x560b56808c00 con 0x560b56924260 2024-04-15T15:16:40.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.997+0000 7f69c4cb7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b56923e80 0x560b568f6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.997+0000 7f69c4cb7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b56923e80 0x560b568f6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60538/0 (socket says 172.21.15.72:60538) 2024-04-15T15:16:40.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.997+0000 7f69c4cb7640 1 -- 172.21.15.72:0/1099975164 learned_addr learned my addr 172.21.15.72:0/1099975164 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:40.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.997+0000 7f69bffff640 1 --2- 172.21.15.72:0/1099975164 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x560b56924260 0x560b56926c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.997+0000 7f69c54b8640 1 --2- 172.21.15.72:0/1099975164 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560b568f7360 0x560b56927150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.998+0000 7f69c4cb7640 1 -- 172.21.15.72:0/1099975164 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x560b56924260 msgr2=0x560b56926c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:40.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.998+0000 7f69c4cb7640 1 --2- 172.21.15.72:0/1099975164 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x560b56924260 0x560b56926c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.002 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.998+0000 7f69c4cb7640 1 -- 172.21.15.72:0/1099975164 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560b568f7360 msgr2=0x560b56927150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:40.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.998+0000 7f69c4cb7640 1 --2- 172.21.15.72:0/1099975164 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560b568f7360 0x560b56927150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.998+0000 7f69c4cb7640 1 -- 172.21.15.72:0/1099975164 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b56641030 con 0x560b56923e80 2024-04-15T15:16:40.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.998+0000 7f69c54b8640 1 --2- 172.21.15.72:0/1099975164 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560b568f7360 0x560b56927150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:40.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.998+0000 7f69c4cb7640 1 --2- 172.21.15.72:0/1099975164 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b56923e80 0x560b568f6e20 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f69b400fbd0 tx=0x7f69b4012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bb39938a9568b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.998+0000 7f69aa7fc640 1 -- 172.21.15.72:0/1099975164 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b4025070 con 0x560b56923e80 2024-04-15T15:16:40.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.998+0000 7f69aa7fc640 1 -- 172.21.15.72:0/1099975164 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69b401b050 con 0x560b56923e80 2024-04-15T15:16:40.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.998+0000 7f69aa7fc640 1 -- 172.21.15.72:0/1099975164 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b4020410 con 0x560b56923e80 2024-04-15T15:16:40.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.998+0000 7f69c9bd7740 1 -- 172.21.15.72:0/1099975164 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b56923e80 msgr2=0x560b568f6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.998+0000 7f69c9bd7740 1 --2- 172.21.15.72:0/1099975164 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b56923e80 0x560b568f6e20 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f69b400fbd0 tx=0x7f69b4012000 comp rx=0 tx=0).stop 2024-04-15T15:16:40.003 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.998+0000 7f69c9bd7740 1 -- 172.21.15.72:0/1099975164 shutdown_connections 2024-04-15T15:16:40.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.998+0000 7f69c9bd7740 1 --2- 172.21.15.72:0/1099975164 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560b568f7360 0x560b56927150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.998+0000 7f69c9bd7740 1 --2- 172.21.15.72:0/1099975164 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x560b56924260 0x560b56926c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.998+0000 7f69c9bd7740 1 --2- 172.21.15.72:0/1099975164 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b56923e80 0x560b568f6e20 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.998+0000 7f69c9bd7740 1 -- 172.21.15.72:0/1099975164 >> 172.21.15.72:0/1099975164 conn(0x560b568f9090 msgr2=0x560b56941c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:40.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.999+0000 7f69c9bd7740 1 -- 172.21.15.72:0/1099975164 shutdown_connections 2024-04-15T15:16:40.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.999+0000 7f69c9bd7740 1 -- 172.21.15.72:0/1099975164 wait complete. 2024-04-15T15:16:40.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.999+0000 7f69c9bd7740 1 -- start start 2024-04-15T15:16:40.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:39.999+0000 7f69c9bd7740 1 Processor -- start 2024-04-15T15:16:40.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.000+0000 7f69c9bd7740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b56923e80 0x560b56922bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.000+0000 7f69c9bd7740 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560b56924260 0x560b5691f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.000+0000 7f69c9bd7740 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x560b569215e0 0x560b5691cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.000+0000 7f69c9bd7740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x560b5681c110 con 0x560b56923e80 2024-04-15T15:16:40.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.000+0000 7f69c9bd7740 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x560b5685e520 con 0x560b56924260 2024-04-15T15:16:40.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.000+0000 7f69c9bd7740 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x560b568fb730 con 0x560b569215e0 2024-04-15T15:16:40.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.000+0000 7f69c9bd7740 10 client.0 _ll_register_callbacks cb 0x560b56574410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:16:40.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.000+0000 7f69c4cb7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b56923e80 0x560b56922bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.000+0000 7f69c4cb7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b56923e80 0x560b56922bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60554/0 (socket says 172.21.15.72:60554) 2024-04-15T15:16:40.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.000+0000 7f69c4cb7640 1 -- 172.21.15.72:0/1336336146 learned_addr learned my addr 172.21.15.72:0/1336336146 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:40.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.000+0000 7f69bffff640 1 --2- 172.21.15.72:0/1336336146 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560b56924260 0x560b5691f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.000+0000 7f69c54b8640 1 --2- 172.21.15.72:0/1336336146 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x560b569215e0 0x560b5691cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.000+0000 7f69c4cb7640 1 -- 172.21.15.72:0/1336336146 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x560b569215e0 msgr2=0x560b5691cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:40.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.000+0000 7f69c4cb7640 1 --2- 172.21.15.72:0/1336336146 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x560b569215e0 0x560b5691cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.000+0000 7f69c4cb7640 1 -- 172.21.15.72:0/1336336146 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560b56924260 msgr2=0x560b5691f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:40.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.000+0000 7f69c4cb7640 1 --2- 172.21.15.72:0/1336336146 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560b56924260 0x560b5691f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.000+0000 7f69c4cb7640 1 -- 172.21.15.72:0/1336336146 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b56574c20 con 0x560b56923e80 2024-04-15T15:16:40.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.000+0000 7f69bffff640 1 --2- 172.21.15.72:0/1336336146 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560b56924260 0x560b5691f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:40.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.001+0000 7f69c4cb7640 1 --2- 172.21.15.72:0/1336336146 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b56923e80 0x560b56922bf0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f69b400f5a0 tx=0x7f69b4001150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84766231de7a0a39 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.001+0000 7f699f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-04-15T15:16:40.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.001+0000 7f699f7fe640 1 -- 172.21.15.72:0/1336336146 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b4025040 con 0x560b56923e80 2024-04-15T15:16:40.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.001+0000 7f699f7fe640 1 -- 172.21.15.72:0/1336336146 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(40..40 src has 1..40) ==== 6130+0+0 (secure 0 0 0) 0x7f69b4007220 con 0x560b56923e80 2024-04-15T15:16:40.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.001+0000 7f69c9bd7740 1 -- 172.21.15.72:0/1336336146 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69b401b050 con 0x560b56923e80 2024-04-15T15:16:40.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.001+0000 7f699f7fe640 1 -- 172.21.15.72:0/1336336146 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f69b4036050 con 0x560b56923e80 2024-04-15T15:16:40.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.001+0000 7f69c9bd7740 10 client.6015 fetch_fsmap learned FSMap version 16 2024-04-15T15:16:40.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.001+0000 7f69c9bd7740 1 -- 172.21.15.72:0/1336336146 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x560b56534920 con 0x560b56923e80 2024-04-15T15:16:40.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.001+0000 7f699f7fe640 1 -- 172.21.15.72:0/1336336146 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69b401b050 con 0x560b56923e80 2024-04-15T15:16:40.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.001+0000 7f699f7fe640 1 -- 172.21.15.72:0/1336336146 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 16) ==== 3494+0+0 (secure 0 0 0) 0x7f69b4039020 con 0x560b56923e80 2024-04-15T15:16:40.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.001+0000 7f69c9bd7740 10 client.6015 fetch_fsmap finished waiting for FSMap version 16 2024-04-15T15:16:40.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.001+0000 7f69c9bd7740 10 client.6015 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-04-15T15:16:40.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.001+0000 7f69c9bd7740 20 client.6015 populate_metadata read hostname 'smithi072' 2024-04-15T15:16:40.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.001+0000 7f69c9bd7740 1 --2- 172.21.15.72:0/1336336146 >> [v2:172.21.15.72:6834/4270793613,v1:172.21.15.72:6836/4270793613] conn(0x560b56928e50 0x560b56929270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.001+0000 7f69c9bd7740 4 client.6015 mds_command: new command op to 5559 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-15T15:16:40.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.001+0000 7f69c9bd7740 1 -- 172.21.15.72:0/1336336146 --> [v2:172.21.15.72:6834/4270793613,v1:172.21.15.72:6836/4270793613] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560b5686ff80 con 0x560b56928e50 2024-04-15T15:16:40.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.001+0000 7f69bffff640 1 --2- 172.21.15.72:0/1336336146 >> [v2:172.21.15.72:6834/4270793613,v1:172.21.15.72:6836/4270793613] conn(0x560b56928e50 0x560b56929270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.002+0000 7f699f7fe640 10 client.6015 ms_handle_connect on v2:172.21.15.72:6834/4270793613 2024-04-15T15:16:40.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.002+0000 7f69bffff640 1 --2- 172.21.15.72:0/1336336146 >> [v2:172.21.15.72:6834/4270793613,v1:172.21.15.72:6836/4270793613] conn(0x560b56928e50 0x560b56929270 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6ac1d9020e6324 server_cookie=fe40ae7d18007fb7 in_seq=0 out_seq=0 2024-04-15T15:16:40.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.002+0000 7f699f7fe640 1 -- 172.21.15.72:0/1336336146 <== mds.0 v2:172.21.15.72:6834/4270793613 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560b5686ff80 con 0x560b56928e50 2024-04-15T15:16:40.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.002+0000 7f699f7fe640 10 client.6015 handle_command_reply: tid=0 2024-04-15T15:16:40.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.002+0000 7f69c9bd7740 1 client.6015 shutdown 2024-04-15T15:16:40.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.002+0000 7f69c9bd7740 1 -- 172.21.15.72:0/1336336146 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b56923e80 msgr2=0x560b56922bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.002+0000 7f69c9bd7740 1 --2- 172.21.15.72:0/1336336146 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b56923e80 0x560b56922bf0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f69b400f5a0 tx=0x7f69b4001150 comp rx=0 tx=0).stop 2024-04-15T15:16:40.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.002+0000 7f69c9bd7740 1 -- 172.21.15.72:0/1336336146 shutdown_connections 2024-04-15T15:16:40.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.002+0000 7f69c9bd7740 1 --2- 172.21.15.72:0/1336336146 >> [v2:172.21.15.72:6834/4270793613,v1:172.21.15.72:6836/4270793613] conn(0x560b56928e50 0x560b56929270 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.002+0000 7f69c9bd7740 1 --2- 172.21.15.72:0/1336336146 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x560b569215e0 0x560b5691cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.002+0000 7f69c9bd7740 1 --2- 172.21.15.72:0/1336336146 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560b56924260 0x560b5691f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.002+0000 7f69c9bd7740 1 --2- 172.21.15.72:0/1336336146 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b56923e80 0x560b56922bf0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.002+0000 7f699f7fe640 0 client.6015 ms_handle_reset on v2:172.21.15.72:6834/4270793613 2024-04-15T15:16:40.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.002+0000 7f69c9bd7740 1 -- 172.21.15.72:0/1336336146 >> 172.21.15.72:0/1336336146 conn(0x560b568f7360 msgr2=0x560b568faf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:40.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.002+0000 7f69c9bd7740 1 -- 172.21.15.72:0/1336336146 shutdown_connections 2024-04-15T15:16:40.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.002+0000 7f69c9bd7740 1 -- 172.21.15.72:0/1336336146 wait complete. 2024-04-15T15:16:40.012 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.003+0000 7f69c9bd7740 20 client.6015 trim_cache size 0 max 16384 2024-04-15T15:16:40.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.019+0000 7f69c9bd7740 1 Processor -- start 2024-04-15T15:16:40.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.019+0000 7f69c9bd7740 1 -- start start 2024-04-15T15:16:40.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.020+0000 7f69c9bd7740 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560b569215e0 0x560b56929ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.020+0000 7f69c9bd7740 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x560b56923e80 0x560b56926650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.020+0000 7f69c9bd7740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b56924260 0x560b56926b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.020+0000 7f69c9bd7740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f69c01ff050 con 0x560b56924260 2024-04-15T15:16:40.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.020+0000 7f69c9bd7740 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f69c01fe070 con 0x560b569215e0 2024-04-15T15:16:40.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.020+0000 7f69c9bd7740 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f69c01fd4f0 con 0x560b56923e80 2024-04-15T15:16:40.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.020+0000 7f69c54b8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b56924260 0x560b56926b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.020+0000 7f69c54b8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b56924260 0x560b56926b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60562/0 (socket says 172.21.15.72:60562) 2024-04-15T15:16:40.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.020+0000 7f69c54b8640 1 -- 172.21.15.72:0/1771929841 learned_addr learned my addr 172.21.15.72:0/1771929841 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:40.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.021+0000 7f69c54b8640 1 -- 172.21.15.72:0/1771929841 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x560b56923e80 msgr2=0x560b56926650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:40.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.021+0000 7f69bffff640 1 --2- 172.21.15.72:0/1771929841 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x560b56923e80 0x560b56926650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.021+0000 7f69c4cb7640 1 --2- 172.21.15.72:0/1771929841 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560b569215e0 0x560b56929ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.028 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.021+0000 7f69c54b8640 1 --2- 172.21.15.72:0/1771929841 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x560b56923e80 0x560b56926650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.021+0000 7f69c54b8640 1 -- 172.21.15.72:0/1771929841 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560b569215e0 msgr2=0x560b56929ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:40.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.021+0000 7f69c54b8640 1 --2- 172.21.15.72:0/1771929841 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560b569215e0 0x560b56929ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.021+0000 7f69c54b8640 1 -- 172.21.15.72:0/1771929841 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b56922660 con 0x560b56924260 2024-04-15T15:16:40.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.021+0000 7f69bffff640 1 --2- 172.21.15.72:0/1771929841 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x560b56923e80 0x560b56926650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:40.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.021+0000 7f69c4cb7640 1 --2- 172.21.15.72:0/1771929841 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560b569215e0 0x560b56929ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:40.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.021+0000 7f69c54b8640 1 --2- 172.21.15.72:0/1771929841 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b56924260 0x560b56926b90 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f69b8007240 tx=0x7f69b8007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebc4faa5f33729f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.021+0000 7f69a9ffb640 1 -- 172.21.15.72:0/1771929841 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b801a070 con 0x560b56924260 2024-04-15T15:16:40.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.021+0000 7f69a9ffb640 1 -- 172.21.15.72:0/1771929841 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69b8013760 con 0x560b56924260 2024-04-15T15:16:40.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.021+0000 7f69a9ffb640 1 -- 172.21.15.72:0/1771929841 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b8013b10 con 0x560b56924260 2024-04-15T15:16:40.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.021+0000 7f69c9bd7740 1 -- 172.21.15.72:0/1771929841 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b56924260 msgr2=0x560b56926b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.021+0000 7f69c9bd7740 1 --2- 172.21.15.72:0/1771929841 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b56924260 0x560b56926b90 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f69b8007240 tx=0x7f69b8007700 comp rx=0 tx=0).stop 2024-04-15T15:16:40.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.021+0000 7f69c9bd7740 1 -- 172.21.15.72:0/1771929841 shutdown_connections 2024-04-15T15:16:40.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.021+0000 7f69c9bd7740 1 --2- 172.21.15.72:0/1771929841 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b56924260 0x560b56926b90 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.021+0000 7f69c9bd7740 1 --2- 172.21.15.72:0/1771929841 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x560b56923e80 0x560b56926650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.029 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.021+0000 7f69c9bd7740 1 --2- 172.21.15.72:0/1771929841 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560b569215e0 0x560b56929ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.021+0000 7f69c9bd7740 1 -- 172.21.15.72:0/1771929841 >> 172.21.15.72:0/1771929841 conn(0x560b568f7360 msgr2=0x560b5691bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:40.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.022+0000 7f69c9bd7740 1 -- 172.21.15.72:0/1771929841 shutdown_connections 2024-04-15T15:16:40.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.022+0000 7f69c9bd7740 1 -- 172.21.15.72:0/1771929841 wait complete. 2024-04-15T15:16:40.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.022+0000 7f69c9bd7740 1 -- start start 2024-04-15T15:16:40.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.022+0000 7f69c9bd7740 1 Processor -- start 2024-04-15T15:16:40.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.023+0000 7f69c9bd7740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b569215e0 0x560b56963a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.023+0000 7f69c9bd7740 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x560b56923e80 0x560b569501e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.023+0000 7f69c9bd7740 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560b56924260 0x560b56950720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.023+0000 7f69c9bd7740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x560b56921020 con 0x560b569215e0 2024-04-15T15:16:40.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.023+0000 7f69c9bd7740 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f69c01fda80 con 0x560b56924260 2024-04-15T15:16:40.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.023+0000 7f69c9bd7740 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x560b568f2db0 con 0x560b56923e80 2024-04-15T15:16:40.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.023+0000 7f69c9bd7740 10 client.0 _ll_register_callbacks cb 0x7f69b4023070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:16:40.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.023+0000 7f69c4cb7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b569215e0 0x560b56963a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.023+0000 7f69c4cb7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b569215e0 0x560b56963a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60572/0 (socket says 172.21.15.72:60572) 2024-04-15T15:16:40.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.023+0000 7f69c4cb7640 1 -- 172.21.15.72:0/2603222177 learned_addr learned my addr 172.21.15.72:0/2603222177 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:40.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.023+0000 7f69c54b8640 1 --2- 172.21.15.72:0/2603222177 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560b56924260 0x560b56950720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.023+0000 7f69bffff640 1 --2- 172.21.15.72:0/2603222177 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x560b56923e80 0x560b569501e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.023+0000 7f69c4cb7640 1 -- 172.21.15.72:0/2603222177 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x560b56923e80 msgr2=0x560b569501e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:40.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.023+0000 7f69c4cb7640 1 --2- 172.21.15.72:0/2603222177 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x560b56923e80 0x560b569501e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.023+0000 7f69c4cb7640 1 -- 172.21.15.72:0/2603222177 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560b56924260 msgr2=0x560b56950720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:40.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.023+0000 7f69c4cb7640 1 --2- 172.21.15.72:0/2603222177 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560b56924260 0x560b56950720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.023+0000 7f69c4cb7640 1 -- 172.21.15.72:0/2603222177 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b5692a480 con 0x560b569215e0 2024-04-15T15:16:40.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.023+0000 7f69bffff640 1 --2- 172.21.15.72:0/2603222177 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x560b56923e80 0x560b569501e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:40.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.024+0000 7f69c4cb7640 1 --2- 172.21.15.72:0/2603222177 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b569215e0 0x560b56963a10 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f69b400f990 tx=0x7f69b400a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39e8295f0a485128 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.024+0000 7f699f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-04-15T15:16:40.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.024+0000 7f699f7fe640 1 -- 172.21.15.72:0/2603222177 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b4025030 con 0x560b569215e0 2024-04-15T15:16:40.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.024+0000 7f699f7fe640 1 -- 172.21.15.72:0/2603222177 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(40..40 src has 1..40) ==== 6130+0+0 (secure 0 0 0) 0x7f69b402c620 con 0x560b569215e0 2024-04-15T15:16:40.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.024+0000 7f69c9bd7740 1 -- 172.21.15.72:0/2603222177 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f69b8013760 con 0x560b569215e0 2024-04-15T15:16:40.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.024+0000 7f699f7fe640 1 -- 172.21.15.72:0/2603222177 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f69b4044050 con 0x560b569215e0 2024-04-15T15:16:40.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.024+0000 7f69c9bd7740 10 client.6021 fetch_fsmap learned FSMap version 16 2024-04-15T15:16:40.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.024+0000 7f69c9bd7740 1 -- 172.21.15.72:0/2603222177 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x560b569644d0 con 0x560b569215e0 2024-04-15T15:16:40.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.024+0000 7f699f7fe640 1 -- 172.21.15.72:0/2603222177 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69b8013760 con 0x560b569215e0 2024-04-15T15:16:40.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.024+0000 7f699f7fe640 1 -- 172.21.15.72:0/2603222177 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 16) ==== 3494+0+0 (secure 0 0 0) 0x7f69b4047020 con 0x560b569215e0 2024-04-15T15:16:40.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.024+0000 7f69c9bd7740 10 client.6021 fetch_fsmap finished waiting for FSMap version 16 2024-04-15T15:16:40.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.024+0000 7f69c9bd7740 10 client.6021 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-04-15T15:16:40.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.024+0000 7f69c9bd7740 20 client.6021 populate_metadata read hostname 'smithi072' 2024-04-15T15:16:40.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.024+0000 7f69c9bd7740 1 --2- 172.21.15.72:0/2603222177 >> [v2:172.21.15.72:6834/4270793613,v1:172.21.15.72:6836/4270793613] conn(0x560b5691b700 0x560b569bacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.024+0000 7f69c9bd7740 4 client.6021 mds_command: new command op to 5559 tid=0 multi_id=0 [{"prefix": "quiesce path", "path": "/volumes/_nogroup/subvol_quiesce/6bdc4ba4-8793-4ca5-8487-64cfa9a984f4", "wait": true}] 2024-04-15T15:16:40.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.024+0000 7f69c9bd7740 1 -- 172.21.15.72:0/2603222177 --> [v2:172.21.15.72:6834/4270793613,v1:172.21.15.72:6836/4270793613] -- command(tid 0: {"prefix": "quiesce path", "path": "/volumes/_nogroup/subvol_quiesce/6bdc4ba4-8793-4ca5-8487-64cfa9a984f4", "wait": true}) -- 0x560b56869df0 con 0x560b5691b700 2024-04-15T15:16:40.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.024+0000 7f69bffff640 1 --2- 172.21.15.72:0/2603222177 >> [v2:172.21.15.72:6834/4270793613,v1:172.21.15.72:6836/4270793613] conn(0x560b5691b700 0x560b569bacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.025+0000 7f699f7fe640 10 client.6021 ms_handle_connect on v2:172.21.15.72:6834/4270793613 2024-04-15T15:16:40.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.025+0000 7f69bffff640 1 --2- 172.21.15.72:0/2603222177 >> [v2:172.21.15.72:6834/4270793613,v1:172.21.15.72:6836/4270793613] conn(0x560b5691b700 0x560b569bacf0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48905806d6c61668 server_cookie=142061816341ca3b in_seq=0 out_seq=0 2024-04-15T15:16:40.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.028+0000 7f699f7fe640 1 -- 172.21.15.72:0/2603222177 <== mds.0 v2:172.21.15.72:6834/4270793613 1 ==== command_reply(tid 0: 0 ) ==== 8+0+2332 (crc 0 0 0) 0x560b56869df0 con 0x560b5691b700 2024-04-15T15:16:40.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.028+0000 7f699f7fe640 10 client.6021 handle_command_reply: tid=0 2024-04-15T15:16:40.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.028+0000 7f69c9bd7740 1 client.6021 shutdown 2024-04-15T15:16:40.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.028+0000 7f69c9bd7740 1 -- 172.21.15.72:0/2603222177 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b569215e0 msgr2=0x560b56963a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.028+0000 7f69c9bd7740 1 --2- 172.21.15.72:0/2603222177 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b569215e0 0x560b56963a10 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f69b400f990 tx=0x7f69b400a000 comp rx=0 tx=0).stop 2024-04-15T15:16:40.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.029+0000 7f69c9bd7740 1 -- 172.21.15.72:0/2603222177 shutdown_connections 2024-04-15T15:16:40.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.029+0000 7f69c9bd7740 1 --2- 172.21.15.72:0/2603222177 >> [v2:172.21.15.72:6834/4270793613,v1:172.21.15.72:6836/4270793613] conn(0x560b5691b700 0x560b569bacf0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.029+0000 7f69c9bd7740 1 --2- 172.21.15.72:0/2603222177 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x560b56924260 0x560b56950720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.029+0000 7f699f7fe640 0 client.6021 ms_handle_reset on v2:172.21.15.72:6834/4270793613 2024-04-15T15:16:40.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.029+0000 7f69c9bd7740 1 --2- 172.21.15.72:0/2603222177 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x560b56923e80 0x560b569501e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.029+0000 7f69c9bd7740 1 --2- 172.21.15.72:0/2603222177 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x560b569215e0 0x560b56963a10 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.029+0000 7f69c9bd7740 1 -- 172.21.15.72:0/2603222177 >> 172.21.15.72:0/2603222177 conn(0x560b568f7360 msgr2=0x560b569454c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:40.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.029+0000 7f69c9bd7740 1 -- 172.21.15.72:0/2603222177 shutdown_connections 2024-04-15T15:16:40.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.029+0000 7f69c9bd7740 1 -- 172.21.15.72:0/2603222177 wait complete. 2024-04-15T15:16:40.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.029+0000 7f69c9bd7740 20 client.6021 trim_cache size 0 max 16384 2024-04-15T15:16:40.035 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-04-15T15:16:40.036 INFO:teuthology.orchestra.run.smithi072.stdout: "op": { 2024-04-15T15:16:40.036 INFO:teuthology.orchestra.run.smithi072.stdout: "result": 0, 2024-04-15T15:16:40.036 INFO:teuthology.orchestra.run.smithi072.stdout: "flag_point": "quiesce complete", 2024-04-15T15:16:40.036 INFO:teuthology.orchestra.run.smithi072.stdout: "reqid": { 2024-04-15T15:16:40.036 INFO:teuthology.orchestra.run.smithi072.stdout: "entity": { 2024-04-15T15:16:40.036 INFO:teuthology.orchestra.run.smithi072.stdout: "type": "mds", 2024-04-15T15:16:40.036 INFO:teuthology.orchestra.run.smithi072.stdout: "num": 0 2024-04-15T15:16:40.036 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:40.036 INFO:teuthology.orchestra.run.smithi072.stdout: "tid": 1 2024-04-15T15:16:40.036 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:40.036 INFO:teuthology.orchestra.run.smithi072.stdout: "op_type": "internal_op", 2024-04-15T15:16:40.036 INFO:teuthology.orchestra.run.smithi072.stdout: "internal_op": 5384, 2024-04-15T15:16:40.036 INFO:teuthology.orchestra.run.smithi072.stdout: "op_name": "quiesce_path", 2024-04-15T15:16:40.036 INFO:teuthology.orchestra.run.smithi072.stdout: "events": [ 2024-04-15T15:16:40.037 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:40.037 INFO:teuthology.orchestra.run.smithi072.stdout: "time": "2024-04-15T15:16:40.025943+0000", 2024-04-15T15:16:40.037 INFO:teuthology.orchestra.run.smithi072.stdout: "event": "initiated" 2024-04-15T15:16:40.037 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:40.037 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:40.037 INFO:teuthology.orchestra.run.smithi072.stdout: "time": "2024-04-15T15:16:40.025943+0000", 2024-04-15T15:16:40.037 INFO:teuthology.orchestra.run.smithi072.stdout: "event": "header_read" 2024-04-15T15:16:40.037 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:40.037 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:40.037 INFO:teuthology.orchestra.run.smithi072.stdout: "time": "2024-04-15T15:16:40.025943+0000", 2024-04-15T15:16:40.037 INFO:teuthology.orchestra.run.smithi072.stdout: "event": "throttled" 2024-04-15T15:16:40.037 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:40.037 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:40.037 INFO:teuthology.orchestra.run.smithi072.stdout: "time": "2024-04-15T15:16:40.025943+0000", 2024-04-15T15:16:40.038 INFO:teuthology.orchestra.run.smithi072.stdout: "event": "all_read" 2024-04-15T15:16:40.038 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:40.038 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:40.038 INFO:teuthology.orchestra.run.smithi072.stdout: "time": "2024-04-15T15:16:40.025943+0000", 2024-04-15T15:16:40.038 INFO:teuthology.orchestra.run.smithi072.stdout: "event": "dispatched" 2024-04-15T15:16:40.038 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:40.038 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:40.038 INFO:teuthology.orchestra.run.smithi072.stdout: "time": "2024-04-15T15:16:40.026003+0000", 2024-04-15T15:16:40.038 INFO:teuthology.orchestra.run.smithi072.stdout: "event": "acquired locks" 2024-04-15T15:16:40.038 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:40.038 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:40.039 INFO:teuthology.orchestra.run.smithi072.stdout: "time": "2024-04-15T15:16:40.026023+0000", 2024-04-15T15:16:40.039 INFO:teuthology.orchestra.run.smithi072.stdout: "event": "acquired locks" 2024-04-15T15:16:40.039 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:40.039 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:40.039 INFO:teuthology.orchestra.run.smithi072.stdout: "time": "2024-04-15T15:16:40.026040+0000", 2024-04-15T15:16:40.039 INFO:teuthology.orchestra.run.smithi072.stdout: "event": "acquired locks" 2024-04-15T15:16:40.039 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:40.039 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:40.039 INFO:teuthology.orchestra.run.smithi072.stdout: "time": "2024-04-15T15:16:40.026059+0000", 2024-04-15T15:16:40.039 INFO:teuthology.orchestra.run.smithi072.stdout: "event": "acquired locks" 2024-04-15T15:16:40.039 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:40.039 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:40.039 INFO:teuthology.orchestra.run.smithi072.stdout: "time": "2024-04-15T15:16:40.029262+0000", 2024-04-15T15:16:40.039 INFO:teuthology.orchestra.run.smithi072.stdout: "event": "acquired locks" 2024-04-15T15:16:40.039 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:40.040 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:40.040 INFO:teuthology.orchestra.run.smithi072.stdout: "time": "2024-04-15T15:16:40.029272+0000", 2024-04-15T15:16:40.040 INFO:teuthology.orchestra.run.smithi072.stdout: "event": "acquired locks" 2024-04-15T15:16:40.040 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:40.040 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:40.040 INFO:teuthology.orchestra.run.smithi072.stdout: "time": "2024-04-15T15:16:40.029283+0000", 2024-04-15T15:16:40.040 INFO:teuthology.orchestra.run.smithi072.stdout: "event": "acquired locks" 2024-04-15T15:16:40.040 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:40.040 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:40.040 INFO:teuthology.orchestra.run.smithi072.stdout: "time": "2024-04-15T15:16:40.029293+0000", 2024-04-15T15:16:40.040 INFO:teuthology.orchestra.run.smithi072.stdout: "event": "acquired locks" 2024-04-15T15:16:40.040 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:40.041 INFO:teuthology.orchestra.run.smithi072.stdout: { 2024-04-15T15:16:40.041 INFO:teuthology.orchestra.run.smithi072.stdout: "time": "2024-04-15T15:16:40.029305+0000", 2024-04-15T15:16:40.041 INFO:teuthology.orchestra.run.smithi072.stdout: "event": "quiesce complete" 2024-04-15T15:16:40.041 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:16:40.041 INFO:teuthology.orchestra.run.smithi072.stdout: ], 2024-04-15T15:16:40.041 INFO:teuthology.orchestra.run.smithi072.stdout: "locks": null 2024-04-15T15:16:40.041 INFO:teuthology.orchestra.run.smithi072.stdout: }, 2024-04-15T15:16:40.041 INFO:teuthology.orchestra.run.smithi072.stdout: "state": { 2024-04-15T15:16:40.041 INFO:teuthology.orchestra.run.smithi072.stdout: "inodes": 2, 2024-04-15T15:16:40.041 INFO:teuthology.orchestra.run.smithi072.stdout: "inodes_quiesced": 1, 2024-04-15T15:16:40.041 INFO:teuthology.orchestra.run.smithi072.stdout: "inodes_blocked": 1, 2024-04-15T15:16:40.041 INFO:teuthology.orchestra.run.smithi072.stdout: "inodes_dropped": 0, 2024-04-15T15:16:40.041 INFO:teuthology.orchestra.run.smithi072.stdout: "failed": [] 2024-04-15T15:16:40.041 INFO:teuthology.orchestra.run.smithi072.stdout: } 2024-04-15T15:16:40.041 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-04-15T15:16:40.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.029+0000 7f69c6f1b640 1 -- 172.21.15.72:0/546079836 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f69a406c980 msgr2=0x7f69a406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.029+0000 7f69c6f1b640 1 --2- 172.21.15.72:0/546079836 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f69a406c980 0x7f69a406ee40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f69b40011e0 tx=0x7f69b4009040 comp rx=0 tx=0).stop 2024-04-15T15:16:40.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.029+0000 7f69c6f1b640 1 -- 172.21.15.72:0/546079836 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69c015a0d0 msgr2=0x7f69c009de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.029+0000 7f69c6f1b640 1 --2- 172.21.15.72:0/546079836 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69c015a0d0 0x7f69c009de00 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f69ac000c00 tx=0x7f69ac0046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.029+0000 7f69c6f1b640 1 -- 172.21.15.72:0/546079836 shutdown_connections 2024-04-15T15:16:40.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.029+0000 7f69c6f1b640 1 --2- 172.21.15.72:0/546079836 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f69a406c980 0x7f69a406ee40 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.029+0000 7f69c6f1b640 1 --2- 172.21.15.72:0/546079836 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f69c015ca60 0x7f69c01f0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.029+0000 7f69c6f1b640 1 --2- 172.21.15.72:0/546079836 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f69c015a0d0 0x7f69c009de00 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.029+0000 7f69c6f1b640 1 --2- 172.21.15.72:0/546079836 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f69c006e210 0x7f69c009d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.029+0000 7f69c6f1b640 1 -- 172.21.15.72:0/546079836 >> 172.21.15.72:0/546079836 conn(0x7f69c00b1ee0 msgr2=0x7f69c015bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:40.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.029+0000 7f69c6f1b640 1 -- 172.21.15.72:0/546079836 shutdown_connections 2024-04-15T15:16:40.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.029+0000 7f69c6f1b640 1 -- 172.21.15.72:0/546079836 wait complete. 2024-04-15T15:16:40.044 DEBUG:tasks.cephfs.test_quiesce:{'op': {'result': 0, 'flag_point': 'quiesce complete', 'reqid': {'entity': {'type': 'mds', 'num': 0}, 'tid': 1}, 'op_type': 'internal_op', 'internal_op': 5384, 'op_name': 'quiesce_path', 'events': [{'time': '2024-04-15T15:16:40.025943+0000', 'event': 'initiated'}, {'time': '2024-04-15T15:16:40.025943+0000', 'event': 'header_read'}, {'time': '2024-04-15T15:16:40.025943+0000', 'event': 'throttled'}, {'time': '2024-04-15T15:16:40.025943+0000', 'event': 'all_read'}, {'time': '2024-04-15T15:16:40.025943+0000', 'event': 'dispatched'}, {'time': '2024-04-15T15:16:40.026003+0000', 'event': 'acquired locks'}, {'time': '2024-04-15T15:16:40.026023+0000', 'event': 'acquired locks'}, {'time': '2024-04-15T15:16:40.026040+0000', 'event': 'acquired locks'}, {'time': '2024-04-15T15:16:40.026059+0000', 'event': 'acquired locks'}, {'time': '2024-04-15T15:16:40.029262+0000', 'event': 'acquired locks'}, {'time': '2024-04-15T15:16:40.029272+0000', 'event': 'acquired locks'}, {'time': '2024-04-15T15:16:40.029283+0000', 'event': 'acquired locks'}, {'time': '2024-04-15T15:16:40.029293+0000', 'event': 'acquired locks'}, {'time': '2024-04-15T15:16:40.029305+0000', 'event': 'quiesce complete'}], 'locks': None}, 'state': {'inodes': 2, 'inodes_quiesced': 1, 'inodes_blocked': 1, 'inodes_dropped': 0, 'failed': []}} 2024-04-15T15:16:40.045 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:16:40.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.147+0000 7f70e4441640 1 Processor -- start 2024-04-15T15:16:40.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.147+0000 7f70e4441640 1 -- start start 2024-04-15T15:16:40.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.147+0000 7f70e4441640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f70dc15c710 0x7f70dc0b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.147+0000 7f70e4441640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f70dc0d1790 0x7f70dc0cd950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.148+0000 7f70e4441640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70dc15caf0 con 0x7f70dc0b5790 2024-04-15T15:16:40.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.148+0000 7f70e4441640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f70dc06f510 con 0x7f70dc15c710 2024-04-15T15:16:40.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.148+0000 7f70e4441640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f70dc0ccb90 con 0x7f70dc0d1790 2024-04-15T15:16:40.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.148+0000 7f70e21dd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f70dc15c710 0x7f70dc0b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.148+0000 7f70e21dd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f70dc15c710 0x7f70dc0b66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60586/0 (socket says 172.21.15.72:60586) 2024-04-15T15:16:40.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.148+0000 7f70e21dd640 1 -- 172.21.15.72:0/1061949375 learned_addr learned my addr 172.21.15.72:0/1061949375 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:40.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.148+0000 7f70e29de640 1 --2- 172.21.15.72:0/1061949375 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f70dc0d1790 0x7f70dc0cd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.148+0000 7f70e21dd640 1 -- 172.21.15.72:0/1061949375 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f70dc0d1790 msgr2=0x7f70dc0cd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:40.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.148+0000 7f70e21dd640 1 --2- 172.21.15.72:0/1061949375 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f70dc0d1790 0x7f70dc0cd950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.148+0000 7f70e21dd640 1 -- 172.21.15.72:0/1061949375 >> v1:172.21.15.175:6789/0 conn(0x7f70dc0b5790 legacy=0x7f70dc0d1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.148+0000 7f70e21dd640 1 -- 172.21.15.72:0/1061949375 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70dc0cce70 con 0x7f70dc15c710 2024-04-15T15:16:40.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.148+0000 7f70e29de640 1 --2- 172.21.15.72:0/1061949375 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f70dc0d1790 0x7f70dc0cd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:40.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.149+0000 7f70e21dd640 1 --2- 172.21.15.72:0/1061949375 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f70dc15c710 0x7f70dc0b66b0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f70d4009e50 tx=0x7f70d402bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2ea402ef692557b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.149+0000 7f70e11db640 1 -- 172.21.15.72:0/1061949375 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d402cb90 con 0x7f70dc15c710 2024-04-15T15:16:40.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.149+0000 7f70e11db640 1 -- 172.21.15.72:0/1061949375 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70d402cd30 con 0x7f70dc15c710 2024-04-15T15:16:40.151 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.149+0000 7f70e11db640 1 -- 172.21.15.72:0/1061949375 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d4034610 con 0x7f70dc15c710 2024-04-15T15:16:40.158 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.149+0000 7f70e4441640 1 -- 172.21.15.72:0/1061949375 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f70dc15c710 msgr2=0x7f70dc0b66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.149+0000 7f70e4441640 1 --2- 172.21.15.72:0/1061949375 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f70dc15c710 0x7f70dc0b66b0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f70d4009e50 tx=0x7f70d402bff0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.150+0000 7f70e4441640 1 -- 172.21.15.72:0/1061949375 shutdown_connections 2024-04-15T15:16:40.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.150+0000 7f70e4441640 1 --2- 172.21.15.72:0/1061949375 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f70dc0d1790 0x7f70dc0cd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.150+0000 7f70e4441640 1 --2- 172.21.15.72:0/1061949375 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f70dc15c710 0x7f70dc0b66b0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.150+0000 7f70e4441640 1 -- 172.21.15.72:0/1061949375 >> 172.21.15.72:0/1061949375 conn(0x7f70dc0b3f90 msgr2=0x7f70dc0b43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:40.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.150+0000 7f70e4441640 1 -- 172.21.15.72:0/1061949375 shutdown_connections 2024-04-15T15:16:40.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.150+0000 7f70e4441640 1 -- 172.21.15.72:0/1061949375 wait complete. 2024-04-15T15:16:40.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.151+0000 7f70e4441640 1 Processor -- start 2024-04-15T15:16:40.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.151+0000 7f70e4441640 1 -- start start 2024-04-15T15:16:40.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.151+0000 7f70e4441640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f70dc0b5790 0x7f70dc14aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.151+0000 7f70e4441640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f70dc0d1790 0x7f70dc15cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.152+0000 7f70e4441640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f70dc15c710 0x7f70dc0c7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.152+0000 7f70e4441640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f70dc0d1cd0 con 0x7f70dc15c710 2024-04-15T15:16:40.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.152+0000 7f70e4441640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f70dc0cd430 con 0x7f70dc0b5790 2024-04-15T15:16:40.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.152+0000 7f70e4441640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f70dc0b6cd0 con 0x7f70dc0d1790 2024-04-15T15:16:40.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.152+0000 7f70e29de640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f70dc15c710 0x7f70dc0c7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.160 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.152+0000 7f70e29de640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f70dc15c710 0x7f70dc0c7ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60596/0 (socket says 172.21.15.72:60596) 2024-04-15T15:16:40.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.152+0000 7f70e29de640 1 -- 172.21.15.72:0/4023659708 learned_addr learned my addr 172.21.15.72:0/4023659708 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:40.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.152+0000 7f70e21dd640 1 --2- 172.21.15.72:0/4023659708 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f70dc0b5790 0x7f70dc14aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.152+0000 7f70e19dc640 1 --2- 172.21.15.72:0/4023659708 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f70dc0d1790 0x7f70dc15cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.152+0000 7f70e29de640 1 -- 172.21.15.72:0/4023659708 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f70dc0d1790 msgr2=0x7f70dc15cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.152+0000 7f70e29de640 1 --2- 172.21.15.72:0/4023659708 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f70dc0d1790 0x7f70dc15cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.152+0000 7f70e29de640 1 -- 172.21.15.72:0/4023659708 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f70dc0b5790 msgr2=0x7f70dc14aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.152+0000 7f70e29de640 1 --2- 172.21.15.72:0/4023659708 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f70dc0b5790 0x7f70dc14aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.152+0000 7f70e29de640 1 -- 172.21.15.72:0/4023659708 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70dc15e6f0 con 0x7f70dc15c710 2024-04-15T15:16:40.161 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.152+0000 7f70e29de640 1 --2- 172.21.15.72:0/4023659708 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f70dc15c710 0x7f70dc0c7ed0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f70d800e5b0 tx=0x7f70d800ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.152+0000 7f70d2ffd640 1 -- 172.21.15.72:0/4023659708 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d8017070 con 0x7f70dc15c710 2024-04-15T15:16:40.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.152+0000 7f70d2ffd640 1 -- 172.21.15.72:0/4023659708 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70d8013610 con 0x7f70dc15c710 2024-04-15T15:16:40.162 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.152+0000 7f70e4441640 1 -- 172.21.15.72:0/4023659708 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70dc15d650 con 0x7f70dc15c710 2024-04-15T15:16:40.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.152+0000 7f70d2ffd640 1 -- 172.21.15.72:0/4023659708 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d800f040 con 0x7f70dc15c710 2024-04-15T15:16:40.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.152+0000 7f70e4441640 1 -- 172.21.15.72:0/4023659708 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70dc15dc10 con 0x7f70dc15c710 2024-04-15T15:16:40.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.153+0000 7f70e4441640 1 -- 172.21.15.72:0/4023659708 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70dc0689e0 con 0x7f70dc15c710 2024-04-15T15:16:40.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.154+0000 7f70d2ffd640 1 -- 172.21.15.72:0/4023659708 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f70d801a070 con 0x7f70dc15c710 2024-04-15T15:16:40.163 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.154+0000 7f70d2ffd640 1 --2- 172.21.15.72:0/4023659708 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f70c006c980 0x7f70c006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.154+0000 7f70d2ffd640 1 -- 172.21.15.72:0/4023659708 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6130+0+0 (secure 0 0 0) 0x7f70d808b8e0 con 0x7f70dc15c710 2024-04-15T15:16:40.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.154+0000 7f70e21dd640 1 --2- 172.21.15.72:0/4023659708 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f70c006c980 0x7f70c006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.155+0000 7f70e21dd640 1 --2- 172.21.15.72:0/4023659708 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f70c006c980 0x7f70c006ee40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f70d4035000 tx=0x7f70d4030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.158+0000 7f70d2ffd640 1 -- 172.21.15.72:0/4023659708 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f70d8059410 con 0x7f70dc15c710 2024-04-15T15:16:40.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.370+0000 7f70e4441640 1 -- 172.21.15.72:0/4023659708 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f70dc0008d0 con 0x7f70dc15c710 2024-04-15T15:16:40.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.370+0000 7f70d2ffd640 1 -- 172.21.15.72:0/4023659708 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5184 (secure 0 0 0) 0x7f70d805d710 con 0x7f70dc15c710 2024-04-15T15:16:40.372 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:16:40.372 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4883,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3363272836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3363272836},{"type":"v1","addr":"172.21.15.175:6834","nonce":3363272836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5134,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6835/1059187505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":1059187505},{"type":"v1","addr":"172.21.15.175:6835","nonce":1059187505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5556,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/690132275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":690132275},{"type":"v1","addr":"172.21.15.72:6837","nonce":690132275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:16:07.050078+0000","modified":"2024-04-15T15:16:09.052120+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5559},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5559":{"gid":5559,"name":"a","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.72:6836/4270793613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":4270793613},{"type":"v1","addr":"172.21.15.72:6836","nonce":4270793613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5559,"qdb_cluster":[5559]},"id":2}]} 2024-04-15T15:16:40.372 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 16 2024-04-15T15:16:40.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.372+0000 7f70e4441640 1 -- 172.21.15.72:0/4023659708 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f70c006c980 msgr2=0x7f70c006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.372+0000 7f70e4441640 1 --2- 172.21.15.72:0/4023659708 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f70c006c980 0x7f70c006ee40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f70d4035000 tx=0x7f70d4030000 comp rx=0 tx=0).stop 2024-04-15T15:16:40.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.372+0000 7f70e4441640 1 -- 172.21.15.72:0/4023659708 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f70dc15c710 msgr2=0x7f70dc0c7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.372+0000 7f70e4441640 1 --2- 172.21.15.72:0/4023659708 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f70dc15c710 0x7f70dc0c7ed0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f70d800e5b0 tx=0x7f70d800ea70 comp rx=0 tx=0).stop 2024-04-15T15:16:40.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.372+0000 7f70e4441640 1 -- 172.21.15.72:0/4023659708 shutdown_connections 2024-04-15T15:16:40.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.372+0000 7f70e4441640 1 --2- 172.21.15.72:0/4023659708 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f70c006c980 0x7f70c006ee40 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.372+0000 7f70e4441640 1 --2- 172.21.15.72:0/4023659708 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f70dc15c710 0x7f70dc0c7ed0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.372+0000 7f70e4441640 1 --2- 172.21.15.72:0/4023659708 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f70dc0d1790 0x7f70dc15cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.372+0000 7f70e4441640 1 --2- 172.21.15.72:0/4023659708 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f70dc0b5790 0x7f70dc14aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.372+0000 7f70e4441640 1 -- 172.21.15.72:0/4023659708 >> 172.21.15.72:0/4023659708 conn(0x7f70dc0b3f90 msgr2=0x7f70dc0b4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:40.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.372+0000 7f70e4441640 1 -- 172.21.15.72:0/4023659708 shutdown_connections 2024-04-15T15:16:40.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.372+0000 7f70e4441640 1 -- 172.21.15.72:0/4023659708 wait complete. 2024-04-15T15:16:40.385 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json dump tree /volumes/_nogroup/subvol_quiesce/6bdc4ba4-8793-4ca5-8487-64cfa9a984f4 0 2024-04-15T15:16:40.503 INFO:teuthology.orchestra.run.smithi072.stdout:[{"ino":1099511627779,"rdev":0,"ctime":"2024-04-15T15:16:39.811476+0000","btime":"2024-04-15T15:16:20.823399+0000","mode":16895,"uid":0,"gid":0,"nlink":1,"dir_layout":{"dir_hash":2,"unused1":0,"unused2":0,"unused3":0},"layout":{"stripe_unit":4194304,"stripe_count":1,"object_size":4194304,"pool_id":6,"pool_ns":""},"old_pools":[],"size":0,"truncate_seq":1,"truncate_size":18446744073709551615,"truncate_from":0,"truncate_pending":0,"mtime":"2024-04-15T15:16:39.811476+0000","atime":"2024-04-15T15:16:20.823399+0000","time_warp_seq":0,"change_attr":2,"export_pin":-1,"export_ephemeral_random_pin":0,"export_ephemeral_distributed_pin":false,"quiesce_block":false,"client_ranges":[],"dirstat":{"version":0,"mtime":"2024-04-15T15:16:39.811476+0000","num_files":1,"num_subdirs":0,"change_attr":1},"rstat":{"version":0,"rbytes":0,"rfiles":1,"rsubdirs":1,"rsnaps":0,"rctime":"2024-04-15T15:16:39.837590+0000"},"accounted_rstat":{"version":0,"rbytes":0,"rfiles":1,"rsubdirs":1,"rsnaps":0,"rctime":"2024-04-15T15:16:39.837590+0000"},"version":20,"file_data_version":0,"xattr_version":1,"backtrace_version":2,"stray_prior_path":"","max_size_ever":0,"quota":{"max_bytes":0,"max_files":0},"last_scrub_stamp":"0.000000","last_scrub_version":0,"symlink":"","xattrs":[],"dirfragtree":{"splits":[]},"old_inodes":[],"oldest_snap":18446744073709551614,"damage_flags":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":1,"is_frozen":false,"is_freezing":false,"pins":{"request":1,"lock":2,"dirfrag":1,"caps":1,"dirtyparent":1,"dirty":1,"waiter":0,"authpin":1},"nref":8,"versionlock":{"gather_set":[],"state":"lock","type":"iversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":1,"num_xlocks":0,"xlock_by":{}},"authlock":{},"linklock":{},"dirfragtreelock":{},"filelock":{},"xattrlock":{},"snaplock":{},"nestlock":{"gather_set":[],"state":"lock","type":"inest","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"flocklock":{},"policylock":{},"quiescelock":{"gather_set":[],"state":"lock","type":"iquiesce","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":1,"xlock_by":{"reqid":{"entity":{"type":"mds","num":0},"tid":2}}},"states":["auth","dirty","dirtyparent","dirtypool"],"client_caps":[{"client_id":5075,"pending":"pAsLsXs","issued":"pAsLsXs","wanted":"-","last_sent":2},{"client_id":5090,"pending":"pAsLsXs","issued":"pAsLsXs","wanted":"-","last_sent":2},{"client_id":5817,"pending":"pAsLsXs","issued":"pAsLsXs","wanted":"-","last_sent":7},{"client_id":5970,"pending":"pAsLsXs","issued":"pAsLsXs","wanted":"-","last_sent":3},{"client_id":5991,"pending":"pAsLsXs","issued":"pAsLsXs","wanted":"-","last_sent":2}],"loner":-1,"want_loner":-1,"mds_caps_wanted":[],"dirfrags":[{"path":"/volumes/_nogroup/subvol_quiesce/6bdc4ba4-8793-4ca5-8487-64cfa9a984f4","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"7","version":"7","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"dirty":1,"authpin":0},"nref":2,"dentries":[{"path":"volumes/_nogroup/subvol_quiesce/6bdc4ba4-8793-4ca5-8487-64cfa9a984f4/file","path_ino":1,"snap_first":2,"snap_last":18446744073709551614,"is_primary":true,"is_remote":false,"is_null":false,"is_new":false,"inode":1099511629282,"remote_type":"","version":6,"projected_version":6,"auth_pins":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"request":0,"lock":0,"inodepin":1,"dirty":1,"authpin":0},"nref":2,"lock":{},"versionlock":{"gather_set":[],"state":"lock","type":"dversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"states":["auth","dirty"]}]}],"is_subvolume":false}] 2024-04-15T15:16:40.504 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "accounted_rstat": { "rbytes": 0, "rctime": "2024-04-15T15:16:39.837590+0000", "rfiles": 1, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "atime": "2024-04-15T15:16:20.823399+0000", "auth_pins": 1, "auth_state": { "replicas": {} }, "authlock": {}, "backtrace_version": 2, "btime": "2024-04-15T15:16:20.823399+0000", "change_attr": 2, "client_caps": [ { "client_id": 5075, "issued": "pAsLsXs", "last_sent": 2, "pending": "pAsLsXs", "wanted": "-" }, { "client_id": 5090, "issued": "pAsLsXs", "last_sent": 2, "pending": "pAsLsXs", "wanted": "-" }, { "client_id": 5817, "issued": "pAsLsXs", "last_sent": 7, "pending": "pAsLsXs", "wanted": "-" }, { "client_id": 5970, "issued": "pAsLsXs", "last_sent": 3, "pending": "pAsLsXs", "wanted": "-" }, { "client_id": 5991, "issued": "pAsLsXs", "last_sent": 2, "pending": "pAsLsXs", "wanted": "-" } ], "client_ranges": [], "ctime": "2024-04-15T15:16:39.811476+0000", "damage_flags": 0, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "dirfrags": [ { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dentries": [ { "auth_pins": 0, "auth_state": { "replicas": {} }, "inode": 1099511629282, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_new": false, "is_null": false, "is_primary": true, "is_remote": false, "lock": {}, "nref": 2, "path": "volumes/_nogroup/subvol_quiesce/6bdc4ba4-8793-4ca5-8487-64cfa9a984f4/file", "path_ino": 1, "pins": { "authpin": 0, "dirty": 1, "inodepin": 1, "lock": 0, "request": 0 }, "projected_version": 6, "remote_type": "", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snap_first": 2, "snap_last": 18446744073709551614, "states": [ "auth", "dirty" ], "version": 6, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "dversion", "xlock_by": {} } } ], "dir_auth": "", "dirfrag": "0x10000000003", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/volumes/_nogroup/subvol_quiesce/6bdc4ba4-8793-4ca5-8487-64cfa9a984f4", "pins": { "authpin": 0, "child": 1, "dirty": 1 }, "projected_version": "7", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "7" } ], "dirfragtree": { "splits": [] }, "dirfragtreelock": {}, "dirstat": { "change_attr": 1, "mtime": "2024-04-15T15:16:39.811476+0000", "num_files": 1, "num_subdirs": 0, "version": 0 }, "export_ephemeral_distributed_pin": false, "export_ephemeral_random_pin": 0, "export_pin": -1, "file_data_version": 0, "filelock": {}, "flocklock": {}, "gid": 0, "ino": 1099511627779, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_subvolume": false, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "layout": { "object_size": 4194304, "pool_id": 6, "pool_ns": "", "stripe_count": 1, "stripe_unit": 4194304 }, "linklock": {}, "loner": -1, "max_size_ever": 0, "mds_caps_wanted": [], "mode": 16895, "mtime": "2024-04-15T15:16:39.811476+0000", "nestlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "inest", "xlock_by": {} }, "nlink": 1, "nref": 8, "old_inodes": [], "old_pools": [], "oldest_snap": 18446744073709551614, "pins": { "authpin": 1, "caps": 1, "dirfrag": 1, "dirty": 1, "dirtyparent": 1, "lock": 2, "request": 1, "waiter": 0 }, "policylock": {}, "quiesce_block": false, "quiescelock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "state": "lock", "type": "iquiesce", "xlock_by": { "reqid": { "entity": { "num": 0, "type": "mds" }, "tid": 2 } } }, "quota": { "max_bytes": 0, "max_files": 0 }, "rdev": 0, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "rstat": { "rbytes": 0, "rctime": "2024-04-15T15:16:39.837590+0000", "rfiles": 1, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "size": 0, "snaplock": {}, "states": [ "auth", "dirty", "dirtyparent", "dirtypool" ], "stray_prior_path": "", "symlink": "", "time_warp_seq": 0, "truncate_from": 0, "truncate_pending": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "uid": 0, "version": 20, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 1, "num_xlocks": 0, "state": "lock", "type": "iversion", "xlock_by": {} }, "want_loner": -1, "xattr_version": 1, "xattrlock": {}, "xattrs": [] } ] 2024-04-15T15:16:40.505 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.2:0 ops --flags=locks --path=/tmp/mds.0-ops 2024-04-15T15:16:40.610 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.609+0000 7f047f100640 1 Processor -- start 2024-04-15T15:16:40.611 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.609+0000 7f047f100640 1 -- start start 2024-04-15T15:16:40.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.610+0000 7f047f100640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f047815c250 0x7f047806d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.610+0000 7f047f100640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f047815f9f0 0x7f04780cca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.610+0000 7f047f100640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f047806df30 con 0x7f04780d0010 2024-04-15T15:16:40.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.610+0000 7f047f100640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f047806e210 con 0x7f047815f9f0 2024-04-15T15:16:40.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.610+0000 7f047f100640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f047806e4f0 con 0x7f047815c250 2024-04-15T15:16:40.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.610+0000 7f047d69d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f04780d0010 0x7f04780b3e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:55994/0 (socket says 172.21.15.72:55994) 2024-04-15T15:16:40.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.610+0000 7f047d69d640 1 -- 172.21.15.72:0/2023828960 learned_addr learned my addr 172.21.15.72:0/2023828960 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:40.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.610+0000 7f047ce9c640 1 --2- 172.21.15.72:0/2023828960 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f047815c250 0x7f047806d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.615 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.610+0000 7f0477fff640 1 --2- 172.21.15.72:0/2023828960 >> v2:172.21.15.175:3300/0 conn(0x7f047815f9f0 0x7f04780cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.610+0000 7f04777fe640 1 -- 172.21.15.72:0/2023828960 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3001268743 0 0) 0x7f047806df30 con 0x7f04780d0010 2024-04-15T15:16:40.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.610+0000 7f04777fe640 1 -- 172.21.15.72:0/2023828960 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0468003520 con 0x7f04780d0010 2024-04-15T15:16:40.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.610+0000 7f04777fe640 1 -- 172.21.15.72:0/2023828960 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1354673713 0 0) 0x7f0468003520 con 0x7f04780d0010 2024-04-15T15:16:40.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.610+0000 7f04777fe640 1 -- 172.21.15.72:0/2023828960 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f047815c250 msgr2=0x7f047806d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:40.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.610+0000 7f04777fe640 1 --2- 172.21.15.72:0/2023828960 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f047815c250 0x7f047806d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.610+0000 7f04777fe640 1 -- 172.21.15.72:0/2023828960 >> v2:172.21.15.175:3300/0 conn(0x7f047815f9f0 msgr2=0x7f04780cca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:40.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.610+0000 7f04777fe640 1 --2- 172.21.15.72:0/2023828960 >> v2:172.21.15.175:3300/0 conn(0x7f047815f9f0 0x7f04780cca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.610+0000 7f04777fe640 1 -- 172.21.15.72:0/2023828960 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04780b3610 con 0x7f04780d0010 2024-04-15T15:16:40.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.610+0000 7f04777fe640 1 -- 172.21.15.72:0/2023828960 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f04700019f0 con 0x7f04780d0010 2024-04-15T15:16:40.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.610+0000 7f047ce9c640 1 --2- 172.21.15.72:0/2023828960 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f047815c250 0x7f047806d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:40.616 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.610+0000 7f04777fe640 1 -- 172.21.15.72:0/2023828960 >> v1:172.21.15.72:6789/0 conn(0x7f04780d0010 legacy=0x7f04780b3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.610+0000 7f04777fe640 1 --2- 172.21.15.72:0/2023828960 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04680039b0 0x7f0468003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.610+0000 7f04777fe640 1 --2- 172.21.15.72:0/2023828960 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0468004800 0x7f0468004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.610+0000 7f04777fe640 1 --2- 172.21.15.72:0/2023828960 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f047815c250 0x7f04680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.610+0000 7f04777fe640 1 -- 172.21.15.72:0/2023828960 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f047806e210 con 0x7f04680039b0 2024-04-15T15:16:40.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.610+0000 7f04777fe640 1 -- 172.21.15.72:0/2023828960 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f047806e4f0 con 0x7f0468004800 2024-04-15T15:16:40.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.610+0000 7f04777fe640 1 -- 172.21.15.72:0/2023828960 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f046800b9c0 con 0x7f047815c250 2024-04-15T15:16:40.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.610+0000 7f047ce9c640 1 --2- 172.21.15.72:0/2023828960 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04680039b0 0x7f0468003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.610+0000 7f0477fff640 1 --2- 172.21.15.72:0/2023828960 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0468004800 0x7f0468004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.610+0000 7f047d69d640 1 --2- 172.21.15.72:0/2023828960 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f047815c250 0x7f04680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.611+0000 7f047ce9c640 1 -- 172.21.15.72:0/2023828960 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f047815c250 msgr2=0x7f04680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:40.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.611+0000 7f047ce9c640 1 --2- 172.21.15.72:0/2023828960 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f047815c250 0x7f04680051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.611+0000 7f047ce9c640 1 -- 172.21.15.72:0/2023828960 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0468004800 msgr2=0x7f0468004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:40.617 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.611+0000 7f047ce9c640 1 --2- 172.21.15.72:0/2023828960 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0468004800 0x7f0468004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.611+0000 7f047ce9c640 1 -- 172.21.15.72:0/2023828960 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0468003520 con 0x7f04680039b0 2024-04-15T15:16:40.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.611+0000 7f047ce9c640 1 --2- 172.21.15.72:0/2023828960 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04680039b0 0x7f0468003dd0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f046c009c90 tx=0x7f0468005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9176f214ea4519d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.612+0000 7f04777fe640 1 -- 172.21.15.72:0/2023828960 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f046c039070 con 0x7f04680039b0 2024-04-15T15:16:40.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.612+0000 7f04777fe640 1 -- 172.21.15.72:0/2023828960 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f046c02c8f0 con 0x7f04680039b0 2024-04-15T15:16:40.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.612+0000 7f04777fe640 1 -- 172.21.15.72:0/2023828960 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f046c009860 con 0x7f04680039b0 2024-04-15T15:16:40.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.612+0000 7f047f100640 1 -- 172.21.15.72:0/2023828960 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04680039b0 msgr2=0x7f0468003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.612+0000 7f047f100640 1 --2- 172.21.15.72:0/2023828960 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04680039b0 0x7f0468003dd0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f046c009c90 tx=0x7f0468005830 comp rx=0 tx=0).stop 2024-04-15T15:16:40.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.612+0000 7f047d69d640 1 -- 172.21.15.72:0/2023828960 reap_dead start 2024-04-15T15:16:40.618 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.612+0000 7f047f100640 1 -- 172.21.15.72:0/2023828960 shutdown_connections 2024-04-15T15:16:40.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.612+0000 7f047f100640 1 -- 172.21.15.72:0/2023828960 >> 172.21.15.72:0/2023828960 conn(0x7f04780ceec0 msgr2=0x7f04780cf2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:40.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.612+0000 7f047f100640 1 -- 172.21.15.72:0/2023828960 shutdown_connections 2024-04-15T15:16:40.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.612+0000 7f047f100640 1 -- 172.21.15.72:0/2023828960 wait complete. 2024-04-15T15:16:40.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.613+0000 7f047f100640 1 Processor -- start 2024-04-15T15:16:40.619 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.613+0000 7f047f100640 1 -- start start 2024-04-15T15:16:40.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.614+0000 7f047f100640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f04680039b0 0x7f0478160360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.614+0000 7f047f100640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04780b2520 0x7f04781608a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.614+0000 7f047f100640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04780b5660 0x7f04780b5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.614+0000 7f047f100640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f047815c7b0 con 0x7f04780b5660 2024-04-15T15:16:40.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.614+0000 7f047f100640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f047815c630 con 0x7f04780b2520 2024-04-15T15:16:40.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.614+0000 7f047f100640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f04780ccfd0 con 0x7f04680039b0 2024-04-15T15:16:40.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.614+0000 7f047d69d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04780b5660 0x7f04780b5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.614+0000 7f047d69d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04780b5660 0x7f04780b5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60616/0 (socket says 172.21.15.72:60616) 2024-04-15T15:16:40.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.614+0000 7f047d69d640 1 -- 172.21.15.72:0/1214551952 learned_addr learned my addr 172.21.15.72:0/1214551952 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:40.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.615+0000 7f047ce9c640 1 --2- 172.21.15.72:0/1214551952 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f04680039b0 0x7f0478160360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.615+0000 7f0477fff640 1 --2- 172.21.15.72:0/1214551952 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04780b2520 0x7f04781608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.615+0000 7f047d69d640 1 -- 172.21.15.72:0/1214551952 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f04680039b0 msgr2=0x7f0478160360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.615+0000 7f047d69d640 1 --2- 172.21.15.72:0/1214551952 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f04680039b0 0x7f0478160360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.615+0000 7f047d69d640 1 -- 172.21.15.72:0/1214551952 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04780b2520 msgr2=0x7f04781608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.615+0000 7f047d69d640 1 --2- 172.21.15.72:0/1214551952 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04780b2520 0x7f04781608a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.615+0000 7f047d69d640 1 -- 172.21.15.72:0/1214551952 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04781f43a0 con 0x7f04780b5660 2024-04-15T15:16:40.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.615+0000 7f047d69d640 1 --2- 172.21.15.72:0/1214551952 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04780b5660 0x7f04780b5ae0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f0470007740 tx=0x7f0470007c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.615+0000 7f04757fa640 1 -- 172.21.15.72:0/1214551952 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f047000ad10 con 0x7f04780b5660 2024-04-15T15:16:40.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.615+0000 7f04757fa640 1 -- 172.21.15.72:0/1214551952 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0470018070 con 0x7f04780b5660 2024-04-15T15:16:40.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.615+0000 7f047f100640 1 -- 172.21.15.72:0/1214551952 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04780b6300 con 0x7f04780b5660 2024-04-15T15:16:40.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.615+0000 7f04757fa640 1 -- 172.21.15.72:0/1214551952 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0470013630 con 0x7f04780b5660 2024-04-15T15:16:40.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.615+0000 7f047f100640 1 -- 172.21.15.72:0/1214551952 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04780b68c0 con 0x7f04780b5660 2024-04-15T15:16:40.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.617+0000 7f04757fa640 1 -- 172.21.15.72:0/1214551952 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f04700137d0 con 0x7f04780b5660 2024-04-15T15:16:40.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.617+0000 7f04757fa640 1 --2- 172.21.15.72:0/1214551952 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f045c06c980 0x7f045c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.617+0000 7f04757fa640 1 -- 172.21.15.72:0/1214551952 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6130+0+0 (secure 0 0 0) 0x7f047008c3e0 con 0x7f04780b5660 2024-04-15T15:16:40.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.617+0000 7f047ce9c640 1 --2- 172.21.15.72:0/1214551952 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f045c06c980 0x7f045c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.618+0000 7f047ce9c640 1 --2- 172.21.15.72:0/1214551952 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f045c06c980 0x7f045c06ee40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f046c036000 tx=0x7f046c030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.618+0000 7f0481dc0740 1 Processor -- start 2024-04-15T15:16:40.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.618+0000 7f0481dc0740 1 -- start start 2024-04-15T15:16:40.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.619+0000 7f0481dc0740 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56273cbc6da0 0x56273cbc7180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.619+0000 7f0481dc0740 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x56273cbcab60 0x56273cbcc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.619+0000 7f0481dc0740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56273cbf68a0 0x56273cbf6d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.619+0000 7f0481dc0740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x56273cb31a80 con 0x56273cbf68a0 2024-04-15T15:16:40.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.619+0000 7f0481dc0740 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x56273cad8580 con 0x56273cbc6da0 2024-04-15T15:16:40.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.619+0000 7f0481dc0740 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x56273cb2da40 con 0x56273cbcab60 2024-04-15T15:16:40.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.619+0000 7f047d69d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56273cbf68a0 0x56273cbf6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.619+0000 7f047d69d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56273cbf68a0 0x56273cbf6d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60620/0 (socket says 172.21.15.72:60620) 2024-04-15T15:16:40.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.619+0000 7f047d69d640 1 -- 172.21.15.72:0/3685970419 learned_addr learned my addr 172.21.15.72:0/3685970419 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:40.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.619+0000 7f0477fff640 1 --2- 172.21.15.72:0/3685970419 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x56273cbcab60 0x56273cbcc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.619+0000 7f047ce9c640 1 --2- 172.21.15.72:0/3685970419 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56273cbc6da0 0x56273cbc7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.619+0000 7f047d69d640 1 -- 172.21.15.72:0/3685970419 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x56273cbcab60 msgr2=0x56273cbcc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:40.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.619+0000 7f047d69d640 1 --2- 172.21.15.72:0/3685970419 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x56273cbcab60 0x56273cbcc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.619+0000 7f047d69d640 1 -- 172.21.15.72:0/3685970419 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56273cbc6da0 msgr2=0x56273cbc7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:40.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.619+0000 7f047d69d640 1 --2- 172.21.15.72:0/3685970419 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56273cbc6da0 0x56273cbc7180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.619+0000 7f047d69d640 1 -- 172.21.15.72:0/3685970419 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56273caecad0 con 0x56273cbf68a0 2024-04-15T15:16:40.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.619+0000 7f047d69d640 1 --2- 172.21.15.72:0/3685970419 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56273cbf68a0 0x56273cbf6d20 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f04700013d0 tx=0x7f047005a670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3420dac961751384 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.620+0000 7f04627fc640 1 -- 172.21.15.72:0/3685970419 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f047005e320 con 0x56273cbf68a0 2024-04-15T15:16:40.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.620+0000 7f04627fc640 1 -- 172.21.15.72:0/3685970419 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0470062b50 con 0x56273cbf68a0 2024-04-15T15:16:40.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.620+0000 7f04627fc640 1 -- 172.21.15.72:0/3685970419 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04700565e0 con 0x56273cbf68a0 2024-04-15T15:16:40.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.620+0000 7f0481dc0740 1 -- 172.21.15.72:0/3685970419 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56273cbf68a0 msgr2=0x56273cbf6d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.620+0000 7f0481dc0740 1 --2- 172.21.15.72:0/3685970419 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56273cbf68a0 0x56273cbf6d20 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f04700013d0 tx=0x7f047005a670 comp rx=0 tx=0).stop 2024-04-15T15:16:40.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.620+0000 7f0481dc0740 1 -- 172.21.15.72:0/3685970419 shutdown_connections 2024-04-15T15:16:40.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.620+0000 7f0481dc0740 1 --2- 172.21.15.72:0/3685970419 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56273cbf68a0 0x56273cbf6d20 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.620+0000 7f0481dc0740 1 --2- 172.21.15.72:0/3685970419 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x56273cbcab60 0x56273cbcc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.620+0000 7f0481dc0740 1 --2- 172.21.15.72:0/3685970419 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56273cbc6da0 0x56273cbc7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.620+0000 7f0481dc0740 1 -- 172.21.15.72:0/3685970419 >> 172.21.15.72:0/3685970419 conn(0x56273cbc8e20 msgr2=0x56273cbca450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:40.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.620+0000 7f0481dc0740 1 -- 172.21.15.72:0/3685970419 shutdown_connections 2024-04-15T15:16:40.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.620+0000 7f0481dc0740 1 -- 172.21.15.72:0/3685970419 wait complete. 2024-04-15T15:16:40.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.621+0000 7f0481dc0740 1 -- start start 2024-04-15T15:16:40.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.621+0000 7f0481dc0740 1 Processor -- start 2024-04-15T15:16:40.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.622+0000 7f0481dc0740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56273cbf68a0 0x56273cbed9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.622+0000 7f0481dc0740 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x56273cbea650 0x56273cbcb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.622+0000 7f0481dc0740 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56273cbe9c40 0x56273cbcbf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.622+0000 7f0481dc0740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x56273cbcaff0 con 0x56273cbf68a0 2024-04-15T15:16:40.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.622+0000 7f0481dc0740 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x56273caeb940 con 0x56273cbe9c40 2024-04-15T15:16:40.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.622+0000 7f0481dc0740 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x56273cbcb2d0 con 0x56273cbea650 2024-04-15T15:16:40.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.622+0000 7f047ce9c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56273cbf68a0 0x56273cbed9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.622+0000 7f0481dc0740 10 client.0 _ll_register_callbacks cb 0x56273caeb350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:16:40.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.622+0000 7f047ce9c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56273cbf68a0 0x56273cbed9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60630/0 (socket says 172.21.15.72:60630) 2024-04-15T15:16:40.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.622+0000 7f047ce9c640 1 -- 172.21.15.72:0/1060434744 learned_addr learned my addr 172.21.15.72:0/1060434744 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:40.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.622+0000 7f0477fff640 1 --2- 172.21.15.72:0/1060434744 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x56273cbea650 0x56273cbcb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.622+0000 7f047d69d640 1 --2- 172.21.15.72:0/1060434744 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56273cbe9c40 0x56273cbcbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.622+0000 7f047ce9c640 1 -- 172.21.15.72:0/1060434744 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x56273cbea650 msgr2=0x56273cbcb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:40.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.622+0000 7f047ce9c640 1 --2- 172.21.15.72:0/1060434744 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x56273cbea650 0x56273cbcb9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.622+0000 7f047ce9c640 1 -- 172.21.15.72:0/1060434744 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56273cbe9c40 msgr2=0x56273cbcbf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:40.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.622+0000 7f047ce9c640 1 --2- 172.21.15.72:0/1060434744 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56273cbe9c40 0x56273cbcbf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.622+0000 7f047ce9c640 1 -- 172.21.15.72:0/1060434744 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56273c844420 con 0x56273cbf68a0 2024-04-15T15:16:40.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.622+0000 7f047ce9c640 1 --2- 172.21.15.72:0/1060434744 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56273cbf68a0 0x56273cbed9e0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f046c009c90 tx=0x7f046c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2982cf4d6fe542c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.622+0000 7f04577fe640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-04-15T15:16:40.634 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.622+0000 7f04577fe640 1 -- 172.21.15.72:0/1060434744 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f046c050060 con 0x56273cbf68a0 2024-04-15T15:16:40.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.622+0000 7f04577fe640 1 -- 172.21.15.72:0/1060434744 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(40..40 src has 1..40) ==== 6130+0+0 (secure 0 0 0) 0x7f046c005990 con 0x56273cbf68a0 2024-04-15T15:16:40.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.622+0000 7f0481dc0740 1 -- 172.21.15.72:0/1060434744 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0470062b50 con 0x56273cbf68a0 2024-04-15T15:16:40.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.623+0000 7f04577fe640 1 -- 172.21.15.72:0/1060434744 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f046c056050 con 0x56273cbf68a0 2024-04-15T15:16:40.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.623+0000 7f0481dc0740 10 client.6039 fetch_fsmap learned FSMap version 16 2024-04-15T15:16:40.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.623+0000 7f0481dc0740 1 -- 172.21.15.72:0/1060434744 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x56273c910830 con 0x56273cbf68a0 2024-04-15T15:16:40.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.623+0000 7f04577fe640 1 -- 172.21.15.72:0/1060434744 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0470062b50 con 0x56273cbf68a0 2024-04-15T15:16:40.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.623+0000 7f04577fe640 1 -- 172.21.15.72:0/1060434744 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 16) ==== 3494+0+0 (secure 0 0 0) 0x7f046c059020 con 0x56273cbf68a0 2024-04-15T15:16:40.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.623+0000 7f0481dc0740 10 client.6039 fetch_fsmap finished waiting for FSMap version 16 2024-04-15T15:16:40.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.623+0000 7f0481dc0740 10 client.6039 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-04-15T15:16:40.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.623+0000 7f0481dc0740 20 client.6039 populate_metadata read hostname 'smithi072' 2024-04-15T15:16:40.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.623+0000 7f0481dc0740 1 --2- 172.21.15.72:0/1060434744 >> [v2:172.21.15.72:6834/4270793613,v1:172.21.15.72:6836/4270793613] conn(0x56273cbf8c90 0x56273cbf2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.623+0000 7f0481dc0740 4 client.6039 mds_command: new command op to 5559 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-15T15:16:40.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.623+0000 7f0481dc0740 1 -- 172.21.15.72:0/1060434744 --> [v2:172.21.15.72:6834/4270793613,v1:172.21.15.72:6836/4270793613] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56273cb3f4a0 con 0x56273cbf8c90 2024-04-15T15:16:40.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.623+0000 7f0477fff640 1 --2- 172.21.15.72:0/1060434744 >> [v2:172.21.15.72:6834/4270793613,v1:172.21.15.72:6836/4270793613] conn(0x56273cbf8c90 0x56273cbf2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.635 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.623+0000 7f04577fe640 10 client.6039 ms_handle_connect on v2:172.21.15.72:6834/4270793613 2024-04-15T15:16:40.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.623+0000 7f0477fff640 1 --2- 172.21.15.72:0/1060434744 >> [v2:172.21.15.72:6834/4270793613,v1:172.21.15.72:6836/4270793613] conn(0x56273cbf8c90 0x56273cbf2900 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e7d81b949ab361d server_cookie=d41929c5afa9cf49 in_seq=0 out_seq=0 2024-04-15T15:16:40.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.624+0000 7f04577fe640 1 -- 172.21.15.72:0/1060434744 <== mds.0 v2:172.21.15.72:6834/4270793613 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56273cb3f4a0 con 0x56273cbf8c90 2024-04-15T15:16:40.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.624+0000 7f04577fe640 10 client.6039 handle_command_reply: tid=0 2024-04-15T15:16:40.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.624+0000 7f0481dc0740 1 client.6039 shutdown 2024-04-15T15:16:40.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.624+0000 7f0481dc0740 1 -- 172.21.15.72:0/1060434744 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56273cbf68a0 msgr2=0x56273cbed9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.624+0000 7f0481dc0740 1 --2- 172.21.15.72:0/1060434744 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56273cbf68a0 0x56273cbed9e0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f046c009c90 tx=0x7f046c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.624+0000 7f0481dc0740 1 -- 172.21.15.72:0/1060434744 shutdown_connections 2024-04-15T15:16:40.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.624+0000 7f0481dc0740 1 --2- 172.21.15.72:0/1060434744 >> [v2:172.21.15.72:6834/4270793613,v1:172.21.15.72:6836/4270793613] conn(0x56273cbf8c90 0x56273cbf2900 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.624+0000 7f0481dc0740 1 --2- 172.21.15.72:0/1060434744 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56273cbe9c40 0x56273cbcbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.624+0000 7f04577fe640 0 client.6039 ms_handle_reset on v2:172.21.15.72:6834/4270793613 2024-04-15T15:16:40.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.624+0000 7f0481dc0740 1 --2- 172.21.15.72:0/1060434744 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x56273cbea650 0x56273cbcb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.624+0000 7f0481dc0740 1 --2- 172.21.15.72:0/1060434744 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56273cbf68a0 0x56273cbed9e0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.624+0000 7f0481dc0740 1 -- 172.21.15.72:0/1060434744 >> 172.21.15.72:0/1060434744 conn(0x56273cbc6da0 msgr2=0x56273cbf4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:40.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.624+0000 7f0481dc0740 1 -- 172.21.15.72:0/1060434744 shutdown_connections 2024-04-15T15:16:40.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.624+0000 7f0481dc0740 1 -- 172.21.15.72:0/1060434744 wait complete. 2024-04-15T15:16:40.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.624+0000 7f0481dc0740 20 client.6039 trim_cache size 0 max 16384 2024-04-15T15:16:40.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.640+0000 7f0481dc0740 1 Processor -- start 2024-04-15T15:16:40.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.640+0000 7f0481dc0740 1 -- start start 2024-04-15T15:16:40.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.641+0000 7f0481dc0740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56273cbe9c40 0x56273cbe92d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.641+0000 7f0481dc0740 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56273cbea650 0x56273cbeb070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.641+0000 7f0481dc0740 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x56273cbf68a0 0x56273cbf2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.641+0000 7f0481dc0740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f0478201840 con 0x56273cbe9c40 2024-04-15T15:16:40.644 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.641+0000 7f0481dc0740 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f0478200860 con 0x56273cbea650 2024-04-15T15:16:40.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.641+0000 7f0481dc0740 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f04781ffbf0 con 0x56273cbf68a0 2024-04-15T15:16:40.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.641+0000 7f047ce9c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56273cbe9c40 0x56273cbe92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.641+0000 7f047ce9c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56273cbe9c40 0x56273cbe92d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60636/0 (socket says 172.21.15.72:60636) 2024-04-15T15:16:40.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.641+0000 7f047ce9c640 1 -- 172.21.15.72:0/3401739490 learned_addr learned my addr 172.21.15.72:0/3401739490 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:40.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.641+0000 7f0477fff640 1 --2- 172.21.15.72:0/3401739490 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56273cbea650 0x56273cbeb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.641+0000 7f047d69d640 1 --2- 172.21.15.72:0/3401739490 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x56273cbf68a0 0x56273cbf2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.641+0000 7f047ce9c640 1 -- 172.21.15.72:0/3401739490 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x56273cbf68a0 msgr2=0x56273cbf2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:40.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.641+0000 7f047ce9c640 1 --2- 172.21.15.72:0/3401739490 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x56273cbf68a0 0x56273cbf2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.641+0000 7f047ce9c640 1 -- 172.21.15.72:0/3401739490 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56273cbea650 msgr2=0x56273cbeb070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:40.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.641+0000 7f047ce9c640 1 --2- 172.21.15.72:0/3401739490 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56273cbea650 0x56273cbeb070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.641+0000 7f047ce9c640 1 -- 172.21.15.72:0/3401739490 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56273cbf3880 con 0x56273cbe9c40 2024-04-15T15:16:40.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.641+0000 7f0477fff640 1 --2- 172.21.15.72:0/3401739490 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56273cbea650 0x56273cbeb070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:40.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.641+0000 7f047d69d640 1 --2- 172.21.15.72:0/3401739490 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x56273cbf68a0 0x56273cbf2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:40.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.641+0000 7f047ce9c640 1 --2- 172.21.15.72:0/3401739490 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56273cbe9c40 0x56273cbe92d0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f046c02c4a0 tx=0x7f046c049000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba2c62510726aa92 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.641+0000 7f0461ffb640 1 -- 172.21.15.72:0/3401739490 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f046c039040 con 0x56273cbe9c40 2024-04-15T15:16:40.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.641+0000 7f0461ffb640 1 -- 172.21.15.72:0/3401739490 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f046c005990 con 0x56273cbe9c40 2024-04-15T15:16:40.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.641+0000 7f0461ffb640 1 -- 172.21.15.72:0/3401739490 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f046c03fa50 con 0x56273cbe9c40 2024-04-15T15:16:40.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.642+0000 7f0481dc0740 1 -- 172.21.15.72:0/3401739490 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56273cbe9c40 msgr2=0x56273cbe92d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.642+0000 7f0481dc0740 1 --2- 172.21.15.72:0/3401739490 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56273cbe9c40 0x56273cbe92d0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f046c02c4a0 tx=0x7f046c049000 comp rx=0 tx=0).stop 2024-04-15T15:16:40.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.642+0000 7f0481dc0740 1 -- 172.21.15.72:0/3401739490 shutdown_connections 2024-04-15T15:16:40.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.642+0000 7f0481dc0740 1 --2- 172.21.15.72:0/3401739490 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x56273cbf68a0 0x56273cbf2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.642+0000 7f0481dc0740 1 --2- 172.21.15.72:0/3401739490 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56273cbea650 0x56273cbeb070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.642+0000 7f0481dc0740 1 --2- 172.21.15.72:0/3401739490 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56273cbe9c40 0x56273cbe92d0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.642+0000 7f0481dc0740 1 -- 172.21.15.72:0/3401739490 >> 172.21.15.72:0/3401739490 conn(0x56273cbc6da0 msgr2=0x56273cbca320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:40.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.642+0000 7f0481dc0740 1 -- 172.21.15.72:0/3401739490 shutdown_connections 2024-04-15T15:16:40.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.642+0000 7f0481dc0740 1 -- 172.21.15.72:0/3401739490 wait complete. 2024-04-15T15:16:40.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.643+0000 7f0481dc0740 1 -- start start 2024-04-15T15:16:40.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.643+0000 7f0481dc0740 1 Processor -- start 2024-04-15T15:16:40.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.643+0000 7f0481dc0740 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x56273cbe9c40 0x56273cc8ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.643+0000 7f0481dc0740 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56273cbea650 0x56273cbf2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.644+0000 7f0481dc0740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56273cbf68a0 0x56273cbf2e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.644+0000 7f0481dc0740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f0478200210 con 0x56273cbf68a0 2024-04-15T15:16:40.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.644+0000 7f0481dc0740 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x56273cc35cb0 con 0x56273cbea650 2024-04-15T15:16:40.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.644+0000 7f0481dc0740 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x56273cc8f8e0 con 0x56273cbe9c40 2024-04-15T15:16:40.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.644+0000 7f0481dc0740 10 client.0 _ll_register_callbacks cb 0x7f046c037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:16:40.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.644+0000 7f047d69d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56273cbf68a0 0x56273cbf2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.644+0000 7f047d69d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56273cbf68a0 0x56273cbf2e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60642/0 (socket says 172.21.15.72:60642) 2024-04-15T15:16:40.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.644+0000 7f047d69d640 1 -- 172.21.15.72:0/3044373691 learned_addr learned my addr 172.21.15.72:0/3044373691 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:40.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.644+0000 7f047ce9c640 1 --2- 172.21.15.72:0/3044373691 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x56273cbe9c40 0x56273cc8ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.644+0000 7f0477fff640 1 --2- 172.21.15.72:0/3044373691 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56273cbea650 0x56273cbf2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.644+0000 7f047d69d640 1 -- 172.21.15.72:0/3044373691 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x56273cbe9c40 msgr2=0x56273cc8ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:40.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.644+0000 7f047d69d640 1 --2- 172.21.15.72:0/3044373691 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x56273cbe9c40 0x56273cc8ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.644+0000 7f047d69d640 1 -- 172.21.15.72:0/3044373691 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56273cbea650 msgr2=0x56273cbf2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:40.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.644+0000 7f047d69d640 1 --2- 172.21.15.72:0/3044373691 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56273cbea650 0x56273cbf2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.644+0000 7f047d69d640 1 -- 172.21.15.72:0/3044373691 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56273cbc5c10 con 0x56273cbf68a0 2024-04-15T15:16:40.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.644+0000 7f047d69d640 1 --2- 172.21.15.72:0/3044373691 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56273cbf68a0 0x56273cbf2e40 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f047008a7b0 tx=0x7f047008c050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=113b8059ede17b58 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.644+0000 7f04577fe640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-04-15T15:16:40.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.644+0000 7f04577fe640 1 -- 172.21.15.72:0/3044373691 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0470062b50 con 0x56273cbf68a0 2024-04-15T15:16:40.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.644+0000 7f04577fe640 1 -- 172.21.15.72:0/3044373691 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(40..40 src has 1..40) ==== 6130+0+0 (secure 0 0 0) 0x7f047008a330 con 0x56273cbf68a0 2024-04-15T15:16:40.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.644+0000 7f0481dc0740 1 -- 172.21.15.72:0/3044373691 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f046c005990 con 0x56273cbf68a0 2024-04-15T15:16:40.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.644+0000 7f04577fe640 1 -- 172.21.15.72:0/3044373691 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f047005e320 con 0x56273cbf68a0 2024-04-15T15:16:40.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.644+0000 7f0481dc0740 10 client.6045 fetch_fsmap learned FSMap version 16 2024-04-15T15:16:40.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.644+0000 7f0481dc0740 1 -- 172.21.15.72:0/3044373691 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x56273cbcc7b0 con 0x56273cbf68a0 2024-04-15T15:16:40.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.644+0000 7f04577fe640 1 -- 172.21.15.72:0/3044373691 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f046c005990 con 0x56273cbf68a0 2024-04-15T15:16:40.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.644+0000 7f04577fe640 1 -- 172.21.15.72:0/3044373691 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 16) ==== 3494+0+0 (secure 0 0 0) 0x7f04700593c0 con 0x56273cbf68a0 2024-04-15T15:16:40.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.645+0000 7f0481dc0740 10 client.6045 fetch_fsmap finished waiting for FSMap version 16 2024-04-15T15:16:40.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.645+0000 7f0481dc0740 10 client.6045 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.a 2024-04-15T15:16:40.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.645+0000 7f0481dc0740 20 client.6045 populate_metadata read hostname 'smithi072' 2024-04-15T15:16:40.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.645+0000 7f0481dc0740 1 --2- 172.21.15.72:0/3044373691 >> [v2:172.21.15.72:6834/4270793613,v1:172.21.15.72:6836/4270793613] conn(0x56273cc8f1e0 0x56273cbe8950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.645+0000 7f0481dc0740 4 client.6045 mds_command: new command op to 5559 tid=0 multi_id=0 [{"prefix": "ops", "flags": ["locks"], "path": "/tmp/mds.0-ops"}] 2024-04-15T15:16:40.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.645+0000 7f0481dc0740 1 -- 172.21.15.72:0/3044373691 --> [v2:172.21.15.72:6834/4270793613,v1:172.21.15.72:6836/4270793613] -- command(tid 0: {"prefix": "ops", "flags": ["locks"], "path": "/tmp/mds.0-ops"}) -- 0x56273cbc9580 con 0x56273cc8f1e0 2024-04-15T15:16:40.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.645+0000 7f047ce9c640 1 --2- 172.21.15.72:0/3044373691 >> [v2:172.21.15.72:6834/4270793613,v1:172.21.15.72:6836/4270793613] conn(0x56273cc8f1e0 0x56273cbe8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.645+0000 7f04577fe640 10 client.6045 ms_handle_connect on v2:172.21.15.72:6834/4270793613 2024-04-15T15:16:40.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.645+0000 7f047ce9c640 1 --2- 172.21.15.72:0/3044373691 >> [v2:172.21.15.72:6834/4270793613,v1:172.21.15.72:6836/4270793613] conn(0x56273cc8f1e0 0x56273cbe8950 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d91fe6a316418352 server_cookie=91c90acb04a560c3 in_seq=0 out_seq=0 2024-04-15T15:16:40.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.646+0000 7f04577fe640 1 -- 172.21.15.72:0/3044373691 <== mds.0 v2:172.21.15.72:6834/4270793613 1 ==== command_reply(tid 0: 0 ) ==== 8+0+33 (crc 0 0 0) 0x56273cbc9580 con 0x56273cc8f1e0 2024-04-15T15:16:40.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.646+0000 7f04577fe640 10 client.6045 handle_command_reply: tid=0 2024-04-15T15:16:40.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.646+0000 7f0481dc0740 1 client.6045 shutdown 2024-04-15T15:16:40.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.646+0000 7f0481dc0740 1 -- 172.21.15.72:0/3044373691 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56273cbf68a0 msgr2=0x56273cbf2e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.646+0000 7f0481dc0740 1 --2- 172.21.15.72:0/3044373691 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56273cbf68a0 0x56273cbf2e40 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f047008a7b0 tx=0x7f047008c050 comp rx=0 tx=0).stop 2024-04-15T15:16:40.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.646+0000 7f0481dc0740 1 -- 172.21.15.72:0/3044373691 shutdown_connections 2024-04-15T15:16:40.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.646+0000 7f0481dc0740 1 --2- 172.21.15.72:0/3044373691 >> [v2:172.21.15.72:6834/4270793613,v1:172.21.15.72:6836/4270793613] conn(0x56273cc8f1e0 0x56273cbe8950 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.646+0000 7f0481dc0740 1 --2- 172.21.15.72:0/3044373691 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x56273cbf68a0 0x56273cbf2e40 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.646+0000 7f0481dc0740 1 --2- 172.21.15.72:0/3044373691 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x56273cbea650 0x56273cbf2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.646+0000 7f04577fe640 0 client.6045 ms_handle_reset on v2:172.21.15.72:6834/4270793613 2024-04-15T15:16:40.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.646+0000 7f0481dc0740 1 --2- 172.21.15.72:0/3044373691 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x56273cbe9c40 0x56273cc8ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.646+0000 7f0481dc0740 1 -- 172.21.15.72:0/3044373691 >> 172.21.15.72:0/3044373691 conn(0x56273cbc6da0 msgr2=0x56273cbe7830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:40.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.646+0000 7f0481dc0740 1 -- 172.21.15.72:0/3044373691 shutdown_connections 2024-04-15T15:16:40.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.646+0000 7f0481dc0740 1 -- 172.21.15.72:0/3044373691 wait complete. 2024-04-15T15:16:40.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.646+0000 7f0481dc0740 20 client.6045 trim_cache size 0 max 16384 2024-04-15T15:16:40.654 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-04-15T15:16:40.654 INFO:teuthology.orchestra.run.smithi072.stdout: "path": "/tmp/mds.0-ops" 2024-04-15T15:16:40.654 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-04-15T15:16:40.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.646+0000 7f047f100640 1 -- 172.21.15.72:0/1214551952 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f045c06c980 msgr2=0x7f045c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.646+0000 7f047f100640 1 --2- 172.21.15.72:0/1214551952 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f045c06c980 0x7f045c06ee40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f046c036000 tx=0x7f046c030000 comp rx=0 tx=0).stop 2024-04-15T15:16:40.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.646+0000 7f047f100640 1 -- 172.21.15.72:0/1214551952 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04780b5660 msgr2=0x7f04780b5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.646+0000 7f047f100640 1 --2- 172.21.15.72:0/1214551952 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04780b5660 0x7f04780b5ae0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f0470007740 tx=0x7f0470007c00 comp rx=0 tx=0).stop 2024-04-15T15:16:40.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.647+0000 7f047f100640 1 -- 172.21.15.72:0/1214551952 shutdown_connections 2024-04-15T15:16:40.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.647+0000 7f047f100640 1 --2- 172.21.15.72:0/1214551952 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f045c06c980 0x7f045c06ee40 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.647+0000 7f047f100640 1 --2- 172.21.15.72:0/1214551952 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f04780b5660 0x7f04780b5ae0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.647+0000 7f047f100640 1 --2- 172.21.15.72:0/1214551952 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f04780b2520 0x7f04781608a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.647+0000 7f047f100640 1 --2- 172.21.15.72:0/1214551952 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f04680039b0 0x7f0478160360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.647+0000 7f047f100640 1 -- 172.21.15.72:0/1214551952 >> 172.21.15.72:0/1214551952 conn(0x7f04780ceec0 msgr2=0x7f04780cdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:40.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.647+0000 7f047f100640 1 -- 172.21.15.72:0/1214551952 shutdown_connections 2024-04-15T15:16:40.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:40.647+0000 7f047f100640 1 -- 172.21.15.72:0/1214551952 wait complete. 2024-04-15T15:16:40.659 DEBUG:teuthology.orchestra.run.smithi072:> mktemp 2024-04-15T15:16:40.675 INFO:teuthology.orchestra.run.smithi072.stdout:/tmp/tmp.ssJ9PtjoBA 2024-04-15T15:16:40.675 DEBUG:teuthology.orchestra.run.smithi072:> sudo cp /tmp/mds.0-ops /tmp/tmp.ssJ9PtjoBA 2024-04-15T15:16:40.744 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0666 /tmp/tmp.ssJ9PtjoBA 2024-04-15T15:16:40.872 DEBUG:teuthology.orchestra.remote:smithi072:/tmp/tmp.ssJ9PtjoBA is 8KB 2024-04-15T15:16:40.922 DEBUG:teuthology.orchestra.run.smithi072:> rm -fr /tmp/tmp.ssJ9PtjoBA 2024-04-15T15:16:40.937 DEBUG:tasks.cephfs.filesystem:read 7765B of ops 2024-04-15T15:16:40.938 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json dump tree /volumes/_nogroup/subvol_quiesce/6bdc4ba4-8793-4ca5-8487-64cfa9a984f4 --path=/tmp/mds.0-cache 2024-04-15T15:16:41.095 INFO:teuthology.orchestra.run.smithi072.stdout:{"path":"/tmp/mds.0-cache"} 2024-04-15T15:16:41.096 DEBUG:tasks.cephfs.filesystem:_json_asok output { "path": "/tmp/mds.0-cache" } 2024-04-15T15:16:41.096 DEBUG:teuthology.orchestra.run.smithi072:> mktemp 2024-04-15T15:16:41.112 INFO:teuthology.orchestra.run.smithi072.stdout:/tmp/tmp.gKA9mITW2S 2024-04-15T15:16:41.112 DEBUG:teuthology.orchestra.run.smithi072:> sudo cp /tmp/mds.0-cache /tmp/tmp.gKA9mITW2S 2024-04-15T15:16:41.181 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0666 /tmp/tmp.gKA9mITW2S 2024-04-15T15:16:41.309 DEBUG:teuthology.orchestra.remote:smithi072:/tmp/tmp.gKA9mITW2S is 6KB 2024-04-15T15:16:41.330 DEBUG:teuthology.orchestra.run.smithi072:> rm -fr /tmp/tmp.gKA9mITW2S 2024-04-15T15:16:41.347 DEBUG:tasks.cephfs.filesystem:read 6360B of cache 2024-04-15T15:16:41.348 INFO:tasks.cephfs.test_quiesce:qp = 1; qi = 1; qib = 1; qina = 0 2024-04-15T15:16:41.348 ERROR:tasks.cephfs.test_quiesce:inode: { "ino": 1099511627779, "rdev": 0, "ctime": "2024-04-15T15:16:39.811476+0000", "btime": "2024-04-15T15:16:20.823399+0000", "mode": 16895, "uid": 0, "gid": 0, "nlink": 1, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "layout": { "stripe_unit": 4194304, "stripe_count": 1, "object_size": 4194304, "pool_id": 6, "pool_ns": "" }, "old_pools": [], "size": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "truncate_from": 0, "truncate_pending": 0, "mtime": "2024-04-15T15:16:39.811476+0000", "atime": "2024-04-15T15:16:20.823399+0000", "time_warp_seq": 0, "change_attr": 2, "export_pin": -1, "export_ephemeral_random_pin": 0, "export_ephemeral_distributed_pin": false, "quiesce_block": false, "client_ranges": [], "dirstat": { "version": 0, "mtime": "2024-04-15T15:16:39.811476+0000", "num_files": 1, "num_subdirs": 0, "change_attr": 1 }, "rstat": { "version": 0, "rbytes": 0, "rfiles": 1, "rsubdirs": 1, "rsnaps": 0, "rctime": "2024-04-15T15:16:39.837590+0000" }, "accounted_rstat": { "version": 0, "rbytes": 0, "rfiles": 1, "rsubdirs": 1, "rsnaps": 0, "rctime": "2024-04-15T15:16:39.837590+0000" }, "version": 20, "file_data_version": 0, "xattr_version": 1, "backtrace_version": 2, "stray_prior_path": "", "max_size_ever": 0, "quota": { "max_bytes": 0, "max_files": 0 }, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "symlink": "", "xattrs": [], "dirfragtree": { "splits": [] }, "old_inodes": [], "oldest_snap": 18446744073709551614, "damage_flags": 0, "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 2, "dirfrag": 1, "caps": 1, "dirtyparent": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 8, "versionlock": { "gather_set": [], "state": "lock", "type": "iversion", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 1, "num_xlocks": 0, "xlock_by": {} }, "authlock": {}, "linklock": {}, "dirfragtreelock": {}, "filelock": {}, "xattrlock": {}, "snaplock": {}, "nestlock": { "gather_set": [], "state": "lock", "type": "inest", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "xlock_by": {} }, "flocklock": {}, "policylock": {}, "quiescelock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 2 } } }, "states": [ "auth", "dirty", "dirtyparent", "dirtypool" ], "client_caps": [ { "client_id": 5075, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 2 }, { "client_id": 5090, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 2 }, { "client_id": 5817, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 7 }, { "client_id": 5970, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 3 }, { "client_id": 5991, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 2 } ], "loner": -1, "want_loner": -1, "mds_caps_wanted": [], "dirfrags": [ { "path": "/volumes/_nogroup/subvol_quiesce/6bdc4ba4-8793-4ca5-8487-64cfa9a984f4", "dirfrag": "0x10000000003", "snapid_first": 2, "projected_version": "7", "version": "7", "committing_version": "0", "committed_version": "0", "is_rep": false, "dir_auth": "", "states": [ "auth", "dirty", "complete" ], "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "child": 1, "dirty": 1, "authpin": 0 }, "nref": 2, "dentries": [ { "path": "volumes/_nogroup/subvol_quiesce/6bdc4ba4-8793-4ca5-8487-64cfa9a984f4/file", "path_ino": 1, "snap_first": 2, "snap_last": 18446744073709551614, "is_primary": true, "is_remote": false, "is_null": false, "is_new": false, "inode": 1099511629282, "remote_type": "", "version": 6, "projected_version": 6, "auth_pins": 0, "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "is_frozen": false, "is_freezing": false, "pins": { "request": 0, "lock": 0, "inodepin": 1, "dirty": 1, "authpin": 0 }, "nref": 2, "lock": {}, "versionlock": { "gather_set": [], "state": "lock", "type": "dversion", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "xlock_by": {} }, "states": [ "auth", "dirty" ] } ] } ], "is_subvolume": false } 2024-04-15T15:16:41.349 ERROR:tasks.cephfs.test_quiesce:op: { "description": "internal op quiesce_inode:mds.0:2 fp=#0x10000000003", "initiated_at": "2024-04-15T15:16:40.026071+0000", "age": 0.620385737, "duration": 0.620515564, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 2 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-15T15:16:40.026071+0000", "event": "initiated" }, { "time": "2024-04-15T15:16:40.026071+0000", "event": "header_read" }, { "time": "2024-04-15T15:16:40.026071+0000", "event": "throttled" }, { "time": "2024-04-15T15:16:40.026071+0000", "event": "all_read" }, { "time": "2024-04-15T15:16:40.026071+0000", "event": "dispatched" }, { "time": "2024-04-15T15:16:40.026256+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-15T15:16:40.026643+0000", "event": "acquired locks" }, { "time": "2024-04-15T15:16:40.029240+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 2, "dirfrag": 1, "caps": 1, "dirtyparent": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 8 }, "object_string": "[inode 0x10000000003 [...2,head] /volumes/_nogroup/subvol_quiesce/6bdc4ba4-8793-4ca5-8487-64cfa9a984f4/ auth v20 ap=1 DIRTYPARENT f(v0 m2024-04-15T15:16:39.811476+0000 1=1+0) n(v0 rc2024-04-15T15:16:39.837590+0000 2=1+1) (inest lock) (iversion lock w=1) (iquiesce lock x=1 by request(mds.0:2 nref=3)) caps={5075=pAsLsXs/-@2,5090=pAsLsXs/-@2,5817=pAsLsXs/-@7,5970=pAsLsXs/-@3,5991=pAsLsXs/-@2} | request=1 lock=2 dirfrag=1 caps=1 dirtyparent=1 dirty=1 waiter=0 authpin=1 0x5614a7fd5600]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 2 } } }, "flags": 4, "wrlock_target": -1 }, { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 2, "dirfrag": 1, "caps": 1, "dirtyparent": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 8 }, "object_string": "[inode 0x10000000003 [...2,head] /volumes/_nogroup/subvol_quiesce/6bdc4ba4-8793-4ca5-8487-64cfa9a984f4/ auth v20 ap=1 DIRTYPARENT f(v0 m2024-04-15T15:16:39.811476+0000 1=1+0) n(v0 rc2024-04-15T15:16:39.837590+0000 2=1+1) (inest lock) (iversion lock w=1) (iquiesce lock x=1 by request(mds.0:2 nref=3)) caps={5075=pAsLsXs/-@2,5090=pAsLsXs/-@2,5817=pAsLsXs/-@7,5970=pAsLsXs/-@3,5991=pAsLsXs/-@2} | request=1 lock=2 dirfrag=1 caps=1 dirtyparent=1 dirty=1 waiter=0 authpin=1 0x5614a7fd5600]", "lock": { "gather_set": [], "state": "lock", "type": "iversion", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 1, "num_xlocks": 0, "xlock_by": {} }, "flags": 2, "wrlock_target": -1 } ] } } 2024-04-15T15:16:41.349 ERROR:tasks.cephfs.test_quiesce:lock_type: iversion 2024-04-15T15:16:41.349 ERROR:tasks.cephfs.test_quiesce:cache: [ { "ino": 1099511629282, "rdev": 0, "ctime": "2024-04-15T15:16:39.837590+0000", "btime": "2024-04-15T15:16:39.811476+0000", "mode": 33188, "uid": 1000, "gid": 1301, "nlink": 1, "dir_layout": { "dir_hash": 0, "unused1": 0, "unused2": 0, "unused3": 0 }, "layout": { "stripe_unit": 4194304, "stripe_count": 1, "object_size": 4194304, "pool_id": 6, "pool_ns": "" }, "old_pools": [], "size": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "truncate_from": 0, "truncate_pending": 0, "mtime": "2024-04-15T15:16:39.815262+0000", "atime": "2024-04-15T15:16:39.815262+0000", "time_warp_seq": 2, "change_attr": 1, "export_pin": -1, "export_ephemeral_random_pin": 0, "export_ephemeral_distributed_pin": false, "quiesce_block": true, "client_ranges": [], "dirstat": { "version": 0, "mtime": "0.000000", "num_files": 0, "num_subdirs": 0, "change_attr": 0 }, "rstat": { "version": 0, "rbytes": 0, "rfiles": 1, "rsubdirs": 0, "rsnaps": 0, "rctime": "2024-04-15T15:16:39.837590+0000" }, "accounted_rstat": { "version": 0, "rbytes": 0, "rfiles": 1, "rsubdirs": 0, "rsnaps": 0, "rctime": "2024-04-15T15:16:39.837590+0000" }, "version": 6, "file_data_version": 0, "xattr_version": 1, "backtrace_version": 4, "stray_prior_path": "", "max_size_ever": 0, "quota": { "max_bytes": 0, "max_files": 0 }, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "symlink": "", "xattrs": [], "dirfragtree": { "splits": [] }, "old_inodes": [], "oldest_snap": 18446744073709551614, "damage_flags": 0, "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 0, "caps": 1, "dirtyparent": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 5, "versionlock": { "gather_set": [], "state": "lock", "type": "iversion", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "xlock_by": {} }, "authlock": {}, "linklock": {}, "dirfragtreelock": {}, "filelock": {}, "xattrlock": {}, "snaplock": {}, "nestlock": { "gather_set": [], "state": "lock", "type": "inest", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "xlock_by": {} }, "flocklock": {}, "policylock": {}, "quiescelock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "xlock_by": {} }, "states": [ "auth", "dirty", "dirtyparent", "dirtypool" ], "client_caps": [ { "client_id": 5970, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 3 } ], "loner": -1, "want_loner": -1, "mds_caps_wanted": [], "dirfrags": [], "is_subvolume": false }, { "ino": 1099511627779, "rdev": 0, "ctime": "2024-04-15T15:16:39.811476+0000", "btime": "2024-04-15T15:16:20.823399+0000", "mode": 16895, "uid": 0, "gid": 0, "nlink": 1, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "layout": { "stripe_unit": 4194304, "stripe_count": 1, "object_size": 4194304, "pool_id": 6, "pool_ns": "" }, "old_pools": [], "size": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "truncate_from": 0, "truncate_pending": 0, "mtime": "2024-04-15T15:16:39.811476+0000", "atime": "2024-04-15T15:16:20.823399+0000", "time_warp_seq": 0, "change_attr": 2, "export_pin": -1, "export_ephemeral_random_pin": 0, "export_ephemeral_distributed_pin": false, "quiesce_block": false, "client_ranges": [], "dirstat": { "version": 0, "mtime": "2024-04-15T15:16:39.811476+0000", "num_files": 1, "num_subdirs": 0, "change_attr": 1 }, "rstat": { "version": 0, "rbytes": 0, "rfiles": 1, "rsubdirs": 1, "rsnaps": 0, "rctime": "2024-04-15T15:16:39.837590+0000" }, "accounted_rstat": { "version": 0, "rbytes": 0, "rfiles": 1, "rsubdirs": 1, "rsnaps": 0, "rctime": "2024-04-15T15:16:39.837590+0000" }, "version": 20, "file_data_version": 0, "xattr_version": 1, "backtrace_version": 2, "stray_prior_path": "", "max_size_ever": 0, "quota": { "max_bytes": 0, "max_files": 0 }, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "symlink": "", "xattrs": [], "dirfragtree": { "splits": [] }, "old_inodes": [], "oldest_snap": 18446744073709551614, "damage_flags": 0, "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 2, "dirfrag": 1, "caps": 1, "dirtyparent": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 8, "versionlock": { "gather_set": [], "state": "lock", "type": "iversion", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 1, "num_xlocks": 0, "xlock_by": {} }, "authlock": {}, "linklock": {}, "dirfragtreelock": {}, "filelock": {}, "xattrlock": {}, "snaplock": {}, "nestlock": { "gather_set": [], "state": "lock", "type": "inest", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "xlock_by": {} }, "flocklock": {}, "policylock": {}, "quiescelock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 2 } } }, "states": [ "auth", "dirty", "dirtyparent", "dirtypool" ], "client_caps": [ { "client_id": 5075, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 2 }, { "client_id": 5090, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 2 }, { "client_id": 5817, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 7 }, { "client_id": 5970, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 3 }, { "client_id": 5991, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 2 } ], "loner": -1, "want_loner": -1, "mds_caps_wanted": [], "dirfrags": [ { "path": "/volumes/_nogroup/subvol_quiesce/6bdc4ba4-8793-4ca5-8487-64cfa9a984f4", "dirfrag": "0x10000000003", "snapid_first": 2, "projected_version": "7", "version": "7", "committing_version": "0", "committed_version": "0", "is_rep": false, "dir_auth": "", "states": [ "auth", "dirty", "complete" ], "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "child": 1, "dirty": 1, "authpin": 0 }, "nref": 2, "dentries": [ { "path": "volumes/_nogroup/subvol_quiesce/6bdc4ba4-8793-4ca5-8487-64cfa9a984f4/file", "path_ino": 1, "snap_first": 2, "snap_last": 18446744073709551614, "is_primary": true, "is_remote": false, "is_null": false, "is_new": false, "inode": 1099511629282, "remote_type": "", "version": 6, "projected_version": 6, "auth_pins": 0, "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "is_frozen": false, "is_freezing": false, "pins": { "request": 0, "lock": 0, "inodepin": 1, "dirty": 1, "authpin": 0 }, "nref": 2, "lock": {}, "versionlock": { "gather_set": [], "state": "lock", "type": "dversion", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "xlock_by": {} }, "states": [ "auth", "dirty" ] } ] } ], "is_subvolume": false } ] 2024-04-15T15:16:41.350 ERROR:tasks.cephfs.test_quiesce:ops: { "1099511627779": { "description": "internal op quiesce_inode:mds.0:2 fp=#0x10000000003", "initiated_at": "2024-04-15T15:16:40.026071+0000", "age": 0.620385737, "duration": 0.620515564, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 2 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-15T15:16:40.026071+0000", "event": "initiated" }, { "time": "2024-04-15T15:16:40.026071+0000", "event": "header_read" }, { "time": "2024-04-15T15:16:40.026071+0000", "event": "throttled" }, { "time": "2024-04-15T15:16:40.026071+0000", "event": "all_read" }, { "time": "2024-04-15T15:16:40.026071+0000", "event": "dispatched" }, { "time": "2024-04-15T15:16:40.026256+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-15T15:16:40.026643+0000", "event": "acquired locks" }, { "time": "2024-04-15T15:16:40.029240+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 2, "dirfrag": 1, "caps": 1, "dirtyparent": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 8 }, "object_string": "[inode 0x10000000003 [...2,head] /volumes/_nogroup/subvol_quiesce/6bdc4ba4-8793-4ca5-8487-64cfa9a984f4/ auth v20 ap=1 DIRTYPARENT f(v0 m2024-04-15T15:16:39.811476+0000 1=1+0) n(v0 rc2024-04-15T15:16:39.837590+0000 2=1+1) (inest lock) (iversion lock w=1) (iquiesce lock x=1 by request(mds.0:2 nref=3)) caps={5075=pAsLsXs/-@2,5090=pAsLsXs/-@2,5817=pAsLsXs/-@7,5970=pAsLsXs/-@3,5991=pAsLsXs/-@2} | request=1 lock=2 dirfrag=1 caps=1 dirtyparent=1 dirty=1 waiter=0 authpin=1 0x5614a7fd5600]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 2 } } }, "flags": 4, "wrlock_target": -1 }, { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 2, "dirfrag": 1, "caps": 1, "dirtyparent": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 8 }, "object_string": "[inode 0x10000000003 [...2,head] /volumes/_nogroup/subvol_quiesce/6bdc4ba4-8793-4ca5-8487-64cfa9a984f4/ auth v20 ap=1 DIRTYPARENT f(v0 m2024-04-15T15:16:39.811476+0000 1=1+0) n(v0 rc2024-04-15T15:16:39.837590+0000 2=1+1) (inest lock) (iversion lock w=1) (iquiesce lock x=1 by request(mds.0:2 nref=3)) caps={5075=pAsLsXs/-@2,5090=pAsLsXs/-@2,5817=pAsLsXs/-@7,5970=pAsLsXs/-@3,5991=pAsLsXs/-@2} | request=1 lock=2 dirfrag=1 caps=1 dirtyparent=1 dirty=1 waiter=0 authpin=1 0x5614a7fd5600]", "lock": { "gather_set": [], "state": "lock", "type": "iversion", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 1, "num_xlocks": 0, "xlock_by": {} }, "flags": 2, "wrlock_target": -1 } ] } }, "1099511629282": { "description": "internal op quiesce_inode:mds.0:3 fp=#0x100000005e2", "initiated_at": "2024-04-15T15:16:40.026794+0000", "age": 0.619663349, "duration": 0.619879702, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce blocked", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 3 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-15T15:16:40.026794+0000", "event": "initiated" }, { "time": "2024-04-15T15:16:40.026794+0000", "event": "header_read" }, { "time": "2024-04-15T15:16:40.026794+0000", "event": "throttled" }, { "time": "2024-04-15T15:16:40.026794+0000", "event": "all_read" }, { "time": "2024-04-15T15:16:40.026794+0000", "event": "dispatched" }, { "time": "2024-04-15T15:16:40.026906+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-15T15:16:40.027524+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-15T15:16:40.029120+0000", "event": "acquired locks" }, { "time": "2024-04-15T15:16:40.029121+0000", "event": "quiesce blocked" } ], "locks": [] } } } 2024-04-15T15:16:41.350 INFO:tasks.cephfs.test_quiesce:making archive directory /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657120/quiesce 2024-04-15T15:16:41.352 ERROR:tasks.cephfs.test_quiesce:cache written to /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657120/quiesce/cachelywu1_gn 2024-04-15T15:16:41.354 ERROR:tasks.cephfs.test_quiesce:ops written to /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657120/quiesce/ops6icygeww 2024-04-15T15:16:41.354 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-15T15:16:41.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.488+0000 7f5024e22640 1 Processor -- start 2024-04-15T15:16:41.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.488+0000 7f5024e22640 1 -- start start 2024-04-15T15:16:41.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.489+0000 7f5024e22640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f50200b5410 0x7f50200b57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.489+0000 7f5024e22640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f50200afae0 0x7f50200cf960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.489+0000 7f5024e22640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50200d07c0 con 0x7f50200b0050 2024-04-15T15:16:41.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.489+0000 7f5024e22640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f50200d0330 con 0x7f50200b5410 2024-04-15T15:16:41.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.489+0000 7f5024e22640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f50200b5d30 con 0x7f50200afae0 2024-04-15T15:16:41.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.489+0000 7f501e59c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f50200b5410 0x7f50200b57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.489+0000 7f501e59c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f50200b5410 0x7f50200b57f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60644/0 (socket says 172.21.15.72:60644) 2024-04-15T15:16:41.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.489+0000 7f501e59c640 1 -- 172.21.15.72:0/2414959896 learned_addr learned my addr 172.21.15.72:0/2414959896 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:41.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.489+0000 7f501ed9d640 1 --2- 172.21.15.72:0/2414959896 >> v2:172.21.15.175:3300/0 conn(0x7f50200afae0 0x7f50200cf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.489+0000 7f501e59c640 1 -- 172.21.15.72:0/2414959896 >> v2:172.21.15.175:3300/0 conn(0x7f50200afae0 msgr2=0x7f50200cf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:41.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.489+0000 7f501e59c640 1 --2- 172.21.15.72:0/2414959896 >> v2:172.21.15.175:3300/0 conn(0x7f50200afae0 0x7f50200cf960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.489+0000 7f501e59c640 1 -- 172.21.15.72:0/2414959896 >> v1:172.21.15.175:6789/0 conn(0x7f50200b0050 legacy=0x7f502006c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.489+0000 7f501e59c640 1 -- 172.21.15.72:0/2414959896 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50200b3aa0 con 0x7f50200b5410 2024-04-15T15:16:41.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.490+0000 7f501e59c640 1 --2- 172.21.15.72:0/2414959896 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50200b5410 0x7f50200b57f0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f5020070430 tx=0x7f501002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22e04d61dc6fe9c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:41.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.490+0000 7f501dd9b640 1 -- 172.21.15.72:0/2414959896 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5010035020 con 0x7f50200b5410 2024-04-15T15:16:41.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.490+0000 7f501dd9b640 1 -- 172.21.15.72:0/2414959896 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5010031870 con 0x7f50200b5410 2024-04-15T15:16:41.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.490+0000 7f501dd9b640 1 -- 172.21.15.72:0/2414959896 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5010031c40 con 0x7f50200b5410 2024-04-15T15:16:41.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.490+0000 7f5024e22640 1 -- 172.21.15.72:0/2414959896 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50200b5410 msgr2=0x7f50200b57f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.491+0000 7f5024e22640 1 --2- 172.21.15.72:0/2414959896 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50200b5410 0x7f50200b57f0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f5020070430 tx=0x7f501002b910 comp rx=0 tx=0).stop 2024-04-15T15:16:41.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.491+0000 7f5024e22640 1 -- 172.21.15.72:0/2414959896 shutdown_connections 2024-04-15T15:16:41.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.491+0000 7f5024e22640 1 --2- 172.21.15.72:0/2414959896 >> v2:172.21.15.175:3300/0 conn(0x7f50200afae0 0x7f50200cf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.491+0000 7f5024e22640 1 --2- 172.21.15.72:0/2414959896 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50200b5410 0x7f50200b57f0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.491+0000 7f5024e22640 1 -- 172.21.15.72:0/2414959896 >> 172.21.15.72:0/2414959896 conn(0x7f50200d0a50 msgr2=0x7f50200d0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:41.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.491+0000 7f5024e22640 1 -- 172.21.15.72:0/2414959896 shutdown_connections 2024-04-15T15:16:41.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.491+0000 7f5024e22640 1 -- 172.21.15.72:0/2414959896 wait complete. 2024-04-15T15:16:41.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.492+0000 7f5024e22640 1 Processor -- start 2024-04-15T15:16:41.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.492+0000 7f5024e22640 1 -- start start 2024-04-15T15:16:41.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.493+0000 7f5024e22640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f50200afae0 0x7f502015d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.493+0000 7f5024e22640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f50200b0050 0x7f5020145190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.493+0000 7f5024e22640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50200b5410 0x7f50200b0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.493+0000 7f5024e22640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f50200d18f0 con 0x7f50200b5410 2024-04-15T15:16:41.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.493+0000 7f5024e22640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f50200d1610 con 0x7f50200b0050 2024-04-15T15:16:41.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.493+0000 7f5024e22640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f50200cfea0 con 0x7f50200afae0 2024-04-15T15:16:41.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.493+0000 7f501ed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50200b5410 0x7f50200b0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.493+0000 7f501ed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50200b5410 0x7f50200b0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:60650/0 (socket says 172.21.15.72:60650) 2024-04-15T15:16:41.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.493+0000 7f501ed9d640 1 -- 172.21.15.72:0/3708037303 learned_addr learned my addr 172.21.15.72:0/3708037303 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:41.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.493+0000 7f501e59c640 1 --2- 172.21.15.72:0/3708037303 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f50200afae0 0x7f502015d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.493+0000 7f5015d9b640 1 --2- 172.21.15.72:0/3708037303 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f50200b0050 0x7f5020145190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.493+0000 7f501ed9d640 1 -- 172.21.15.72:0/3708037303 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f50200afae0 msgr2=0x7f502015d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.493+0000 7f501ed9d640 1 --2- 172.21.15.72:0/3708037303 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f50200afae0 0x7f502015d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.493+0000 7f501ed9d640 1 -- 172.21.15.72:0/3708037303 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f50200b0050 msgr2=0x7f5020145190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.493+0000 7f501ed9d640 1 --2- 172.21.15.72:0/3708037303 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f50200b0050 0x7f5020145190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.493+0000 7f501ed9d640 1 -- 172.21.15.72:0/3708037303 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50201456d0 con 0x7f50200b5410 2024-04-15T15:16:41.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.493+0000 7f501ed9d640 1 --2- 172.21.15.72:0/3708037303 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50200b5410 0x7f50200b0530 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f5018007e60 tx=0x7f501800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:41.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.493+0000 7f50177fe640 1 -- 172.21.15.72:0/3708037303 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5018014420 con 0x7f50200b5410 2024-04-15T15:16:41.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.494+0000 7f50177fe640 1 -- 172.21.15.72:0/3708037303 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5018019070 con 0x7f50200b5410 2024-04-15T15:16:41.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.494+0000 7f5024e22640 1 -- 172.21.15.72:0/3708037303 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50200b0a70 con 0x7f50200b5410 2024-04-15T15:16:41.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.494+0000 7f50177fe640 1 -- 172.21.15.72:0/3708037303 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5018014820 con 0x7f50200b5410 2024-04-15T15:16:41.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.494+0000 7f5024e22640 1 -- 172.21.15.72:0/3708037303 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50200b12d0 con 0x7f50200b5410 2024-04-15T15:16:41.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.495+0000 7f5024e22640 1 -- 172.21.15.72:0/3708037303 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5020075c50 con 0x7f50200b5410 2024-04-15T15:16:41.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.495+0000 7f50177fe640 1 -- 172.21.15.72:0/3708037303 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f50180149c0 con 0x7f50200b5410 2024-04-15T15:16:41.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.495+0000 7f50177fe640 1 --2- 172.21.15.72:0/3708037303 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f500006c980 0x7f500006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.496+0000 7f50177fe640 1 -- 172.21.15.72:0/3708037303 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6130+0+0 (secure 0 0 0) 0x7f501808c5b0 con 0x7f50200b5410 2024-04-15T15:16:41.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.496+0000 7f501e59c640 1 --2- 172.21.15.72:0/3708037303 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f500006c980 0x7f500006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.496+0000 7f501e59c640 1 --2- 172.21.15.72:0/3708037303 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f500006c980 0x7f500006ee40 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f5010036000 tx=0x7f501003c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:41.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.500+0000 7f50177fe640 1 -- 172.21.15.72:0/3708037303 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f501805a840 con 0x7f50200b5410 2024-04-15T15:16:41.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.713+0000 7f5024e22640 1 -- 172.21.15.72:0/3708037303 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f502006d610 con 0x7f50200b5410 2024-04-15T15:16:41.715 INFO:tasks.ceph.mon.a.smithi072.stderr:2024-04-15T15:16:41.714+0000 7f01d0611640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-15T15:16:41.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.715+0000 7f50177fe640 1 -- 172.21.15.72:0/3708037303 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v17) ==== 161+0+0 (secure 0 0 0) 0x7f501805a140 con 0x7f50200b5410 2024-04-15T15:16:41.717 INFO:teuthology.orchestra.run.smithi072.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-15T15:16:41.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.718+0000 7f5024e22640 1 -- 172.21.15.72:0/3708037303 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f500006c980 msgr2=0x7f500006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.718+0000 7f5024e22640 1 --2- 172.21.15.72:0/3708037303 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f500006c980 0x7f500006ee40 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f5010036000 tx=0x7f501003c040 comp rx=0 tx=0).stop 2024-04-15T15:16:41.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.718+0000 7f5024e22640 1 -- 172.21.15.72:0/3708037303 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50200b5410 msgr2=0x7f50200b0530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.718+0000 7f5024e22640 1 --2- 172.21.15.72:0/3708037303 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50200b5410 0x7f50200b0530 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f5018007e60 tx=0x7f501800e4b0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.718+0000 7f5024e22640 1 -- 172.21.15.72:0/3708037303 shutdown_connections 2024-04-15T15:16:41.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.718+0000 7f5024e22640 1 --2- 172.21.15.72:0/3708037303 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f500006c980 0x7f500006ee40 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.718+0000 7f5024e22640 1 --2- 172.21.15.72:0/3708037303 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f50200b5410 0x7f50200b0530 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.718+0000 7f5024e22640 1 --2- 172.21.15.72:0/3708037303 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f50200b0050 0x7f5020145190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.718+0000 7f5024e22640 1 --2- 172.21.15.72:0/3708037303 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f50200afae0 0x7f502015d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.718+0000 7f5024e22640 1 -- 172.21.15.72:0/3708037303 >> 172.21.15.72:0/3708037303 conn(0x7f50200d0a50 msgr2=0x7f5020072af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:41.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.719+0000 7f5024e22640 1 -- 172.21.15.72:0/3708037303 shutdown_connections 2024-04-15T15:16:41.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.719+0000 7f5024e22640 1 -- 172.21.15.72:0/3708037303 wait complete. 2024-04-15T15:16:41.731 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-04-15T15:16:41.757 INFO:tasks.ceph.mds.a.smithi072.stdout:starting mds.a at 2024-04-15T15:16:41.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.837+0000 7f8383da8640 1 Processor -- start 2024-04-15T15:16:41.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.838+0000 7f8383da8640 1 -- start start 2024-04-15T15:16:41.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.838+0000 7f8383da8640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f837c15c550 0x7f837c160b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.838+0000 7f8383da8640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f837c06f510 0x7f837c0d1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.838+0000 7f8383da8640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f837c06f970 0x7f837c0d1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.838+0000 7f8383da8640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f837c072790 con 0x7f837c15c550 2024-04-15T15:16:41.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.838+0000 7f8383da8640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f837c161060 con 0x7f837c06f970 2024-04-15T15:16:41.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.838+0000 7f8383da8640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f837c0cce20 con 0x7f837c06f510 2024-04-15T15:16:41.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.839+0000 7f8381b44640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f837c15c550 0x7f837c160b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.839+0000 7f8381b44640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f837c15c550 0x7f837c160b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43482/0 (socket says 172.21.15.72:43482) 2024-04-15T15:16:41.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.839+0000 7f8381b44640 1 -- 172.21.15.72:0/241322107 learned_addr learned my addr 172.21.15.72:0/241322107 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:41.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.839+0000 7f8382345640 1 --2- 172.21.15.72:0/241322107 >> v2:172.21.15.175:3300/0 conn(0x7f837c06f970 0x7f837c0d1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.839+0000 7f8381343640 1 --2- 172.21.15.72:0/241322107 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f837c06f510 0x7f837c0d1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.839+0000 7f8381b44640 1 -- 172.21.15.72:0/241322107 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f837c06f510 msgr2=0x7f837c0d1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:41.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.839+0000 7f8381b44640 1 --2- 172.21.15.72:0/241322107 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f837c06f510 0x7f837c0d1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.839+0000 7f8381b44640 1 -- 172.21.15.72:0/241322107 >> v2:172.21.15.175:3300/0 conn(0x7f837c06f970 msgr2=0x7f837c0d1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:41.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.839+0000 7f8381b44640 1 --2- 172.21.15.72:0/241322107 >> v2:172.21.15.175:3300/0 conn(0x7f837c06f970 0x7f837c0d1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.839+0000 7f8381b44640 1 -- 172.21.15.72:0/241322107 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f837c0cd100 con 0x7f837c15c550 2024-04-15T15:16:41.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.840+0000 7f8381b44640 1 --2- 172.21.15.72:0/241322107 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f837c15c550 0x7f837c160b20 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f837c0b5360 tx=0x7f837802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc19968ab7721770 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:41.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.840+0000 7f8380b42640 1 -- 172.21.15.72:0/241322107 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8378035020 con 0x7f837c15c550 2024-04-15T15:16:41.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.840+0000 7f8380b42640 1 -- 172.21.15.72:0/241322107 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8378031870 con 0x7f837c15c550 2024-04-15T15:16:41.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.840+0000 7f8380b42640 1 -- 172.21.15.72:0/241322107 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8378031c40 con 0x7f837c15c550 2024-04-15T15:16:41.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.840+0000 7f8383da8640 1 -- 172.21.15.72:0/241322107 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f837c15c550 msgr2=0x7f837c160b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.840+0000 7f8383da8640 1 --2- 172.21.15.72:0/241322107 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f837c15c550 0x7f837c160b20 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f837c0b5360 tx=0x7f837802b910 comp rx=0 tx=0).stop 2024-04-15T15:16:41.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.840+0000 7f8383da8640 1 -- 172.21.15.72:0/241322107 shutdown_connections 2024-04-15T15:16:41.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.840+0000 7f8383da8640 1 --2- 172.21.15.72:0/241322107 >> v2:172.21.15.175:3300/0 conn(0x7f837c06f970 0x7f837c0d1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.840+0000 7f8383da8640 1 --2- 172.21.15.72:0/241322107 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f837c06f510 0x7f837c0d1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.840+0000 7f8383da8640 1 --2- 172.21.15.72:0/241322107 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f837c15c550 0x7f837c160b20 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.840+0000 7f8383da8640 1 -- 172.21.15.72:0/241322107 >> 172.21.15.72:0/241322107 conn(0x7f837c0b3f50 msgr2=0x7f837c0b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:41.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.840+0000 7f8383da8640 1 -- 172.21.15.72:0/241322107 shutdown_connections 2024-04-15T15:16:41.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.840+0000 7f8383da8640 1 -- 172.21.15.72:0/241322107 wait complete. 2024-04-15T15:16:41.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.842+0000 7f8383da8640 1 Processor -- start 2024-04-15T15:16:41.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.842+0000 7f8383da8640 1 -- start start 2024-04-15T15:16:41.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.842+0000 7f8383da8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f837c06f510 0x7f837c0c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.842+0000 7f8383da8640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f837c06f970 0x7f837c0cb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.842+0000 7f8383da8640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f837c15c550 0x7f837c0cb920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.842+0000 7f8383da8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f837c0cd9a0 con 0x7f837c06f510 2024-04-15T15:16:41.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.842+0000 7f8383da8640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f837c0cd6c0 con 0x7f837c06f970 2024-04-15T15:16:41.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.842+0000 7f8383da8640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f837c15c930 con 0x7f837c15c550 2024-04-15T15:16:41.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.842+0000 7f8381b44640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f837c06f510 0x7f837c0c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.842+0000 7f8381b44640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f837c06f510 0x7f837c0c53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43494/0 (socket says 172.21.15.72:43494) 2024-04-15T15:16:41.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.843+0000 7f8381b44640 1 -- 172.21.15.72:0/2364080449 learned_addr learned my addr 172.21.15.72:0/2364080449 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:41.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.843+0000 7f8381b44640 1 -- 172.21.15.72:0/2364080449 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f837c15c550 msgr2=0x7f837c0cb920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.843+0000 7f8381b44640 1 --2- 172.21.15.72:0/2364080449 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f837c15c550 0x7f837c0cb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.843+0000 7f8381b44640 1 -- 172.21.15.72:0/2364080449 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f837c06f970 msgr2=0x7f837c0cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.843+0000 7f8381b44640 1 --2- 172.21.15.72:0/2364080449 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f837c06f970 0x7f837c0cb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.843+0000 7f8381b44640 1 -- 172.21.15.72:0/2364080449 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f837c15dad0 con 0x7f837c06f510 2024-04-15T15:16:41.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.843+0000 7f8381b44640 1 --2- 172.21.15.72:0/2364080449 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f837c06f510 0x7f837c0c53f0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f8378000c00 tx=0x7f83780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:41.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.843+0000 7f836e7fc640 1 -- 172.21.15.72:0/2364080449 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f837803ad60 con 0x7f837c06f510 2024-04-15T15:16:41.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.843+0000 7f836e7fc640 1 -- 172.21.15.72:0/2364080449 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83780443f0 con 0x7f837c06f510 2024-04-15T15:16:41.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.843+0000 7f836e7fc640 1 -- 172.21.15.72:0/2364080449 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83780447e0 con 0x7f837c06f510 2024-04-15T15:16:41.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.843+0000 7f8383da8640 1 -- 172.21.15.72:0/2364080449 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f837c15cd10 con 0x7f837c06f510 2024-04-15T15:16:41.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.843+0000 7f8383da8640 1 -- 172.21.15.72:0/2364080449 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f837c15d2d0 con 0x7f837c06f510 2024-04-15T15:16:41.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.844+0000 7f8383da8640 1 -- 172.21.15.72:0/2364080449 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f837c075c50 con 0x7f837c06f510 2024-04-15T15:16:41.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.845+0000 7f836e7fc640 1 -- 172.21.15.72:0/2364080449 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f837804b060 con 0x7f837c06f510 2024-04-15T15:16:41.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.845+0000 7f836e7fc640 1 --2- 172.21.15.72:0/2364080449 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f836006c980 0x7f836006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.845+0000 7f836e7fc640 1 -- 172.21.15.72:0/2364080449 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7f83780ae000 con 0x7f837c06f510 2024-04-15T15:16:41.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.845+0000 7f8381343640 1 --2- 172.21.15.72:0/2364080449 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f836006c980 0x7f836006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.846+0000 7f8381343640 1 --2- 172.21.15.72:0/2364080449 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f836006c980 0x7f836006ee40 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f8370006fd0 tx=0x7f8370008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:41.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:41.849+0000 7f836e7fc640 1 -- 172.21.15.72:0/2364080449 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f837803f070 con 0x7f837c06f510 2024-04-15T15:16:42.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.064+0000 7f8383da8640 1 -- 172.21.15.72:0/2364080449 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f837c0b58e0 con 0x7f837c06f510 2024-04-15T15:16:42.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.717+0000 7f836e7fc640 1 -- 172.21.15.72:0/2364080449 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v18) ==== 162+0+0 (secure 0 0 0) 0x7f837802f320 con 0x7f837c06f510 2024-04-15T15:16:42.719 INFO:teuthology.orchestra.run.smithi072.stderr:cephfs marked joinable; MDS may join as newly active. 2024-04-15T15:16:42.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.721+0000 7f8383da8640 1 -- 172.21.15.72:0/2364080449 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f836006c980 msgr2=0x7f836006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.721+0000 7f8383da8640 1 --2- 172.21.15.72:0/2364080449 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f836006c980 0x7f836006ee40 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f8370006fd0 tx=0x7f8370008040 comp rx=0 tx=0).stop 2024-04-15T15:16:42.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.721+0000 7f8383da8640 1 -- 172.21.15.72:0/2364080449 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f837c06f510 msgr2=0x7f837c0c53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.721+0000 7f8383da8640 1 --2- 172.21.15.72:0/2364080449 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f837c06f510 0x7f837c0c53f0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f8378000c00 tx=0x7f83780046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.721+0000 7f8383da8640 1 -- 172.21.15.72:0/2364080449 shutdown_connections 2024-04-15T15:16:42.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.721+0000 7f8383da8640 1 --2- 172.21.15.72:0/2364080449 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f836006c980 0x7f836006ee40 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.721+0000 7f8383da8640 1 --2- 172.21.15.72:0/2364080449 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f837c15c550 0x7f837c0cb920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.721+0000 7f8383da8640 1 --2- 172.21.15.72:0/2364080449 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f837c06f970 0x7f837c0cb3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.721+0000 7f8383da8640 1 --2- 172.21.15.72:0/2364080449 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f837c06f510 0x7f837c0c53f0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.721+0000 7f8383da8640 1 -- 172.21.15.72:0/2364080449 >> 172.21.15.72:0/2364080449 conn(0x7f837c0b3f50 msgr2=0x7f837c0c9530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:42.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.721+0000 7f8383da8640 1 -- 172.21.15.72:0/2364080449 shutdown_connections 2024-04-15T15:16:42.723 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.721+0000 7f8383da8640 1 -- 172.21.15.72:0/2364080449 wait complete. 2024-04-15T15:16:42.738 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:16:42.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.888+0000 7ff75c2c8640 1 Processor -- start 2024-04-15T15:16:42.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.888+0000 7ff75c2c8640 1 -- start start 2024-04-15T15:16:42.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.889+0000 7ff75c2c8640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff7540c9570 0x7ff754162b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.889+0000 7ff75c2c8640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7540ccd40 con 0x7ff7540d12f0 2024-04-15T15:16:42.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.889+0000 7ff75c2c8640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7540cd020 con 0x7ff754161fb0 2024-04-15T15:16:42.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.889+0000 7ff75c2c8640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7ff7540ccaf0 con 0x7ff7540c9570 2024-04-15T15:16:42.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.889+0000 7ff75a064640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7ff7540d12f0 0x7ff7541618a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49858/0 (socket says 172.21.15.72:49858) 2024-04-15T15:16:42.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.889+0000 7ff75a064640 1 -- 172.21.15.72:0/1290836581 learned_addr learned my addr 172.21.15.72:0/1290836581 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:42.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.890+0000 7ff75a865640 1 --2- 172.21.15.72:0/1290836581 >> v2:172.21.15.175:3300/0 conn(0x7ff7540c9570 0x7ff754162b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.890+0000 7ff759062640 1 -- 172.21.15.72:0/1290836581 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1522584677 0 0) 0x7ff7540ccd40 con 0x7ff7540d12f0 2024-04-15T15:16:42.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.890+0000 7ff759062640 1 -- 172.21.15.72:0/1290836581 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff740003520 con 0x7ff7540d12f0 2024-04-15T15:16:42.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.890+0000 7ff759062640 1 -- 172.21.15.72:0/1290836581 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3245757409 0 0) 0x7ff740003520 con 0x7ff7540d12f0 2024-04-15T15:16:42.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.890+0000 7ff759062640 1 -- 172.21.15.72:0/1290836581 >> v2:172.21.15.175:3300/0 conn(0x7ff7540c9570 msgr2=0x7ff754162b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:42.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.890+0000 7ff759062640 1 --2- 172.21.15.72:0/1290836581 >> v2:172.21.15.175:3300/0 conn(0x7ff7540c9570 0x7ff754162b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.890+0000 7ff759062640 1 -- 172.21.15.72:0/1290836581 >> v1:172.21.15.175:6789/0 conn(0x7ff754161fb0 legacy=0x7ff754162450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.890+0000 7ff759062640 1 -- 172.21.15.72:0/1290836581 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7540cd300 con 0x7ff7540d12f0 2024-04-15T15:16:42.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.890+0000 7ff759062640 1 -- 172.21.15.72:0/1290836581 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7ff750001670 con 0x7ff7540d12f0 2024-04-15T15:16:42.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.890+0000 7ff759062640 1 -- 172.21.15.72:0/1290836581 >> v1:172.21.15.72:6789/0 conn(0x7ff7540d12f0 legacy=0x7ff7541618a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.890+0000 7ff759062640 1 --2- 172.21.15.72:0/1290836581 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff7400039b0 0x7ff740003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.890+0000 7ff759062640 1 --2- 172.21.15.72:0/1290836581 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff740004800 0x7ff740004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.890+0000 7ff759062640 1 --2- 172.21.15.72:0/1290836581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7400051c0 0x7ff74000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.890+0000 7ff759062640 1 -- 172.21.15.72:0/1290836581 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ff7540ccaf0 con 0x7ff7400051c0 2024-04-15T15:16:42.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.890+0000 7ff759062640 1 -- 172.21.15.72:0/1290836581 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7ff74000c060 con 0x7ff740004800 2024-04-15T15:16:42.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.890+0000 7ff759062640 1 -- 172.21.15.72:0/1290836581 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7ff74000c260 con 0x7ff7400039b0 2024-04-15T15:16:42.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.890+0000 7ff75a865640 1 --2- 172.21.15.72:0/1290836581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7400051c0 0x7ff74000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.890+0000 7ff75a064640 1 --2- 172.21.15.72:0/1290836581 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff7400039b0 0x7ff740003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.890+0000 7ff759863640 1 --2- 172.21.15.72:0/1290836581 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff740004800 0x7ff740004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.890+0000 7ff75a865640 1 -- 172.21.15.72:0/1290836581 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff7400039b0 msgr2=0x7ff740003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:42.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.890+0000 7ff75a865640 1 --2- 172.21.15.72:0/1290836581 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff7400039b0 0x7ff740003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.890+0000 7ff75a865640 1 -- 172.21.15.72:0/1290836581 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff740004800 msgr2=0x7ff740004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:42.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.890+0000 7ff75a865640 1 --2- 172.21.15.72:0/1290836581 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff740004800 0x7ff740004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.890+0000 7ff75a865640 1 -- 172.21.15.72:0/1290836581 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff740003520 con 0x7ff7400051c0 2024-04-15T15:16:42.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.890+0000 7ff759863640 1 --2- 172.21.15.72:0/1290836581 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff740004800 0x7ff740004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:42.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.891+0000 7ff75a865640 1 -- 172.21.15.72:0/1290836581 reap_dead start 2024-04-15T15:16:42.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.891+0000 7ff75a865640 1 --2- 172.21.15.72:0/1290836581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7400051c0 0x7ff74000b9c0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7ff74c009f90 tx=0x7ff740005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ac1c74aa500c2d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:42.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.891+0000 7ff759062640 1 -- 172.21.15.72:0/1290836581 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff74c039070 con 0x7ff7400051c0 2024-04-15T15:16:42.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.891+0000 7ff759062640 1 -- 172.21.15.72:0/1290836581 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff74c02cc20 con 0x7ff7400051c0 2024-04-15T15:16:42.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.891+0000 7ff759062640 1 -- 172.21.15.72:0/1290836581 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff74c009c10 con 0x7ff7400051c0 2024-04-15T15:16:42.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.892+0000 7ff75c2c8640 1 -- 172.21.15.72:0/1290836581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7400051c0 msgr2=0x7ff74000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.892+0000 7ff75c2c8640 1 --2- 172.21.15.72:0/1290836581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7400051c0 0x7ff74000b9c0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7ff74c009f90 tx=0x7ff740005830 comp rx=0 tx=0).stop 2024-04-15T15:16:42.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.892+0000 7ff75c2c8640 1 -- 172.21.15.72:0/1290836581 shutdown_connections 2024-04-15T15:16:42.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.892+0000 7ff75c2c8640 1 --2- 172.21.15.72:0/1290836581 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff7400051c0 0x7ff74000b9c0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.892+0000 7ff75c2c8640 1 -- 172.21.15.72:0/1290836581 >> 172.21.15.72:0/1290836581 conn(0x7ff7540b3f90 msgr2=0x7ff7540b43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:42.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.892+0000 7ff75c2c8640 1 -- 172.21.15.72:0/1290836581 shutdown_connections 2024-04-15T15:16:42.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.892+0000 7ff75c2c8640 1 -- 172.21.15.72:0/1290836581 wait complete. 2024-04-15T15:16:42.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.893+0000 7ff75c2c8640 1 Processor -- start 2024-04-15T15:16:42.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.893+0000 7ff75c2c8640 1 -- start start 2024-04-15T15:16:42.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.894+0000 7ff75c2c8640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff7400051c0 0x7ff7540c7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.894+0000 7ff75c2c8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff75415ac80 0x7ff7541471b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.894+0000 7ff75c2c8640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff754149be0 0x7ff75414a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.894+0000 7ff75c2c8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ff7540d2210 con 0x7ff75415ac80 2024-04-15T15:16:42.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.894+0000 7ff75c2c8640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7ff7540d2090 con 0x7ff7400051c0 2024-04-15T15:16:42.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.894+0000 7ff75c2c8640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7ff7540b68e0 con 0x7ff754149be0 2024-04-15T15:16:42.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.894+0000 7ff759863640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff75415ac80 0x7ff7541471b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.894+0000 7ff759863640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff75415ac80 0x7ff7541471b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43518/0 (socket says 172.21.15.72:43518) 2024-04-15T15:16:42.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.894+0000 7ff759863640 1 -- 172.21.15.72:0/282580691 learned_addr learned my addr 172.21.15.72:0/282580691 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:42.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.894+0000 7ff75a064640 1 --2- 172.21.15.72:0/282580691 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff7400051c0 0x7ff7540c7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.899 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.894+0000 7ff75a865640 1 --2- 172.21.15.72:0/282580691 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff754149be0 0x7ff75414a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.894+0000 7ff759863640 1 -- 172.21.15.72:0/282580691 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff754149be0 msgr2=0x7ff75414a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.894+0000 7ff759863640 1 --2- 172.21.15.72:0/282580691 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff754149be0 0x7ff75414a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.894+0000 7ff759863640 1 -- 172.21.15.72:0/282580691 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff7400051c0 msgr2=0x7ff7540c7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.894+0000 7ff759863640 1 --2- 172.21.15.72:0/282580691 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff7400051c0 0x7ff7540c7e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.894+0000 7ff759863640 1 -- 172.21.15.72:0/282580691 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff75415b520 con 0x7ff75415ac80 2024-04-15T15:16:42.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.894+0000 7ff75a865640 1 --2- 172.21.15.72:0/282580691 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff754149be0 0x7ff75414a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:42.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.894+0000 7ff759863640 1 --2- 172.21.15.72:0/282580691 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff75415ac80 0x7ff7541471b0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7ff744007810 tx=0x7ff74400e910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:42.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.894+0000 7ff74affd640 1 -- 172.21.15.72:0/282580691 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff744019070 con 0x7ff75415ac80 2024-04-15T15:16:42.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.894+0000 7ff74affd640 1 -- 172.21.15.72:0/282580691 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7440052e0 con 0x7ff75415ac80 2024-04-15T15:16:42.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.894+0000 7ff74affd640 1 -- 172.21.15.72:0/282580691 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff744005490 con 0x7ff75415ac80 2024-04-15T15:16:42.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.894+0000 7ff75c2c8640 1 -- 172.21.15.72:0/282580691 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff75415b240 con 0x7ff75415ac80 2024-04-15T15:16:42.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.895+0000 7ff75c2c8640 1 -- 172.21.15.72:0/282580691 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7540c7640 con 0x7ff75415ac80 2024-04-15T15:16:42.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.895+0000 7ff75c2c8640 1 -- 172.21.15.72:0/282580691 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7540689e0 con 0x7ff75415ac80 2024-04-15T15:16:42.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.896+0000 7ff74affd640 1 -- 172.21.15.72:0/282580691 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7ff74401c070 con 0x7ff75415ac80 2024-04-15T15:16:42.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.896+0000 7ff74affd640 1 --2- 172.21.15.72:0/282580691 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff73806c9b0 0x7ff73806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.896+0000 7ff74affd640 1 -- 172.21.15.72:0/282580691 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7ff74408b290 con 0x7ff75415ac80 2024-04-15T15:16:42.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.896+0000 7ff75a064640 1 --2- 172.21.15.72:0/282580691 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff73806c9b0 0x7ff73806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.897+0000 7ff75a064640 1 --2- 172.21.15.72:0/282580691 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff73806c9b0 0x7ff73806ee70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff750007cb0 tx=0x7ff750009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:42.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:42.901+0000 7ff74affd640 1 -- 172.21.15.72:0/282580691 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff75415b520 con 0x7ff75415ac80 2024-04-15T15:16:43.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:43.114+0000 7ff75c2c8640 1 -- 172.21.15.72:0/282580691 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff7540b5c90 con 0x7ff75415ac80 2024-04-15T15:16:43.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:43.115+0000 7ff74affd640 1 -- 172.21.15.72:0/282580691 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+5178 (secure 0 0 0) 0x7ff744058e80 con 0x7ff75415ac80 2024-04-15T15:16:43.116 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:16:43.117 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":19,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4883,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3363272836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3363272836},{"type":"v1","addr":"172.21.15.175:6834","nonce":3363272836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5134,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6835/1059187505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":1059187505},{"type":"v1","addr":"172.21.15.175:6835","nonce":1059187505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5556,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/690132275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":690132275},{"type":"v1","addr":"172.21.15.72:6837","nonce":690132275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:16:07.050078+0000","modified":"2024-04-15T15:16:42.721279+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"a","rank":0,"incarnation":19,"state":"up:replay","state_seq":1,"addr":"172.21.15.72:6836/3407328263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3407328263},{"type":"v1","addr":"172.21.15.72:6836","nonce":3407328263}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-15T15:16:43.117 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 19 2024-04-15T15:16:43.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:43.117+0000 7ff75c2c8640 1 -- 172.21.15.72:0/282580691 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff73806c9b0 msgr2=0x7ff73806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:43.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:43.117+0000 7ff75c2c8640 1 --2- 172.21.15.72:0/282580691 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff73806c9b0 0x7ff73806ee70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff750007cb0 tx=0x7ff750009040 comp rx=0 tx=0).stop 2024-04-15T15:16:43.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:43.117+0000 7ff75c2c8640 1 -- 172.21.15.72:0/282580691 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff75415ac80 msgr2=0x7ff7541471b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:43.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:43.117+0000 7ff75c2c8640 1 --2- 172.21.15.72:0/282580691 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff75415ac80 0x7ff7541471b0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7ff744007810 tx=0x7ff74400e910 comp rx=0 tx=0).stop 2024-04-15T15:16:43.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:43.117+0000 7ff75c2c8640 1 -- 172.21.15.72:0/282580691 shutdown_connections 2024-04-15T15:16:43.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:43.117+0000 7ff75c2c8640 1 --2- 172.21.15.72:0/282580691 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff73806c9b0 0x7ff73806ee70 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:43.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:43.117+0000 7ff75c2c8640 1 --2- 172.21.15.72:0/282580691 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff754149be0 0x7ff75414a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:43.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:43.117+0000 7ff75c2c8640 1 --2- 172.21.15.72:0/282580691 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff75415ac80 0x7ff7541471b0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:43.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:43.117+0000 7ff75c2c8640 1 --2- 172.21.15.72:0/282580691 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff7400051c0 0x7ff7540c7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:43.119 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:43.117+0000 7ff75c2c8640 1 -- 172.21.15.72:0/282580691 >> 172.21.15.72:0/282580691 conn(0x7ff7540b3f90 msgr2=0x7ff7540b3880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:43.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:43.117+0000 7ff75c2c8640 1 -- 172.21.15.72:0/282580691 shutdown_connections 2024-04-15T15:16:43.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:43.117+0000 7ff75c2c8640 1 -- 172.21.15.72:0/282580691 wait complete. 2024-04-15T15:16:43.533 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 19, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-15T15:16:07.050078+0000', 'modified': '2024-04-15T15:16:42.721279+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 41, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6054}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6054': {'gid': 6054, 'name': 'a', 'rank': 0, 'incarnation': 19, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.72:6836/3407328263', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 3407328263}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 3407328263}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-15T15:16:43.533 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6054:up:replay 2024-04-15T15:16:44.534 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:16:44.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.647+0000 7f8355f04640 1 Processor -- start 2024-04-15T15:16:44.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.647+0000 7f8355f04640 1 -- start start 2024-04-15T15:16:44.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.647+0000 7f8355f04640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8350161e40 0x7f83500b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:44.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.647+0000 7f8355f04640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f83500d45b0 0x7f83500d4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:44.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.647+0000 7f8355f04640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8350162220 con 0x7f83500d12b0 2024-04-15T15:16:44.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.648+0000 7f8355f04640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f83500d4f60 con 0x7f8350161e40 2024-04-15T15:16:44.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.648+0000 7f8355f04640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f83500d21f0 con 0x7f83500d45b0 2024-04-15T15:16:44.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.648+0000 7f834f7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8350161e40 0x7f83500b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:44.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.648+0000 7f834effd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f83500d12b0 0x7f83500d2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49864/0 (socket says 172.21.15.72:49864) 2024-04-15T15:16:44.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.648+0000 7f834f7fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8350161e40 0x7f83500b37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43522/0 (socket says 172.21.15.72:43522) 2024-04-15T15:16:44.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.648+0000 7f834effd640 1 -- 172.21.15.72:0/2368236893 learned_addr learned my addr 172.21.15.72:0/2368236893 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:44.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.648+0000 7f834f7fe640 1 -- 172.21.15.72:0/2368236893 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f83500d45b0 msgr2=0x7f83500d4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:44.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.648+0000 7f834f7fe640 1 --2- 172.21.15.72:0/2368236893 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f83500d45b0 0x7f83500d4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.648+0000 7f834f7fe640 1 -- 172.21.15.72:0/2368236893 >> v1:172.21.15.72:6789/0 conn(0x7f83500d12b0 legacy=0x7f83500d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:44.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.648+0000 7f834f7fe640 1 -- 172.21.15.72:0/2368236893 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83500d3070 con 0x7f8350161e40 2024-04-15T15:16:44.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.648+0000 7f834e7fc640 1 -- 172.21.15.72:0/2368236893 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4047389495 0 0) 0x7f8350162220 con 0x7f83500d12b0 2024-04-15T15:16:44.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.649+0000 7f834f7fe640 1 --2- 172.21.15.72:0/2368236893 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8350161e40 0x7f83500b37f0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f83400097d0 tx=0x7f834002b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f7a34e8e2674ac3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:44.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.649+0000 7f834e7fc640 1 -- 172.21.15.72:0/2368236893 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8340031770 con 0x7f8350161e40 2024-04-15T15:16:44.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.649+0000 7f834e7fc640 1 -- 172.21.15.72:0/2368236893 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8340037070 con 0x7f8350161e40 2024-04-15T15:16:44.651 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.649+0000 7f834e7fc640 1 -- 172.21.15.72:0/2368236893 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8340031b30 con 0x7f8350161e40 2024-04-15T15:16:44.652 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.649+0000 7f8355f04640 1 -- 172.21.15.72:0/2368236893 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8350161e40 msgr2=0x7f83500b37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:44.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.649+0000 7f8355f04640 1 --2- 172.21.15.72:0/2368236893 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8350161e40 0x7f83500b37f0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f83400097d0 tx=0x7f834002b810 comp rx=0 tx=0).stop 2024-04-15T15:16:44.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.649+0000 7f8355f04640 1 -- 172.21.15.72:0/2368236893 shutdown_connections 2024-04-15T15:16:44.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.649+0000 7f8355f04640 1 --2- 172.21.15.72:0/2368236893 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f83500d45b0 0x7f83500d4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.649+0000 7f8355f04640 1 --2- 172.21.15.72:0/2368236893 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8350161e40 0x7f83500b37f0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.649+0000 7f8355f04640 1 -- 172.21.15.72:0/2368236893 >> 172.21.15.72:0/2368236893 conn(0x7f835009d970 msgr2=0x7f83500b95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:44.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.649+0000 7f8355f04640 1 -- 172.21.15.72:0/2368236893 shutdown_connections 2024-04-15T15:16:44.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.650+0000 7f8355f04640 1 -- 172.21.15.72:0/2368236893 wait complete. 2024-04-15T15:16:44.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.651+0000 7f8355f04640 1 Processor -- start 2024-04-15T15:16:44.653 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.651+0000 7f8355f04640 1 -- start start 2024-04-15T15:16:44.655 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.652+0000 7f8355f04640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f83500d12b0 0x7f835014eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:44.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.652+0000 7f8355f04640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83500d45b0 0x7f83500b5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:44.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.652+0000 7f8355f04640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8350161e40 0x7f83500cd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:44.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.652+0000 7f8355f04640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f83500d1de0 con 0x7f83500d45b0 2024-04-15T15:16:44.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.652+0000 7f8355f04640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f83500d1b00 con 0x7f8350161e40 2024-04-15T15:16:44.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.652+0000 7f8355f04640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f83500d2440 con 0x7f83500d12b0 2024-04-15T15:16:44.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.652+0000 7f834effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83500d45b0 0x7f83500b5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:44.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.652+0000 7f834effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83500d45b0 0x7f83500b5ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43526/0 (socket says 172.21.15.72:43526) 2024-04-15T15:16:44.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.652+0000 7f834effd640 1 -- 172.21.15.72:0/553558776 learned_addr learned my addr 172.21.15.72:0/553558776 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:44.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.652+0000 7f834f7fe640 1 --2- 172.21.15.72:0/553558776 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f83500d12b0 0x7f835014eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:44.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.652+0000 7f834ffff640 1 --2- 172.21.15.72:0/553558776 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8350161e40 0x7f83500cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:44.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.652+0000 7f834effd640 1 -- 172.21.15.72:0/553558776 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f83500d12b0 msgr2=0x7f835014eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:44.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.652+0000 7f834effd640 1 --2- 172.21.15.72:0/553558776 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f83500d12b0 0x7f835014eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.652+0000 7f834effd640 1 -- 172.21.15.72:0/553558776 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8350161e40 msgr2=0x7f83500cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:44.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.652+0000 7f834effd640 1 --2- 172.21.15.72:0/553558776 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8350161e40 0x7f83500cd700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.652+0000 7f834effd640 1 -- 172.21.15.72:0/553558776 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f835014f0a0 con 0x7f83500d45b0 2024-04-15T15:16:44.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.652+0000 7f834effd640 1 --2- 172.21.15.72:0/553558776 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83500d45b0 0x7f83500b5ed0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f834400c430 tx=0x7f834400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:44.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.652+0000 7f8337fff640 1 -- 172.21.15.72:0/553558776 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83440074e0 con 0x7f83500d45b0 2024-04-15T15:16:44.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.652+0000 7f8337fff640 1 -- 172.21.15.72:0/553558776 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83440047e0 con 0x7f83500d45b0 2024-04-15T15:16:44.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.652+0000 7f8337fff640 1 -- 172.21.15.72:0/553558776 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8344004bd0 con 0x7f83500d45b0 2024-04-15T15:16:44.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.652+0000 7f8355f04640 1 -- 172.21.15.72:0/553558776 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f835015e170 con 0x7f83500d45b0 2024-04-15T15:16:44.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.653+0000 7f8355f04640 1 -- 172.21.15.72:0/553558776 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f835015e730 con 0x7f83500d45b0 2024-04-15T15:16:44.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.654+0000 7f8355f04640 1 -- 172.21.15.72:0/553558776 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f835015f790 con 0x7f83500d45b0 2024-04-15T15:16:44.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.654+0000 7f8337fff640 1 -- 172.21.15.72:0/553558776 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f8344011070 con 0x7f83500d45b0 2024-04-15T15:16:44.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.654+0000 7f8337fff640 1 --2- 172.21.15.72:0/553558776 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f833c06c980 0x7f833c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:44.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.655+0000 7f8337fff640 1 -- 172.21.15.72:0/553558776 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7f834408ba00 con 0x7f83500d45b0 2024-04-15T15:16:44.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.655+0000 7f834f7fe640 1 --2- 172.21.15.72:0/553558776 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f833c06c980 0x7f833c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:44.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.655+0000 7f834f7fe640 1 --2- 172.21.15.72:0/553558776 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f833c06c980 0x7f833c06ee40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8340036000 tx=0x7f8340008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:44.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.659+0000 7f8337fff640 1 -- 172.21.15.72:0/553558776 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8344059c40 con 0x7f83500d45b0 2024-04-15T15:16:44.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.874+0000 7f8355f04640 1 -- 172.21.15.72:0/553558776 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8350161660 con 0x7f83500d45b0 2024-04-15T15:16:44.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.874+0000 7f8337fff640 1 -- 172.21.15.72:0/553558776 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+5178 (secure 0 0 0) 0x7f8344059540 con 0x7f83500d45b0 2024-04-15T15:16:44.876 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:16:44.876 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":21,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4883,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3363272836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3363272836},{"type":"v1","addr":"172.21.15.175:6834","nonce":3363272836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5134,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6835/1059187505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":1059187505},{"type":"v1","addr":"172.21.15.175:6835","nonce":1059187505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5556,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/690132275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":690132275},{"type":"v1","addr":"172.21.15.72:6837","nonce":690132275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:16:07.050078+0000","modified":"2024-04-15T15:16:43.727403+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"a","rank":0,"incarnation":19,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.72:6836/3407328263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3407328263},{"type":"v1","addr":"172.21.15.72:6836","nonce":3407328263}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-15T15:16:44.876 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 21 2024-04-15T15:16:44.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.876+0000 7f8355f04640 1 -- 172.21.15.72:0/553558776 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f833c06c980 msgr2=0x7f833c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:44.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.876+0000 7f8355f04640 1 --2- 172.21.15.72:0/553558776 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f833c06c980 0x7f833c06ee40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8340036000 tx=0x7f8340008030 comp rx=0 tx=0).stop 2024-04-15T15:16:44.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.876+0000 7f8355f04640 1 -- 172.21.15.72:0/553558776 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83500d45b0 msgr2=0x7f83500b5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:44.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.876+0000 7f8355f04640 1 --2- 172.21.15.72:0/553558776 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83500d45b0 0x7f83500b5ed0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f834400c430 tx=0x7f834400c8f0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.876+0000 7f8355f04640 1 -- 172.21.15.72:0/553558776 shutdown_connections 2024-04-15T15:16:44.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.876+0000 7f8355f04640 1 --2- 172.21.15.72:0/553558776 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f833c06c980 0x7f833c06ee40 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.876+0000 7f8355f04640 1 --2- 172.21.15.72:0/553558776 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8350161e40 0x7f83500cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.876+0000 7f8355f04640 1 --2- 172.21.15.72:0/553558776 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83500d45b0 0x7f83500b5ed0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.876+0000 7f8355f04640 1 --2- 172.21.15.72:0/553558776 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f83500d12b0 0x7f835014eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:44.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.876+0000 7f8355f04640 1 -- 172.21.15.72:0/553558776 >> 172.21.15.72:0/553558776 conn(0x7f835009d970 msgr2=0x7f83500b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:44.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.877+0000 7f8355f04640 1 -- 172.21.15.72:0/553558776 shutdown_connections 2024-04-15T15:16:44.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:44.877+0000 7f8355f04640 1 -- 172.21.15.72:0/553558776 wait complete. 2024-04-15T15:16:45.292 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-15T15:16:07.050078+0000', 'modified': '2024-04-15T15:16:43.727403+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 41, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6054}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6054': {'gid': 6054, 'name': 'a', 'rank': 0, 'incarnation': 19, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.72:6836/3407328263', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 3407328263}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 3407328263}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-15T15:16:45.293 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6054:up:rejoin 2024-04-15T15:16:46.294 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:16:46.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.409+0000 7fc8934c5640 1 Processor -- start 2024-04-15T15:16:46.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.409+0000 7fc8934c5640 1 -- start start 2024-04-15T15:16:46.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.410+0000 7fc8934c5640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc88c15c480 0x7fc88c06ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:46.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.410+0000 7fc8934c5640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc88c0d1d80 con 0x7fc88c0ce670 2024-04-15T15:16:46.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.410+0000 7fc8934c5640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc88c0d2060 con 0x7fc88c0cf4d0 2024-04-15T15:16:46.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.410+0000 7fc8934c5640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fc88c0d1b30 con 0x7fc88c15c480 2024-04-15T15:16:46.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.410+0000 7fc890a60640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fc88c0ce670 0x7fc88c0cedc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49868/0 (socket says 172.21.15.72:49868) 2024-04-15T15:16:46.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.410+0000 7fc890a60640 1 -- 172.21.15.72:0/2628035423 learned_addr learned my addr 172.21.15.72:0/2628035423 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:46.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.410+0000 7fc883fff640 1 -- 172.21.15.72:0/2628035423 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 965666710 0 0) 0x7fc88c0d1d80 con 0x7fc88c0ce670 2024-04-15T15:16:46.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.410+0000 7fc891261640 1 --2- 172.21.15.72:0/2628035423 >> v2:172.21.15.175:3300/0 conn(0x7fc88c15c480 0x7fc88c06ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:46.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.410+0000 7fc883fff640 1 -- 172.21.15.72:0/2628035423 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc878003520 con 0x7fc88c0ce670 2024-04-15T15:16:46.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.410+0000 7fc883fff640 1 -- 172.21.15.72:0/2628035423 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2066513553 0 0) 0x7fc878003520 con 0x7fc88c0ce670 2024-04-15T15:16:46.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.410+0000 7fc883fff640 1 -- 172.21.15.72:0/2628035423 >> v2:172.21.15.175:3300/0 conn(0x7fc88c15c480 msgr2=0x7fc88c06ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:46.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.410+0000 7fc883fff640 1 --2- 172.21.15.72:0/2628035423 >> v2:172.21.15.175:3300/0 conn(0x7fc88c15c480 0x7fc88c06ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.410+0000 7fc883fff640 1 -- 172.21.15.72:0/2628035423 >> v1:172.21.15.175:6789/0 conn(0x7fc88c0cf4d0 legacy=0x7fc88c0b52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:46.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.410+0000 7fc883fff640 1 -- 172.21.15.72:0/2628035423 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc88c0ccc50 con 0x7fc88c0ce670 2024-04-15T15:16:46.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.410+0000 7fc883fff640 1 -- 172.21.15.72:0/2628035423 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7fc87c001670 con 0x7fc88c0ce670 2024-04-15T15:16:46.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.411+0000 7fc883fff640 1 -- 172.21.15.72:0/2628035423 >> v1:172.21.15.72:6789/0 conn(0x7fc88c0ce670 legacy=0x7fc88c0cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:46.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.411+0000 7fc883fff640 1 --2- 172.21.15.72:0/2628035423 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc8780039b0 0x7fc878003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:46.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.411+0000 7fc883fff640 1 --2- 172.21.15.72:0/2628035423 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc878004800 0x7fc878004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:46.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.411+0000 7fc883fff640 1 --2- 172.21.15.72:0/2628035423 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8780051c0 0x7fc87800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:46.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.411+0000 7fc883fff640 1 -- 172.21.15.72:0/2628035423 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c0d1b30 con 0x7fc878004800 2024-04-15T15:16:46.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.411+0000 7fc883fff640 1 -- 172.21.15.72:0/2628035423 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fc87800c060 con 0x7fc8780051c0 2024-04-15T15:16:46.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.411+0000 7fc883fff640 1 -- 172.21.15.72:0/2628035423 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fc87800c260 con 0x7fc8780039b0 2024-04-15T15:16:46.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.411+0000 7fc890a60640 1 --2- 172.21.15.72:0/2628035423 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc878004800 0x7fc878004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:46.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.411+0000 7fc891261640 1 --2- 172.21.15.72:0/2628035423 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc8780039b0 0x7fc878003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:46.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.411+0000 7fc891a62640 1 --2- 172.21.15.72:0/2628035423 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8780051c0 0x7fc87800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:46.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.411+0000 7fc890a60640 1 -- 172.21.15.72:0/2628035423 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc8780039b0 msgr2=0x7fc878003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:46.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.411+0000 7fc890a60640 1 --2- 172.21.15.72:0/2628035423 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc8780039b0 0x7fc878003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.411+0000 7fc890a60640 1 -- 172.21.15.72:0/2628035423 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8780051c0 msgr2=0x7fc87800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:46.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.411+0000 7fc890a60640 1 --2- 172.21.15.72:0/2628035423 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc8780051c0 0x7fc87800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.411+0000 7fc891a62640 1 -- 172.21.15.72:0/2628035423 reap_dead start 2024-04-15T15:16:46.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.411+0000 7fc890a60640 1 -- 172.21.15.72:0/2628035423 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc878003520 con 0x7fc878004800 2024-04-15T15:16:46.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.412+0000 7fc890a60640 1 --2- 172.21.15.72:0/2628035423 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc878004800 0x7fc878004c80 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fc87c00c240 tx=0x7fc878005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e27b3e4e88985f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:46.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.412+0000 7fc883fff640 1 -- 172.21.15.72:0/2628035423 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc87c007d10 con 0x7fc878004800 2024-04-15T15:16:46.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.412+0000 7fc883fff640 1 -- 172.21.15.72:0/2628035423 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc87c030070 con 0x7fc878004800 2024-04-15T15:16:46.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.412+0000 7fc883fff640 1 -- 172.21.15.72:0/2628035423 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc87c02cdc0 con 0x7fc878004800 2024-04-15T15:16:46.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.412+0000 7fc8934c5640 1 -- 172.21.15.72:0/2628035423 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc878004800 msgr2=0x7fc878004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:46.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.412+0000 7fc8934c5640 1 --2- 172.21.15.72:0/2628035423 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc878004800 0x7fc878004c80 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fc87c00c240 tx=0x7fc878005830 comp rx=0 tx=0).stop 2024-04-15T15:16:46.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.412+0000 7fc8934c5640 1 -- 172.21.15.72:0/2628035423 shutdown_connections 2024-04-15T15:16:46.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.412+0000 7fc8934c5640 1 --2- 172.21.15.72:0/2628035423 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc878004800 0x7fc878004c80 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.412+0000 7fc8934c5640 1 -- 172.21.15.72:0/2628035423 >> 172.21.15.72:0/2628035423 conn(0x7fc88c0b3fa0 msgr2=0x7fc88c0b43b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:46.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.413+0000 7fc8934c5640 1 -- 172.21.15.72:0/2628035423 shutdown_connections 2024-04-15T15:16:46.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.413+0000 7fc8934c5640 1 -- 172.21.15.72:0/2628035423 wait complete. 2024-04-15T15:16:46.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.414+0000 7fc8934c5640 1 Processor -- start 2024-04-15T15:16:46.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.414+0000 7fc8934c5640 1 -- start start 2024-04-15T15:16:46.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.414+0000 7fc8934c5640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc878004800 0x7fc88c158e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:46.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.414+0000 7fc8934c5640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc88c0c8190 0x7fc88c146bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:46.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.415+0000 7fc8934c5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc88c1e17e0 0x7fc88c1e1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:46.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.415+0000 7fc8934c5640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c0cd170 con 0x7fc88c1e17e0 2024-04-15T15:16:46.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.415+0000 7fc8934c5640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c0cca00 con 0x7fc878004800 2024-04-15T15:16:46.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.415+0000 7fc8934c5640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fc88c15c860 con 0x7fc88c0c8190 2024-04-15T15:16:46.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.415+0000 7fc891a62640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc88c1e17e0 0x7fc88c1e1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:46.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.415+0000 7fc891a62640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc88c1e17e0 0x7fc88c1e1c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43544/0 (socket says 172.21.15.72:43544) 2024-04-15T15:16:46.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.415+0000 7fc891a62640 1 -- 172.21.15.72:0/958186334 learned_addr learned my addr 172.21.15.72:0/958186334 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:46.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.415+0000 7fc891261640 1 --2- 172.21.15.72:0/958186334 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc878004800 0x7fc88c158e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:46.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.415+0000 7fc890a60640 1 --2- 172.21.15.72:0/958186334 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc88c0c8190 0x7fc88c146bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:46.421 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.415+0000 7fc891a62640 1 -- 172.21.15.72:0/958186334 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc88c0c8190 msgr2=0x7fc88c146bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:46.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.415+0000 7fc891a62640 1 --2- 172.21.15.72:0/958186334 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc88c0c8190 0x7fc88c146bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.415+0000 7fc891a62640 1 -- 172.21.15.72:0/958186334 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc878004800 msgr2=0x7fc88c158e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:46.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.415+0000 7fc891a62640 1 --2- 172.21.15.72:0/958186334 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc878004800 0x7fc88c158e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.415+0000 7fc891a62640 1 -- 172.21.15.72:0/958186334 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc88c147410 con 0x7fc88c1e17e0 2024-04-15T15:16:46.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.415+0000 7fc891261640 1 --2- 172.21.15.72:0/958186334 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc878004800 0x7fc88c158e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:46.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.415+0000 7fc891a62640 1 --2- 172.21.15.72:0/958186334 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc88c1e17e0 0x7fc88c1e1c60 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fc88c074a90 tx=0x7fc88400fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:46.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.415+0000 7fc881ffb640 1 -- 172.21.15.72:0/958186334 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc884003b80 con 0x7fc88c1e17e0 2024-04-15T15:16:46.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.415+0000 7fc881ffb640 1 -- 172.21.15.72:0/958186334 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc884018070 con 0x7fc88c1e17e0 2024-04-15T15:16:46.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.415+0000 7fc8934c5640 1 -- 172.21.15.72:0/958186334 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc88c1476f0 con 0x7fc88c1e17e0 2024-04-15T15:16:46.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.415+0000 7fc881ffb640 1 -- 172.21.15.72:0/958186334 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8840050d0 con 0x7fc88c1e17e0 2024-04-15T15:16:46.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.415+0000 7fc8934c5640 1 -- 172.21.15.72:0/958186334 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc88c147cb0 con 0x7fc88c1e17e0 2024-04-15T15:16:46.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.416+0000 7fc8934c5640 1 -- 172.21.15.72:0/958186334 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc88c0690f0 con 0x7fc88c1e17e0 2024-04-15T15:16:46.422 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.417+0000 7fc881ffb640 1 -- 172.21.15.72:0/958186334 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fc884005270 con 0x7fc88c1e17e0 2024-04-15T15:16:46.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.417+0000 7fc881ffb640 1 --2- 172.21.15.72:0/958186334 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc87006c9b0 0x7fc87006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:46.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.418+0000 7fc891261640 1 --2- 172.21.15.72:0/958186334 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc87006c9b0 0x7fc87006ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:46.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.418+0000 7fc881ffb640 1 -- 172.21.15.72:0/958186334 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7fc88408b8b0 con 0x7fc88c1e17e0 2024-04-15T15:16:46.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.418+0000 7fc891261640 1 --2- 172.21.15.72:0/958186334 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc87006c9b0 0x7fc87006ee70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc888009c40 tx=0x7fc888008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:46.423 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.422+0000 7fc881ffb640 1 -- 172.21.15.72:0/958186334 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc8840112b0 con 0x7fc88c1e17e0 2024-04-15T15:16:46.636 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.635+0000 7fc8934c5640 1 -- 172.21.15.72:0/958186334 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc88c0c8570 con 0x7fc88c1e17e0 2024-04-15T15:16:46.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.635+0000 7fc881ffb640 1 -- 172.21.15.72:0/958186334 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+5185 (secure 0 0 0) 0x7fc884059a70 con 0x7fc88c1e17e0 2024-04-15T15:16:46.637 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:16:46.637 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":22,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4883,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3363272836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3363272836},{"type":"v1","addr":"172.21.15.175:6834","nonce":3363272836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5134,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6835/1059187505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":1059187505},{"type":"v1","addr":"172.21.15.175:6835","nonce":1059187505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5556,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/690132275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":690132275},{"type":"v1","addr":"172.21.15.72:6837","nonce":690132275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:16:07.050078+0000","modified":"2024-04-15T15:16:45.725971+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.72:6836/3407328263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3407328263},{"type":"v1","addr":"172.21.15.72:6836","nonce":3407328263}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-04-15T15:16:46.637 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 22 2024-04-15T15:16:46.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.637+0000 7fc8934c5640 1 -- 172.21.15.72:0/958186334 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc87006c9b0 msgr2=0x7fc87006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:46.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.637+0000 7fc8934c5640 1 --2- 172.21.15.72:0/958186334 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc87006c9b0 0x7fc87006ee70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fc888009c40 tx=0x7fc888008040 comp rx=0 tx=0).stop 2024-04-15T15:16:46.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.637+0000 7fc8934c5640 1 -- 172.21.15.72:0/958186334 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc88c1e17e0 msgr2=0x7fc88c1e1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:46.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.637+0000 7fc8934c5640 1 --2- 172.21.15.72:0/958186334 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc88c1e17e0 0x7fc88c1e1c60 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fc88c074a90 tx=0x7fc88400fc00 comp rx=0 tx=0).stop 2024-04-15T15:16:46.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.637+0000 7fc8934c5640 1 -- 172.21.15.72:0/958186334 shutdown_connections 2024-04-15T15:16:46.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.637+0000 7fc8934c5640 1 --2- 172.21.15.72:0/958186334 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc87006c9b0 0x7fc87006ee70 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.637+0000 7fc8934c5640 1 --2- 172.21.15.72:0/958186334 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc88c1e17e0 0x7fc88c1e1c60 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.637+0000 7fc8934c5640 1 --2- 172.21.15.72:0/958186334 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc88c0c8190 0x7fc88c146bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.637+0000 7fc8934c5640 1 --2- 172.21.15.72:0/958186334 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc878004800 0x7fc88c158e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:46.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.637+0000 7fc8934c5640 1 -- 172.21.15.72:0/958186334 >> 172.21.15.72:0/958186334 conn(0x7fc88c0b3fa0 msgr2=0x7fc88c159850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:46.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.637+0000 7fc8934c5640 1 -- 172.21.15.72:0/958186334 shutdown_connections 2024-04-15T15:16:46.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:46.637+0000 7fc8934c5640 1 -- 172.21.15.72:0/958186334 wait complete. 2024-04-15T15:16:47.053 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-15T15:16:07.050078+0000', 'modified': '2024-04-15T15:16:45.725971+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 41, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6054}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6054': {'gid': 6054, 'name': 'a', 'rank': 0, 'incarnation': 19, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.72:6836/3407328263', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.72:6834', 'nonce': 3407328263}, {'type': 'v1', 'addr': '172.21.15.72:6836', 'nonce': 3407328263}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 6054, 'qdb_cluster': [6054]} 2024-04-15T15:16:47.053 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-15T15:16:47.053 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-15T15:16:47.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.158+0000 7f59e6740640 1 Processor -- start 2024-04-15T15:16:47.159 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.158+0000 7f59e6740640 1 -- start start 2024-04-15T15:16:47.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.158+0000 7f59e6740640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f59e015d830 0x7f59e00b57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.159+0000 7f59e6740640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59e00b61e0 con 0x7f59e00b5d30 2024-04-15T15:16:47.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.159+0000 7f59e6740640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59e00b64c0 con 0x7f59e00cf870 2024-04-15T15:16:47.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.159+0000 7f59e6740640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f59e00b67a0 con 0x7f59e015d830 2024-04-15T15:16:47.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.159+0000 7f59df7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f59e00b5d30 0x7f59e00cf160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49872/0 (socket says 172.21.15.72:49872) 2024-04-15T15:16:47.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.159+0000 7f59df7fe640 1 -- 172.21.15.72:0/1533760583 learned_addr learned my addr 172.21.15.72:0/1533760583 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:47.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.159+0000 7f59deffd640 1 -- 172.21.15.72:0/1533760583 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1084136552 0 0) 0x7f59e00b61e0 con 0x7f59e00b5d30 2024-04-15T15:16:47.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.159+0000 7f59deffd640 1 -- 172.21.15.72:0/1533760583 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59c8003520 con 0x7f59e00b5d30 2024-04-15T15:16:47.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.159+0000 7f59dffff640 1 --2- 172.21.15.72:0/1533760583 >> v2:172.21.15.175:3300/0 conn(0x7f59e015d830 0x7f59e00b57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.159+0000 7f59deffd640 1 -- 172.21.15.72:0/1533760583 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 598176263 0 0) 0x7f59c8003520 con 0x7f59e00b5d30 2024-04-15T15:16:47.164 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.159+0000 7f59deffd640 1 -- 172.21.15.72:0/1533760583 >> v2:172.21.15.175:3300/0 conn(0x7f59e015d830 msgr2=0x7f59e00b57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:47.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.159+0000 7f59deffd640 1 --2- 172.21.15.72:0/1533760583 >> v2:172.21.15.175:3300/0 conn(0x7f59e015d830 0x7f59e00b57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.159+0000 7f59deffd640 1 -- 172.21.15.72:0/1533760583 >> v1:172.21.15.175:6789/0 conn(0x7f59e00cf870 legacy=0x7f59e015e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:47.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.159+0000 7f59deffd640 1 -- 172.21.15.72:0/1533760583 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59e00b6a80 con 0x7f59e00b5d30 2024-04-15T15:16:47.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.159+0000 7f59deffd640 1 -- 172.21.15.72:0/1533760583 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f59d4001670 con 0x7f59e00b5d30 2024-04-15T15:16:47.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.159+0000 7f59dffff640 1 --2- 172.21.15.72:0/1533760583 >> v2:172.21.15.175:3300/0 conn(0x7f59e015d830 0x7f59e00b57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:47.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.159+0000 7f59deffd640 1 -- 172.21.15.72:0/1533760583 >> v1:172.21.15.72:6789/0 conn(0x7f59e00b5d30 legacy=0x7f59e00cf160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.159+0000 7f59deffd640 1 --2- 172.21.15.72:0/1533760583 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f59c80039b0 0x7f59c8003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.159+0000 7f59deffd640 1 --2- 172.21.15.72:0/1533760583 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59c8004800 0x7f59c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.159+0000 7f59deffd640 1 --2- 172.21.15.72:0/1533760583 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59c80052e0 0x7f59c800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.159+0000 7f59deffd640 1 -- 172.21.15.72:0/1533760583 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f59e00b67a0 con 0x7f59c80052e0 2024-04-15T15:16:47.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.159+0000 7f59deffd640 1 -- 172.21.15.72:0/1533760583 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f59c800c190 con 0x7f59c8004800 2024-04-15T15:16:47.165 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.159+0000 7f59deffd640 1 -- 172.21.15.72:0/1533760583 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f59c800c390 con 0x7f59c80039b0 2024-04-15T15:16:47.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.159+0000 7f59e4cdd640 1 --2- 172.21.15.72:0/1533760583 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59c80052e0 0x7f59c800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.159+0000 7f59df7fe640 1 --2- 172.21.15.72:0/1533760583 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59c8004800 0x7f59c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.159+0000 7f59dffff640 1 --2- 172.21.15.72:0/1533760583 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f59c80039b0 0x7f59c8003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.160+0000 7f59e4cdd640 1 -- 172.21.15.72:0/1533760583 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f59c80039b0 msgr2=0x7f59c8003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:47.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.160+0000 7f59e4cdd640 1 --2- 172.21.15.72:0/1533760583 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f59c80039b0 0x7f59c8003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.160+0000 7f59e4cdd640 1 -- 172.21.15.72:0/1533760583 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59c8004800 msgr2=0x7f59c8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:47.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.160+0000 7f59e4cdd640 1 --2- 172.21.15.72:0/1533760583 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59c8004800 0x7f59c8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.160+0000 7f59e4cdd640 1 -- 172.21.15.72:0/1533760583 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59c8003520 con 0x7f59c80052e0 2024-04-15T15:16:47.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.160+0000 7f59e4cdd640 1 -- 172.21.15.72:0/1533760583 reap_dead start 2024-04-15T15:16:47.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.160+0000 7f59e4cdd640 1 --2- 172.21.15.72:0/1533760583 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59c80052e0 0x7f59c800b9c0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f59d8004480 tx=0x7f59c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=718839ba06ec9a77 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:47.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.161+0000 7f59deffd640 1 -- 172.21.15.72:0/1533760583 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d8034070 con 0x7f59c80052e0 2024-04-15T15:16:47.166 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.161+0000 7f59deffd640 1 -- 172.21.15.72:0/1533760583 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59d8006b50 con 0x7f59c80052e0 2024-04-15T15:16:47.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.161+0000 7f59deffd640 1 -- 172.21.15.72:0/1533760583 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d8004080 con 0x7f59c80052e0 2024-04-15T15:16:47.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.161+0000 7f59e6740640 1 -- 172.21.15.72:0/1533760583 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59c80052e0 msgr2=0x7f59c800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.161+0000 7f59e6740640 1 --2- 172.21.15.72:0/1533760583 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59c80052e0 0x7f59c800b9c0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f59d8004480 tx=0x7f59c8005830 comp rx=0 tx=0).stop 2024-04-15T15:16:47.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.161+0000 7f59e6740640 1 -- 172.21.15.72:0/1533760583 shutdown_connections 2024-04-15T15:16:47.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.161+0000 7f59e6740640 1 --2- 172.21.15.72:0/1533760583 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59c80052e0 0x7f59c800b9c0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.161+0000 7f59e6740640 1 -- 172.21.15.72:0/1533760583 >> 172.21.15.72:0/1533760583 conn(0x7f59e00d24d0 msgr2=0x7f59e00d28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:47.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.161+0000 7f59e6740640 1 -- 172.21.15.72:0/1533760583 shutdown_connections 2024-04-15T15:16:47.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.161+0000 7f59e6740640 1 -- 172.21.15.72:0/1533760583 wait complete. 2024-04-15T15:16:47.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.162+0000 7f59e6740640 1 Processor -- start 2024-04-15T15:16:47.167 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.162+0000 7f59e6740640 1 -- start start 2024-04-15T15:16:47.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.163+0000 7f59e6740640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59c80052e0 0x7f59e0157f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.163+0000 7f59e6740640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59e0148270 0x7f59e0159b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.163+0000 7f59e6740640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f59e0146650 0x7f59e0146ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.163+0000 7f59e6740640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f59e015ced0 con 0x7f59c80052e0 2024-04-15T15:16:47.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.163+0000 7f59e6740640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f59e015cd50 con 0x7f59e0148270 2024-04-15T15:16:47.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.163+0000 7f59e6740640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f59e00b57f0 con 0x7f59e0146650 2024-04-15T15:16:47.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.163+0000 7f59dffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59c80052e0 0x7f59e0157f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.163+0000 7f59dffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59c80052e0 0x7f59e0157f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43566/0 (socket says 172.21.15.72:43566) 2024-04-15T15:16:47.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.163+0000 7f59dffff640 1 -- 172.21.15.72:0/4148302283 learned_addr learned my addr 172.21.15.72:0/4148302283 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:47.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.163+0000 7f59df7fe640 1 --2- 172.21.15.72:0/4148302283 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59e0148270 0x7f59e0159b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.163+0000 7f59e4cdd640 1 --2- 172.21.15.72:0/4148302283 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f59e0146650 0x7f59e0146ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.164+0000 7f59dffff640 1 -- 172.21.15.72:0/4148302283 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f59e0146650 msgr2=0x7f59e0146ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.164+0000 7f59dffff640 1 --2- 172.21.15.72:0/4148302283 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f59e0146650 0x7f59e0146ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.164+0000 7f59dffff640 1 -- 172.21.15.72:0/4148302283 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59e0148270 msgr2=0x7f59e0159b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.164+0000 7f59dffff640 1 --2- 172.21.15.72:0/4148302283 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59e0148270 0x7f59e0159b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.164+0000 7f59dffff640 1 -- 172.21.15.72:0/4148302283 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59e01587e0 con 0x7f59c80052e0 2024-04-15T15:16:47.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.164+0000 7f59df7fe640 1 --2- 172.21.15.72:0/4148302283 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59e0148270 0x7f59e0159b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:47.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.164+0000 7f59dffff640 1 --2- 172.21.15.72:0/4148302283 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59c80052e0 0x7f59e0157f00 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f59d00115c0 tx=0x7f59d0011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:47.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.164+0000 7f59dcff9640 1 -- 172.21.15.72:0/4148302283 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d0019070 con 0x7f59c80052e0 2024-04-15T15:16:47.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.164+0000 7f59dcff9640 1 -- 172.21.15.72:0/4148302283 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59d0004220 con 0x7f59c80052e0 2024-04-15T15:16:47.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.164+0000 7f59e6740640 1 -- 172.21.15.72:0/4148302283 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59e0158ac0 con 0x7f59c80052e0 2024-04-15T15:16:47.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.164+0000 7f59dcff9640 1 -- 172.21.15.72:0/4148302283 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d000f040 con 0x7f59c80052e0 2024-04-15T15:16:47.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.164+0000 7f59e6740640 1 -- 172.21.15.72:0/4148302283 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59e0158440 con 0x7f59c80052e0 2024-04-15T15:16:47.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.165+0000 7f59dcff9640 1 -- 172.21.15.72:0/4148302283 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f59d000f6f0 con 0x7f59c80052e0 2024-04-15T15:16:47.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.166+0000 7f59dcff9640 1 --2- 172.21.15.72:0/4148302283 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f59cc06c9b0 0x7f59cc06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.166+0000 7f59dcff9640 1 -- 172.21.15.72:0/4148302283 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7f59d008c0b0 con 0x7f59c80052e0 2024-04-15T15:16:47.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.166+0000 7f59df7fe640 1 --2- 172.21.15.72:0/4148302283 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f59cc06c9b0 0x7f59cc06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.166+0000 7f59df7fe640 1 --2- 172.21.15.72:0/4148302283 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f59cc06c9b0 0x7f59cc06ee70 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f59e0155ee0 tx=0x7f59d4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:47.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.167+0000 7f59e9409740 1 Processor -- start 2024-04-15T15:16:47.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.167+0000 7f59e9409740 1 -- start start 2024-04-15T15:16:47.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.168+0000 7f59e9409740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dbc5097e10 0x55dbc506adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.168+0000 7f59e9409740 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55dbc50981f0 0x55dbc509aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.168+0000 7f59e9409740 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55dbc506b2f0 0x55dbc509b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.168+0000 7f59e9409740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55dbc4fd59c0 con 0x55dbc5097e10 2024-04-15T15:16:47.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.168+0000 7f59e9409740 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x55dbc4f7c650 con 0x55dbc50981f0 2024-04-15T15:16:47.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.168+0000 7f59e9409740 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x55dbc4fd1980 con 0x55dbc506b2f0 2024-04-15T15:16:47.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.168+0000 7f59dffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dbc5097e10 0x55dbc506adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.168+0000 7f59dffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dbc5097e10 0x55dbc506adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43580/0 (socket says 172.21.15.72:43580) 2024-04-15T15:16:47.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.168+0000 7f59dffff640 1 -- 172.21.15.72:0/3411841982 learned_addr learned my addr 172.21.15.72:0/3411841982 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:47.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.168+0000 7f59dffff640 1 -- 172.21.15.72:0/3411841982 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55dbc506b2f0 msgr2=0x55dbc509b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:47.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.168+0000 7f59dffff640 1 --2- 172.21.15.72:0/3411841982 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55dbc506b2f0 0x55dbc509b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.168+0000 7f59dffff640 1 -- 172.21.15.72:0/3411841982 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55dbc50981f0 msgr2=0x55dbc509aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:47.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.168+0000 7f59dffff640 1 --2- 172.21.15.72:0/3411841982 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55dbc50981f0 0x55dbc509aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.168+0000 7f59dffff640 1 -- 172.21.15.72:0/3411841982 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbc4ce82d0 con 0x55dbc5097e10 2024-04-15T15:16:47.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.168+0000 7f59dffff640 1 --2- 172.21.15.72:0/3411841982 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dbc5097e10 0x55dbc506adb0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f59d008a290 tx=0x7f59d0059570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d5fb18a9584e9f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:47.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.168+0000 7f59c5ffb640 1 -- 172.21.15.72:0/3411841982 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d0022040 con 0x55dbc5097e10 2024-04-15T15:16:47.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.168+0000 7f59c5ffb640 1 -- 172.21.15.72:0/3411841982 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59d00626c0 con 0x55dbc5097e10 2024-04-15T15:16:47.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.168+0000 7f59c5ffb640 1 -- 172.21.15.72:0/3411841982 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d0058f30 con 0x55dbc5097e10 2024-04-15T15:16:47.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.169+0000 7f59e9409740 1 -- 172.21.15.72:0/3411841982 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dbc5097e10 msgr2=0x55dbc506adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.169+0000 7f59e9409740 1 --2- 172.21.15.72:0/3411841982 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dbc5097e10 0x55dbc506adb0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f59d008a290 tx=0x7f59d0059570 comp rx=0 tx=0).stop 2024-04-15T15:16:47.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.169+0000 7f59e9409740 1 -- 172.21.15.72:0/3411841982 shutdown_connections 2024-04-15T15:16:47.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.169+0000 7f59e9409740 1 --2- 172.21.15.72:0/3411841982 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55dbc506b2f0 0x55dbc509b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.169+0000 7f59e9409740 1 --2- 172.21.15.72:0/3411841982 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55dbc50981f0 0x55dbc509aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.169+0000 7f59e9409740 1 --2- 172.21.15.72:0/3411841982 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dbc5097e10 0x55dbc506adb0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.169+0000 7f59e9409740 1 -- 172.21.15.72:0/3411841982 >> 172.21.15.72:0/3411841982 conn(0x55dbc506d020 msgr2=0x55dbc50b5ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:47.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.169+0000 7f59e9409740 1 -- 172.21.15.72:0/3411841982 shutdown_connections 2024-04-15T15:16:47.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.169+0000 7f59e9409740 1 -- 172.21.15.72:0/3411841982 wait complete. 2024-04-15T15:16:47.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.169+0000 7f59e9409740 1 -- start start 2024-04-15T15:16:47.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.170+0000 7f59e9409740 1 Processor -- start 2024-04-15T15:16:47.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.170+0000 7f59e9409740 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55dbc5097e10 0x55dbc5085910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.170+0000 7f59e9409740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dbc50981f0 0x55dbc509dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.170+0000 7f59e9409740 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55dbc509e2e0 0x55dbc50b01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.170+0000 7f59e9409740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55dbc506f5c0 con 0x55dbc50981f0 2024-04-15T15:16:47.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.170+0000 7f59e9409740 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x55dbc4f8fa10 con 0x55dbc5097e10 2024-04-15T15:16:47.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.170+0000 7f59e9409740 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x55dbc506e9b0 con 0x55dbc509e2e0 2024-04-15T15:16:47.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.170+0000 7f59e9409740 10 client.0 _ll_register_callbacks cb 0x55dbc4fcb250 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:16:47.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.170+0000 7f59df7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dbc50981f0 0x55dbc509dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.170+0000 7f59df7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dbc50981f0 0x55dbc509dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43590/0 (socket says 172.21.15.72:43590) 2024-04-15T15:16:47.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.170+0000 7f59df7fe640 1 -- 172.21.15.72:0/2191264175 learned_addr learned my addr 172.21.15.72:0/2191264175 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:47.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.171+0000 7f59dffff640 1 --2- 172.21.15.72:0/2191264175 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55dbc5097e10 0x55dbc5085910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.171+0000 7f59e4cdd640 1 --2- 172.21.15.72:0/2191264175 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55dbc509e2e0 0x55dbc50b01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.171+0000 7f59df7fe640 1 -- 172.21.15.72:0/2191264175 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55dbc509e2e0 msgr2=0x55dbc50b01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:47.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.171+0000 7f59df7fe640 1 --2- 172.21.15.72:0/2191264175 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55dbc509e2e0 0x55dbc50b01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.171+0000 7f59df7fe640 1 -- 172.21.15.72:0/2191264175 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55dbc5097e10 msgr2=0x55dbc5085910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:47.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.171+0000 7f59df7fe640 1 --2- 172.21.15.72:0/2191264175 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55dbc5097e10 0x55dbc5085910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.171+0000 7f59df7fe640 1 -- 172.21.15.72:0/2191264175 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbc4db26b0 con 0x55dbc50981f0 2024-04-15T15:16:47.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.171+0000 7f59e4cdd640 1 --2- 172.21.15.72:0/2191264175 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55dbc509e2e0 0x55dbc50b01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:47.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.171+0000 7f59df7fe640 1 --2- 172.21.15.72:0/2191264175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dbc50981f0 0x55dbc509dda0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f59d4015690 tx=0x7f59d40078f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fce43e8f66015985 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:47.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.171+0000 7f59beffd640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-04-15T15:16:47.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.171+0000 7f59beffd640 1 -- 172.21.15.72:0/2191264175 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d40145e0 con 0x55dbc50981f0 2024-04-15T15:16:47.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.171+0000 7f59beffd640 1 -- 172.21.15.72:0/2191264175 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7f59d40068b0 con 0x55dbc50981f0 2024-04-15T15:16:47.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.171+0000 7f59e9409740 1 -- 172.21.15.72:0/2191264175 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59d00626c0 con 0x55dbc50981f0 2024-04-15T15:16:47.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.171+0000 7f59beffd640 1 -- 172.21.15.72:0/2191264175 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f59d4006ab0 con 0x55dbc50981f0 2024-04-15T15:16:47.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.171+0000 7f59e9409740 10 client.6093 fetch_fsmap learned FSMap version 22 2024-04-15T15:16:47.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.171+0000 7f59e9409740 1 -- 172.21.15.72:0/2191264175 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=2+}) -- 0x55dbc5086f00 con 0x55dbc50981f0 2024-04-15T15:16:47.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.171+0000 7f59beffd640 1 -- 172.21.15.72:0/2191264175 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59d00626c0 con 0x55dbc50981f0 2024-04-15T15:16:47.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.171+0000 7f59beffd640 1 -- 172.21.15.72:0/2191264175 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 22) ==== 3494+0+0 (secure 0 0 0) 0x7f59d4006c50 con 0x55dbc50981f0 2024-04-15T15:16:47.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.172+0000 7f59e9409740 10 client.6093 fetch_fsmap finished waiting for FSMap version 22 2024-04-15T15:16:47.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.172+0000 7f59e9409740 10 client.6093 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-15T15:16:47.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.172+0000 7f59e9409740 20 client.6093 populate_metadata read hostname 'smithi072' 2024-04-15T15:16:47.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.172+0000 7f59e9409740 1 --2- 172.21.15.72:0/2191264175 >> [v2:172.21.15.72:6834/3407328263,v1:172.21.15.72:6836/3407328263] conn(0x55dbc5087780 0x55dbc5088900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.172+0000 7f59e9409740 4 client.6093 mds_command: new command op to 6054 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-15T15:16:47.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.172+0000 7f59e9409740 1 -- 172.21.15.72:0/2191264175 --> [v2:172.21.15.72:6834/3407328263,v1:172.21.15.72:6836/3407328263] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dbc4fe33e0 con 0x55dbc5087780 2024-04-15T15:16:47.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.172+0000 7f59dffff640 1 --2- 172.21.15.72:0/2191264175 >> [v2:172.21.15.72:6834/3407328263,v1:172.21.15.72:6836/3407328263] conn(0x55dbc5087780 0x55dbc5088900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.172+0000 7f59beffd640 10 client.6093 ms_handle_connect on v2:172.21.15.72:6834/3407328263 2024-04-15T15:16:47.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.172+0000 7f59dffff640 1 --2- 172.21.15.72:0/2191264175 >> [v2:172.21.15.72:6834/3407328263,v1:172.21.15.72:6836/3407328263] conn(0x55dbc5087780 0x55dbc5088900 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d3592685b6a3882 server_cookie=e01bc13d7bebcaa5 in_seq=0 out_seq=0 2024-04-15T15:16:47.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.173+0000 7f59beffd640 1 -- 172.21.15.72:0/2191264175 <== mds.0 v2:172.21.15.72:6834/3407328263 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dbc4fe33e0 con 0x55dbc5087780 2024-04-15T15:16:47.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.173+0000 7f59beffd640 10 client.6093 handle_command_reply: tid=0 2024-04-15T15:16:47.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.173+0000 7f59e9409740 1 client.6093 shutdown 2024-04-15T15:16:47.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.173+0000 7f59e9409740 1 -- 172.21.15.72:0/2191264175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dbc50981f0 msgr2=0x55dbc509dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.173+0000 7f59e9409740 1 --2- 172.21.15.72:0/2191264175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dbc50981f0 0x55dbc509dda0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f59d4015690 tx=0x7f59d40078f0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.173+0000 7f59e9409740 1 -- 172.21.15.72:0/2191264175 shutdown_connections 2024-04-15T15:16:47.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.173+0000 7f59e9409740 1 --2- 172.21.15.72:0/2191264175 >> [v2:172.21.15.72:6834/3407328263,v1:172.21.15.72:6836/3407328263] conn(0x55dbc5087780 0x55dbc5088900 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.173+0000 7f59e9409740 1 --2- 172.21.15.72:0/2191264175 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55dbc509e2e0 0x55dbc50b01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.173+0000 7f59beffd640 0 client.6093 ms_handle_reset on v2:172.21.15.72:6834/3407328263 2024-04-15T15:16:47.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.173+0000 7f59e9409740 1 --2- 172.21.15.72:0/2191264175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dbc50981f0 0x55dbc509dda0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.173+0000 7f59e9409740 1 --2- 172.21.15.72:0/2191264175 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55dbc5097e10 0x55dbc5085910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.173+0000 7f59e9409740 1 -- 172.21.15.72:0/2191264175 >> 172.21.15.72:0/2191264175 conn(0x55dbc506b2f0 msgr2=0x55dbc50b6df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:47.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.173+0000 7f59e9409740 1 -- 172.21.15.72:0/2191264175 shutdown_connections 2024-04-15T15:16:47.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.173+0000 7f59e9409740 1 -- 172.21.15.72:0/2191264175 wait complete. 2024-04-15T15:16:47.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.173+0000 7f59e9409740 20 client.6093 trim_cache size 0 max 16384 2024-04-15T15:16:47.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.191+0000 7f59e9409740 1 Processor -- start 2024-04-15T15:16:47.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.191+0000 7f59e9409740 1 -- start start 2024-04-15T15:16:47.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.192+0000 7f59e9409740 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55dbc5097e10 0x55dbc506e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.192+0000 7f59e9409740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dbc50981f0 0x55dbc5069eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.192+0000 7f59e9409740 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55dbc509e2e0 0x55dbc50cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.192+0000 7f59e9409740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f59e01fe950 con 0x55dbc50981f0 2024-04-15T15:16:47.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.192+0000 7f59e9409740 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f59e01fdda0 con 0x55dbc5097e10 2024-04-15T15:16:47.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.192+0000 7f59e9409740 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f59e01fe2c0 con 0x55dbc509e2e0 2024-04-15T15:16:47.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.192+0000 7f59df7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dbc50981f0 0x55dbc5069eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.192+0000 7f59df7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dbc50981f0 0x55dbc5069eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43596/0 (socket says 172.21.15.72:43596) 2024-04-15T15:16:47.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.192+0000 7f59df7fe640 1 -- 172.21.15.72:0/3503037267 learned_addr learned my addr 172.21.15.72:0/3503037267 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:47.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.192+0000 7f59df7fe640 1 -- 172.21.15.72:0/3503037267 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55dbc509e2e0 msgr2=0x55dbc50cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:47.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.192+0000 7f59df7fe640 1 --2- 172.21.15.72:0/3503037267 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55dbc509e2e0 0x55dbc50cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.192+0000 7f59df7fe640 1 -- 172.21.15.72:0/3503037267 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55dbc5097e10 msgr2=0x55dbc506e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:47.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.192+0000 7f59df7fe640 1 --2- 172.21.15.72:0/3503037267 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55dbc5097e10 0x55dbc506e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.192+0000 7f59df7fe640 1 -- 172.21.15.72:0/3503037267 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbc509e110 con 0x55dbc50981f0 2024-04-15T15:16:47.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.192+0000 7f59df7fe640 1 --2- 172.21.15.72:0/3503037267 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dbc50981f0 0x55dbc5069eb0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f59d4007610 tx=0x7f59d4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea793f04d24f4772 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:47.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.192+0000 7f59c57fa640 1 -- 172.21.15.72:0/3503037267 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d40145e0 con 0x55dbc50981f0 2024-04-15T15:16:47.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.192+0000 7f59c57fa640 1 -- 172.21.15.72:0/3503037267 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59d401c050 con 0x55dbc50981f0 2024-04-15T15:16:47.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.192+0000 7f59c57fa640 1 -- 172.21.15.72:0/3503037267 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d4014b20 con 0x55dbc50981f0 2024-04-15T15:16:47.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.193+0000 7f59e9409740 1 -- 172.21.15.72:0/3503037267 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dbc50981f0 msgr2=0x55dbc5069eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.193+0000 7f59e9409740 1 --2- 172.21.15.72:0/3503037267 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dbc50981f0 0x55dbc5069eb0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f59d4007610 tx=0x7f59d4017000 comp rx=0 tx=0).stop 2024-04-15T15:16:47.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.193+0000 7f59e9409740 1 -- 172.21.15.72:0/3503037267 shutdown_connections 2024-04-15T15:16:47.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.193+0000 7f59e9409740 1 --2- 172.21.15.72:0/3503037267 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55dbc509e2e0 0x55dbc50cedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.193+0000 7f59e9409740 1 --2- 172.21.15.72:0/3503037267 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dbc50981f0 0x55dbc5069eb0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.193+0000 7f59e9409740 1 --2- 172.21.15.72:0/3503037267 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55dbc5097e10 0x55dbc506e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.193+0000 7f59e9409740 1 -- 172.21.15.72:0/3503037267 >> 172.21.15.72:0/3503037267 conn(0x55dbc506b2f0 msgr2=0x55dbc509d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:47.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.193+0000 7f59e9409740 1 -- 172.21.15.72:0/3503037267 shutdown_connections 2024-04-15T15:16:47.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.193+0000 7f59e9409740 1 -- 172.21.15.72:0/3503037267 wait complete. 2024-04-15T15:16:47.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.193+0000 7f59e9409740 1 -- start start 2024-04-15T15:16:47.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.194+0000 7f59e9409740 1 Processor -- start 2024-04-15T15:16:47.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.194+0000 7f59e9409740 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dbc5097e10 0x55dbc50cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.194+0000 7f59e9409740 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55dbc50981f0 0x55dbc50cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.194+0000 7f59e9409740 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55dbc509e2e0 0x55dbc50c40b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.194+0000 7f59e9409740 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x55dbc5087600 con 0x55dbc5097e10 2024-04-15T15:16:47.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.194+0000 7f59e9409740 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x55dbc50add60 con 0x55dbc509e2e0 2024-04-15T15:16:47.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.194+0000 7f59e9409740 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x55dbc5087bc0 con 0x55dbc50981f0 2024-04-15T15:16:47.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.194+0000 7f59e9409740 10 client.0 _ll_register_callbacks cb 0x7f59d4018040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:16:47.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.194+0000 7f59dffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dbc5097e10 0x55dbc50cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.195+0000 7f59dffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dbc5097e10 0x55dbc50cedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43612/0 (socket says 172.21.15.72:43612) 2024-04-15T15:16:47.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.195+0000 7f59dffff640 1 -- 172.21.15.72:0/1081271735 learned_addr learned my addr 172.21.15.72:0/1081271735 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:47.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.195+0000 7f59e4cdd640 1 --2- 172.21.15.72:0/1081271735 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55dbc509e2e0 0x55dbc50c40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.195+0000 7f59df7fe640 1 --2- 172.21.15.72:0/1081271735 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55dbc50981f0 0x55dbc50cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.195+0000 7f59dffff640 1 -- 172.21.15.72:0/1081271735 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55dbc50981f0 msgr2=0x55dbc50cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:47.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.195+0000 7f59dffff640 1 --2- 172.21.15.72:0/1081271735 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55dbc50981f0 0x55dbc50cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.195+0000 7f59dffff640 1 -- 172.21.15.72:0/1081271735 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55dbc509e2e0 msgr2=0x55dbc50c40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:47.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.195+0000 7f59dffff640 1 --2- 172.21.15.72:0/1081271735 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55dbc509e2e0 0x55dbc50c40b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.195+0000 7f59dffff640 1 -- 172.21.15.72:0/1081271735 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbc50b7d50 con 0x55dbc5097e10 2024-04-15T15:16:47.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.195+0000 7f59e4cdd640 1 --2- 172.21.15.72:0/1081271735 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55dbc509e2e0 0x55dbc50c40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:47.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.195+0000 7f59df7fe640 1 --2- 172.21.15.72:0/1081271735 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55dbc50981f0 0x55dbc50cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:47.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.195+0000 7f59dffff640 1 --2- 172.21.15.72:0/1081271735 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dbc5097e10 0x55dbc50cedb0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f59d0004630 tx=0x7f59d0006f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d17e7f98c9df949b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:47.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.195+0000 7f59beffd640 10 client.0 ms_handle_connect on v2:172.21.15.72:3300/0 2024-04-15T15:16:47.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.195+0000 7f59beffd640 1 -- 172.21.15.72:0/1081271735 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d0089040 con 0x55dbc5097e10 2024-04-15T15:16:47.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.195+0000 7f59beffd640 1 -- 172.21.15.72:0/1081271735 <== mon.0 v2:172.21.15.72:3300/0 2 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7f59d0058f30 con 0x55dbc5097e10 2024-04-15T15:16:47.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.195+0000 7f59e9409740 1 -- 172.21.15.72:0/1081271735 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59d401c050 con 0x55dbc5097e10 2024-04-15T15:16:47.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.195+0000 7f59beffd640 1 -- 172.21.15.72:0/1081271735 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) ==== 24+0+0 (secure 0 0 0) 0x7f59d0022040 con 0x55dbc5097e10 2024-04-15T15:16:47.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.195+0000 7f59e9409740 10 client.6099 fetch_fsmap learned FSMap version 22 2024-04-15T15:16:47.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.195+0000 7f59e9409740 1 -- 172.21.15.72:0/1081271735 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=2+}) -- 0x55dbc50b7890 con 0x55dbc5097e10 2024-04-15T15:16:47.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.195+0000 7f59beffd640 1 -- 172.21.15.72:0/1081271735 <== mon.0 v2:172.21.15.72:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59d401c050 con 0x55dbc5097e10 2024-04-15T15:16:47.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.195+0000 7f59beffd640 1 -- 172.21.15.72:0/1081271735 <== mon.0 v2:172.21.15.72:3300/0 5 ==== fsmap(e 22) ==== 3494+0+0 (secure 0 0 0) 0x7f59d0060090 con 0x55dbc5097e10 2024-04-15T15:16:47.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.196+0000 7f59e9409740 10 client.6099 fetch_fsmap finished waiting for FSMap version 22 2024-04-15T15:16:47.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.196+0000 7f59e9409740 10 client.6099 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-15T15:16:47.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.196+0000 7f59e9409740 20 client.6099 populate_metadata read hostname 'smithi072' 2024-04-15T15:16:47.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.196+0000 7f59e9409740 1 --2- 172.21.15.72:0/1081271735 >> [v2:172.21.15.72:6834/3407328263,v1:172.21.15.72:6836/3407328263] conn(0x55dbc50c46e0 0x55dbc50b84e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.196+0000 7f59e9409740 4 client.6099 mds_command: new command op to 6054 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-15T15:16:47.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.196+0000 7f59e9409740 1 -- 172.21.15.72:0/1081271735 --> [v2:172.21.15.72:6834/3407328263,v1:172.21.15.72:6836/3407328263] -- command(tid 0: {"prefix": "status"}) -- 0x55dbc50b1f10 con 0x55dbc50c46e0 2024-04-15T15:16:47.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.196+0000 7f59df7fe640 1 --2- 172.21.15.72:0/1081271735 >> [v2:172.21.15.72:6834/3407328263,v1:172.21.15.72:6836/3407328263] conn(0x55dbc50c46e0 0x55dbc50b84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.196+0000 7f59beffd640 10 client.6099 ms_handle_connect on v2:172.21.15.72:6834/3407328263 2024-04-15T15:16:47.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.196+0000 7f59df7fe640 1 --2- 172.21.15.72:0/1081271735 >> [v2:172.21.15.72:6834/3407328263,v1:172.21.15.72:6836/3407328263] conn(0x55dbc50c46e0 0x55dbc50b84e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8728cb5ef2098ff server_cookie=5d3f669e65d82b90 in_seq=0 out_seq=0 2024-04-15T15:16:47.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.196+0000 7f59beffd640 1 -- 172.21.15.72:0/1081271735 <== mds.0 v2:172.21.15.72:6834/3407328263 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55dbc50b1f10 con 0x55dbc50c46e0 2024-04-15T15:16:47.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.196+0000 7f59beffd640 10 client.6099 handle_command_reply: tid=0 2024-04-15T15:16:47.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.196+0000 7f59e9409740 1 client.6099 shutdown 2024-04-15T15:16:47.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.196+0000 7f59e9409740 1 -- 172.21.15.72:0/1081271735 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dbc5097e10 msgr2=0x55dbc50cedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.196+0000 7f59e9409740 1 --2- 172.21.15.72:0/1081271735 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dbc5097e10 0x55dbc50cedb0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f59d0004630 tx=0x7f59d0006f60 comp rx=0 tx=0).stop 2024-04-15T15:16:47.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.196+0000 7f59e9409740 1 -- 172.21.15.72:0/1081271735 shutdown_connections 2024-04-15T15:16:47.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.196+0000 7f59e9409740 1 --2- 172.21.15.72:0/1081271735 >> [v2:172.21.15.72:6834/3407328263,v1:172.21.15.72:6836/3407328263] conn(0x55dbc50c46e0 0x55dbc50b84e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.196+0000 7f59e9409740 1 --2- 172.21.15.72:0/1081271735 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x55dbc509e2e0 0x55dbc50c40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.196+0000 7f59e9409740 1 --2- 172.21.15.72:0/1081271735 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x55dbc50981f0 0x55dbc50cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.196+0000 7f59beffd640 0 client.6099 ms_handle_reset on v2:172.21.15.72:6834/3407328263 2024-04-15T15:16:47.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.196+0000 7f59e9409740 1 --2- 172.21.15.72:0/1081271735 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x55dbc5097e10 0x55dbc50cedb0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.196+0000 7f59e9409740 1 -- 172.21.15.72:0/1081271735 >> 172.21.15.72:0/1081271735 conn(0x55dbc506b2f0 msgr2=0x55dbc5069eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:47.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.197+0000 7f59e9409740 1 -- 172.21.15.72:0/1081271735 shutdown_connections 2024-04-15T15:16:47.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.197+0000 7f59e9409740 1 -- 172.21.15.72:0/1081271735 wait complete. 2024-04-15T15:16:47.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.197+0000 7f59e9409740 20 client.6099 trim_cache size 0 max 16384 2024-04-15T15:16:47.206 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-04-15T15:16:47.206 INFO:teuthology.orchestra.run.smithi072.stdout: "cluster_fsid": "05091e39-b18a-4545-8b68-c0ae4699e116", 2024-04-15T15:16:47.206 INFO:teuthology.orchestra.run.smithi072.stdout: "whoami": 0, 2024-04-15T15:16:47.206 INFO:teuthology.orchestra.run.smithi072.stdout: "id": 6054, 2024-04-15T15:16:47.206 INFO:teuthology.orchestra.run.smithi072.stdout: "want_state": "up:active", 2024-04-15T15:16:47.206 INFO:teuthology.orchestra.run.smithi072.stdout: "state": "up:active", 2024-04-15T15:16:47.206 INFO:teuthology.orchestra.run.smithi072.stdout: "fs_name": "cephfs", 2024-04-15T15:16:47.207 INFO:teuthology.orchestra.run.smithi072.stdout: "rank_uptime": 4.4706618110000003, 2024-04-15T15:16:47.207 INFO:teuthology.orchestra.run.smithi072.stdout: "mdsmap_epoch": 22, 2024-04-15T15:16:47.207 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch": 41, 2024-04-15T15:16:47.207 INFO:teuthology.orchestra.run.smithi072.stdout: "osdmap_epoch_barrier": 41, 2024-04-15T15:16:47.207 INFO:teuthology.orchestra.run.smithi072.stdout: "uptime": 5.4394629070000002 2024-04-15T15:16:47.207 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-04-15T15:16:47.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.197+0000 7f59e6740640 1 -- 172.21.15.72:0/4148302283 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f59cc06c9b0 msgr2=0x7f59cc06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.197+0000 7f59e6740640 1 --2- 172.21.15.72:0/4148302283 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f59cc06c9b0 0x7f59cc06ee70 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f59e0155ee0 tx=0x7f59d4009040 comp rx=0 tx=0).stop 2024-04-15T15:16:47.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.197+0000 7f59e6740640 1 -- 172.21.15.72:0/4148302283 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59c80052e0 msgr2=0x7f59e0157f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.197+0000 7f59e6740640 1 --2- 172.21.15.72:0/4148302283 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59c80052e0 0x7f59e0157f00 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f59d00115c0 tx=0x7f59d0011a80 comp rx=0 tx=0).stop 2024-04-15T15:16:47.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.197+0000 7f59e6740640 1 -- 172.21.15.72:0/4148302283 shutdown_connections 2024-04-15T15:16:47.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.197+0000 7f59e6740640 1 --2- 172.21.15.72:0/4148302283 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f59cc06c9b0 0x7f59cc06ee70 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.197+0000 7f59e6740640 1 --2- 172.21.15.72:0/4148302283 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f59e0146650 0x7f59e0146ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.197+0000 7f59e6740640 1 --2- 172.21.15.72:0/4148302283 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59e0148270 0x7f59e0159b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.197+0000 7f59e6740640 1 --2- 172.21.15.72:0/4148302283 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59c80052e0 0x7f59e0157f00 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.197+0000 7f59e6740640 1 -- 172.21.15.72:0/4148302283 >> 172.21.15.72:0/4148302283 conn(0x7f59e00d24d0 msgr2=0x7f59e0153120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:47.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.197+0000 7f59e6740640 1 -- 172.21.15.72:0/4148302283 shutdown_connections 2024-04-15T15:16:47.209 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.197+0000 7f59e6740640 1 -- 172.21.15.72:0/4148302283 wait complete. 2024-04-15T15:16:47.210 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-15T15:16:47.210 DEBUG:teuthology.orchestra.run.smithi072:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-15T15:16:47.266 DEBUG:teuthology.orchestra.run.smithi175:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-15T15:16:47.317 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-15T15:16:47.318 INFO:tasks.cephfs.mount:Terminating background process 2024-04-15T15:16:47.318 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:16:47.318 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:16:47.335 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:47.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.477+0000 7f9ba3231640 1 Processor -- start 2024-04-15T15:16:47.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.478+0000 7f9ba3231640 1 -- start start 2024-04-15T15:16:47.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.478+0000 7f9ba3231640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f9b9c161e40 0x7f9b9c0b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.478+0000 7f9ba3231640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9b9c0d45b0 0x7f9b9c0d49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.478+0000 7f9ba3231640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b9c162220 con 0x7f9b9c0d12b0 2024-04-15T15:16:47.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.478+0000 7f9ba3231640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f9b9c0d21f0 con 0x7f9b9c0d45b0 2024-04-15T15:16:47.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.478+0000 7f9ba3231640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f9b9c0d4f20 con 0x7f9b9c161e40 2024-04-15T15:16:47.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.478+0000 7f9ba17ce640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9b9c0d45b0 0x7f9b9c0d49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.479+0000 7f9ba17ce640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9b9c0d45b0 0x7f9b9c0d49e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43624/0 (socket says 172.21.15.72:43624) 2024-04-15T15:16:47.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.479+0000 7f9b9bfff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f9b9c0d12b0 0x7f9b9c0d2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49884/0 (socket says 172.21.15.72:49884) 2024-04-15T15:16:47.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.479+0000 7f9ba17ce640 1 -- 172.21.15.72:0/2543561527 learned_addr learned my addr 172.21.15.72:0/2543561527 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:47.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.479+0000 7f9ba0fcd640 1 --2- 172.21.15.72:0/2543561527 >> v2:172.21.15.175:3300/0 conn(0x7f9b9c161e40 0x7f9b9c0b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.479+0000 7f9ba17ce640 1 -- 172.21.15.72:0/2543561527 >> v2:172.21.15.175:3300/0 conn(0x7f9b9c161e40 msgr2=0x7f9b9c0b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:47.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.479+0000 7f9ba17ce640 1 --2- 172.21.15.72:0/2543561527 >> v2:172.21.15.175:3300/0 conn(0x7f9b9c161e40 0x7f9b9c0b37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.479+0000 7f9ba17ce640 1 -- 172.21.15.72:0/2543561527 >> v1:172.21.15.72:6789/0 conn(0x7f9b9c0d12b0 legacy=0x7f9b9c0d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.479+0000 7f9ba17ce640 1 -- 172.21.15.72:0/2543561527 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b9c0d3070 con 0x7f9b9c0d45b0 2024-04-15T15:16:47.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.479+0000 7f9ba17ce640 1 --2- 172.21.15.72:0/2543561527 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b9c0d45b0 0x7f9b9c0d49e0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f9b94009870 tx=0x7f9b9402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d5b9aa8bca04427 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:47.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.480+0000 7f9b9b7fe640 1 -- 172.21.15.72:0/2543561527 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b94035020 con 0x7f9b9c0d45b0 2024-04-15T15:16:47.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.480+0000 7f9b9b7fe640 1 -- 172.21.15.72:0/2543561527 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b94031870 con 0x7f9b9c0d45b0 2024-04-15T15:16:47.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.480+0000 7f9b9b7fe640 1 -- 172.21.15.72:0/2543561527 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b94031c40 con 0x7f9b9c0d45b0 2024-04-15T15:16:47.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.480+0000 7f9ba3231640 1 -- 172.21.15.72:0/2543561527 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b9c0d45b0 msgr2=0x7f9b9c0d49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.480+0000 7f9ba3231640 1 --2- 172.21.15.72:0/2543561527 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b9c0d45b0 0x7f9b9c0d49e0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f9b94009870 tx=0x7f9b9402b910 comp rx=0 tx=0).stop 2024-04-15T15:16:47.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.480+0000 7f9ba3231640 1 -- 172.21.15.72:0/2543561527 shutdown_connections 2024-04-15T15:16:47.483 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.480+0000 7f9ba3231640 1 --2- 172.21.15.72:0/2543561527 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b9c0d45b0 0x7f9b9c0d49e0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.480+0000 7f9ba3231640 1 --2- 172.21.15.72:0/2543561527 >> v2:172.21.15.175:3300/0 conn(0x7f9b9c161e40 0x7f9b9c0b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.480+0000 7f9ba3231640 1 -- 172.21.15.72:0/2543561527 >> 172.21.15.72:0/2543561527 conn(0x7f9b9c09d970 msgr2=0x7f9b9c0b95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:47.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.480+0000 7f9ba3231640 1 -- 172.21.15.72:0/2543561527 shutdown_connections 2024-04-15T15:16:47.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.480+0000 7f9ba3231640 1 -- 172.21.15.72:0/2543561527 wait complete. 2024-04-15T15:16:47.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.481+0000 7f9ba3231640 1 Processor -- start 2024-04-15T15:16:47.484 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.481+0000 7f9ba3231640 1 -- start start 2024-04-15T15:16:47.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.482+0000 7f9ba3231640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b9c0d12b0 0x7f9b9c151d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.482+0000 7f9ba3231640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9b9c0d45b0 0x7f9b9c0ca7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.482+0000 7f9ba3231640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9b9c161e40 0x7f9b9c1511a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.482+0000 7f9ba3231640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f9b9c0d1de0 con 0x7f9b9c0d12b0 2024-04-15T15:16:47.486 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.482+0000 7f9ba3231640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f9b9c0d1b00 con 0x7f9b9c0d45b0 2024-04-15T15:16:47.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.482+0000 7f9ba3231640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f9b9c0d2440 con 0x7f9b9c161e40 2024-04-15T15:16:47.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.482+0000 7f9ba0fcd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b9c0d12b0 0x7f9b9c151d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.482+0000 7f9ba0fcd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b9c0d12b0 0x7f9b9c151d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43638/0 (socket says 172.21.15.72:43638) 2024-04-15T15:16:47.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.482+0000 7f9ba0fcd640 1 -- 172.21.15.72:0/3956713824 learned_addr learned my addr 172.21.15.72:0/3956713824 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:47.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.483+0000 7f9b9bfff640 1 --2- 172.21.15.72:0/3956713824 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9b9c0d45b0 0x7f9b9c0ca7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.483+0000 7f9ba17ce640 1 --2- 172.21.15.72:0/3956713824 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9b9c161e40 0x7f9b9c1511a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.483+0000 7f9ba0fcd640 1 -- 172.21.15.72:0/3956713824 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9b9c161e40 msgr2=0x7f9b9c1511a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.483+0000 7f9ba0fcd640 1 --2- 172.21.15.72:0/3956713824 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9b9c161e40 0x7f9b9c1511a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.483+0000 7f9ba0fcd640 1 -- 172.21.15.72:0/3956713824 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9b9c0d45b0 msgr2=0x7f9b9c0ca7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.483+0000 7f9ba0fcd640 1 --2- 172.21.15.72:0/3956713824 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9b9c0d45b0 0x7f9b9c0ca7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.483+0000 7f9ba0fcd640 1 -- 172.21.15.72:0/3956713824 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b9c15b030 con 0x7f9b9c0d12b0 2024-04-15T15:16:47.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.483+0000 7f9b9bfff640 1 --2- 172.21.15.72:0/3956713824 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9b9c0d45b0 0x7f9b9c0ca7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:47.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.483+0000 7f9ba17ce640 1 --2- 172.21.15.72:0/3956713824 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9b9c161e40 0x7f9b9c1511a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:47.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.483+0000 7f9ba0fcd640 1 --2- 172.21.15.72:0/3956713824 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b9c0d12b0 0x7f9b9c151d30 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c00d5f0 tx=0x7f9b8c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:47.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.483+0000 7f9b997fa640 1 -- 172.21.15.72:0/3956713824 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c0125f0 con 0x7f9b9c0d12b0 2024-04-15T15:16:47.487 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.483+0000 7f9b997fa640 1 -- 172.21.15.72:0/3956713824 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b8c019070 con 0x7f9b9c0d12b0 2024-04-15T15:16:47.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.483+0000 7f9ba3231640 1 -- 172.21.15.72:0/3956713824 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b9c0cc2f0 con 0x7f9b9c0d12b0 2024-04-15T15:16:47.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.483+0000 7f9b997fa640 1 -- 172.21.15.72:0/3956713824 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c0129d0 con 0x7f9b9c0d12b0 2024-04-15T15:16:47.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.483+0000 7f9ba3231640 1 -- 172.21.15.72:0/3956713824 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b9c0cc8b0 con 0x7f9b9c0d12b0 2024-04-15T15:16:47.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.484+0000 7f9ba3231640 1 -- 172.21.15.72:0/3956713824 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b9c0689e0 con 0x7f9b9c0d12b0 2024-04-15T15:16:47.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.485+0000 7f9b997fa640 1 -- 172.21.15.72:0/3956713824 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f9b8c012b70 con 0x7f9b9c0d12b0 2024-04-15T15:16:47.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.485+0000 7f9b997fa640 1 --2- 172.21.15.72:0/3956713824 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f9b7c06c980 0x7f9b7c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.485+0000 7f9b9bfff640 1 --2- 172.21.15.72:0/3956713824 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f9b7c06c980 0x7f9b7c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.485+0000 7f9b997fa640 1 -- 172.21.15.72:0/3956713824 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7f9b8c08c0f0 con 0x7f9b9c0d12b0 2024-04-15T15:16:47.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.486+0000 7f9b9bfff640 1 --2- 172.21.15.72:0/3956713824 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f9b7c06c980 0x7f9b7c06ee40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9b84007fd0 tx=0x7f9b84009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:47.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.490+0000 7f9b997fa640 1 -- 172.21.15.72:0/3956713824 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9b8c05a3c0 con 0x7f9b9c0d12b0 2024-04-15T15:16:47.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.664+0000 7f9ba3231640 1 -- 172.21.15.72:0/3956713824 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9b9c0d51b0 con 0x7f9b9c0d12b0 2024-04-15T15:16:47.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.665+0000 7f9b997fa640 1 -- 172.21.15.72:0/3956713824 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) ==== 74+0+17120 (secure 0 0 0) 0x7f9b8c083e20 con 0x7f9b9c0d12b0 2024-04-15T15:16:47.668 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:16:47.669 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":41,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:16:41.714841+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:05.551499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:06.003152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.26","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:6836/4270793613":"2024-04-16T15:16:41.714817+0000","172.21.15.72:6834/4270793613":"2024-04-16T15:16:41.714817+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:47.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.667+0000 7f9ba3231640 1 -- 172.21.15.72:0/3956713824 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f9b7c06c980 msgr2=0x7f9b7c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.667+0000 7f9ba3231640 1 --2- 172.21.15.72:0/3956713824 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f9b7c06c980 0x7f9b7c06ee40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9b84007fd0 tx=0x7f9b84009040 comp rx=0 tx=0).stop 2024-04-15T15:16:47.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.667+0000 7f9ba3231640 1 -- 172.21.15.72:0/3956713824 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b9c0d12b0 msgr2=0x7f9b9c151d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.667+0000 7f9ba3231640 1 --2- 172.21.15.72:0/3956713824 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b9c0d12b0 0x7f9b9c151d30 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c00d5f0 tx=0x7f9b8c00dab0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.667+0000 7f9ba3231640 1 -- 172.21.15.72:0/3956713824 shutdown_connections 2024-04-15T15:16:47.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.667+0000 7f9ba3231640 1 --2- 172.21.15.72:0/3956713824 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f9b7c06c980 0x7f9b7c06ee40 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.667+0000 7f9ba3231640 1 --2- 172.21.15.72:0/3956713824 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9b9c161e40 0x7f9b9c1511a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.667+0000 7f9ba3231640 1 --2- 172.21.15.72:0/3956713824 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9b9c0d45b0 0x7f9b9c0ca7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.667+0000 7f9ba3231640 1 --2- 172.21.15.72:0/3956713824 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b9c0d12b0 0x7f9b9c151d30 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.667+0000 7f9ba3231640 1 -- 172.21.15.72:0/3956713824 >> 172.21.15.72:0/3956713824 conn(0x7f9b9c09d970 msgr2=0x7f9b9c0b7240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:47.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.667+0000 7f9ba3231640 1 -- 172.21.15.72:0/3956713824 shutdown_connections 2024-04-15T15:16:47.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.667+0000 7f9ba3231640 1 -- 172.21.15.72:0/3956713824 wait complete. 2024-04-15T15:16:47.680 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:39.359102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:16:05.551499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:16:06.003152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:16:47.681 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:47.728 INFO:tasks.ceph.mds.a.smithi072.stderr:2024-04-15T15:16:47.727+0000 7f68bd49d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-15T15:16:47.788 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.787+0000 7f1cc7590640 1 Processor -- start 2024-04-15T15:16:47.789 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.787+0000 7f1cc7590640 1 -- start start 2024-04-15T15:16:47.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.788+0000 7f1cc7590640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1cc00ce2f0 0x7f1cc015c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.788+0000 7f1cc7590640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1cc00cc7c0 0x7f1cc0155600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.788+0000 7f1cc7590640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cc006bbb0 con 0x7f1cc00d08f0 2024-04-15T15:16:47.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.788+0000 7f1cc7590640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f1cc006b960 con 0x7f1cc00cc7c0 2024-04-15T15:16:47.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.788+0000 7f1cc7590640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f1cc006be90 con 0x7f1cc00ce2f0 2024-04-15T15:16:47.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.788+0000 7f1cc5b2d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1cc00cc7c0 0x7f1cc0155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.788+0000 7f1cc5b2d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1cc00cc7c0 0x7f1cc0155600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43640/0 (socket says 172.21.15.72:43640) 2024-04-15T15:16:47.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.788+0000 7f1cc5b2d640 1 -- 172.21.15.72:0/2952856371 learned_addr learned my addr 172.21.15.72:0/2952856371 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:47.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.788+0000 7f1cc5b2d640 1 -- 172.21.15.72:0/2952856371 >> v2:172.21.15.175:3300/0 conn(0x7f1cc00ce2f0 msgr2=0x7f1cc015c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:47.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.788+0000 7f1cc5b2d640 1 --2- 172.21.15.72:0/2952856371 >> v2:172.21.15.175:3300/0 conn(0x7f1cc00ce2f0 0x7f1cc015c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.788+0000 7f1cc5b2d640 1 -- 172.21.15.72:0/2952856371 >> v1:172.21.15.175:6789/0 conn(0x7f1cc00d08f0 legacy=0x7f1cc00cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:47.790 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.788+0000 7f1cc5b2d640 1 -- 172.21.15.72:0/2952856371 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cc006c170 con 0x7f1cc00cc7c0 2024-04-15T15:16:47.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.789+0000 7f1cc5b2d640 1 --2- 172.21.15.72:0/2952856371 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1cc00cc7c0 0x7f1cc0155600 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f1cc00cdf10 tx=0x7f1cb002e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2f6008f7d684c54 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:47.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.789+0000 7f1cbffff640 1 -- 172.21.15.72:0/2952856371 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb002ea90 con 0x7f1cc00cc7c0 2024-04-15T15:16:47.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.789+0000 7f1cbffff640 1 -- 172.21.15.72:0/2952856371 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cb002ec30 con 0x7f1cc00cc7c0 2024-04-15T15:16:47.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.789+0000 7f1cbffff640 1 -- 172.21.15.72:0/2952856371 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb0009600 con 0x7f1cc00cc7c0 2024-04-15T15:16:47.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.790+0000 7f1cc7590640 1 -- 172.21.15.72:0/2952856371 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1cc00cc7c0 msgr2=0x7f1cc0155600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.790+0000 7f1cc7590640 1 --2- 172.21.15.72:0/2952856371 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1cc00cc7c0 0x7f1cc0155600 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f1cc00cdf10 tx=0x7f1cb002e0f0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.790+0000 7f1cc7590640 1 -- 172.21.15.72:0/2952856371 shutdown_connections 2024-04-15T15:16:47.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.790+0000 7f1cc7590640 1 --2- 172.21.15.72:0/2952856371 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1cc00cc7c0 0x7f1cc0155600 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.790+0000 7f1cc7590640 1 --2- 172.21.15.72:0/2952856371 >> v2:172.21.15.175:3300/0 conn(0x7f1cc00ce2f0 0x7f1cc015c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.790+0000 7f1cc7590640 1 -- 172.21.15.72:0/2952856371 >> 172.21.15.72:0/2952856371 conn(0x7f1cc00b31d0 msgr2=0x7f1cc00b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:47.792 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.790+0000 7f1cc7590640 1 -- 172.21.15.72:0/2952856371 shutdown_connections 2024-04-15T15:16:47.793 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.790+0000 7f1cc7590640 1 -- 172.21.15.72:0/2952856371 wait complete. 2024-04-15T15:16:47.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.791+0000 7f1cc7590640 1 Processor -- start 2024-04-15T15:16:47.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.791+0000 7f1cc7590640 1 -- start start 2024-04-15T15:16:47.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.792+0000 7f1cc7590640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1cc00cc7c0 0x7f1cc013e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.792+0000 7f1cc7590640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1cc00ce2f0 0x7f1cc01f39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.792+0000 7f1cc7590640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1cc00d08f0 0x7f1cc01f3f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.792+0000 7f1cc7590640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc00ce6d0 con 0x7f1cc00d08f0 2024-04-15T15:16:47.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.792+0000 7f1cc7590640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc015ca70 con 0x7f1cc00ce2f0 2024-04-15T15:16:47.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.792+0000 7f1cc7590640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f1cc00ce850 con 0x7f1cc00cc7c0 2024-04-15T15:16:47.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.792+0000 7f1cc5b2d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1cc00d08f0 0x7f1cc01f3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.792+0000 7f1cc5b2d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1cc00d08f0 0x7f1cc01f3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43646/0 (socket says 172.21.15.72:43646) 2024-04-15T15:16:47.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.792+0000 7f1cc5b2d640 1 -- 172.21.15.72:0/630185578 learned_addr learned my addr 172.21.15.72:0/630185578 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:47.795 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.792+0000 7f1cc4b2b640 1 --2- 172.21.15.72:0/630185578 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1cc00ce2f0 0x7f1cc01f39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.792+0000 7f1cc532c640 1 --2- 172.21.15.72:0/630185578 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1cc00cc7c0 0x7f1cc013e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.792+0000 7f1cc5b2d640 1 -- 172.21.15.72:0/630185578 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1cc00cc7c0 msgr2=0x7f1cc013e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.792+0000 7f1cc5b2d640 1 --2- 172.21.15.72:0/630185578 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1cc00cc7c0 0x7f1cc013e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.792+0000 7f1cc5b2d640 1 -- 172.21.15.72:0/630185578 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1cc00ce2f0 msgr2=0x7f1cc01f39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.792+0000 7f1cc5b2d640 1 --2- 172.21.15.72:0/630185578 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1cc00ce2f0 0x7f1cc01f39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.792+0000 7f1cc5b2d640 1 -- 172.21.15.72:0/630185578 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cc01e51f0 con 0x7f1cc00d08f0 2024-04-15T15:16:47.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.792+0000 7f1cc532c640 1 --2- 172.21.15.72:0/630185578 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1cc00cc7c0 0x7f1cc013e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:47.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.792+0000 7f1cc5b2d640 1 --2- 172.21.15.72:0/630185578 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1cc00d08f0 0x7f1cc01f3f30 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0038040 tx=0x7f1cb0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:47.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.793+0000 7f1cbdffb640 1 -- 172.21.15.72:0/630185578 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb0045020 con 0x7f1cc00d08f0 2024-04-15T15:16:47.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.793+0000 7f1cbdffb640 1 -- 172.21.15.72:0/630185578 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cb0040660 con 0x7f1cc00d08f0 2024-04-15T15:16:47.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.793+0000 7f1cc7590640 1 -- 172.21.15.72:0/630185578 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cc01e54d0 con 0x7f1cc00d08f0 2024-04-15T15:16:47.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.793+0000 7f1cbdffb640 1 -- 172.21.15.72:0/630185578 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb002ea90 con 0x7f1cc00d08f0 2024-04-15T15:16:47.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.793+0000 7f1cc7590640 1 -- 172.21.15.72:0/630185578 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cc01e5a90 con 0x7f1cc00d08f0 2024-04-15T15:16:47.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.794+0000 7f1cc7590640 1 -- 172.21.15.72:0/630185578 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cc0075c50 con 0x7f1cc00d08f0 2024-04-15T15:16:47.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.794+0000 7f1cbdffb640 1 -- 172.21.15.72:0/630185578 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f1cb0048020 con 0x7f1cc00d08f0 2024-04-15T15:16:47.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.795+0000 7f1cbdffb640 1 --2- 172.21.15.72:0/630185578 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1cac06c980 0x7f1cac06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:47.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.795+0000 7f1cbdffb640 1 -- 172.21.15.72:0/630185578 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7f1cb00b0860 con 0x7f1cc00d08f0 2024-04-15T15:16:47.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.795+0000 7f1cc532c640 1 --2- 172.21.15.72:0/630185578 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1cac06c980 0x7f1cac06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:47.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.795+0000 7f1cc532c640 1 --2- 172.21.15.72:0/630185578 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1cac06c980 0x7f1cac06ee40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8009930 tx=0x7f1cb8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:47.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.799+0000 7f1cbdffb640 1 -- 172.21.15.72:0/630185578 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1cb004f030 con 0x7f1cc00d08f0 2024-04-15T15:16:47.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.977+0000 7f1cc7590640 1 -- 172.21.15.72:0/630185578 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1cc006cab0 con 0x7f1cc00d08f0 2024-04-15T15:16:47.979 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.977+0000 7f1cbdffb640 1 -- 172.21.15.72:0/630185578 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) ==== 93+0+10 (secure 0 0 0) 0x7f1cb007eab0 con 0x7f1cc00d08f0 2024-04-15T15:16:47.979 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-15T15:16:47.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.979+0000 7f1cc7590640 1 -- 172.21.15.72:0/630185578 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1cac06c980 msgr2=0x7f1cac06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.979+0000 7f1cc7590640 1 --2- 172.21.15.72:0/630185578 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1cac06c980 0x7f1cac06ee40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8009930 tx=0x7f1cb8008040 comp rx=0 tx=0).stop 2024-04-15T15:16:47.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.979+0000 7f1cc7590640 1 -- 172.21.15.72:0/630185578 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1cc00d08f0 msgr2=0x7f1cc01f3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:47.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.979+0000 7f1cc7590640 1 --2- 172.21.15.72:0/630185578 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1cc00d08f0 0x7f1cc01f3f30 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0038040 tx=0x7f1cb0004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:47.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.980+0000 7f1cc7590640 1 -- 172.21.15.72:0/630185578 shutdown_connections 2024-04-15T15:16:47.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.980+0000 7f1cc7590640 1 --2- 172.21.15.72:0/630185578 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1cac06c980 0x7f1cac06ee40 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.980+0000 7f1cc7590640 1 --2- 172.21.15.72:0/630185578 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1cc00d08f0 0x7f1cc01f3f30 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.980+0000 7f1cc7590640 1 --2- 172.21.15.72:0/630185578 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1cc00ce2f0 0x7f1cc01f39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.980+0000 7f1cc7590640 1 --2- 172.21.15.72:0/630185578 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1cc00cc7c0 0x7f1cc013e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:47.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.980+0000 7f1cc7590640 1 -- 172.21.15.72:0/630185578 >> 172.21.15.72:0/630185578 conn(0x7f1cc00b31d0 msgr2=0x7f1cc015d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:47.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.980+0000 7f1cc7590640 1 -- 172.21.15.72:0/630185578 shutdown_connections 2024-04-15T15:16:47.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:47.980+0000 7f1cc7590640 1 -- 172.21.15.72:0/630185578 wait complete. 2024-04-15T15:16:47.992 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:16:48.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.097+0000 7fe3fcf59640 1 Processor -- start 2024-04-15T15:16:48.098 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.097+0000 7fe3fcf59640 1 -- start start 2024-04-15T15:16:48.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.098+0000 7fe3fcf59640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe3f80d08f0 0x7fe3f80cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.098+0000 7fe3fcf59640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe3f80ce6d0 0x7fe3f80cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.098+0000 7fe3fcf59640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3f806bbb0 con 0x7fe3f80ce2f0 2024-04-15T15:16:48.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.098+0000 7fe3fcf59640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fe3f806b960 con 0x7fe3f80ce6d0 2024-04-15T15:16:48.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.098+0000 7fe3fcf59640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fe3f806be90 con 0x7fe3f80d08f0 2024-04-15T15:16:48.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.098+0000 7fe3f6d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe3f80ce6d0 0x7fe3f80cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.098+0000 7fe3f6d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe3f80ce6d0 0x7fe3f80cc5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43652/0 (socket says 172.21.15.72:43652) 2024-04-15T15:16:48.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.098+0000 7fe3f6d9d640 1 -- 172.21.15.72:0/3858442531 learned_addr learned my addr 172.21.15.72:0/3858442531 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:48.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.098+0000 7fe3effff640 1 --2- 172.21.15.72:0/3858442531 >> v2:172.21.15.175:3300/0 conn(0x7fe3f80d08f0 0x7fe3f80cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.098+0000 7fe3f6d9d640 1 -- 172.21.15.72:0/3858442531 >> v2:172.21.15.175:3300/0 conn(0x7fe3f80d08f0 msgr2=0x7fe3f80cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:48.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.098+0000 7fe3f6d9d640 1 --2- 172.21.15.72:0/3858442531 >> v2:172.21.15.175:3300/0 conn(0x7fe3f80d08f0 0x7fe3f80cc0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.098+0000 7fe3f6d9d640 1 -- 172.21.15.72:0/3858442531 >> v1:172.21.15.175:6789/0 conn(0x7fe3f80ce2f0 legacy=0x7fe3f815c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:48.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.098+0000 7fe3f6d9d640 1 -- 172.21.15.72:0/3858442531 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3f806c170 con 0x7fe3f80ce6d0 2024-04-15T15:16:48.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.098+0000 7fe3effff640 1 --2- 172.21.15.72:0/3858442531 >> v2:172.21.15.175:3300/0 conn(0x7fe3f80d08f0 0x7fe3f80cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:48.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.099+0000 7fe3f6d9d640 1 --2- 172.21.15.72:0/3858442531 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3f80ce6d0 0x7fe3f80cc5f0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fe3f80b56b0 tx=0x7fe3f002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfe3da125077f6fe server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:48.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.099+0000 7fe3f5d9b640 1 -- 172.21.15.72:0/3858442531 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f002ca80 con 0x7fe3f80ce6d0 2024-04-15T15:16:48.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.099+0000 7fe3f5d9b640 1 -- 172.21.15.72:0/3858442531 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3f002cc20 con 0x7fe3f80ce6d0 2024-04-15T15:16:48.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.099+0000 7fe3f5d9b640 1 -- 172.21.15.72:0/3858442531 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f00343f0 con 0x7fe3f80ce6d0 2024-04-15T15:16:48.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.099+0000 7fe3fcf59640 1 -- 172.21.15.72:0/3858442531 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3f80ce6d0 msgr2=0x7fe3f80cc5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.099+0000 7fe3fcf59640 1 --2- 172.21.15.72:0/3858442531 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3f80ce6d0 0x7fe3f80cc5f0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fe3f80b56b0 tx=0x7fe3f002bee0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.100+0000 7fe3fcf59640 1 -- 172.21.15.72:0/3858442531 shutdown_connections 2024-04-15T15:16:48.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.100+0000 7fe3fcf59640 1 --2- 172.21.15.72:0/3858442531 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3f80ce6d0 0x7fe3f80cc5f0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.100+0000 7fe3fcf59640 1 --2- 172.21.15.72:0/3858442531 >> v2:172.21.15.175:3300/0 conn(0x7fe3f80d08f0 0x7fe3f80cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.100+0000 7fe3fcf59640 1 -- 172.21.15.72:0/3858442531 >> 172.21.15.72:0/3858442531 conn(0x7fe3f80b31d0 msgr2=0x7fe3f80b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:48.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.100+0000 7fe3fcf59640 1 -- 172.21.15.72:0/3858442531 shutdown_connections 2024-04-15T15:16:48.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.100+0000 7fe3fcf59640 1 -- 172.21.15.72:0/3858442531 wait complete. 2024-04-15T15:16:48.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.101+0000 7fe3fcf59640 1 Processor -- start 2024-04-15T15:16:48.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.101+0000 7fe3fcf59640 1 -- start start 2024-04-15T15:16:48.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.102+0000 7fe3fcf59640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe3f80ce2f0 0x7fe3f80b0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.102+0000 7fe3fcf59640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3f80ce6d0 0x7fe3f81f23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.102+0000 7fe3fcf59640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3f80d08f0 0x7fe3f8157790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.102+0000 7fe3fcf59640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f8155780 con 0x7fe3f80ce6d0 2024-04-15T15:16:48.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.102+0000 7fe3fcf59640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fe3f8155600 con 0x7fe3f80d08f0 2024-04-15T15:16:48.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.102+0000 7fe3fcf59640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fe3f8155900 con 0x7fe3f80ce2f0 2024-04-15T15:16:48.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.102+0000 7fe3effff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3f80ce6d0 0x7fe3f81f23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.102+0000 7fe3effff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3f80ce6d0 0x7fe3f81f23e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43654/0 (socket says 172.21.15.72:43654) 2024-04-15T15:16:48.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.102+0000 7fe3effff640 1 -- 172.21.15.72:0/587642898 learned_addr learned my addr 172.21.15.72:0/587642898 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:48.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.102+0000 7fe3f6d9d640 1 --2- 172.21.15.72:0/587642898 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3f80d08f0 0x7fe3f8157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.102+0000 7fe3f659c640 1 --2- 172.21.15.72:0/587642898 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe3f80ce2f0 0x7fe3f80b0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.102+0000 7fe3effff640 1 -- 172.21.15.72:0/587642898 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe3f80ce2f0 msgr2=0x7fe3f80b0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.102+0000 7fe3effff640 1 --2- 172.21.15.72:0/587642898 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe3f80ce2f0 0x7fe3f80b0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.102+0000 7fe3effff640 1 -- 172.21.15.72:0/587642898 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3f80d08f0 msgr2=0x7fe3f8157790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.102+0000 7fe3effff640 1 --2- 172.21.15.72:0/587642898 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3f80d08f0 0x7fe3f8157790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.102+0000 7fe3effff640 1 -- 172.21.15.72:0/587642898 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3f81f2dc0 con 0x7fe3f80ce6d0 2024-04-15T15:16:48.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.102+0000 7fe3effff640 1 --2- 172.21.15.72:0/587642898 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3f80ce6d0 0x7fe3f81f23e0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fe3e000c7f0 tx=0x7fe3e000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:48.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.102+0000 7fe3eeffd640 1 -- 172.21.15.72:0/587642898 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e0015070 con 0x7fe3f80ce6d0 2024-04-15T15:16:48.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.102+0000 7fe3eeffd640 1 -- 172.21.15.72:0/587642898 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3e0007820 con 0x7fe3f80ce6d0 2024-04-15T15:16:48.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.102+0000 7fe3fcf59640 1 -- 172.21.15.72:0/587642898 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3f81f30a0 con 0x7fe3f80ce6d0 2024-04-15T15:16:48.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.102+0000 7fe3eeffd640 1 -- 172.21.15.72:0/587642898 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e0007bd0 con 0x7fe3f80ce6d0 2024-04-15T15:16:48.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.102+0000 7fe3fcf59640 1 -- 172.21.15.72:0/587642898 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3f80b05a0 con 0x7fe3f80ce6d0 2024-04-15T15:16:48.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.103+0000 7fe3fcf59640 1 -- 172.21.15.72:0/587642898 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3f8075c50 con 0x7fe3f80ce6d0 2024-04-15T15:16:48.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.104+0000 7fe3eeffd640 1 -- 172.21.15.72:0/587642898 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fe3e0018070 con 0x7fe3f80ce6d0 2024-04-15T15:16:48.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.104+0000 7fe3eeffd640 1 --2- 172.21.15.72:0/587642898 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe3d806c9b0 0x7fe3d806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.105+0000 7fe3f659c640 1 --2- 172.21.15.72:0/587642898 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe3d806c9b0 0x7fe3d806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.105+0000 7fe3eeffd640 1 -- 172.21.15.72:0/587642898 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7fe3e008bf10 con 0x7fe3f80ce6d0 2024-04-15T15:16:48.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.105+0000 7fe3f659c640 1 --2- 172.21.15.72:0/587642898 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe3d806c9b0 0x7fe3d806ee70 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8003d90 tx=0x7fe3e8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:48.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.109+0000 7fe3eeffd640 1 -- 172.21.15.72:0/587642898 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe3e005a0d0 con 0x7fe3f80ce6d0 2024-04-15T15:16:48.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.281+0000 7fe3fcf59640 1 -- 172.21.15.72:0/587642898 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe3f8159ad0 con 0x7fe3f80ce6d0 2024-04-15T15:16:48.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.281+0000 7fe3eeffd640 1 -- 172.21.15.72:0/587642898 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v41) ==== 104+0+11 (secure 0 0 0) 0x7fe3e00599d0 con 0x7fe3f80ce6d0 2024-04-15T15:16:48.283 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:16:48.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.283+0000 7fe3fcf59640 1 -- 172.21.15.72:0/587642898 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe3d806c9b0 msgr2=0x7fe3d806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.283+0000 7fe3fcf59640 1 --2- 172.21.15.72:0/587642898 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe3d806c9b0 0x7fe3d806ee70 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8003d90 tx=0x7fe3e8009040 comp rx=0 tx=0).stop 2024-04-15T15:16:48.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.283+0000 7fe3fcf59640 1 -- 172.21.15.72:0/587642898 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3f80ce6d0 msgr2=0x7fe3f81f23e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.283+0000 7fe3fcf59640 1 --2- 172.21.15.72:0/587642898 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3f80ce6d0 0x7fe3f81f23e0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fe3e000c7f0 tx=0x7fe3e000ccb0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.283+0000 7fe3fcf59640 1 -- 172.21.15.72:0/587642898 shutdown_connections 2024-04-15T15:16:48.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.283+0000 7fe3fcf59640 1 --2- 172.21.15.72:0/587642898 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe3d806c9b0 0x7fe3d806ee70 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.283+0000 7fe3fcf59640 1 --2- 172.21.15.72:0/587642898 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe3f80d08f0 0x7fe3f8157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.283+0000 7fe3fcf59640 1 --2- 172.21.15.72:0/587642898 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe3f80ce6d0 0x7fe3f81f23e0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.283+0000 7fe3fcf59640 1 --2- 172.21.15.72:0/587642898 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe3f80ce2f0 0x7fe3f80b0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.283+0000 7fe3fcf59640 1 -- 172.21.15.72:0/587642898 >> 172.21.15.72:0/587642898 conn(0x7fe3f80b31d0 msgr2=0x7fe3f806ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:48.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.284+0000 7fe3fcf59640 1 -- 172.21.15.72:0/587642898 shutdown_connections 2024-04-15T15:16:48.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.284+0000 7fe3fcf59640 1 -- 172.21.15.72:0/587642898 wait complete. 2024-04-15T15:16:48.296 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:16:48.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.400+0000 7f671547b640 1 Processor -- start 2024-04-15T15:16:48.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.400+0000 7f671547b640 1 -- start start 2024-04-15T15:16:48.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.401+0000 7f671547b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f671015e610 0x7f67100d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.401+0000 7f671547b640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f671015e9f0 0x7f67100d3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.401+0000 7f671547b640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67100d2320 con 0x7f671006c400 2024-04-15T15:16:48.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.401+0000 7f671547b640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f67100d31d0 con 0x7f671015e610 2024-04-15T15:16:48.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.401+0000 7f671547b640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f67100d25c0 con 0x7f671015e9f0 2024-04-15T15:16:48.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.401+0000 7f670effd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f671015e610 0x7f67100d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.401+0000 7f670effd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f671015e610 0x7f67100d0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43662/0 (socket says 172.21.15.72:43662) 2024-04-15T15:16:48.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.401+0000 7f670effd640 1 -- 172.21.15.72:0/716482449 learned_addr learned my addr 172.21.15.72:0/716482449 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:48.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.402+0000 7f670e7fc640 1 --2- 172.21.15.72:0/716482449 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f671015e9f0 0x7f67100d3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.402+0000 7f670effd640 1 -- 172.21.15.72:0/716482449 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f671015e9f0 msgr2=0x7f67100d3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:48.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.402+0000 7f670effd640 1 --2- 172.21.15.72:0/716482449 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f671015e9f0 0x7f67100d3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.402+0000 7f670effd640 1 -- 172.21.15.72:0/716482449 >> v1:172.21.15.175:6789/0 conn(0x7f671006c400 legacy=0x7f67100b7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.402+0000 7f670effd640 1 -- 172.21.15.72:0/716482449 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67100d28a0 con 0x7f671015e610 2024-04-15T15:16:48.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.403+0000 7f670effd640 1 --2- 172.21.15.72:0/716482449 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f671015e610 0x7f67100d0690 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f6700009e50 tx=0x7f670002bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c089b92021a1334 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:48.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.403+0000 7f670dffb640 1 -- 172.21.15.72:0/716482449 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f670002cb90 con 0x7f671015e610 2024-04-15T15:16:48.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.403+0000 7f670dffb640 1 -- 172.21.15.72:0/716482449 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f670002cd30 con 0x7f671015e610 2024-04-15T15:16:48.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.403+0000 7f670dffb640 1 -- 172.21.15.72:0/716482449 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6700034610 con 0x7f671015e610 2024-04-15T15:16:48.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.403+0000 7f671547b640 1 -- 172.21.15.72:0/716482449 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f671015e610 msgr2=0x7f67100d0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.403+0000 7f671547b640 1 --2- 172.21.15.72:0/716482449 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f671015e610 0x7f67100d0690 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f6700009e50 tx=0x7f670002bff0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.403+0000 7f671547b640 1 -- 172.21.15.72:0/716482449 shutdown_connections 2024-04-15T15:16:48.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.403+0000 7f671547b640 1 --2- 172.21.15.72:0/716482449 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f671015e9f0 0x7f67100d3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.403+0000 7f671547b640 1 --2- 172.21.15.72:0/716482449 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f671015e610 0x7f67100d0690 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.403+0000 7f671547b640 1 -- 172.21.15.72:0/716482449 >> 172.21.15.72:0/716482449 conn(0x7f67100d4750 msgr2=0x7f67100d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:48.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.403+0000 7f671547b640 1 -- 172.21.15.72:0/716482449 shutdown_connections 2024-04-15T15:16:48.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.403+0000 7f671547b640 1 -- 172.21.15.72:0/716482449 wait complete. 2024-04-15T15:16:48.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.405+0000 7f671547b640 1 Processor -- start 2024-04-15T15:16:48.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.405+0000 7f671547b640 1 -- start start 2024-04-15T15:16:48.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.405+0000 7f671547b640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f671006c400 0x7f671015b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.405+0000 7f671547b640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f671015e610 0x7f67100cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.405+0000 7f671547b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f671015e9f0 0x7f671015fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.405+0000 7f671547b640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f671006d130 con 0x7f671015e9f0 2024-04-15T15:16:48.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.405+0000 7f671547b640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f67100d2e60 con 0x7f671015e610 2024-04-15T15:16:48.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.405+0000 7f671547b640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f67100d0bd0 con 0x7f671006c400 2024-04-15T15:16:48.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.405+0000 7f670f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f671015e9f0 0x7f671015fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.406+0000 7f670f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f671015e9f0 0x7f671015fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43668/0 (socket says 172.21.15.72:43668) 2024-04-15T15:16:48.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.406+0000 7f670f7fe640 1 -- 172.21.15.72:0/2079035065 learned_addr learned my addr 172.21.15.72:0/2079035065 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:48.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.406+0000 7f670f7fe640 1 -- 172.21.15.72:0/2079035065 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f671006c400 msgr2=0x7f671015b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.406+0000 7f670f7fe640 1 --2- 172.21.15.72:0/2079035065 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f671006c400 0x7f671015b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.406+0000 7f670f7fe640 1 -- 172.21.15.72:0/2079035065 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f671015e610 msgr2=0x7f67100cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.406+0000 7f670f7fe640 1 --2- 172.21.15.72:0/2079035065 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f671015e610 0x7f67100cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.406+0000 7f670f7fe640 1 -- 172.21.15.72:0/2079035065 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67100ccb40 con 0x7f671015e9f0 2024-04-15T15:16:48.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.406+0000 7f670f7fe640 1 --2- 172.21.15.72:0/2079035065 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f671015e9f0 0x7f671015fd40 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f670800c400 tx=0x7f670800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:48.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.406+0000 7f67077fe640 1 -- 172.21.15.72:0/2079035065 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6708017070 con 0x7f671015e9f0 2024-04-15T15:16:48.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.406+0000 7f67077fe640 1 -- 172.21.15.72:0/2079035065 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67080052d0 con 0x7f671015e9f0 2024-04-15T15:16:48.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.406+0000 7f671547b640 1 -- 172.21.15.72:0/2079035065 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67100cce20 con 0x7f671015e9f0 2024-04-15T15:16:48.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.406+0000 7f67077fe640 1 -- 172.21.15.72:0/2079035065 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6708003810 con 0x7f671015e9f0 2024-04-15T15:16:48.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.406+0000 7f671547b640 1 -- 172.21.15.72:0/2079035065 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67100cd3e0 con 0x7f671015e9f0 2024-04-15T15:16:48.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.407+0000 7f671547b640 1 -- 172.21.15.72:0/2079035065 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6710075c50 con 0x7f671015e9f0 2024-04-15T15:16:48.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.408+0000 7f67077fe640 1 -- 172.21.15.72:0/2079035065 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f670801a070 con 0x7f671015e9f0 2024-04-15T15:16:48.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.408+0000 7f67077fe640 1 --2- 172.21.15.72:0/2079035065 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f66f006c980 0x7f66f006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.408+0000 7f67077fe640 1 -- 172.21.15.72:0/2079035065 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7f670808b110 con 0x7f671015e9f0 2024-04-15T15:16:48.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.408+0000 7f670effd640 1 --2- 172.21.15.72:0/2079035065 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f66f006c980 0x7f66f006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.409+0000 7f670effd640 1 --2- 172.21.15.72:0/2079035065 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f66f006c980 0x7f66f006ee40 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f6700035000 tx=0x7f670002e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:48.414 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.413+0000 7f67077fe640 1 -- 172.21.15.72:0/2079035065 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6708059360 con 0x7f671015e9f0 2024-04-15T15:16:48.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.592+0000 7f671547b640 1 -- 172.21.15.72:0/2079035065 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f67100d3dd0 con 0x7f671015e9f0 2024-04-15T15:16:48.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.592+0000 7f67077fe640 1 -- 172.21.15.72:0/2079035065 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v41) ==== 100+0+11 (secure 0 0 0) 0x7f6708058c60 con 0x7f671015e9f0 2024-04-15T15:16:48.594 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-04-15T15:16:48.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.594+0000 7f671547b640 1 -- 172.21.15.72:0/2079035065 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f66f006c980 msgr2=0x7f66f006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.594+0000 7f671547b640 1 --2- 172.21.15.72:0/2079035065 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f66f006c980 0x7f66f006ee40 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f6700035000 tx=0x7f670002e000 comp rx=0 tx=0).stop 2024-04-15T15:16:48.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.594+0000 7f671547b640 1 -- 172.21.15.72:0/2079035065 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f671015e9f0 msgr2=0x7f671015fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.594+0000 7f671547b640 1 --2- 172.21.15.72:0/2079035065 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f671015e9f0 0x7f671015fd40 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f670800c400 tx=0x7f670800c8c0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.594+0000 7f671547b640 1 -- 172.21.15.72:0/2079035065 shutdown_connections 2024-04-15T15:16:48.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.594+0000 7f671547b640 1 --2- 172.21.15.72:0/2079035065 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f66f006c980 0x7f66f006ee40 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.594+0000 7f671547b640 1 --2- 172.21.15.72:0/2079035065 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f671015e9f0 0x7f671015fd40 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.594+0000 7f671547b640 1 --2- 172.21.15.72:0/2079035065 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f671015e610 0x7f67100cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.594+0000 7f671547b640 1 --2- 172.21.15.72:0/2079035065 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f671006c400 0x7f671015b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.594+0000 7f671547b640 1 -- 172.21.15.72:0/2079035065 >> 172.21.15.72:0/2079035065 conn(0x7f67100d4750 msgr2=0x7f671015dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:48.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.594+0000 7f671547b640 1 -- 172.21.15.72:0/2079035065 shutdown_connections 2024-04-15T15:16:48.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.594+0000 7f671547b640 1 -- 172.21.15.72:0/2079035065 wait complete. 2024-04-15T15:16:48.607 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:16:48.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.710+0000 7f2e2c042640 1 Processor -- start 2024-04-15T15:16:48.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.710+0000 7f2e2c042640 1 -- start start 2024-04-15T15:16:48.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.711+0000 7f2e2c042640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2e240b5410 0x7f2e240b57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.711+0000 7f2e2c042640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2e240afae0 0x7f2e240cf960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.711+0000 7f2e2c042640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e240d07c0 con 0x7f2e240b0050 2024-04-15T15:16:48.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.711+0000 7f2e2c042640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f2e240d0330 con 0x7f2e240afae0 2024-04-15T15:16:48.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.711+0000 7f2e2c042640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f2e240b5d30 con 0x7f2e240b5410 2024-04-15T15:16:48.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.712+0000 7f2e2a5df640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2e240afae0 0x7f2e240cf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.712+0000 7f2e29dde640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2e240b5410 0x7f2e240b57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.712+0000 7f2e295dd640 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f2e240b0050 0x7f2e2406c340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.72:43684/0 (socket says 172.21.15.72:43684) 2024-04-15T15:16:48.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.712+0000 7f2e295dd640 1 -- 172.21.15.72:0/3525518254 learned_addr learned my addr 172.21.15.72:0/3525518254 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:48.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.712+0000 7f2e2a5df640 1 -- 172.21.15.72:0/3525518254 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2e240b5410 msgr2=0x7f2e240b57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:48.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.712+0000 7f2e2a5df640 1 --2- 172.21.15.72:0/3525518254 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2e240b5410 0x7f2e240b57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.712+0000 7f2e2a5df640 1 -- 172.21.15.72:0/3525518254 >> v1:172.21.15.175:6789/0 conn(0x7f2e240b0050 legacy=0x7f2e2406c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.712+0000 7f2e2a5df640 1 -- 172.21.15.72:0/3525518254 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e240b3aa0 con 0x7f2e240afae0 2024-04-15T15:16:48.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.712+0000 7f2e29dde640 1 --2- 172.21.15.72:0/3525518254 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2e240b5410 0x7f2e240b57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:48.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.713+0000 7f2e2a5df640 1 --2- 172.21.15.72:0/3525518254 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2e240afae0 0x7f2e240cf960 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c0097a0 tx=0x7f2e1c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81581080f3e8d6dc server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:48.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.713+0000 7f2e28ddc640 1 -- 172.21.15.72:0/3525518254 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c0315a0 con 0x7f2e240afae0 2024-04-15T15:16:48.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.713+0000 7f2e28ddc640 1 -- 172.21.15.72:0/3525518254 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e1c037070 con 0x7f2e240afae0 2024-04-15T15:16:48.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.713+0000 7f2e28ddc640 1 -- 172.21.15.72:0/3525518254 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e1c031960 con 0x7f2e240afae0 2024-04-15T15:16:48.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.714+0000 7f2e2c042640 1 -- 172.21.15.72:0/3525518254 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2e240afae0 msgr2=0x7f2e240cf960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.714+0000 7f2e2c042640 1 --2- 172.21.15.72:0/3525518254 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2e240afae0 0x7f2e240cf960 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c0097a0 tx=0x7f2e1c02b840 comp rx=0 tx=0).stop 2024-04-15T15:16:48.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.714+0000 7f2e2c042640 1 -- 172.21.15.72:0/3525518254 shutdown_connections 2024-04-15T15:16:48.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.714+0000 7f2e2c042640 1 --2- 172.21.15.72:0/3525518254 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2e240afae0 0x7f2e240cf960 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.714+0000 7f2e2c042640 1 --2- 172.21.15.72:0/3525518254 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2e240b5410 0x7f2e240b57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.714+0000 7f2e2c042640 1 -- 172.21.15.72:0/3525518254 >> 172.21.15.72:0/3525518254 conn(0x7f2e240d0a50 msgr2=0x7f2e240d0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:48.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.714+0000 7f2e2c042640 1 -- 172.21.15.72:0/3525518254 shutdown_connections 2024-04-15T15:16:48.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.714+0000 7f2e2c042640 1 -- 172.21.15.72:0/3525518254 wait complete. 2024-04-15T15:16:48.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.715+0000 7f2e2c042640 1 Processor -- start 2024-04-15T15:16:48.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.715+0000 7f2e2c042640 1 -- start start 2024-04-15T15:16:48.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.716+0000 7f2e2c042640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2e240afae0 0x7f2e241475e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.716+0000 7f2e2c042640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e240b0050 0x7f2e2415b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.716+0000 7f2e2c042640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2e240b5410 0x7f2e24149fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.716+0000 7f2e2c042640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2e240d18f0 con 0x7f2e240b0050 2024-04-15T15:16:48.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.716+0000 7f2e2c042640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f2e240d1610 con 0x7f2e240afae0 2024-04-15T15:16:48.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.716+0000 7f2e2c042640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f2e240cfea0 con 0x7f2e240b5410 2024-04-15T15:16:48.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.716+0000 7f2e295dd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e240b0050 0x7f2e2415b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.716+0000 7f2e295dd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e240b0050 0x7f2e2415b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43682/0 (socket says 172.21.15.72:43682) 2024-04-15T15:16:48.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.716+0000 7f2e295dd640 1 -- 172.21.15.72:0/4094139880 learned_addr learned my addr 172.21.15.72:0/4094139880 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:48.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.716+0000 7f2e29dde640 1 --2- 172.21.15.72:0/4094139880 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2e240afae0 0x7f2e241475e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.716+0000 7f2e2a5df640 1 --2- 172.21.15.72:0/4094139880 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2e240b5410 0x7f2e24149fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.716+0000 7f2e295dd640 1 -- 172.21.15.72:0/4094139880 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2e240b5410 msgr2=0x7f2e24149fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.716+0000 7f2e295dd640 1 --2- 172.21.15.72:0/4094139880 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2e240b5410 0x7f2e24149fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.716+0000 7f2e295dd640 1 -- 172.21.15.72:0/4094139880 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2e240afae0 msgr2=0x7f2e241475e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.716+0000 7f2e295dd640 1 --2- 172.21.15.72:0/4094139880 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2e240afae0 0x7f2e241475e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.716+0000 7f2e295dd640 1 -- 172.21.15.72:0/4094139880 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e2415bb20 con 0x7f2e240b0050 2024-04-15T15:16:48.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.716+0000 7f2e29dde640 1 --2- 172.21.15.72:0/4094139880 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2e240afae0 0x7f2e241475e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:48.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.716+0000 7f2e2a5df640 1 --2- 172.21.15.72:0/4094139880 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2e240b5410 0x7f2e24149fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:48.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.716+0000 7f2e295dd640 1 --2- 172.21.15.72:0/4094139880 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e240b0050 0x7f2e2415b060 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f2e1400c4b0 tx=0x7f2e1400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:48.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.716+0000 7f2e1a7fc640 1 -- 172.21.15.72:0/4094139880 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e14017070 con 0x7f2e240b0050 2024-04-15T15:16:48.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.716+0000 7f2e1a7fc640 1 -- 172.21.15.72:0/4094139880 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e14005ce0 con 0x7f2e240b0050 2024-04-15T15:16:48.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.716+0000 7f2e2c042640 1 -- 172.21.15.72:0/4094139880 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e2415be00 con 0x7f2e240b0050 2024-04-15T15:16:48.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.716+0000 7f2e1a7fc640 1 -- 172.21.15.72:0/4094139880 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e14007270 con 0x7f2e240b0050 2024-04-15T15:16:48.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.717+0000 7f2e2c042640 1 -- 172.21.15.72:0/4094139880 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e24157550 con 0x7f2e240b0050 2024-04-15T15:16:48.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.717+0000 7f2e2c042640 1 -- 172.21.15.72:0/4094139880 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e24161880 con 0x7f2e240b0050 2024-04-15T15:16:48.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.718+0000 7f2e1a7fc640 1 -- 172.21.15.72:0/4094139880 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f2e1401a070 con 0x7f2e240b0050 2024-04-15T15:16:48.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.718+0000 7f2e1a7fc640 1 --2- 172.21.15.72:0/4094139880 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2e0806c980 0x7f2e0806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.718+0000 7f2e1a7fc640 1 -- 172.21.15.72:0/4094139880 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7f2e1408b010 con 0x7f2e240b0050 2024-04-15T15:16:48.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.718+0000 7f2e29dde640 1 --2- 172.21.15.72:0/4094139880 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2e0806c980 0x7f2e0806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.722 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.719+0000 7f2e29dde640 1 --2- 172.21.15.72:0/4094139880 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2e0806c980 0x7f2e0806ee40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2e200074d0 tx=0x7f2e2000b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:48.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.723+0000 7f2e1a7fc640 1 -- 172.21.15.72:0/4094139880 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2e24161880 con 0x7f2e240b0050 2024-04-15T15:16:48.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.911+0000 7f2e2c042640 1 -- 172.21.15.72:0/4094139880 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2e24075c50 con 0x7f2e240b0050 2024-04-15T15:16:48.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.912+0000 7f2e1a7fc640 1 -- 172.21.15.72:0/4094139880 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v41) ==== 80+0+122 (secure 0 0 0) 0x7f2e14058c00 con 0x7f2e240b0050 2024-04-15T15:16:48.913 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6836/4270793613 2024-04-16T15:16:41.714817+0000 2024-04-15T15:16:48.913 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6834/4270793613 2024-04-16T15:16:41.714817+0000 2024-04-15T15:16:48.914 INFO:teuthology.orchestra.run.smithi072.stderr:listed 2 entries 2024-04-15T15:16:48.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.913+0000 7f2e2c042640 1 -- 172.21.15.72:0/4094139880 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2e0806c980 msgr2=0x7f2e0806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.913+0000 7f2e2c042640 1 --2- 172.21.15.72:0/4094139880 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2e0806c980 0x7f2e0806ee40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2e200074d0 tx=0x7f2e2000b040 comp rx=0 tx=0).stop 2024-04-15T15:16:48.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.913+0000 7f2e2c042640 1 -- 172.21.15.72:0/4094139880 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e240b0050 msgr2=0x7f2e2415b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.915 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.913+0000 7f2e2c042640 1 --2- 172.21.15.72:0/4094139880 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e240b0050 0x7f2e2415b060 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f2e1400c4b0 tx=0x7f2e1400c970 comp rx=0 tx=0).stop 2024-04-15T15:16:48.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.914+0000 7f2e2c042640 1 -- 172.21.15.72:0/4094139880 shutdown_connections 2024-04-15T15:16:48.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.914+0000 7f2e2c042640 1 --2- 172.21.15.72:0/4094139880 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2e0806c980 0x7f2e0806ee40 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.914+0000 7f2e2c042640 1 --2- 172.21.15.72:0/4094139880 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2e240b5410 0x7f2e24149fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.914+0000 7f2e2c042640 1 --2- 172.21.15.72:0/4094139880 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2e240b0050 0x7f2e2415b060 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.914+0000 7f2e2c042640 1 --2- 172.21.15.72:0/4094139880 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2e240afae0 0x7f2e241475e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.914+0000 7f2e2c042640 1 -- 172.21.15.72:0/4094139880 >> 172.21.15.72:0/4094139880 conn(0x7f2e240d0a50 msgr2=0x7f2e2415e670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:48.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.914+0000 7f2e2c042640 1 -- 172.21.15.72:0/4094139880 shutdown_connections 2024-04-15T15:16:48.916 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:48.914+0000 7f2e2c042640 1 -- 172.21.15.72:0/4094139880 wait complete. 2024-04-15T15:16:48.926 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi072.front.sepia.ceph.com... 2024-04-15T15:16:48.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:48.926 DEBUG:teuthology.orchestra.run.smithi072:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-15T15:16:48.954 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi072.stderr:ceph-fuse[56093]: fuse finished with error 0 and tester_r 0 2024-04-15T15:16:48.956 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi072.front.sepia.ceph.com 2024-04-15T15:16:48.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:48.956 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-15T15:16:49.124 INFO:tasks.cephfs.mount:Terminating background process 2024-04-15T15:16:49.125 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:16:49.125 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:16:49.145 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:49.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.267+0000 7f666c4b2640 1 Processor -- start 2024-04-15T15:16:49.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.268+0000 7f666c4b2640 1 -- start start 2024-04-15T15:16:49.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.268+0000 7f666c4b2640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f66640cd2a0 0x7f66640ad0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.269+0000 7f666c4b2640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f666406f8f0 con 0x7f66640af400 2024-04-15T15:16:49.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.269+0000 7f666c4b2640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f666415ef80 con 0x7f666415eba0 2024-04-15T15:16:49.270 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.269+0000 7f666c4b2640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f66640ce2b0 con 0x7f66640cd2a0 2024-04-15T15:16:49.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.269+0000 7f666aa4f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f66640cd2a0 0x7f66640ad0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.269+0000 7f666aa4f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f66640cd2a0 0x7f66640ad0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43686/0 (socket says 172.21.15.72:43686) 2024-04-15T15:16:49.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.269+0000 7f666aa4f640 1 -- 172.21.15.72:0/1396626395 learned_addr learned my addr 172.21.15.72:0/1396626395 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:49.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.269+0000 7f666aa4f640 1 -- 172.21.15.72:0/1396626395 >> v1:172.21.15.175:6789/0 conn(0x7f666415eba0 legacy=0x7f666406f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:49.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.269+0000 7f666aa4f640 1 -- 172.21.15.72:0/1396626395 >> v1:172.21.15.72:6789/0 conn(0x7f66640af400 legacy=0x7f66640ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.269+0000 7f666aa4f640 1 -- 172.21.15.72:0/1396626395 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66640ce590 con 0x7f66640cd2a0 2024-04-15T15:16:49.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.270+0000 7f666aa4f640 1 --2- 172.21.15.72:0/1396626395 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66640cd2a0 0x7f66640ad0b0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f665c009e50 tx=0x7f665c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23f66f1f8019e5cb server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:49.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.270+0000 7f666924c640 1 -- 172.21.15.72:0/1396626395 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f665c02ca80 con 0x7f66640cd2a0 2024-04-15T15:16:49.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.270+0000 7f666924c640 1 -- 172.21.15.72:0/1396626395 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f665c02cc20 con 0x7f66640cd2a0 2024-04-15T15:16:49.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.270+0000 7f666924c640 1 -- 172.21.15.72:0/1396626395 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f665c0343f0 con 0x7f66640cd2a0 2024-04-15T15:16:49.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.270+0000 7f666c4b2640 1 -- 172.21.15.72:0/1396626395 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66640cd2a0 msgr2=0x7f66640ad0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.270+0000 7f666c4b2640 1 --2- 172.21.15.72:0/1396626395 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66640cd2a0 0x7f66640ad0b0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f665c009e50 tx=0x7f665c02bee0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.270+0000 7f666c4b2640 1 -- 172.21.15.72:0/1396626395 shutdown_connections 2024-04-15T15:16:49.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.270+0000 7f666c4b2640 1 --2- 172.21.15.72:0/1396626395 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66640cd2a0 0x7f66640ad0b0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.270+0000 7f666c4b2640 1 -- 172.21.15.72:0/1396626395 >> 172.21.15.72:0/1396626395 conn(0x7f66640b1dd0 msgr2=0x7f66640b21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:49.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.271+0000 7f666c4b2640 1 -- 172.21.15.72:0/1396626395 shutdown_connections 2024-04-15T15:16:49.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.271+0000 7f666c4b2640 1 -- 172.21.15.72:0/1396626395 wait complete. 2024-04-15T15:16:49.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.272+0000 7f666c4b2640 1 Processor -- start 2024-04-15T15:16:49.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.272+0000 7f666c4b2640 1 -- start start 2024-04-15T15:16:49.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.272+0000 7f666c4b2640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f66640af400 0x7f6664149a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.272+0000 7f666c4b2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66640cd2a0 0x7f666415c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.272+0000 7f666c4b2640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f666415eba0 0x7f666415c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.272+0000 7f666c4b2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f66640b2b80 con 0x7f66640cd2a0 2024-04-15T15:16:49.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.272+0000 7f666c4b2640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f666406ecc0 con 0x7f666415eba0 2024-04-15T15:16:49.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.272+0000 7f666c4b2640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f66640ceb50 con 0x7f66640af400 2024-04-15T15:16:49.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.273+0000 7f6669a4d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66640cd2a0 0x7f666415c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.273+0000 7f6669a4d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66640cd2a0 0x7f666415c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43696/0 (socket says 172.21.15.72:43696) 2024-04-15T15:16:49.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.273+0000 7f6669a4d640 1 -- 172.21.15.72:0/205154602 learned_addr learned my addr 172.21.15.72:0/205154602 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:49.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.273+0000 7f666aa4f640 1 --2- 172.21.15.72:0/205154602 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f666415eba0 0x7f666415c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.273+0000 7f666a24e640 1 --2- 172.21.15.72:0/205154602 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f66640af400 0x7f6664149a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.273+0000 7f6669a4d640 1 -- 172.21.15.72:0/205154602 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f66640af400 msgr2=0x7f6664149a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.273+0000 7f6669a4d640 1 --2- 172.21.15.72:0/205154602 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f66640af400 0x7f6664149a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.273+0000 7f6669a4d640 1 -- 172.21.15.72:0/205154602 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f666415eba0 msgr2=0x7f666415c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.273+0000 7f6669a4d640 1 --2- 172.21.15.72:0/205154602 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f666415eba0 0x7f666415c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.273+0000 7f6669a4d640 1 -- 172.21.15.72:0/205154602 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66641588f0 con 0x7f66640cd2a0 2024-04-15T15:16:49.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.273+0000 7f666a24e640 1 --2- 172.21.15.72:0/205154602 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f66640af400 0x7f6664149a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:49.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.273+0000 7f6669a4d640 1 --2- 172.21.15.72:0/205154602 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66640cd2a0 0x7f666415c310 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f666000c3c0 tx=0x7f666000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:49.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.273+0000 7f665affd640 1 -- 172.21.15.72:0/205154602 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66600052d0 con 0x7f66640cd2a0 2024-04-15T15:16:49.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.273+0000 7f665affd640 1 -- 172.21.15.72:0/205154602 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6660005490 con 0x7f66640cd2a0 2024-04-15T15:16:49.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.273+0000 7f666c4b2640 1 -- 172.21.15.72:0/205154602 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6664158bd0 con 0x7f66640cd2a0 2024-04-15T15:16:49.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.273+0000 7f665affd640 1 -- 172.21.15.72:0/205154602 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6660007260 con 0x7f66640cd2a0 2024-04-15T15:16:49.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.273+0000 7f666c4b2640 1 -- 172.21.15.72:0/205154602 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6664154000 con 0x7f66640cd2a0 2024-04-15T15:16:49.278 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.274+0000 7f666c4b2640 1 -- 172.21.15.72:0/205154602 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6664159a20 con 0x7f66640cd2a0 2024-04-15T15:16:49.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.275+0000 7f665affd640 1 -- 172.21.15.72:0/205154602 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f6660007400 con 0x7f66640cd2a0 2024-04-15T15:16:49.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.275+0000 7f665affd640 1 --2- 172.21.15.72:0/205154602 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f664806c9b0 0x7f664806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.275+0000 7f665affd640 1 -- 172.21.15.72:0/205154602 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7f666008b1a0 con 0x7f66640cd2a0 2024-04-15T15:16:49.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.275+0000 7f666a24e640 1 --2- 172.21.15.72:0/205154602 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f664806c9b0 0x7f664806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.279 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.276+0000 7f666a24e640 1 --2- 172.21.15.72:0/205154602 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f664806c9b0 0x7f664806ee70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6654003eb0 tx=0x7f6654009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:49.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.279+0000 7f665affd640 1 -- 172.21.15.72:0/205154602 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6664159a20 con 0x7f66640cd2a0 2024-04-15T15:16:49.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.457+0000 7f666c4b2640 1 -- 172.21.15.72:0/205154602 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6664154200 con 0x7f66640cd2a0 2024-04-15T15:16:49.459 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.458+0000 7f665affd640 1 -- 172.21.15.72:0/205154602 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) ==== 74+0+17120 (secure 0 0 0) 0x7f6664154200 con 0x7f66640cd2a0 2024-04-15T15:16:49.459 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:16:49.460 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":41,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:16:41.714841+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:05.551499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:06.003152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.26","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:6836/4270793613":"2024-04-16T15:16:41.714817+0000","172.21.15.72:6834/4270793613":"2024-04-16T15:16:41.714817+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:49.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.460+0000 7f666c4b2640 1 -- 172.21.15.72:0/205154602 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f664806c9b0 msgr2=0x7f664806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.460+0000 7f666c4b2640 1 --2- 172.21.15.72:0/205154602 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f664806c9b0 0x7f664806ee70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6654003eb0 tx=0x7f6654009040 comp rx=0 tx=0).stop 2024-04-15T15:16:49.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.460+0000 7f666c4b2640 1 -- 172.21.15.72:0/205154602 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66640cd2a0 msgr2=0x7f666415c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.460+0000 7f666c4b2640 1 --2- 172.21.15.72:0/205154602 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66640cd2a0 0x7f666415c310 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f666000c3c0 tx=0x7f666000c880 comp rx=0 tx=0).stop 2024-04-15T15:16:49.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.460+0000 7f666c4b2640 1 -- 172.21.15.72:0/205154602 shutdown_connections 2024-04-15T15:16:49.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.460+0000 7f666c4b2640 1 --2- 172.21.15.72:0/205154602 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f664806c9b0 0x7f664806ee70 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.460+0000 7f666c4b2640 1 --2- 172.21.15.72:0/205154602 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f666415eba0 0x7f666415c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.460+0000 7f666c4b2640 1 --2- 172.21.15.72:0/205154602 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f66640cd2a0 0x7f666415c310 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.460+0000 7f666c4b2640 1 --2- 172.21.15.72:0/205154602 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f66640af400 0x7f6664149a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.460+0000 7f666c4b2640 1 -- 172.21.15.72:0/205154602 >> 172.21.15.72:0/205154602 conn(0x7f66640b1dd0 msgr2=0x7f666406b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:49.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.460+0000 7f666c4b2640 1 -- 172.21.15.72:0/205154602 shutdown_connections 2024-04-15T15:16:49.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.460+0000 7f666c4b2640 1 -- 172.21.15.72:0/205154602 wait complete. 2024-04-15T15:16:49.473 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:39.359102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:16:05.551499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:16:06.003152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:16:49.473 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:49.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.578+0000 7fad87382640 1 Processor -- start 2024-04-15T15:16:49.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.578+0000 7fad87382640 1 -- start start 2024-04-15T15:16:49.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.579+0000 7fad87382640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fad800b5410 0x7fad800b57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.579+0000 7fad87382640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fad800b0050 0x7fad8006c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.579+0000 7fad87382640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad800d07c0 con 0x7fad800afae0 2024-04-15T15:16:49.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.579+0000 7fad87382640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fad800d0330 con 0x7fad800b0050 2024-04-15T15:16:49.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.579+0000 7fad87382640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fad800b5d30 con 0x7fad800b5410 2024-04-15T15:16:49.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.579+0000 7fad8491d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fad800b0050 0x7fad8006c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.579+0000 7fad8491d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fad800b0050 0x7fad8006c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43706/0 (socket says 172.21.15.72:43706) 2024-04-15T15:16:49.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.579+0000 7fad8491d640 1 -- 172.21.15.72:0/3753529252 learned_addr learned my addr 172.21.15.72:0/3753529252 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:49.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.579+0000 7fad8491d640 1 -- 172.21.15.72:0/3753529252 >> v2:172.21.15.175:3300/0 conn(0x7fad800b5410 msgr2=0x7fad800b57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:49.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.579+0000 7fad8491d640 1 --2- 172.21.15.72:0/3753529252 >> v2:172.21.15.175:3300/0 conn(0x7fad800b5410 0x7fad800b57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.579+0000 7fad8491d640 1 -- 172.21.15.72:0/3753529252 >> v1:172.21.15.175:6789/0 conn(0x7fad800afae0 legacy=0x7fad800ce950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:49.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.579+0000 7fad8491d640 1 -- 172.21.15.72:0/3753529252 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad800b3aa0 con 0x7fad800b0050 2024-04-15T15:16:49.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.580+0000 7fad8491d640 1 --2- 172.21.15.72:0/3753529252 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad800b0050 0x7fad8006c340 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fad80073da0 tx=0x7fad7402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=950ea08c9264b28d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:49.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.580+0000 7fad73fff640 1 -- 172.21.15.72:0/3753529252 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad7402c9f0 con 0x7fad800b0050 2024-04-15T15:16:49.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.580+0000 7fad73fff640 1 -- 172.21.15.72:0/3753529252 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad7402cb90 con 0x7fad800b0050 2024-04-15T15:16:49.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.580+0000 7fad73fff640 1 -- 172.21.15.72:0/3753529252 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad740343f0 con 0x7fad800b0050 2024-04-15T15:16:49.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.580+0000 7fad87382640 1 -- 172.21.15.72:0/3753529252 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad800b0050 msgr2=0x7fad8006c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.580+0000 7fad87382640 1 --2- 172.21.15.72:0/3753529252 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad800b0050 0x7fad8006c340 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fad80073da0 tx=0x7fad7402be50 comp rx=0 tx=0).stop 2024-04-15T15:16:49.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.581+0000 7fad87382640 1 -- 172.21.15.72:0/3753529252 shutdown_connections 2024-04-15T15:16:49.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.581+0000 7fad87382640 1 --2- 172.21.15.72:0/3753529252 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad800b0050 0x7fad8006c340 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.581+0000 7fad87382640 1 --2- 172.21.15.72:0/3753529252 >> v2:172.21.15.175:3300/0 conn(0x7fad800b5410 0x7fad800b57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.581+0000 7fad87382640 1 -- 172.21.15.72:0/3753529252 >> 172.21.15.72:0/3753529252 conn(0x7fad800d0a50 msgr2=0x7fad800d0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:49.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.581+0000 7fad87382640 1 -- 172.21.15.72:0/3753529252 shutdown_connections 2024-04-15T15:16:49.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.581+0000 7fad87382640 1 -- 172.21.15.72:0/3753529252 wait complete. 2024-04-15T15:16:49.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.582+0000 7fad87382640 1 Processor -- start 2024-04-15T15:16:49.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.582+0000 7fad87382640 1 -- start start 2024-04-15T15:16:49.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.582+0000 7fad87382640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad800afae0 0x7fad80145620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.582+0000 7fad87382640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fad800b0050 0x7fad80144e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.582+0000 7fad87382640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fad800b5410 0x7fad8015c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.582+0000 7fad87382640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fad800d18f0 con 0x7fad800afae0 2024-04-15T15:16:49.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.582+0000 7fad87382640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fad800d1610 con 0x7fad800b5410 2024-04-15T15:16:49.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.582+0000 7fad87382640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fad8006db40 con 0x7fad800b0050 2024-04-15T15:16:49.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.583+0000 7fad8511e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad800afae0 0x7fad80145620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.586 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.583+0000 7fad8511e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad800afae0 0x7fad80145620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43714/0 (socket says 172.21.15.72:43714) 2024-04-15T15:16:49.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.583+0000 7fad8511e640 1 -- 172.21.15.72:0/3348197820 learned_addr learned my addr 172.21.15.72:0/3348197820 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:49.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.583+0000 7fad8511e640 1 -- 172.21.15.72:0/3348197820 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fad800b0050 msgr2=0x7fad80144e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.583+0000 7fad8511e640 1 --2- 172.21.15.72:0/3348197820 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fad800b0050 0x7fad80144e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.583+0000 7fad8511e640 1 -- 172.21.15.72:0/3348197820 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fad800b5410 msgr2=0x7fad8015c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.583+0000 7fad8511e640 1 --2- 172.21.15.72:0/3348197820 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fad800b5410 0x7fad8015c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.583+0000 7fad8511e640 1 -- 172.21.15.72:0/3348197820 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad8015cf90 con 0x7fad800afae0 2024-04-15T15:16:49.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.583+0000 7fad8511e640 1 --2- 172.21.15.72:0/3348197820 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad800afae0 0x7fad80145620 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fad7800b4a0 tx=0x7fad7800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:49.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.583+0000 7fad71ffb640 1 -- 172.21.15.72:0/3348197820 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad78014070 con 0x7fad800afae0 2024-04-15T15:16:49.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.583+0000 7fad71ffb640 1 -- 172.21.15.72:0/3348197820 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad78010610 con 0x7fad800afae0 2024-04-15T15:16:49.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.583+0000 7fad87382640 1 -- 172.21.15.72:0/3348197820 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad8015d270 con 0x7fad800afae0 2024-04-15T15:16:49.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.583+0000 7fad71ffb640 1 -- 172.21.15.72:0/3348197820 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad78010a00 con 0x7fad800afae0 2024-04-15T15:16:49.587 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.583+0000 7fad87382640 1 -- 172.21.15.72:0/3348197820 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad801e80f0 con 0x7fad800afae0 2024-04-15T15:16:49.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.584+0000 7fad87382640 1 -- 172.21.15.72:0/3348197820 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad80075c50 con 0x7fad800afae0 2024-04-15T15:16:49.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.585+0000 7fad71ffb640 1 -- 172.21.15.72:0/3348197820 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fad78017070 con 0x7fad800afae0 2024-04-15T15:16:49.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.585+0000 7fad71ffb640 1 --2- 172.21.15.72:0/3348197820 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fad6406c980 0x7fad6406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.585+0000 7fad71ffb640 1 -- 172.21.15.72:0/3348197820 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7fad7808b820 con 0x7fad800afae0 2024-04-15T15:16:49.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.585+0000 7fad8491d640 1 --2- 172.21.15.72:0/3348197820 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fad6406c980 0x7fad6406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.586+0000 7fad8491d640 1 --2- 172.21.15.72:0/3348197820 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fad6406c980 0x7fad6406ee40 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fad74035000 tx=0x7fad7402e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:49.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.589+0000 7fad71ffb640 1 -- 172.21.15.72:0/3348197820 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fad78059a60 con 0x7fad800afae0 2024-04-15T15:16:49.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.764+0000 7fad87382640 1 -- 172.21.15.72:0/3348197820 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fad8006d610 con 0x7fad800afae0 2024-04-15T15:16:49.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.764+0000 7fad71ffb640 1 -- 172.21.15.72:0/3348197820 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) ==== 93+0+10 (secure 0 0 0) 0x7fad78059360 con 0x7fad800afae0 2024-04-15T15:16:49.766 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-15T15:16:49.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.766+0000 7fad87382640 1 -- 172.21.15.72:0/3348197820 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fad6406c980 msgr2=0x7fad6406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.766+0000 7fad87382640 1 --2- 172.21.15.72:0/3348197820 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fad6406c980 0x7fad6406ee40 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fad74035000 tx=0x7fad7402e000 comp rx=0 tx=0).stop 2024-04-15T15:16:49.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.766+0000 7fad87382640 1 -- 172.21.15.72:0/3348197820 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad800afae0 msgr2=0x7fad80145620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.766+0000 7fad87382640 1 --2- 172.21.15.72:0/3348197820 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad800afae0 0x7fad80145620 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fad7800b4a0 tx=0x7fad7800b960 comp rx=0 tx=0).stop 2024-04-15T15:16:49.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.766+0000 7fad87382640 1 -- 172.21.15.72:0/3348197820 shutdown_connections 2024-04-15T15:16:49.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.766+0000 7fad87382640 1 --2- 172.21.15.72:0/3348197820 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fad6406c980 0x7fad6406ee40 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.766+0000 7fad87382640 1 --2- 172.21.15.72:0/3348197820 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fad800b5410 0x7fad8015c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.766+0000 7fad87382640 1 --2- 172.21.15.72:0/3348197820 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fad800b0050 0x7fad80144e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.766+0000 7fad87382640 1 --2- 172.21.15.72:0/3348197820 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fad800afae0 0x7fad80145620 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.766+0000 7fad87382640 1 -- 172.21.15.72:0/3348197820 >> 172.21.15.72:0/3348197820 conn(0x7fad800d0a50 msgr2=0x7fad801612f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:49.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.766+0000 7fad87382640 1 -- 172.21.15.72:0/3348197820 shutdown_connections 2024-04-15T15:16:49.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.766+0000 7fad87382640 1 -- 172.21.15.72:0/3348197820 wait complete. 2024-04-15T15:16:49.779 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:16:49.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.884+0000 7f74e6a88640 1 Processor -- start 2024-04-15T15:16:49.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.884+0000 7f74e6a88640 1 -- start start 2024-04-15T15:16:49.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.885+0000 7f74e6a88640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f74e00cd2a0 0x7f74e006cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.885+0000 7f74e6a88640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f74e006e620 0x7f74e006ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.885+0000 7f74e6a88640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74e006bd50 con 0x7f74e015c630 2024-04-15T15:16:49.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.885+0000 7f74e6a88640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f74e006d2f0 con 0x7f74e006e620 2024-04-15T15:16:49.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.885+0000 7f74e6a88640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f74e006c920 con 0x7f74e00cd2a0 2024-04-15T15:16:49.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.885+0000 7f74e5025640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f74e006e620 0x7f74e006ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.885+0000 7f74e5025640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f74e006e620 0x7f74e006ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43730/0 (socket says 172.21.15.72:43730) 2024-04-15T15:16:49.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.885+0000 7f74e5025640 1 -- 172.21.15.72:0/3543204569 learned_addr learned my addr 172.21.15.72:0/3543204569 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:49.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.885+0000 7f74e4824640 1 --1- 172.21.15.72:0/3543204569 >> v1:172.21.15.72:6789/0 conn(0x7f74e015c630 0x7f74e00ccb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49896/0 (socket says 172.21.15.72:49896) 2024-04-15T15:16:49.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.885+0000 7f74e5025640 1 -- 172.21.15.72:0/3543204569 >> v2:172.21.15.175:3300/0 conn(0x7f74e00cd2a0 msgr2=0x7f74e006cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:49.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.885+0000 7f74e5025640 1 --2- 172.21.15.72:0/3543204569 >> v2:172.21.15.175:3300/0 conn(0x7f74e00cd2a0 0x7f74e006cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.885+0000 7f74e5025640 1 -- 172.21.15.72:0/3543204569 >> v1:172.21.15.72:6789/0 conn(0x7f74e015c630 legacy=0x7f74e00ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.885+0000 7f74e5025640 1 -- 172.21.15.72:0/3543204569 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74e006bff0 con 0x7f74e006e620 2024-04-15T15:16:49.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.886+0000 7f74e5025640 1 --2- 172.21.15.72:0/3543204569 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74e006e620 0x7f74e006ea70 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f74e00700c0 tx=0x7f74d802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4fd9d9b58f98c6a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:49.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.886+0000 7f74df7fe640 1 -- 172.21.15.72:0/3543204569 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d802cb20 con 0x7f74e006e620 2024-04-15T15:16:49.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.886+0000 7f74df7fe640 1 -- 172.21.15.72:0/3543204569 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74d802ccc0 con 0x7f74e006e620 2024-04-15T15:16:49.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.886+0000 7f74df7fe640 1 -- 172.21.15.72:0/3543204569 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d8034610 con 0x7f74e006e620 2024-04-15T15:16:49.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.887+0000 7f74e6a88640 1 -- 172.21.15.72:0/3543204569 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74e006e620 msgr2=0x7f74e006ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.887+0000 7f74e6a88640 1 --2- 172.21.15.72:0/3543204569 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74e006e620 0x7f74e006ea70 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f74e00700c0 tx=0x7f74d802bf80 comp rx=0 tx=0).stop 2024-04-15T15:16:49.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.887+0000 7f74e6a88640 1 -- 172.21.15.72:0/3543204569 shutdown_connections 2024-04-15T15:16:49.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.887+0000 7f74e6a88640 1 --2- 172.21.15.72:0/3543204569 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74e006e620 0x7f74e006ea70 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.887+0000 7f74e6a88640 1 --2- 172.21.15.72:0/3543204569 >> v2:172.21.15.175:3300/0 conn(0x7f74e00cd2a0 0x7f74e006cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.887+0000 7f74e6a88640 1 -- 172.21.15.72:0/3543204569 >> 172.21.15.72:0/3543204569 conn(0x7f74e00b31d0 msgr2=0x7f74e00b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:49.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.887+0000 7f74e6a88640 1 -- 172.21.15.72:0/3543204569 shutdown_connections 2024-04-15T15:16:49.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.887+0000 7f74e6a88640 1 -- 172.21.15.72:0/3543204569 wait complete. 2024-04-15T15:16:49.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.888+0000 7f74e6a88640 1 Processor -- start 2024-04-15T15:16:49.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.888+0000 7f74e6a88640 1 -- start start 2024-04-15T15:16:49.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.889+0000 7f74e6a88640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74e006e620 0x7f74e00c5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.889+0000 7f74e6a88640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f74e00cd2a0 0x7f74e014d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.889+0000 7f74e6a88640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74e015c630 0x7f74e0149620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.889+0000 7f74e6a88640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f74e015ca10 con 0x7f74e006e620 2024-04-15T15:16:49.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.889+0000 7f74e6a88640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f74e006c5b0 con 0x7f74e015c630 2024-04-15T15:16:49.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.889+0000 7f74e6a88640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f74e006efb0 con 0x7f74e00cd2a0 2024-04-15T15:16:49.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.889+0000 7f74e4824640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74e006e620 0x7f74e00c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.889+0000 7f74e4824640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74e006e620 0x7f74e00c5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43742/0 (socket says 172.21.15.72:43742) 2024-04-15T15:16:49.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.889+0000 7f74e4824640 1 -- 172.21.15.72:0/872105186 learned_addr learned my addr 172.21.15.72:0/872105186 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:49.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.889+0000 7f74e5025640 1 --2- 172.21.15.72:0/872105186 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74e015c630 0x7f74e0149620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.889+0000 7f74dffff640 1 --2- 172.21.15.72:0/872105186 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f74e00cd2a0 0x7f74e014d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.889+0000 7f74e4824640 1 -- 172.21.15.72:0/872105186 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f74e00cd2a0 msgr2=0x7f74e014d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.889+0000 7f74e4824640 1 --2- 172.21.15.72:0/872105186 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f74e00cd2a0 0x7f74e014d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.889+0000 7f74e4824640 1 -- 172.21.15.72:0/872105186 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74e015c630 msgr2=0x7f74e0149620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.889+0000 7f74e4824640 1 --2- 172.21.15.72:0/872105186 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74e015c630 0x7f74e0149620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.889+0000 7f74e4824640 1 -- 172.21.15.72:0/872105186 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74e0149b60 con 0x7f74e006e620 2024-04-15T15:16:49.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.889+0000 7f74e5025640 1 --2- 172.21.15.72:0/872105186 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74e015c630 0x7f74e0149620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:49.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.889+0000 7f74dffff640 1 --2- 172.21.15.72:0/872105186 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f74e00cd2a0 0x7f74e014d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:49.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.890+0000 7f74e4824640 1 --2- 172.21.15.72:0/872105186 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74e006e620 0x7f74e00c5040 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f74d000c400 tx=0x7f74d000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:49.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.890+0000 7f74dd7fa640 1 -- 172.21.15.72:0/872105186 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d00074e0 con 0x7f74e006e620 2024-04-15T15:16:49.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.890+0000 7f74dd7fa640 1 -- 172.21.15.72:0/872105186 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74d0007680 con 0x7f74e006e620 2024-04-15T15:16:49.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.890+0000 7f74e6a88640 1 -- 172.21.15.72:0/872105186 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74e0149ff0 con 0x7f74e006e620 2024-04-15T15:16:49.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.890+0000 7f74dd7fa640 1 -- 172.21.15.72:0/872105186 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d0004a00 con 0x7f74e006e620 2024-04-15T15:16:49.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.890+0000 7f74e6a88640 1 -- 172.21.15.72:0/872105186 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74e014a5b0 con 0x7f74e006e620 2024-04-15T15:16:49.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.891+0000 7f74e6a88640 1 -- 172.21.15.72:0/872105186 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74e0075c50 con 0x7f74e006e620 2024-04-15T15:16:49.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.891+0000 7f74dd7fa640 1 -- 172.21.15.72:0/872105186 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f74d001a020 con 0x7f74e006e620 2024-04-15T15:16:49.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.892+0000 7f74dd7fa640 1 --2- 172.21.15.72:0/872105186 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f74cc06c980 0x7f74cc06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.892+0000 7f74dffff640 1 --2- 172.21.15.72:0/872105186 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f74cc06c980 0x7f74cc06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.892+0000 7f74dd7fa640 1 -- 172.21.15.72:0/872105186 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7f74d008b550 con 0x7f74e006e620 2024-04-15T15:16:49.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.892+0000 7f74dffff640 1 --2- 172.21.15.72:0/872105186 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f74cc06c980 0x7f74cc06ee40 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f74d4006fd0 tx=0x7f74d4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:49.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:49.896+0000 7f74dd7fa640 1 -- 172.21.15.72:0/872105186 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f74d0059820 con 0x7f74e006e620 2024-04-15T15:16:50.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.069+0000 7f74e6a88640 1 -- 172.21.15.72:0/872105186 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f74e00d0f10 con 0x7f74e006e620 2024-04-15T15:16:50.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.070+0000 7f74dd7fa640 1 -- 172.21.15.72:0/872105186 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v41) ==== 104+0+11 (secure 0 0 0) 0x7f74d0059120 con 0x7f74e006e620 2024-04-15T15:16:50.071 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:16:50.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.072+0000 7f74e6a88640 1 -- 172.21.15.72:0/872105186 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f74cc06c980 msgr2=0x7f74cc06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.072+0000 7f74e6a88640 1 --2- 172.21.15.72:0/872105186 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f74cc06c980 0x7f74cc06ee40 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f74d4006fd0 tx=0x7f74d4008040 comp rx=0 tx=0).stop 2024-04-15T15:16:50.073 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.072+0000 7f74e6a88640 1 -- 172.21.15.72:0/872105186 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74e006e620 msgr2=0x7f74e00c5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.072+0000 7f74e6a88640 1 --2- 172.21.15.72:0/872105186 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74e006e620 0x7f74e00c5040 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f74d000c400 tx=0x7f74d000c8c0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.072+0000 7f74e6a88640 1 -- 172.21.15.72:0/872105186 shutdown_connections 2024-04-15T15:16:50.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.072+0000 7f74e6a88640 1 --2- 172.21.15.72:0/872105186 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f74cc06c980 0x7f74cc06ee40 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.072+0000 7f74e6a88640 1 --2- 172.21.15.72:0/872105186 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f74e015c630 0x7f74e0149620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.072+0000 7f74e6a88640 1 --2- 172.21.15.72:0/872105186 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f74e00cd2a0 0x7f74e014d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.072+0000 7f74e6a88640 1 --2- 172.21.15.72:0/872105186 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f74e006e620 0x7f74e00c5040 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.072+0000 7f74e6a88640 1 -- 172.21.15.72:0/872105186 >> 172.21.15.72:0/872105186 conn(0x7f74e00b31d0 msgr2=0x7f74e0157ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:50.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.072+0000 7f74e6a88640 1 -- 172.21.15.72:0/872105186 shutdown_connections 2024-04-15T15:16:50.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.072+0000 7f74e6a88640 1 -- 172.21.15.72:0/872105186 wait complete. 2024-04-15T15:16:50.084 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:16:50.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.188+0000 7f44205f4640 1 Processor -- start 2024-04-15T15:16:50.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.188+0000 7f44205f4640 1 -- start start 2024-04-15T15:16:50.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.189+0000 7f44205f4640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f441806c400 0x7f441806c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.189+0000 7f44205f4640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44180d2320 con 0x7f441815e610 2024-04-15T15:16:50.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.189+0000 7f44205f4640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44180d25c0 con 0x7f441815e9f0 2024-04-15T15:16:50.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.189+0000 7f44205f4640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f44180d31d0 con 0x7f441806c400 2024-04-15T15:16:50.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.189+0000 7f441eb91640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f441806c400 0x7f441806c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.190+0000 7f441eb91640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f441806c400 0x7f441806c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43746/0 (socket says 172.21.15.72:43746) 2024-04-15T15:16:50.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.190+0000 7f441eb91640 1 -- 172.21.15.72:0/1299036849 learned_addr learned my addr 172.21.15.72:0/1299036849 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:50.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.190+0000 7f441eb91640 1 -- 172.21.15.72:0/1299036849 >> v1:172.21.15.175:6789/0 conn(0x7f441815e9f0 legacy=0x7f44180d3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:50.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.190+0000 7f441eb91640 1 -- 172.21.15.72:0/1299036849 >> v1:172.21.15.72:6789/0 conn(0x7f441815e610 legacy=0x7f44180d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.190+0000 7f441eb91640 1 -- 172.21.15.72:0/1299036849 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44180d28a0 con 0x7f441806c400 2024-04-15T15:16:50.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.190+0000 7f441d38e640 1 -- 172.21.15.72:0/1299036849 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 29836181 0 0) 0x7f44180d2320 con 0x7f441815e610 2024-04-15T15:16:50.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.191+0000 7f441eb91640 1 --2- 172.21.15.72:0/1299036849 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f441806c400 0x7f441806c890 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f4410009870 tx=0x7f441002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cabded314b4a039d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:50.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.191+0000 7f441d38e640 1 -- 172.21.15.72:0/1299036849 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441002f840 con 0x7f441806c400 2024-04-15T15:16:50.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.191+0000 7f441d38e640 1 -- 172.21.15.72:0/1299036849 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f441002f9e0 con 0x7f441806c400 2024-04-15T15:16:50.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.191+0000 7f441d38e640 1 -- 172.21.15.72:0/1299036849 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441002fdb0 con 0x7f441806c400 2024-04-15T15:16:50.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.191+0000 7f44205f4640 1 -- 172.21.15.72:0/1299036849 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f441806c400 msgr2=0x7f441806c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.191+0000 7f44205f4640 1 --2- 172.21.15.72:0/1299036849 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f441806c400 0x7f441806c890 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f4410009870 tx=0x7f441002ba20 comp rx=0 tx=0).stop 2024-04-15T15:16:50.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.191+0000 7f44205f4640 1 -- 172.21.15.72:0/1299036849 shutdown_connections 2024-04-15T15:16:50.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.191+0000 7f44205f4640 1 --2- 172.21.15.72:0/1299036849 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f441806c400 0x7f441806c890 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.191+0000 7f44205f4640 1 -- 172.21.15.72:0/1299036849 >> 172.21.15.72:0/1299036849 conn(0x7f44180d4750 msgr2=0x7f44180d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:50.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.191+0000 7f44205f4640 1 -- 172.21.15.72:0/1299036849 shutdown_connections 2024-04-15T15:16:50.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.191+0000 7f44205f4640 1 -- 172.21.15.72:0/1299036849 wait complete. 2024-04-15T15:16:50.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.192+0000 7f44205f4640 1 Processor -- start 2024-04-15T15:16:50.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.192+0000 7f44205f4640 1 -- start start 2024-04-15T15:16:50.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.193+0000 7f44205f4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f441806c400 0x7f441814bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.193+0000 7f44205f4640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f441815e610 0x7f4418161630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.193+0000 7f44205f4640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f441815e9f0 0x7f44180b6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.193+0000 7f44205f4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f441806d130 con 0x7f441806c400 2024-04-15T15:16:50.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.193+0000 7f44205f4640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f44180d2e60 con 0x7f441815e610 2024-04-15T15:16:50.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.193+0000 7f44205f4640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f44180b7940 con 0x7f441815e9f0 2024-04-15T15:16:50.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.193+0000 7f441e390640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f441806c400 0x7f441814bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.193+0000 7f441e390640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f441806c400 0x7f441814bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43748/0 (socket says 172.21.15.72:43748) 2024-04-15T15:16:50.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.193+0000 7f441e390640 1 -- 172.21.15.72:0/688601548 learned_addr learned my addr 172.21.15.72:0/688601548 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:50.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.193+0000 7f441e390640 1 -- 172.21.15.72:0/688601548 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f441815e9f0 msgr2=0x7f44180b6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.193+0000 7f441e390640 1 --2- 172.21.15.72:0/688601548 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f441815e9f0 0x7f44180b6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.193+0000 7f441e390640 1 -- 172.21.15.72:0/688601548 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f441815e610 msgr2=0x7f4418161630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.193+0000 7f441e390640 1 --2- 172.21.15.72:0/688601548 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f441815e610 0x7f4418161630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.194+0000 7f441e390640 1 -- 172.21.15.72:0/688601548 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44181f5a10 con 0x7f441806c400 2024-04-15T15:16:50.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.194+0000 7f441e390640 1 --2- 172.21.15.72:0/688601548 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f441806c400 0x7f441814bbd0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f441400c890 tx=0x7f441400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:50.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.194+0000 7f440effd640 1 -- 172.21.15.72:0/688601548 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4414003b90 con 0x7f441806c400 2024-04-15T15:16:50.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.194+0000 7f440effd640 1 -- 172.21.15.72:0/688601548 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4414003d70 con 0x7f441806c400 2024-04-15T15:16:50.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.194+0000 7f44205f4640 1 -- 172.21.15.72:0/688601548 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44181f4d50 con 0x7f441806c400 2024-04-15T15:16:50.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.194+0000 7f440effd640 1 -- 172.21.15.72:0/688601548 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441400aa90 con 0x7f441806c400 2024-04-15T15:16:50.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.194+0000 7f44205f4640 1 -- 172.21.15.72:0/688601548 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44181f5310 con 0x7f441806c400 2024-04-15T15:16:50.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.195+0000 7f44205f4640 1 -- 172.21.15.72:0/688601548 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4418075c50 con 0x7f441806c400 2024-04-15T15:16:50.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.195+0000 7f440effd640 1 -- 172.21.15.72:0/688601548 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f441400ac30 con 0x7f441806c400 2024-04-15T15:16:50.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.196+0000 7f440effd640 1 --2- 172.21.15.72:0/688601548 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f43fc06c980 0x7f43fc06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.196+0000 7f440effd640 1 -- 172.21.15.72:0/688601548 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7f441408bd30 con 0x7f441806c400 2024-04-15T15:16:50.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.196+0000 7f441db8f640 1 --2- 172.21.15.72:0/688601548 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f43fc06c980 0x7f43fc06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.196+0000 7f441db8f640 1 --2- 172.21.15.72:0/688601548 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f43fc06c980 0x7f43fc06ee40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4408006fd0 tx=0x7f4408006bc0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:50.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.200+0000 7f440effd640 1 -- 172.21.15.72:0/688601548 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f441405a0c0 con 0x7f441806c400 2024-04-15T15:16:50.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.376+0000 7f44205f4640 1 -- 172.21.15.72:0/688601548 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f44180cf740 con 0x7f441806c400 2024-04-15T15:16:50.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.376+0000 7f440effd640 1 -- 172.21.15.72:0/688601548 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v41) ==== 100+0+11 (secure 0 0 0) 0x7f44140599c0 con 0x7f441806c400 2024-04-15T15:16:50.378 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-04-15T15:16:50.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.378+0000 7f44205f4640 1 -- 172.21.15.72:0/688601548 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f43fc06c980 msgr2=0x7f43fc06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.378+0000 7f44205f4640 1 --2- 172.21.15.72:0/688601548 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f43fc06c980 0x7f43fc06ee40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4408006fd0 tx=0x7f4408006bc0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.378+0000 7f44205f4640 1 -- 172.21.15.72:0/688601548 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f441806c400 msgr2=0x7f441814bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.378+0000 7f44205f4640 1 --2- 172.21.15.72:0/688601548 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f441806c400 0x7f441814bbd0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f441400c890 tx=0x7f441400cd50 comp rx=0 tx=0).stop 2024-04-15T15:16:50.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.378+0000 7f44205f4640 1 -- 172.21.15.72:0/688601548 shutdown_connections 2024-04-15T15:16:50.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.378+0000 7f44205f4640 1 --2- 172.21.15.72:0/688601548 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f43fc06c980 0x7f43fc06ee40 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.378+0000 7f44205f4640 1 --2- 172.21.15.72:0/688601548 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f441815e9f0 0x7f44180b6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.378+0000 7f44205f4640 1 --2- 172.21.15.72:0/688601548 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f441815e610 0x7f4418161630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.378+0000 7f44205f4640 1 --2- 172.21.15.72:0/688601548 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f441806c400 0x7f441814bbd0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.378+0000 7f44205f4640 1 -- 172.21.15.72:0/688601548 >> 172.21.15.72:0/688601548 conn(0x7f44180d4750 msgr2=0x7f44180b4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:50.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.378+0000 7f44205f4640 1 -- 172.21.15.72:0/688601548 shutdown_connections 2024-04-15T15:16:50.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.378+0000 7f44205f4640 1 -- 172.21.15.72:0/688601548 wait complete. 2024-04-15T15:16:50.391 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:16:50.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.493+0000 7f595a9b5640 1 Processor -- start 2024-04-15T15:16:50.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.493+0000 7f595a9b5640 1 -- start start 2024-04-15T15:16:50.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.494+0000 7f595a9b5640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f595415c250 0x7f595406d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.494+0000 7f595a9b5640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f595415f9f0 0x7f59540cca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.494+0000 7f595a9b5640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f595406df30 con 0x7f59540d0010 2024-04-15T15:16:50.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.494+0000 7f595a9b5640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f595406e210 con 0x7f595415f9f0 2024-04-15T15:16:50.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.494+0000 7f595a9b5640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f595406e4f0 con 0x7f595415c250 2024-04-15T15:16:50.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.494+0000 7f5958f52640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f59540d0010 0x7f59540b3e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49922/0 (socket says 172.21.15.72:49922) 2024-04-15T15:16:50.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.494+0000 7f5958f52640 1 -- 172.21.15.72:0/2615382194 learned_addr learned my addr 172.21.15.72:0/2615382194 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:50.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.495+0000 7f594bfff640 1 --2- 172.21.15.72:0/2615382194 >> v2:172.21.15.175:3300/0 conn(0x7f595415f9f0 0x7f59540cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.495+0000 7f5953fff640 1 --2- 172.21.15.72:0/2615382194 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f595415c250 0x7f595406d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.495+0000 7f59537fe640 1 -- 172.21.15.72:0/2615382194 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3615807693 0 0) 0x7f595406df30 con 0x7f59540d0010 2024-04-15T15:16:50.499 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.495+0000 7f59537fe640 1 -- 172.21.15.72:0/2615382194 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5940003520 con 0x7f59540d0010 2024-04-15T15:16:50.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.495+0000 7f59537fe640 1 -- 172.21.15.72:0/2615382194 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 615349301 0 0) 0x7f5940003520 con 0x7f59540d0010 2024-04-15T15:16:50.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.495+0000 7f59537fe640 1 -- 172.21.15.72:0/2615382194 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f595415c250 msgr2=0x7f595406d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:50.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.495+0000 7f59537fe640 1 --2- 172.21.15.72:0/2615382194 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f595415c250 0x7f595406d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.495+0000 7f59537fe640 1 -- 172.21.15.72:0/2615382194 >> v2:172.21.15.175:3300/0 conn(0x7f595415f9f0 msgr2=0x7f59540cca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:50.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.495+0000 7f59537fe640 1 --2- 172.21.15.72:0/2615382194 >> v2:172.21.15.175:3300/0 conn(0x7f595415f9f0 0x7f59540cca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.495+0000 7f59537fe640 1 -- 172.21.15.72:0/2615382194 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59540b3610 con 0x7f59540d0010 2024-04-15T15:16:50.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.495+0000 7f59537fe640 1 -- 172.21.15.72:0/2615382194 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f594c0019f0 con 0x7f59540d0010 2024-04-15T15:16:50.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.495+0000 7f59537fe640 1 -- 172.21.15.72:0/2615382194 >> v1:172.21.15.72:6789/0 conn(0x7f59540d0010 legacy=0x7f59540b3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.495+0000 7f59537fe640 1 --2- 172.21.15.72:0/2615382194 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f59400039b0 0x7f5940003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.495+0000 7f59537fe640 1 --2- 172.21.15.72:0/2615382194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59400043a0 0x7f5940004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.495+0000 7f59537fe640 1 --2- 172.21.15.72:0/2615382194 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5940004f90 0x7f594000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.495+0000 7f59537fe640 1 -- 172.21.15.72:0/2615382194 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f595406e210 con 0x7f59400043a0 2024-04-15T15:16:50.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.495+0000 7f59537fe640 1 -- 172.21.15.72:0/2615382194 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f595406e4f0 con 0x7f5940004f90 2024-04-15T15:16:50.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.495+0000 7f59537fe640 1 -- 172.21.15.72:0/2615382194 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f594000bf30 con 0x7f59400039b0 2024-04-15T15:16:50.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.495+0000 7f5953fff640 1 --2- 172.21.15.72:0/2615382194 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f595415c250 0x7f595406d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:50.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.495+0000 7f594bfff640 1 --2- 172.21.15.72:0/2615382194 >> v2:172.21.15.175:3300/0 conn(0x7f595415f9f0 0x7f59540cca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:50.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.495+0000 7f594bfff640 1 --2- 172.21.15.72:0/2615382194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59400043a0 0x7f5940004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.495+0000 7f5958f52640 1 --2- 172.21.15.72:0/2615382194 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5940004f90 0x7f594000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.495+0000 7f5953fff640 1 --2- 172.21.15.72:0/2615382194 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f59400039b0 0x7f5940003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.495+0000 7f594bfff640 1 -- 172.21.15.72:0/2615382194 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f59400039b0 msgr2=0x7f5940003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:50.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.495+0000 7f594bfff640 1 --2- 172.21.15.72:0/2615382194 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f59400039b0 0x7f5940003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.495+0000 7f594bfff640 1 -- 172.21.15.72:0/2615382194 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5940004f90 msgr2=0x7f594000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:50.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.495+0000 7f594bfff640 1 --2- 172.21.15.72:0/2615382194 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f5940004f90 0x7f594000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.495+0000 7f594bfff640 1 -- 172.21.15.72:0/2615382194 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5940003520 con 0x7f59400043a0 2024-04-15T15:16:50.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.496+0000 7f594bfff640 1 --2- 172.21.15.72:0/2615382194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59400043a0 0x7f5940004880 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f593c009c70 tx=0x7f5940005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ec1475deb646e6e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:50.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.496+0000 7f59537fe640 1 -- 172.21.15.72:0/2615382194 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f593c009840 con 0x7f59400043a0 2024-04-15T15:16:50.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.496+0000 7f59537fe640 1 -- 172.21.15.72:0/2615382194 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f593c004020 con 0x7f59400043a0 2024-04-15T15:16:50.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.496+0000 7f59537fe640 1 -- 172.21.15.72:0/2615382194 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f593c004450 con 0x7f59400043a0 2024-04-15T15:16:50.501 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.497+0000 7f595a9b5640 1 -- 172.21.15.72:0/2615382194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59400043a0 msgr2=0x7f5940004880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.497+0000 7f595a9b5640 1 --2- 172.21.15.72:0/2615382194 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f59400043a0 0x7f5940004880 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f593c009c70 tx=0x7f5940005830 comp rx=0 tx=0).stop 2024-04-15T15:16:50.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.497+0000 7f5958f52640 1 -- 172.21.15.72:0/2615382194 reap_dead start 2024-04-15T15:16:50.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.497+0000 7f595a9b5640 1 -- 172.21.15.72:0/2615382194 shutdown_connections 2024-04-15T15:16:50.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.497+0000 7f595a9b5640 1 -- 172.21.15.72:0/2615382194 >> 172.21.15.72:0/2615382194 conn(0x7f59540ceec0 msgr2=0x7f59540cf2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:50.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.497+0000 7f595a9b5640 1 -- 172.21.15.72:0/2615382194 shutdown_connections 2024-04-15T15:16:50.502 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.497+0000 7f595a9b5640 1 -- 172.21.15.72:0/2615382194 wait complete. 2024-04-15T15:16:50.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.498+0000 7f595a9b5640 1 Processor -- start 2024-04-15T15:16:50.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.498+0000 7f595a9b5640 1 -- start start 2024-04-15T15:16:50.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.499+0000 7f595a9b5640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59400043a0 0x7f595415f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.499+0000 7f595a9b5640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f59540b3d10 0x7f59540b4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.499+0000 7f595a9b5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5954147880 0x7f5954147d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.499+0000 7f595a9b5640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f595415c7b0 con 0x7f5954147880 2024-04-15T15:16:50.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.499+0000 7f595a9b5640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f595415c630 con 0x7f59400043a0 2024-04-15T15:16:50.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.499+0000 7f595a9b5640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f59540ccfd0 con 0x7f59540b3d10 2024-04-15T15:16:50.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.499+0000 7f5958f52640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5954147880 0x7f5954147d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.499+0000 7f5958f52640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5954147880 0x7f5954147d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43768/0 (socket says 172.21.15.72:43768) 2024-04-15T15:16:50.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.499+0000 7f5958f52640 1 -- 172.21.15.72:0/3875069135 learned_addr learned my addr 172.21.15.72:0/3875069135 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:50.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.499+0000 7f594bfff640 1 --2- 172.21.15.72:0/3875069135 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f59540b3d10 0x7f59540b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.499+0000 7f5953fff640 1 --2- 172.21.15.72:0/3875069135 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59400043a0 0x7f595415f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.499+0000 7f5958f52640 1 -- 172.21.15.72:0/3875069135 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f59540b3d10 msgr2=0x7f59540b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.499+0000 7f5958f52640 1 --2- 172.21.15.72:0/3875069135 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f59540b3d10 0x7f59540b4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.499+0000 7f5958f52640 1 -- 172.21.15.72:0/3875069135 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59400043a0 msgr2=0x7f595415f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.499+0000 7f5958f52640 1 --2- 172.21.15.72:0/3875069135 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59400043a0 0x7f595415f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.499+0000 7f5958f52640 1 -- 172.21.15.72:0/3875069135 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59541485d0 con 0x7f5954147880 2024-04-15T15:16:50.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.499+0000 7f5958f52640 1 --2- 172.21.15.72:0/3875069135 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5954147880 0x7f5954147d00 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f595406f4b0 tx=0x7f594c007b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:50.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.500+0000 7f59517fa640 1 -- 172.21.15.72:0/3875069135 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f594c00abc0 con 0x7f5954147880 2024-04-15T15:16:50.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.500+0000 7f59517fa640 1 -- 172.21.15.72:0/3875069135 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f594c019070 con 0x7f5954147880 2024-04-15T15:16:50.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.500+0000 7f59517fa640 1 -- 172.21.15.72:0/3875069135 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f594c0133f0 con 0x7f5954147880 2024-04-15T15:16:50.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.500+0000 7f595a9b5640 1 -- 172.21.15.72:0/3875069135 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59540cb4b0 con 0x7f5954147880 2024-04-15T15:16:50.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.500+0000 7f595a9b5640 1 -- 172.21.15.72:0/3875069135 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59540cba70 con 0x7f5954147880 2024-04-15T15:16:50.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.501+0000 7f595a9b5640 1 -- 172.21.15.72:0/3875069135 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59540b46d0 con 0x7f5954147880 2024-04-15T15:16:50.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.501+0000 7f59517fa640 1 -- 172.21.15.72:0/3875069135 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f594c013590 con 0x7f5954147880 2024-04-15T15:16:50.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.502+0000 7f59517fa640 1 --2- 172.21.15.72:0/3875069135 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f593406c980 0x7f593406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.502+0000 7f59517fa640 1 -- 172.21.15.72:0/3875069135 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7f594c08c250 con 0x7f5954147880 2024-04-15T15:16:50.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.502+0000 7f5953fff640 1 --2- 172.21.15.72:0/3875069135 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f593406c980 0x7f593406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.502+0000 7f5953fff640 1 --2- 172.21.15.72:0/3875069135 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f593406c980 0x7f593406ee40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f59540cfcc0 tx=0x7f5944009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:50.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.506+0000 7f59517fa640 1 -- 172.21.15.72:0/3875069135 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f594c059e40 con 0x7f5954147880 2024-04-15T15:16:50.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.693+0000 7f595a9b5640 1 -- 172.21.15.72:0/3875069135 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f59540b83c0 con 0x7f5954147880 2024-04-15T15:16:50.696 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.693+0000 7f59517fa640 1 -- 172.21.15.72:0/3875069135 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v41) ==== 80+0+122 (secure 0 0 0) 0x7f594c05e140 con 0x7f5954147880 2024-04-15T15:16:50.696 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6836/4270793613 2024-04-16T15:16:41.714817+0000 2024-04-15T15:16:50.696 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6834/4270793613 2024-04-16T15:16:41.714817+0000 2024-04-15T15:16:50.697 INFO:teuthology.orchestra.run.smithi072.stderr:listed 2 entries 2024-04-15T15:16:50.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.695+0000 7f595a9b5640 1 -- 172.21.15.72:0/3875069135 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f593406c980 msgr2=0x7f593406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.695+0000 7f595a9b5640 1 --2- 172.21.15.72:0/3875069135 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f593406c980 0x7f593406ee40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f59540cfcc0 tx=0x7f5944009190 comp rx=0 tx=0).stop 2024-04-15T15:16:50.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.695+0000 7f595a9b5640 1 -- 172.21.15.72:0/3875069135 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5954147880 msgr2=0x7f5954147d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.695+0000 7f595a9b5640 1 --2- 172.21.15.72:0/3875069135 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5954147880 0x7f5954147d00 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f595406f4b0 tx=0x7f594c007b20 comp rx=0 tx=0).stop 2024-04-15T15:16:50.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.695+0000 7f595a9b5640 1 -- 172.21.15.72:0/3875069135 shutdown_connections 2024-04-15T15:16:50.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.695+0000 7f595a9b5640 1 --2- 172.21.15.72:0/3875069135 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f593406c980 0x7f593406ee40 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.695+0000 7f595a9b5640 1 --2- 172.21.15.72:0/3875069135 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5954147880 0x7f5954147d00 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.695+0000 7f595a9b5640 1 --2- 172.21.15.72:0/3875069135 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f59540b3d10 0x7f59540b4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.695+0000 7f595a9b5640 1 --2- 172.21.15.72:0/3875069135 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f59400043a0 0x7f595415f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.695+0000 7f595a9b5640 1 -- 172.21.15.72:0/3875069135 >> 172.21.15.72:0/3875069135 conn(0x7f59540ceec0 msgr2=0x7f59540cf2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:50.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.695+0000 7f595a9b5640 1 -- 172.21.15.72:0/3875069135 shutdown_connections 2024-04-15T15:16:50.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:50.695+0000 7f595a9b5640 1 -- 172.21.15.72:0/3875069135 wait complete. 2024-04-15T15:16:50.708 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi175.front.sepia.ceph.com... 2024-04-15T15:16:50.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:50.708 DEBUG:teuthology.orchestra.run.smithi175:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:50.741 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi175.stderr:ceph-fuse[45499]: fuse finished with error 0 and tester_r 0 2024-04-15T15:16:50.745 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi175.front.sepia.ceph.com 2024-04-15T15:16:50.745 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:50.745 DEBUG:teuthology.orchestra.run.smithi175:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-15T15:16:50.916 INFO:tasks.cephfs.mount:Terminating background process 2024-04-15T15:16:50.916 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:16:50.916 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:16:50.933 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:51.073 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:16:51.076 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:16:51.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.075+0000 7f476beb1640 1 Processor -- start 2024-04-15T15:16:51.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.075+0000 7f476beb1640 1 -- start start 2024-04-15T15:16:51.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.076+0000 7f476beb1640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f47640d45b0 0x7f47640d49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.076+0000 7f476beb1640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4764162220 con 0x7f47640d12b0 2024-04-15T15:16:51.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.076+0000 7f476beb1640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47640d3070 con 0x7f4764161e40 2024-04-15T15:16:51.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.076+0000 7f476beb1640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f47640b3f00 con 0x7f47640d45b0 2024-04-15T15:16:51.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.076+0000 7f476944c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f47640d12b0 0x7f47640d2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:49930/0 (socket says 172.21.15.72:49930) 2024-04-15T15:16:51.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.076+0000 7f476944c640 1 -- 172.21.15.72:0/3202873580 learned_addr learned my addr 172.21.15.72:0/3202873580 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:51.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.076+0000 7f476a44e640 1 --2- 172.21.15.72:0/3202873580 >> v2:172.21.15.175:3300/0 conn(0x7f47640d45b0 0x7f47640d49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.076+0000 7f4768c4b640 1 -- 172.21.15.72:0/3202873580 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3720121427 0 0) 0x7f4764162220 con 0x7f47640d12b0 2024-04-15T15:16:51.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.076+0000 7f4768c4b640 1 -- 172.21.15.72:0/3202873580 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4750003520 con 0x7f47640d12b0 2024-04-15T15:16:51.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.077+0000 7f4768c4b640 1 -- 172.21.15.72:0/3202873580 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2206507273 0 0) 0x7f4750003520 con 0x7f47640d12b0 2024-04-15T15:16:51.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.077+0000 7f4768c4b640 1 -- 172.21.15.72:0/3202873580 >> v2:172.21.15.175:3300/0 conn(0x7f47640d45b0 msgr2=0x7f47640d49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:51.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.077+0000 7f4768c4b640 1 --2- 172.21.15.72:0/3202873580 >> v2:172.21.15.175:3300/0 conn(0x7f47640d45b0 0x7f47640d49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.077+0000 7f4768c4b640 1 -- 172.21.15.72:0/3202873580 >> v1:172.21.15.175:6789/0 conn(0x7f4764161e40 legacy=0x7f47640b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.077+0000 7f4768c4b640 1 -- 172.21.15.72:0/3202873580 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47640d1820 con 0x7f47640d12b0 2024-04-15T15:16:51.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.077+0000 7f4768c4b640 1 -- 172.21.15.72:0/3202873580 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f4760001690 con 0x7f47640d12b0 2024-04-15T15:16:51.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.077+0000 7f4768c4b640 1 -- 172.21.15.72:0/3202873580 >> v1:172.21.15.72:6789/0 conn(0x7f47640d12b0 legacy=0x7f47640d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.077+0000 7f4768c4b640 1 --2- 172.21.15.72:0/3202873580 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f47500039b0 0x7f4750003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.077+0000 7f4768c4b640 1 --2- 172.21.15.72:0/3202873580 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f47500043b0 0x7f4750004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.077+0000 7f4768c4b640 1 --2- 172.21.15.72:0/3202873580 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4750005120 0x7f475000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.077+0000 7f4768c4b640 1 -- 172.21.15.72:0/3202873580 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f47640b3f00 con 0x7f4750005120 2024-04-15T15:16:51.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.077+0000 7f4768c4b640 1 -- 172.21.15.72:0/3202873580 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f475000c060 con 0x7f47500039b0 2024-04-15T15:16:51.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.077+0000 7f4768c4b640 1 -- 172.21.15.72:0/3202873580 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f475000c260 con 0x7f47500043b0 2024-04-15T15:16:51.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.077+0000 7f476a44e640 1 --2- 172.21.15.72:0/3202873580 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4750005120 0x7f475000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.077+0000 7f4769c4d640 1 --2- 172.21.15.72:0/3202873580 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f47500039b0 0x7f4750003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.077+0000 7f476944c640 1 --2- 172.21.15.72:0/3202873580 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f47500043b0 0x7f4750004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.077+0000 7f476a44e640 1 -- 172.21.15.72:0/3202873580 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f47500043b0 msgr2=0x7f4750004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:51.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.077+0000 7f476a44e640 1 --2- 172.21.15.72:0/3202873580 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f47500043b0 0x7f4750004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.077+0000 7f476a44e640 1 -- 172.21.15.72:0/3202873580 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f47500039b0 msgr2=0x7f4750003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:51.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.077+0000 7f476a44e640 1 --2- 172.21.15.72:0/3202873580 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f47500039b0 0x7f4750003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.077+0000 7f476a44e640 1 -- 172.21.15.72:0/3202873580 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4750003520 con 0x7f4750005120 2024-04-15T15:16:51.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.078+0000 7f476a44e640 1 -- 172.21.15.72:0/3202873580 reap_dead start 2024-04-15T15:16:51.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.078+0000 7f476a44e640 1 --2- 172.21.15.72:0/3202873580 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4750005120 0x7f475000b9c0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f475c009fc0 tx=0x7f4750005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57775ce35d0bece server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:51.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.078+0000 7f4768c4b640 1 -- 172.21.15.72:0/3202873580 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475c009c10 con 0x7f4750005120 2024-04-15T15:16:51.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.078+0000 7f4768c4b640 1 -- 172.21.15.72:0/3202873580 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f475c02cca0 con 0x7f4750005120 2024-04-15T15:16:51.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.078+0000 7f4768c4b640 1 -- 172.21.15.72:0/3202873580 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475c003bd0 con 0x7f4750005120 2024-04-15T15:16:51.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.079+0000 7f476beb1640 1 -- 172.21.15.72:0/3202873580 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4750005120 msgr2=0x7f475000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.079+0000 7f476beb1640 1 --2- 172.21.15.72:0/3202873580 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4750005120 0x7f475000b9c0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f475c009fc0 tx=0x7f4750005830 comp rx=0 tx=0).stop 2024-04-15T15:16:51.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.079+0000 7f476beb1640 1 -- 172.21.15.72:0/3202873580 shutdown_connections 2024-04-15T15:16:51.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.079+0000 7f476beb1640 1 --2- 172.21.15.72:0/3202873580 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4750005120 0x7f475000b9c0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.079+0000 7f476beb1640 1 -- 172.21.15.72:0/3202873580 >> 172.21.15.72:0/3202873580 conn(0x7f476409d970 msgr2=0x7f47640b95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:51.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.079+0000 7f476beb1640 1 -- 172.21.15.72:0/3202873580 shutdown_connections 2024-04-15T15:16:51.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.079+0000 7f476beb1640 1 -- 172.21.15.72:0/3202873580 wait complete. 2024-04-15T15:16:51.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.080+0000 7f476beb1640 1 Processor -- start 2024-04-15T15:16:51.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.080+0000 7f476beb1640 1 -- start start 2024-04-15T15:16:51.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.081+0000 7f476beb1640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4750005120 0x7f47640b4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.081+0000 7f476beb1640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f47640cc370 0x7f47640b4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.081+0000 7f476beb1640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f47640b5380 0x7f47640b5800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.081+0000 7f476beb1640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f47640d1de0 con 0x7f4750005120 2024-04-15T15:16:51.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.081+0000 7f476beb1640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f47640d21f0 con 0x7f47640b5380 2024-04-15T15:16:51.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.081+0000 7f476beb1640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f47640d2440 con 0x7f47640cc370 2024-04-15T15:16:51.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.081+0000 7f4769c4d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4750005120 0x7f47640b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.081+0000 7f4769c4d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4750005120 0x7f47640b4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43790/0 (socket says 172.21.15.72:43790) 2024-04-15T15:16:51.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.081+0000 7f4769c4d640 1 -- 172.21.15.72:0/737528014 learned_addr learned my addr 172.21.15.72:0/737528014 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:51.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.081+0000 7f476944c640 1 --2- 172.21.15.72:0/737528014 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f47640cc370 0x7f47640b4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.081+0000 7f476a44e640 1 --2- 172.21.15.72:0/737528014 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f47640b5380 0x7f47640b5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.081+0000 7f4769c4d640 1 -- 172.21.15.72:0/737528014 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f47640cc370 msgr2=0x7f47640b4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.081+0000 7f4769c4d640 1 --2- 172.21.15.72:0/737528014 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f47640cc370 0x7f47640b4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.081+0000 7f4769c4d640 1 -- 172.21.15.72:0/737528014 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f47640b5380 msgr2=0x7f47640b5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.081+0000 7f4769c4d640 1 --2- 172.21.15.72:0/737528014 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f47640b5380 0x7f47640b5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.081+0000 7f4769c4d640 1 -- 172.21.15.72:0/737528014 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4764154190 con 0x7f4750005120 2024-04-15T15:16:51.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.081+0000 7f476944c640 1 --2- 172.21.15.72:0/737528014 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f47640cc370 0x7f47640b4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:51.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.081+0000 7f4769c4d640 1 --2- 172.21.15.72:0/737528014 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4750005120 0x7f47640b4900 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f475400f870 tx=0x7f475400fd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:51.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.082+0000 7f475a7fc640 1 -- 172.21.15.72:0/737528014 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475401a070 con 0x7f4750005120 2024-04-15T15:16:51.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.082+0000 7f475a7fc640 1 -- 172.21.15.72:0/737528014 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4754005320 con 0x7f4750005120 2024-04-15T15:16:51.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.082+0000 7f476beb1640 1 -- 172.21.15.72:0/737528014 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47640cc8a0 con 0x7f4750005120 2024-04-15T15:16:51.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.082+0000 7f475a7fc640 1 -- 172.21.15.72:0/737528014 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47540072b0 con 0x7f4750005120 2024-04-15T15:16:51.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.082+0000 7f476beb1640 1 -- 172.21.15.72:0/737528014 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47640cce60 con 0x7f4750005120 2024-04-15T15:16:51.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.083+0000 7f476beb1640 1 -- 172.21.15.72:0/737528014 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47640689e0 con 0x7f4750005120 2024-04-15T15:16:51.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.083+0000 7f475a7fc640 1 -- 172.21.15.72:0/737528014 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f475401d070 con 0x7f4750005120 2024-04-15T15:16:51.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.084+0000 7f475a7fc640 1 --2- 172.21.15.72:0/737528014 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f474806c9b0 0x7f474806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.084+0000 7f475a7fc640 1 -- 172.21.15.72:0/737528014 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7f475408b5b0 con 0x7f4750005120 2024-04-15T15:16:51.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.084+0000 7f476944c640 1 --2- 172.21.15.72:0/737528014 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f474806c9b0 0x7f474806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.084+0000 7f476944c640 1 --2- 172.21.15.72:0/737528014 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f474806c9b0 0x7f474806ee70 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f47640ce600 tx=0x7f4760009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:51.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.088+0000 7f475a7fc640 1 -- 172.21.15.72:0/737528014 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4754059880 con 0x7f4750005120 2024-04-15T15:16:51.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.271+0000 7f476beb1640 1 -- 172.21.15.72:0/737528014 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f47640d51b0 con 0x7f4750005120 2024-04-15T15:16:51.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.272+0000 7f475a7fc640 1 -- 172.21.15.72:0/737528014 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) ==== 74+0+17120 (secure 0 0 0) 0x7f4754059180 con 0x7f4750005120 2024-04-15T15:16:51.273 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:16:51.274 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":41,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:16:41.714841+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:05.551499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:06.003152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.26","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:6836/4270793613":"2024-04-16T15:16:41.714817+0000","172.21.15.72:6834/4270793613":"2024-04-16T15:16:41.714817+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:51.275 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.274+0000 7f476beb1640 1 -- 172.21.15.72:0/737528014 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f474806c9b0 msgr2=0x7f474806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.274+0000 7f476beb1640 1 --2- 172.21.15.72:0/737528014 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f474806c9b0 0x7f474806ee70 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f47640ce600 tx=0x7f4760009040 comp rx=0 tx=0).stop 2024-04-15T15:16:51.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.274+0000 7f476beb1640 1 -- 172.21.15.72:0/737528014 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4750005120 msgr2=0x7f47640b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.274+0000 7f476beb1640 1 --2- 172.21.15.72:0/737528014 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4750005120 0x7f47640b4900 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f475400f870 tx=0x7f475400fd30 comp rx=0 tx=0).stop 2024-04-15T15:16:51.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.274+0000 7f476beb1640 1 -- 172.21.15.72:0/737528014 shutdown_connections 2024-04-15T15:16:51.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.274+0000 7f476beb1640 1 --2- 172.21.15.72:0/737528014 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f474806c9b0 0x7f474806ee70 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.274+0000 7f476beb1640 1 --2- 172.21.15.72:0/737528014 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f47640b5380 0x7f47640b5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.274+0000 7f476beb1640 1 --2- 172.21.15.72:0/737528014 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f47640cc370 0x7f47640b4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.274+0000 7f476beb1640 1 --2- 172.21.15.72:0/737528014 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4750005120 0x7f47640b4900 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.274+0000 7f476beb1640 1 -- 172.21.15.72:0/737528014 >> 172.21.15.72:0/737528014 conn(0x7f476409d970 msgr2=0x7f47640b95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:51.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.274+0000 7f476beb1640 1 -- 172.21.15.72:0/737528014 shutdown_connections 2024-04-15T15:16:51.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.274+0000 7f476beb1640 1 -- 172.21.15.72:0/737528014 wait complete. 2024-04-15T15:16:51.287 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:39.359102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:16:05.551499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:16:06.003152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:16:51.287 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:51.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.392+0000 7ff75ca42640 1 Processor -- start 2024-04-15T15:16:51.393 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.392+0000 7ff75ca42640 1 -- start start 2024-04-15T15:16:51.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.392+0000 7ff75ca42640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7ff75815c5f0 0x7ff7580d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.392+0000 7ff75ca42640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff75806f510 0x7ff7580d1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.393+0000 7ff75ca42640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff758160ff0 con 0x7ff7580ce190 2024-04-15T15:16:51.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.393+0000 7ff75ca42640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7ff758160da0 con 0x7ff75806f510 2024-04-15T15:16:51.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.393+0000 7ff75ca42640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7ff7581612d0 con 0x7ff75815c5f0 2024-04-15T15:16:51.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.393+0000 7ff756d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff75806f510 0x7ff7580d1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.393+0000 7ff756d9d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7ff75806f510 0x7ff7580d1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43802/0 (socket says 172.21.15.72:43802) 2024-04-15T15:16:51.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.393+0000 7ff756d9d640 1 -- 172.21.15.72:0/3776192070 learned_addr learned my addr 172.21.15.72:0/3776192070 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:51.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.393+0000 7ff756d9d640 1 -- 172.21.15.72:0/3776192070 >> v2:172.21.15.175:3300/0 conn(0x7ff75815c5f0 msgr2=0x7ff7580d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:51.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.393+0000 7ff75659c640 1 --2- 172.21.15.72:0/3776192070 >> v2:172.21.15.175:3300/0 conn(0x7ff75815c5f0 0x7ff7580d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.393+0000 7ff756d9d640 1 --2- 172.21.15.72:0/3776192070 >> v2:172.21.15.175:3300/0 conn(0x7ff75815c5f0 0x7ff7580d1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.393+0000 7ff756d9d640 1 -- 172.21.15.72:0/3776192070 >> v1:172.21.15.72:6789/0 conn(0x7ff7580ce190 legacy=0x7ff7580ccb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.393+0000 7ff756d9d640 1 -- 172.21.15.72:0/3776192070 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7580b5840 con 0x7ff75806f510 2024-04-15T15:16:51.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.393+0000 7ff755d9b640 1 -- 172.21.15.72:0/3776192070 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1774313894 0 0) 0x7ff758160ff0 con 0x7ff7580ce190 2024-04-15T15:16:51.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.394+0000 7ff756d9d640 1 --2- 172.21.15.72:0/3776192070 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff75806f510 0x7ff7580d1640 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7ff7580696e0 tx=0x7ff75002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac8279ab21fc6a92 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:51.396 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.394+0000 7ff755d9b640 1 -- 172.21.15.72:0/3776192070 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff75002ca80 con 0x7ff75806f510 2024-04-15T15:16:51.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.394+0000 7ff755d9b640 1 -- 172.21.15.72:0/3776192070 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff75002cc20 con 0x7ff75806f510 2024-04-15T15:16:51.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.394+0000 7ff755d9b640 1 -- 172.21.15.72:0/3776192070 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7500343f0 con 0x7ff75806f510 2024-04-15T15:16:51.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.394+0000 7ff75ca42640 1 -- 172.21.15.72:0/3776192070 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff75806f510 msgr2=0x7ff7580d1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.394+0000 7ff75ca42640 1 --2- 172.21.15.72:0/3776192070 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff75806f510 0x7ff7580d1640 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7ff7580696e0 tx=0x7ff75002bee0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.394+0000 7ff75ca42640 1 -- 172.21.15.72:0/3776192070 shutdown_connections 2024-04-15T15:16:51.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.394+0000 7ff75ca42640 1 --2- 172.21.15.72:0/3776192070 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff75806f510 0x7ff7580d1640 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.394+0000 7ff75ca42640 1 --2- 172.21.15.72:0/3776192070 >> v2:172.21.15.175:3300/0 conn(0x7ff75815c5f0 0x7ff7580d1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.394+0000 7ff75ca42640 1 -- 172.21.15.72:0/3776192070 >> 172.21.15.72:0/3776192070 conn(0x7ff7580b4010 msgr2=0x7ff7580b4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:51.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.394+0000 7ff75ca42640 1 -- 172.21.15.72:0/3776192070 shutdown_connections 2024-04-15T15:16:51.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.394+0000 7ff75ca42640 1 -- 172.21.15.72:0/3776192070 wait complete. 2024-04-15T15:16:51.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.396+0000 7ff75ca42640 1 Processor -- start 2024-04-15T15:16:51.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.396+0000 7ff75ca42640 1 -- start start 2024-04-15T15:16:51.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.397+0000 7ff75ca42640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff75806f510 0x7ff75815dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.397+0000 7ff75ca42640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff75815c5f0 0x7ff7580cc490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.397+0000 7ff75ca42640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff7580ad8d0 0x7ff7580b6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.397+0000 7ff75ca42640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7ff7580b5bb0 con 0x7ff75806f510 2024-04-15T15:16:51.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.397+0000 7ff75ca42640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7ff7580d23a0 con 0x7ff75815c5f0 2024-04-15T15:16:51.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.397+0000 7ff75ca42640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7ff75815c9d0 con 0x7ff7580ad8d0 2024-04-15T15:16:51.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.397+0000 7ff75659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff75806f510 0x7ff75815dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.397+0000 7ff75659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff75806f510 0x7ff75815dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43812/0 (socket says 172.21.15.72:43812) 2024-04-15T15:16:51.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.397+0000 7ff75659c640 1 -- 172.21.15.72:0/834511360 learned_addr learned my addr 172.21.15.72:0/834511360 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:51.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.397+0000 7ff74dd9b640 1 --2- 172.21.15.72:0/834511360 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff75815c5f0 0x7ff7580cc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.397+0000 7ff756d9d640 1 --2- 172.21.15.72:0/834511360 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff7580ad8d0 0x7ff7580b6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.397+0000 7ff75659c640 1 -- 172.21.15.72:0/834511360 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff7580ad8d0 msgr2=0x7ff7580b6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.397+0000 7ff75659c640 1 --2- 172.21.15.72:0/834511360 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff7580ad8d0 0x7ff7580b6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.397+0000 7ff75659c640 1 -- 172.21.15.72:0/834511360 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff75815c5f0 msgr2=0x7ff7580cc490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.397+0000 7ff75659c640 1 --2- 172.21.15.72:0/834511360 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff75815c5f0 0x7ff7580cc490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.397+0000 7ff75659c640 1 -- 172.21.15.72:0/834511360 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff75815e4c0 con 0x7ff75806f510 2024-04-15T15:16:51.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.397+0000 7ff74dd9b640 1 --2- 172.21.15.72:0/834511360 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff75815c5f0 0x7ff7580cc490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:51.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.397+0000 7ff756d9d640 1 --2- 172.21.15.72:0/834511360 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff7580ad8d0 0x7ff7580b6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:51.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.397+0000 7ff75659c640 1 --2- 172.21.15.72:0/834511360 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff75806f510 0x7ff75815dde0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7ff74800c820 tx=0x7ff74800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:51.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.397+0000 7ff74f7fe640 1 -- 172.21.15.72:0/834511360 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff748015070 con 0x7ff75806f510 2024-04-15T15:16:51.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.397+0000 7ff74f7fe640 1 -- 172.21.15.72:0/834511360 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff748004020 con 0x7ff75806f510 2024-04-15T15:16:51.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.397+0000 7ff75ca42640 1 -- 172.21.15.72:0/834511360 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff75815e7a0 con 0x7ff75806f510 2024-04-15T15:16:51.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.397+0000 7ff74f7fe640 1 -- 172.21.15.72:0/834511360 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7480043f0 con 0x7ff75806f510 2024-04-15T15:16:51.402 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.398+0000 7ff75ca42640 1 -- 172.21.15.72:0/834511360 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff75815ed60 con 0x7ff75806f510 2024-04-15T15:16:51.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.399+0000 7ff75ca42640 1 -- 172.21.15.72:0/834511360 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff758075c50 con 0x7ff75806f510 2024-04-15T15:16:51.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.399+0000 7ff74f7fe640 1 -- 172.21.15.72:0/834511360 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7ff748007830 con 0x7ff75806f510 2024-04-15T15:16:51.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.399+0000 7ff74f7fe640 1 --2- 172.21.15.72:0/834511360 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff73806c980 0x7ff73806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.400+0000 7ff74dd9b640 1 --2- 172.21.15.72:0/834511360 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff73806c980 0x7ff73806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.400+0000 7ff74f7fe640 1 -- 172.21.15.72:0/834511360 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7ff74808bc40 con 0x7ff75806f510 2024-04-15T15:16:51.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.400+0000 7ff74dd9b640 1 --2- 172.21.15.72:0/834511360 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff73806c980 0x7ff73806ee40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff7580b34e0 tx=0x7ff740009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:51.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.404+0000 7ff74f7fe640 1 -- 172.21.15.72:0/834511360 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff748059e00 con 0x7ff75806f510 2024-04-15T15:16:51.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.578+0000 7ff75ca42640 1 -- 172.21.15.72:0/834511360 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff758071430 con 0x7ff75806f510 2024-04-15T15:16:51.580 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.578+0000 7ff74f7fe640 1 -- 172.21.15.72:0/834511360 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) ==== 93+0+10 (secure 0 0 0) 0x7ff748059700 con 0x7ff75806f510 2024-04-15T15:16:51.580 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-15T15:16:51.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.580+0000 7ff75ca42640 1 -- 172.21.15.72:0/834511360 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff73806c980 msgr2=0x7ff73806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.580+0000 7ff75ca42640 1 --2- 172.21.15.72:0/834511360 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff73806c980 0x7ff73806ee40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff7580b34e0 tx=0x7ff740009040 comp rx=0 tx=0).stop 2024-04-15T15:16:51.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.580+0000 7ff75ca42640 1 -- 172.21.15.72:0/834511360 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff75806f510 msgr2=0x7ff75815dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.580+0000 7ff75ca42640 1 --2- 172.21.15.72:0/834511360 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff75806f510 0x7ff75815dde0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7ff74800c820 tx=0x7ff74800cce0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.580+0000 7ff75ca42640 1 -- 172.21.15.72:0/834511360 shutdown_connections 2024-04-15T15:16:51.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.580+0000 7ff75ca42640 1 --2- 172.21.15.72:0/834511360 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7ff73806c980 0x7ff73806ee40 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.580+0000 7ff75ca42640 1 --2- 172.21.15.72:0/834511360 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7ff7580ad8d0 0x7ff7580b6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.580+0000 7ff75ca42640 1 --2- 172.21.15.72:0/834511360 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7ff75815c5f0 0x7ff7580cc490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.580+0000 7ff75ca42640 1 --2- 172.21.15.72:0/834511360 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff75806f510 0x7ff75815dde0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.580+0000 7ff75ca42640 1 -- 172.21.15.72:0/834511360 >> 172.21.15.72:0/834511360 conn(0x7ff7580b4010 msgr2=0x7ff7580b35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:51.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.581+0000 7ff75ca42640 1 -- 172.21.15.72:0/834511360 shutdown_connections 2024-04-15T15:16:51.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.581+0000 7ff75ca42640 1 -- 172.21.15.72:0/834511360 wait complete. 2024-04-15T15:16:51.594 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:16:51.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.698+0000 7f1401f0e640 1 Processor -- start 2024-04-15T15:16:51.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.698+0000 7f1401f0e640 1 -- start start 2024-04-15T15:16:51.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.698+0000 7f1401f0e640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f13fc0d08f0 0x7f13fc0cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.698+0000 7f1401f0e640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f13fc0ce6d0 0x7f13fc0cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.698+0000 7f1401f0e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13fc06bbb0 con 0x7f13fc0ce2f0 2024-04-15T15:16:51.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.698+0000 7f1401f0e640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f13fc06b960 con 0x7f13fc0d08f0 2024-04-15T15:16:51.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.698+0000 7f1401f0e640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f13fc06be90 con 0x7f13fc0ce6d0 2024-04-15T15:16:51.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.699+0000 7f13fb7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f13fc0ce2f0 0x7f13fc15c530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60020/0 (socket says 172.21.15.72:60020) 2024-04-15T15:16:51.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.699+0000 7f13fb7fe640 1 -- 172.21.15.72:0/1506813310 learned_addr learned my addr 172.21.15.72:0/1506813310 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:51.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.699+0000 7f13fa7fc640 1 -- 172.21.15.72:0/1506813310 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4242495371 0 0) 0x7f13fc06bbb0 con 0x7f13fc0ce2f0 2024-04-15T15:16:51.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.699+0000 7f13fa7fc640 1 -- 172.21.15.72:0/1506813310 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13e8003520 con 0x7f13fc0ce2f0 2024-04-15T15:16:51.703 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.699+0000 7f13fbfff640 1 --2- 172.21.15.72:0/1506813310 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f13fc0ce6d0 0x7f13fc0cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.699+0000 7f13faffd640 1 --2- 172.21.15.72:0/1506813310 >> v2:172.21.15.175:3300/0 conn(0x7f13fc0d08f0 0x7f13fc0cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.699+0000 7f13fa7fc640 1 -- 172.21.15.72:0/1506813310 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 142680355 0 0) 0x7f13e8003520 con 0x7f13fc0ce2f0 2024-04-15T15:16:51.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.699+0000 7f13fa7fc640 1 -- 172.21.15.72:0/1506813310 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f13fc0ce6d0 msgr2=0x7f13fc0cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:51.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.699+0000 7f13fa7fc640 1 --2- 172.21.15.72:0/1506813310 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f13fc0ce6d0 0x7f13fc0cc5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.699+0000 7f13fa7fc640 1 -- 172.21.15.72:0/1506813310 >> v2:172.21.15.175:3300/0 conn(0x7f13fc0d08f0 msgr2=0x7f13fc0cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:51.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.699+0000 7f13fa7fc640 1 --2- 172.21.15.72:0/1506813310 >> v2:172.21.15.175:3300/0 conn(0x7f13fc0d08f0 0x7f13fc0cc0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.699+0000 7f13fa7fc640 1 -- 172.21.15.72:0/1506813310 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13fc06c170 con 0x7f13fc0ce2f0 2024-04-15T15:16:51.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.699+0000 7f13fa7fc640 1 -- 172.21.15.72:0/1506813310 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f13e4001670 con 0x7f13fc0ce2f0 2024-04-15T15:16:51.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.699+0000 7f13faffd640 1 --2- 172.21.15.72:0/1506813310 >> v2:172.21.15.175:3300/0 conn(0x7f13fc0d08f0 0x7f13fc0cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:51.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.699+0000 7f13fa7fc640 1 -- 172.21.15.72:0/1506813310 >> v1:172.21.15.72:6789/0 conn(0x7f13fc0ce2f0 legacy=0x7f13fc15c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.699+0000 7f13fa7fc640 1 --2- 172.21.15.72:0/1506813310 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13e80039b0 0x7f13e8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.699+0000 7f13fa7fc640 1 --2- 172.21.15.72:0/1506813310 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f13fc0ce6d0 0x7f13e8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.699+0000 7f13fa7fc640 1 --2- 172.21.15.72:0/1506813310 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f13e8004d40 0x7f13e80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.699+0000 7f13fa7fc640 1 -- 172.21.15.72:0/1506813310 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f13fc06b960 con 0x7f13e80039b0 2024-04-15T15:16:51.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.699+0000 7f13fa7fc640 1 -- 172.21.15.72:0/1506813310 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f13fc06be90 con 0x7f13e8004d40 2024-04-15T15:16:51.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.699+0000 7f13fa7fc640 1 -- 172.21.15.72:0/1506813310 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f13e800dc80 con 0x7f13fc0ce6d0 2024-04-15T15:16:51.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.699+0000 7f13fb7fe640 1 --2- 172.21.15.72:0/1506813310 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13e80039b0 0x7f13e8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.699+0000 7f13fbfff640 1 --2- 172.21.15.72:0/1506813310 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f13e8004d40 0x7f13e80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.699+0000 7f13faffd640 1 --2- 172.21.15.72:0/1506813310 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f13fc0ce6d0 0x7f13e8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.699+0000 7f13fb7fe640 1 -- 172.21.15.72:0/1506813310 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f13fc0ce6d0 msgr2=0x7f13e8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:51.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.699+0000 7f13fb7fe640 1 --2- 172.21.15.72:0/1506813310 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f13fc0ce6d0 0x7f13e8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.700+0000 7f13fb7fe640 1 -- 172.21.15.72:0/1506813310 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f13e8004d40 msgr2=0x7f13e80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:51.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.700+0000 7f13fb7fe640 1 --2- 172.21.15.72:0/1506813310 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f13e8004d40 0x7f13e80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.700+0000 7f13fb7fe640 1 -- 172.21.15.72:0/1506813310 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13e8003520 con 0x7f13e80039b0 2024-04-15T15:16:51.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.700+0000 7f13fbfff640 1 --2- 172.21.15.72:0/1506813310 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f13e8004d40 0x7f13e80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:51.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.700+0000 7f13fb7fe640 1 --2- 172.21.15.72:0/1506813310 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13e80039b0 0x7f13e8003dd0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f13e400c240 tx=0x7f13e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=104adb5f83e59a78 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:51.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.700+0000 7f13fa7fc640 1 -- 172.21.15.72:0/1506813310 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e4007d50 con 0x7f13e80039b0 2024-04-15T15:16:51.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.700+0000 7f13fa7fc640 1 -- 172.21.15.72:0/1506813310 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13e4030070 con 0x7f13e80039b0 2024-04-15T15:16:51.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.700+0000 7f13fa7fc640 1 -- 172.21.15.72:0/1506813310 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e402cd80 con 0x7f13e80039b0 2024-04-15T15:16:51.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.701+0000 7f1401f0e640 1 -- 172.21.15.72:0/1506813310 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13e80039b0 msgr2=0x7f13e8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.701+0000 7f1401f0e640 1 --2- 172.21.15.72:0/1506813310 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13e80039b0 0x7f13e8003dd0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f13e400c240 tx=0x7f13e8005830 comp rx=0 tx=0).stop 2024-04-15T15:16:51.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.701+0000 7f13fbfff640 1 -- 172.21.15.72:0/1506813310 reap_dead start 2024-04-15T15:16:51.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.701+0000 7f1401f0e640 1 -- 172.21.15.72:0/1506813310 shutdown_connections 2024-04-15T15:16:51.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.701+0000 7f1401f0e640 1 -- 172.21.15.72:0/1506813310 >> 172.21.15.72:0/1506813310 conn(0x7f13fc0b31d0 msgr2=0x7f13fc0b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:51.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.701+0000 7f1401f0e640 1 -- 172.21.15.72:0/1506813310 shutdown_connections 2024-04-15T15:16:51.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.701+0000 7f1401f0e640 1 -- 172.21.15.72:0/1506813310 wait complete. 2024-04-15T15:16:51.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.702+0000 7f1401f0e640 1 Processor -- start 2024-04-15T15:16:51.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.702+0000 7f1401f0e640 1 -- start start 2024-04-15T15:16:51.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.703+0000 7f1401f0e640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f13e80039b0 0x7f13fc0afd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.703+0000 7f1401f0e640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f13fc14f670 0x7f13fc1445d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.703+0000 7f1401f0e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13fc0b16a0 0x7f13fc0b1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.703+0000 7f1401f0e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f13fc155780 con 0x7f13fc0b16a0 2024-04-15T15:16:51.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.703+0000 7f1401f0e640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f13fc155600 con 0x7f13e80039b0 2024-04-15T15:16:51.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.703+0000 7f1401f0e640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f13fc155900 con 0x7f13fc14f670 2024-04-15T15:16:51.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.703+0000 7f13fbfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13fc0b16a0 0x7f13fc0b1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.703+0000 7f13fbfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13fc0b16a0 0x7f13fc0b1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37516/0 (socket says 172.21.15.72:37516) 2024-04-15T15:16:51.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.703+0000 7f13fbfff640 1 -- 172.21.15.72:0/2269208498 learned_addr learned my addr 172.21.15.72:0/2269208498 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:51.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.703+0000 7f13fb7fe640 1 --2- 172.21.15.72:0/2269208498 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f13e80039b0 0x7f13fc0afd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.703+0000 7f13faffd640 1 --2- 172.21.15.72:0/2269208498 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f13fc14f670 0x7f13fc1445d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.703+0000 7f13fbfff640 1 -- 172.21.15.72:0/2269208498 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f13fc14f670 msgr2=0x7f13fc1445d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.703+0000 7f13fbfff640 1 --2- 172.21.15.72:0/2269208498 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f13fc14f670 0x7f13fc1445d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.703+0000 7f13fbfff640 1 -- 172.21.15.72:0/2269208498 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f13e80039b0 msgr2=0x7f13fc0afd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.703+0000 7f13fbfff640 1 --2- 172.21.15.72:0/2269208498 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f13e80039b0 0x7f13fc0afd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.703+0000 7f13fbfff640 1 -- 172.21.15.72:0/2269208498 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13fc0af600 con 0x7f13fc0b16a0 2024-04-15T15:16:51.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.703+0000 7f13fb7fe640 1 --2- 172.21.15.72:0/2269208498 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f13e80039b0 0x7f13fc0afd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:51.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.703+0000 7f13faffd640 1 --2- 172.21.15.72:0/2269208498 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f13fc14f670 0x7f13fc1445d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:51.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.703+0000 7f13fbfff640 1 --2- 172.21.15.72:0/2269208498 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13fc0b16a0 0x7f13fc0b1b20 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f13fc06d450 tx=0x7f13f4004840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:51.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.704+0000 7f13f3fff640 1 -- 172.21.15.72:0/2269208498 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f4007c80 con 0x7f13fc0b16a0 2024-04-15T15:16:51.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.704+0000 7f13f3fff640 1 -- 172.21.15.72:0/2269208498 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13f4019070 con 0x7f13fc0b16a0 2024-04-15T15:16:51.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.704+0000 7f1401f0e640 1 -- 172.21.15.72:0/2269208498 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13fc0af8a0 con 0x7f13fc0b16a0 2024-04-15T15:16:51.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.704+0000 7f13f3fff640 1 -- 172.21.15.72:0/2269208498 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f4014650 con 0x7f13fc0b16a0 2024-04-15T15:16:51.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.704+0000 7f1401f0e640 1 -- 172.21.15.72:0/2269208498 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13fc073040 con 0x7f13fc0b16a0 2024-04-15T15:16:51.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.705+0000 7f1401f0e640 1 -- 172.21.15.72:0/2269208498 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13fc0d12d0 con 0x7f13fc0b16a0 2024-04-15T15:16:51.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.705+0000 7f13f3fff640 1 -- 172.21.15.72:0/2269208498 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f13f40147f0 con 0x7f13fc0b16a0 2024-04-15T15:16:51.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.706+0000 7f13f3fff640 1 --2- 172.21.15.72:0/2269208498 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f13dc06c980 0x7f13dc06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.706+0000 7f13fb7fe640 1 --2- 172.21.15.72:0/2269208498 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f13dc06c980 0x7f13dc06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.706+0000 7f13f3fff640 1 -- 172.21.15.72:0/2269208498 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7f13f408c4e0 con 0x7f13fc0b16a0 2024-04-15T15:16:51.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.706+0000 7f13fb7fe640 1 --2- 172.21.15.72:0/2269208498 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f13dc06c980 0x7f13dc06ee40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f13e4034000 tx=0x7f13e4039040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:51.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.710+0000 7f13f3fff640 1 -- 172.21.15.72:0/2269208498 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f13fc0d12d0 con 0x7f13fc0b16a0 2024-04-15T15:16:51.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.890+0000 7f1401f0e640 1 -- 172.21.15.72:0/2269208498 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f13fc0b2e20 con 0x7f13fc0b16a0 2024-04-15T15:16:51.892 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.890+0000 7f13f3fff640 1 -- 172.21.15.72:0/2269208498 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v41) ==== 104+0+11 (secure 0 0 0) 0x7f13fc0b2e20 con 0x7f13fc0b16a0 2024-04-15T15:16:51.892 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:16:51.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.891+0000 7f1401f0e640 1 -- 172.21.15.72:0/2269208498 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f13dc06c980 msgr2=0x7f13dc06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.891+0000 7f1401f0e640 1 --2- 172.21.15.72:0/2269208498 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f13dc06c980 0x7f13dc06ee40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f13e4034000 tx=0x7f13e4039040 comp rx=0 tx=0).stop 2024-04-15T15:16:51.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.891+0000 7f1401f0e640 1 -- 172.21.15.72:0/2269208498 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13fc0b16a0 msgr2=0x7f13fc0b1b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.891+0000 7f1401f0e640 1 --2- 172.21.15.72:0/2269208498 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13fc0b16a0 0x7f13fc0b1b20 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f13fc06d450 tx=0x7f13f4004840 comp rx=0 tx=0).stop 2024-04-15T15:16:51.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.891+0000 7f1401f0e640 1 -- 172.21.15.72:0/2269208498 shutdown_connections 2024-04-15T15:16:51.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.891+0000 7f1401f0e640 1 --2- 172.21.15.72:0/2269208498 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f13dc06c980 0x7f13dc06ee40 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.891+0000 7f1401f0e640 1 --2- 172.21.15.72:0/2269208498 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13fc0b16a0 0x7f13fc0b1b20 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.891+0000 7f1401f0e640 1 --2- 172.21.15.72:0/2269208498 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f13fc14f670 0x7f13fc1445d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.891+0000 7f1401f0e640 1 --2- 172.21.15.72:0/2269208498 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f13e80039b0 0x7f13fc0afd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.891+0000 7f1401f0e640 1 -- 172.21.15.72:0/2269208498 >> 172.21.15.72:0/2269208498 conn(0x7f13fc0b31d0 msgr2=0x7f13fc152870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:51.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.892+0000 7f1401f0e640 1 -- 172.21.15.72:0/2269208498 shutdown_connections 2024-04-15T15:16:51.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:51.892+0000 7f1401f0e640 1 -- 172.21.15.72:0/2269208498 wait complete. 2024-04-15T15:16:51.905 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:16:52.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.008+0000 7f151eec8640 1 Processor -- start 2024-04-15T15:16:52.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.009+0000 7f151eec8640 1 -- start start 2024-04-15T15:16:52.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.009+0000 7f151eec8640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f151815e9f0 0x7f15180d3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:52.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.009+0000 7f151eec8640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15180d2320 con 0x7f151806c400 2024-04-15T15:16:52.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.009+0000 7f151eec8640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15180d25c0 con 0x7f151815e610 2024-04-15T15:16:52.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.009+0000 7f151eec8640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f15180d31d0 con 0x7f151815e9f0 2024-04-15T15:16:52.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.010+0000 7f151d465640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f151806c400 0x7f15180b7940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60034/0 (socket says 172.21.15.72:60034) 2024-04-15T15:16:52.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.010+0000 7f151d465640 1 -- 172.21.15.72:0/706093912 learned_addr learned my addr 172.21.15.72:0/706093912 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:52.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.010+0000 7f1517fff640 1 --2- 172.21.15.72:0/706093912 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f151815e9f0 0x7f15180d3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:52.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.010+0000 7f15177fe640 1 -- 172.21.15.72:0/706093912 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3247617736 0 0) 0x7f15180d2320 con 0x7f151806c400 2024-04-15T15:16:52.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.010+0000 7f15177fe640 1 -- 172.21.15.72:0/706093912 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1500003520 con 0x7f151806c400 2024-04-15T15:16:52.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.010+0000 7f15177fe640 1 -- 172.21.15.72:0/706093912 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4240933568 0 0) 0x7f1500003520 con 0x7f151806c400 2024-04-15T15:16:52.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.010+0000 7f15177fe640 1 -- 172.21.15.72:0/706093912 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f151815e9f0 msgr2=0x7f15180d3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:52.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.010+0000 7f15177fe640 1 --2- 172.21.15.72:0/706093912 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f151815e9f0 0x7f15180d3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.010+0000 7f15177fe640 1 -- 172.21.15.72:0/706093912 >> v1:172.21.15.175:6789/0 conn(0x7f151815e610 legacy=0x7f15180d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:52.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.010+0000 7f15177fe640 1 -- 172.21.15.72:0/706093912 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15180d28a0 con 0x7f151806c400 2024-04-15T15:16:52.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.010+0000 7f15177fe640 1 -- 172.21.15.72:0/706093912 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f1510001670 con 0x7f151806c400 2024-04-15T15:16:52.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.010+0000 7f15177fe640 1 -- 172.21.15.72:0/706093912 >> v1:172.21.15.72:6789/0 conn(0x7f151806c400 legacy=0x7f15180b7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:52.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.010+0000 7f15177fe640 1 --2- 172.21.15.72:0/706093912 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f15000039e0 0x7f1500003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:52.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.010+0000 7f15177fe640 1 --2- 172.21.15.72:0/706093912 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f151815e9f0 0x7f1500004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:52.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.010+0000 7f15177fe640 1 --2- 172.21.15.72:0/706093912 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1500004d40 0x7f15000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:52.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.010+0000 7f15177fe640 1 -- 172.21.15.72:0/706093912 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f15180d31d0 con 0x7f1500004d40 2024-04-15T15:16:52.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.010+0000 7f15177fe640 1 -- 172.21.15.72:0/706093912 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f150000ba70 con 0x7f15000039e0 2024-04-15T15:16:52.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.010+0000 7f15177fe640 1 -- 172.21.15.72:0/706093912 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f150000bbf0 con 0x7f151815e9f0 2024-04-15T15:16:52.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.010+0000 7f151d465640 1 --2- 172.21.15.72:0/706093912 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1500004d40 0x7f15000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:52.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.010+0000 7f151cc64640 1 --2- 172.21.15.72:0/706093912 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f15000039e0 0x7f1500003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:52.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.010+0000 7f1517fff640 1 --2- 172.21.15.72:0/706093912 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f151815e9f0 0x7f1500004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:52.016 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.011+0000 7f151d465640 1 -- 172.21.15.72:0/706093912 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f151815e9f0 msgr2=0x7f1500004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:52.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.011+0000 7f151d465640 1 --2- 172.21.15.72:0/706093912 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f151815e9f0 0x7f1500004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.011+0000 7f151d465640 1 -- 172.21.15.72:0/706093912 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f15000039e0 msgr2=0x7f1500003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:52.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.011+0000 7f151d465640 1 --2- 172.21.15.72:0/706093912 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f15000039e0 0x7f1500003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.011+0000 7f151d465640 1 -- 172.21.15.72:0/706093912 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1500003520 con 0x7f1500004d40 2024-04-15T15:16:52.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.011+0000 7f151d465640 1 --2- 172.21.15.72:0/706093912 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1500004d40 0x7f15000051c0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f1510007fd0 tx=0x7f1500005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=772f5501ac305b14 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:52.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.011+0000 7f15177fe640 1 -- 172.21.15.72:0/706093912 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1510007ca0 con 0x7f1500004d40 2024-04-15T15:16:52.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.011+0000 7f15177fe640 1 -- 172.21.15.72:0/706093912 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1510030070 con 0x7f1500004d40 2024-04-15T15:16:52.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.011+0000 7f15177fe640 1 -- 172.21.15.72:0/706093912 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1510005d10 con 0x7f1500004d40 2024-04-15T15:16:52.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.012+0000 7f151eec8640 1 -- 172.21.15.72:0/706093912 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1500004d40 msgr2=0x7f15000051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:52.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.012+0000 7f151eec8640 1 --2- 172.21.15.72:0/706093912 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1500004d40 0x7f15000051c0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f1510007fd0 tx=0x7f1500005830 comp rx=0 tx=0).stop 2024-04-15T15:16:52.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.012+0000 7f151d465640 1 -- 172.21.15.72:0/706093912 reap_dead start 2024-04-15T15:16:52.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.012+0000 7f151eec8640 1 -- 172.21.15.72:0/706093912 shutdown_connections 2024-04-15T15:16:52.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.012+0000 7f151eec8640 1 -- 172.21.15.72:0/706093912 >> 172.21.15.72:0/706093912 conn(0x7f15180d4750 msgr2=0x7f15180d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:52.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.012+0000 7f151eec8640 1 -- 172.21.15.72:0/706093912 shutdown_connections 2024-04-15T15:16:52.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.012+0000 7f151eec8640 1 -- 172.21.15.72:0/706093912 wait complete. 2024-04-15T15:16:52.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.013+0000 7f151eec8640 1 Processor -- start 2024-04-15T15:16:52.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.013+0000 7f151eec8640 1 -- start start 2024-04-15T15:16:52.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.014+0000 7f151eec8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1500004d40 0x7f151814fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:52.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.014+0000 7f151eec8640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1518158e00 0x7f1518159280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:52.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.014+0000 7f151eec8640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f151815d4a0 0x7f151815d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:52.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.014+0000 7f151eec8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f151806d130 con 0x7f1500004d40 2024-04-15T15:16:52.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.014+0000 7f151eec8640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f15180d2e60 con 0x7f1518158e00 2024-04-15T15:16:52.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.014+0000 7f151eec8640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f15180b8050 con 0x7f151815d4a0 2024-04-15T15:16:52.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.014+0000 7f151cc64640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1500004d40 0x7f151814fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:52.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.014+0000 7f151cc64640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1500004d40 0x7f151814fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37534/0 (socket says 172.21.15.72:37534) 2024-04-15T15:16:52.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.014+0000 7f151cc64640 1 -- 172.21.15.72:0/3312776160 learned_addr learned my addr 172.21.15.72:0/3312776160 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:52.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.014+0000 7f151cc64640 1 -- 172.21.15.72:0/3312776160 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f151815d4a0 msgr2=0x7f151815d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:52.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.014+0000 7f1517fff640 1 --2- 172.21.15.72:0/3312776160 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1518158e00 0x7f1518159280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:52.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.014+0000 7f151cc64640 1 --2- 172.21.15.72:0/3312776160 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f151815d4a0 0x7f151815d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.014+0000 7f151cc64640 1 -- 172.21.15.72:0/3312776160 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1518158e00 msgr2=0x7f1518159280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:52.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.014+0000 7f151cc64640 1 --2- 172.21.15.72:0/3312776160 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1518158e00 0x7f1518159280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.014+0000 7f151cc64640 1 -- 172.21.15.72:0/3312776160 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1518160680 con 0x7f1500004d40 2024-04-15T15:16:52.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.015+0000 7f151cc64640 1 --2- 172.21.15.72:0/3312776160 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1500004d40 0x7f151814fc10 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f150800f910 tx=0x7f150800fdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:52.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.015+0000 7f15157fa640 1 -- 172.21.15.72:0/3312776160 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15080052e0 con 0x7f1500004d40 2024-04-15T15:16:52.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.015+0000 7f15157fa640 1 -- 172.21.15.72:0/3312776160 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1508018070 con 0x7f1500004d40 2024-04-15T15:16:52.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.015+0000 7f15157fa640 1 -- 172.21.15.72:0/3312776160 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f150800d9b0 con 0x7f1500004d40 2024-04-15T15:16:52.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.015+0000 7f151eec8640 1 -- 172.21.15.72:0/3312776160 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1518150150 con 0x7f1500004d40 2024-04-15T15:16:52.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.015+0000 7f151eec8640 1 -- 172.21.15.72:0/3312776160 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1518157870 con 0x7f1500004d40 2024-04-15T15:16:52.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.016+0000 7f151eec8640 1 -- 172.21.15.72:0/3312776160 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15180cf740 con 0x7f1500004d40 2024-04-15T15:16:52.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.016+0000 7f15157fa640 1 -- 172.21.15.72:0/3312776160 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f150800db50 con 0x7f1500004d40 2024-04-15T15:16:52.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.017+0000 7f15157fa640 1 --2- 172.21.15.72:0/3312776160 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f150406c980 0x7f150406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:52.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.017+0000 7f15157fa640 1 -- 172.21.15.72:0/3312776160 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7f150808bce0 con 0x7f1500004d40 2024-04-15T15:16:52.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.017+0000 7f1517fff640 1 --2- 172.21.15.72:0/3312776160 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f150406c980 0x7f150406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:52.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.017+0000 7f1517fff640 1 --2- 172.21.15.72:0/3312776160 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f150406c980 0x7f150406ee40 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1518155580 tx=0x7f150c00b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:52.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.021+0000 7f15157fa640 1 -- 172.21.15.72:0/3312776160 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1518150150 con 0x7f1500004d40 2024-04-15T15:16:52.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.196+0000 7f151eec8640 1 -- 172.21.15.72:0/3312776160 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1518075c30 con 0x7f1500004d40 2024-04-15T15:16:52.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.196+0000 7f15157fa640 1 -- 172.21.15.72:0/3312776160 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v41) ==== 100+0+11 (secure 0 0 0) 0x7f1508059fb0 con 0x7f1500004d40 2024-04-15T15:16:52.199 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-04-15T15:16:52.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.198+0000 7f151eec8640 1 -- 172.21.15.72:0/3312776160 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f150406c980 msgr2=0x7f150406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:52.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.198+0000 7f151eec8640 1 --2- 172.21.15.72:0/3312776160 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f150406c980 0x7f150406ee40 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1518155580 tx=0x7f150c00b040 comp rx=0 tx=0).stop 2024-04-15T15:16:52.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.198+0000 7f151eec8640 1 -- 172.21.15.72:0/3312776160 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1500004d40 msgr2=0x7f151814fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:52.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.198+0000 7f151eec8640 1 --2- 172.21.15.72:0/3312776160 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1500004d40 0x7f151814fc10 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f150800f910 tx=0x7f150800fdd0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.198+0000 7f151eec8640 1 -- 172.21.15.72:0/3312776160 shutdown_connections 2024-04-15T15:16:52.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.198+0000 7f151eec8640 1 --2- 172.21.15.72:0/3312776160 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f150406c980 0x7f150406ee40 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.198+0000 7f151eec8640 1 --2- 172.21.15.72:0/3312776160 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f151815d4a0 0x7f151815d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.198+0000 7f151eec8640 1 --2- 172.21.15.72:0/3312776160 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1518158e00 0x7f1518159280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.198+0000 7f151eec8640 1 --2- 172.21.15.72:0/3312776160 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1500004d40 0x7f151814fc10 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.198+0000 7f151eec8640 1 -- 172.21.15.72:0/3312776160 >> 172.21.15.72:0/3312776160 conn(0x7f15180d4750 msgr2=0x7f15180b7940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:52.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.198+0000 7f151eec8640 1 -- 172.21.15.72:0/3312776160 shutdown_connections 2024-04-15T15:16:52.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.198+0000 7f151eec8640 1 -- 172.21.15.72:0/3312776160 wait complete. 2024-04-15T15:16:52.212 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:16:52.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.314+0000 7f950111c640 1 Processor -- start 2024-04-15T15:16:52.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.314+0000 7f950111c640 1 -- start start 2024-04-15T15:16:52.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.315+0000 7f950111c640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f94fc0ce2f0 0x7f94fc15c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:52.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.315+0000 7f950111c640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f94fc0d08f0 0x7f94fc0cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:52.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.315+0000 7f950111c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f94fc0ce6d0 0x7f94fc0cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:52.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.315+0000 7f950111c640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f94fc06b960 con 0x7f94fc0ce6d0 2024-04-15T15:16:52.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.315+0000 7f950111c640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f94fc06bbb0 con 0x7f94fc0ce2f0 2024-04-15T15:16:52.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.315+0000 7f950111c640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f94fc06be90 con 0x7f94fc0d08f0 2024-04-15T15:16:52.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.315+0000 7f94fb59e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f94fc0ce6d0 0x7f94fc0cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:52.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.315+0000 7f94fb59e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f94fc0ce6d0 0x7f94fc0cc5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37548/0 (socket says 172.21.15.72:37548) 2024-04-15T15:16:52.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.315+0000 7f94fb59e640 1 -- 172.21.15.72:0/933300966 learned_addr learned my addr 172.21.15.72:0/933300966 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:52.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.315+0000 7f94fad9d640 1 --2- 172.21.15.72:0/933300966 >> v2:172.21.15.175:3300/0 conn(0x7f94fc0ce2f0 0x7f94fc15c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:52.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.315+0000 7f94f259c640 1 --2- 172.21.15.72:0/933300966 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f94fc0d08f0 0x7f94fc0cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:52.318 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.315+0000 7f94fb59e640 1 -- 172.21.15.72:0/933300966 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f94fc0d08f0 msgr2=0x7f94fc0cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:52.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.315+0000 7f94fb59e640 1 --2- 172.21.15.72:0/933300966 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f94fc0d08f0 0x7f94fc0cc0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.315+0000 7f94fb59e640 1 -- 172.21.15.72:0/933300966 >> v2:172.21.15.175:3300/0 conn(0x7f94fc0ce2f0 msgr2=0x7f94fc15c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:52.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.315+0000 7f94fb59e640 1 --2- 172.21.15.72:0/933300966 >> v2:172.21.15.175:3300/0 conn(0x7f94fc0ce2f0 0x7f94fc15c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.315+0000 7f94fb59e640 1 -- 172.21.15.72:0/933300966 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94fc06c170 con 0x7f94fc0ce6d0 2024-04-15T15:16:52.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.316+0000 7f94fb59e640 1 --2- 172.21.15.72:0/933300966 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94fc0ce6d0 0x7f94fc0cc5f0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f94fc0cdf10 tx=0x7f94f402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8cd686fc7d4ef40 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:52.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.316+0000 7f94fa59c640 1 -- 172.21.15.72:0/933300966 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f402ca70 con 0x7f94fc0ce6d0 2024-04-15T15:16:52.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.316+0000 7f94fa59c640 1 -- 172.21.15.72:0/933300966 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94f402cc10 con 0x7f94fc0ce6d0 2024-04-15T15:16:52.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.316+0000 7f94fa59c640 1 -- 172.21.15.72:0/933300966 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f40343f0 con 0x7f94fc0ce6d0 2024-04-15T15:16:52.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.316+0000 7f950111c640 1 -- 172.21.15.72:0/933300966 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94fc0ce6d0 msgr2=0x7f94fc0cc5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:52.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.316+0000 7f950111c640 1 --2- 172.21.15.72:0/933300966 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94fc0ce6d0 0x7f94fc0cc5f0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f94fc0cdf10 tx=0x7f94f402bed0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.317+0000 7f950111c640 1 -- 172.21.15.72:0/933300966 shutdown_connections 2024-04-15T15:16:52.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.317+0000 7f950111c640 1 --2- 172.21.15.72:0/933300966 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94fc0ce6d0 0x7f94fc0cc5f0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.317+0000 7f950111c640 1 --2- 172.21.15.72:0/933300966 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f94fc0d08f0 0x7f94fc0cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.317+0000 7f950111c640 1 --2- 172.21.15.72:0/933300966 >> v2:172.21.15.175:3300/0 conn(0x7f94fc0ce2f0 0x7f94fc15c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.317+0000 7f950111c640 1 -- 172.21.15.72:0/933300966 >> 172.21.15.72:0/933300966 conn(0x7f94fc0b31d0 msgr2=0x7f94fc0b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:52.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.317+0000 7f950111c640 1 -- 172.21.15.72:0/933300966 shutdown_connections 2024-04-15T15:16:52.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.317+0000 7f950111c640 1 -- 172.21.15.72:0/933300966 wait complete. 2024-04-15T15:16:52.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.318+0000 7f950111c640 1 Processor -- start 2024-04-15T15:16:52.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.318+0000 7f950111c640 1 -- start start 2024-04-15T15:16:52.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.319+0000 7f950111c640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f94fc0ce2f0 0x7f94fc0b1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:52.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.319+0000 7f950111c640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94fc0ce6d0 0x7f94fc0afd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:52.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.319+0000 7f950111c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94fc0d08f0 0x7f94fc073ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:52.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.319+0000 7f950111c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f94fc155600 con 0x7f94fc0d08f0 2024-04-15T15:16:52.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.319+0000 7f950111c640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f94fc15ca70 con 0x7f94fc0ce6d0 2024-04-15T15:16:52.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.319+0000 7f950111c640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f94fc155780 con 0x7f94fc0ce2f0 2024-04-15T15:16:52.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.319+0000 7f94fb59e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94fc0d08f0 0x7f94fc073ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:52.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.319+0000 7f94fb59e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94fc0d08f0 0x7f94fc073ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37554/0 (socket says 172.21.15.72:37554) 2024-04-15T15:16:52.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.319+0000 7f94fb59e640 1 -- 172.21.15.72:0/3577906806 learned_addr learned my addr 172.21.15.72:0/3577906806 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:52.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.319+0000 7f94f259c640 1 --2- 172.21.15.72:0/3577906806 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94fc0ce6d0 0x7f94fc0afd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:52.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.319+0000 7f94fad9d640 1 --2- 172.21.15.72:0/3577906806 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f94fc0ce2f0 0x7f94fc0b1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:52.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.319+0000 7f94fb59e640 1 -- 172.21.15.72:0/3577906806 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f94fc0ce2f0 msgr2=0x7f94fc0b1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:52.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.319+0000 7f94fb59e640 1 --2- 172.21.15.72:0/3577906806 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f94fc0ce2f0 0x7f94fc0b1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.319+0000 7f94fb59e640 1 -- 172.21.15.72:0/3577906806 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94fc0ce6d0 msgr2=0x7f94fc0afd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:52.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.319+0000 7f94fb59e640 1 --2- 172.21.15.72:0/3577906806 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94fc0ce6d0 0x7f94fc0afd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.319+0000 7f94fb59e640 1 -- 172.21.15.72:0/3577906806 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94fc0b03e0 con 0x7f94fc0d08f0 2024-04-15T15:16:52.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.319+0000 7f94fb59e640 1 --2- 172.21.15.72:0/3577906806 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94fc0d08f0 0x7f94fc073ae0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f94f4009f40 tx=0x7f94f4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:52.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.320+0000 7f94f3fff640 1 -- 172.21.15.72:0/3577906806 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f4043020 con 0x7f94fc0d08f0 2024-04-15T15:16:52.324 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.320+0000 7f94f3fff640 1 -- 172.21.15.72:0/3577906806 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94f403e450 con 0x7f94fc0d08f0 2024-04-15T15:16:52.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.320+0000 7f94f3fff640 1 -- 172.21.15.72:0/3577906806 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f4009510 con 0x7f94fc0d08f0 2024-04-15T15:16:52.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.320+0000 7f950111c640 1 -- 172.21.15.72:0/3577906806 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94fc0b06c0 con 0x7f94fc0d08f0 2024-04-15T15:16:52.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.320+0000 7f950111c640 1 -- 172.21.15.72:0/3577906806 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94fc0b0c80 con 0x7f94fc0d08f0 2024-04-15T15:16:52.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.321+0000 7f950111c640 1 -- 172.21.15.72:0/3577906806 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94fc075c50 con 0x7f94fc0d08f0 2024-04-15T15:16:52.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.321+0000 7f94f3fff640 1 -- 172.21.15.72:0/3577906806 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f94f4046020 con 0x7f94fc0d08f0 2024-04-15T15:16:52.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.322+0000 7f94f3fff640 1 --2- 172.21.15.72:0/3577906806 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f94e006c9b0 0x7f94e006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:52.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.322+0000 7f94f3fff640 1 -- 172.21.15.72:0/3577906806 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7f94f40ae570 con 0x7f94fc0d08f0 2024-04-15T15:16:52.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.322+0000 7f94fad9d640 1 --2- 172.21.15.72:0/3577906806 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f94e006c9b0 0x7f94e006ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:52.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.322+0000 7f94fad9d640 1 --2- 172.21.15.72:0/3577906806 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f94e006c9b0 0x7f94e006ee70 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f94ec006fd0 tx=0x7f94ec008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:52.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.326+0000 7f94f3fff640 1 -- 172.21.15.72:0/3577906806 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f94f404d030 con 0x7f94fc0d08f0 2024-04-15T15:16:52.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.513+0000 7f950111c640 1 -- 172.21.15.72:0/3577906806 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f94fc06cab0 con 0x7f94fc0d08f0 2024-04-15T15:16:52.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.513+0000 7f94f3fff640 1 -- 172.21.15.72:0/3577906806 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v41) ==== 80+0+122 (secure 0 0 0) 0x7f94f407c7b0 con 0x7f94fc0d08f0 2024-04-15T15:16:52.516 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6836/4270793613 2024-04-16T15:16:41.714817+0000 2024-04-15T15:16:52.516 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6834/4270793613 2024-04-16T15:16:41.714817+0000 2024-04-15T15:16:52.516 INFO:teuthology.orchestra.run.smithi072.stderr:listed 2 entries 2024-04-15T15:16:52.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.515+0000 7f950111c640 1 -- 172.21.15.72:0/3577906806 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f94e006c9b0 msgr2=0x7f94e006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:52.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.515+0000 7f950111c640 1 --2- 172.21.15.72:0/3577906806 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f94e006c9b0 0x7f94e006ee70 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f94ec006fd0 tx=0x7f94ec008040 comp rx=0 tx=0).stop 2024-04-15T15:16:52.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.515+0000 7f950111c640 1 -- 172.21.15.72:0/3577906806 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94fc0d08f0 msgr2=0x7f94fc073ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:52.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.515+0000 7f950111c640 1 --2- 172.21.15.72:0/3577906806 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94fc0d08f0 0x7f94fc073ae0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f94f4009f40 tx=0x7f94f4004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:52.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.515+0000 7f950111c640 1 -- 172.21.15.72:0/3577906806 shutdown_connections 2024-04-15T15:16:52.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.515+0000 7f950111c640 1 --2- 172.21.15.72:0/3577906806 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f94e006c9b0 0x7f94e006ee70 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.515+0000 7f950111c640 1 --2- 172.21.15.72:0/3577906806 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94fc0d08f0 0x7f94fc073ae0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.515+0000 7f950111c640 1 --2- 172.21.15.72:0/3577906806 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94fc0ce6d0 0x7f94fc0afd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.515+0000 7f950111c640 1 --2- 172.21.15.72:0/3577906806 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f94fc0ce2f0 0x7f94fc0b1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.515+0000 7f950111c640 1 -- 172.21.15.72:0/3577906806 >> 172.21.15.72:0/3577906806 conn(0x7f94fc0b31d0 msgr2=0x7f94fc15aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:52.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.515+0000 7f950111c640 1 -- 172.21.15.72:0/3577906806 shutdown_connections 2024-04-15T15:16:52.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.515+0000 7f950111c640 1 -- 172.21.15.72:0/3577906806 wait complete. 2024-04-15T15:16:52.529 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi072.front.sepia.ceph.com... 2024-04-15T15:16:52.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:52.529 DEBUG:teuthology.orchestra.run.smithi072:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-04-15T15:16:52.556 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi072.stderr:ceph-fuse[56518]: fuse finished with error 0 and tester_r 0 2024-04-15T15:16:52.557 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi072.front.sepia.ceph.com 2024-04-15T15:16:52.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:52.558 DEBUG:teuthology.orchestra.run.smithi072:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-15T15:16:52.726 INFO:tasks.cephfs.mount:Terminating background process 2024-04-15T15:16:52.726 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:16:52.726 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:16:52.746 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:52.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.868+0000 7f36b870e640 1 Processor -- start 2024-04-15T15:16:52.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.868+0000 7f36b870e640 1 -- start start 2024-04-15T15:16:52.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.869+0000 7f36b870e640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f36b015c710 0x7f36b00b66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:52.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.869+0000 7f36b870e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f36b00b5790 0x7f36b00d1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:52.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.869+0000 7f36b870e640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36b015caf0 con 0x7f36b00d15c0 2024-04-15T15:16:52.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.869+0000 7f36b870e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f36b00d19f0 con 0x7f36b00b5790 2024-04-15T15:16:52.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.869+0000 7f36b870e640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f36b006f510 con 0x7f36b015c710 2024-04-15T15:16:52.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.869+0000 7f36b5ca9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f36b00b5790 0x7f36b00d1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:52.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.869+0000 7f36b5ca9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f36b00b5790 0x7f36b00d1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37560/0 (socket says 172.21.15.72:37560) 2024-04-15T15:16:52.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.869+0000 7f36b5ca9640 1 -- 172.21.15.72:0/2398121353 learned_addr learned my addr 172.21.15.72:0/2398121353 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:52.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.869+0000 7f36b5ca9640 1 -- 172.21.15.72:0/2398121353 >> v2:172.21.15.175:3300/0 conn(0x7f36b015c710 msgr2=0x7f36b00b66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:52.872 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.869+0000 7f36b64aa640 1 --2- 172.21.15.72:0/2398121353 >> v2:172.21.15.175:3300/0 conn(0x7f36b015c710 0x7f36b00b66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:52.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.869+0000 7f36b5ca9640 1 --2- 172.21.15.72:0/2398121353 >> v2:172.21.15.175:3300/0 conn(0x7f36b015c710 0x7f36b00b66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.869+0000 7f36b5ca9640 1 -- 172.21.15.72:0/2398121353 >> v1:172.21.15.175:6789/0 conn(0x7f36b00d15c0 legacy=0x7f36b01601e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:52.873 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.869+0000 7f36b5ca9640 1 -- 172.21.15.72:0/2398121353 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36b00ccb90 con 0x7f36b00b5790 2024-04-15T15:16:52.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.870+0000 7f36b5ca9640 1 --2- 172.21.15.72:0/2398121353 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36b00b5790 0x7f36b00d1080 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f36b0073730 tx=0x7f36a002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50e12931992b965e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:52.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.870+0000 7f36b54a8640 1 -- 172.21.15.72:0/2398121353 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a0034070 con 0x7f36b00b5790 2024-04-15T15:16:52.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.870+0000 7f36b54a8640 1 -- 172.21.15.72:0/2398121353 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36a0030860 con 0x7f36b00b5790 2024-04-15T15:16:52.874 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.870+0000 7f36b54a8640 1 -- 172.21.15.72:0/2398121353 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a0030c30 con 0x7f36b00b5790 2024-04-15T15:16:52.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.871+0000 7f36b870e640 1 -- 172.21.15.72:0/2398121353 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36b00b5790 msgr2=0x7f36b00d1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:52.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.871+0000 7f36b870e640 1 --2- 172.21.15.72:0/2398121353 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36b00b5790 0x7f36b00d1080 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f36b0073730 tx=0x7f36a002b9a0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.871+0000 7f36b870e640 1 -- 172.21.15.72:0/2398121353 shutdown_connections 2024-04-15T15:16:52.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.871+0000 7f36b870e640 1 --2- 172.21.15.72:0/2398121353 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36b00b5790 0x7f36b00d1080 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.871+0000 7f36b870e640 1 --2- 172.21.15.72:0/2398121353 >> v2:172.21.15.175:3300/0 conn(0x7f36b015c710 0x7f36b00b66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.871+0000 7f36b870e640 1 -- 172.21.15.72:0/2398121353 >> 172.21.15.72:0/2398121353 conn(0x7f36b00b3f90 msgr2=0x7f36b00b43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:52.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.871+0000 7f36b870e640 1 -- 172.21.15.72:0/2398121353 shutdown_connections 2024-04-15T15:16:52.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.871+0000 7f36b870e640 1 -- 172.21.15.72:0/2398121353 wait complete. 2024-04-15T15:16:52.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.872+0000 7f36b870e640 1 Processor -- start 2024-04-15T15:16:52.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.872+0000 7f36b870e640 1 -- start start 2024-04-15T15:16:52.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.873+0000 7f36b870e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36b00b5790 0x7f36b00c88c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:52.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.873+0000 7f36b870e640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f36b00d15c0 0x7f36b00cb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:52.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.873+0000 7f36b870e640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f36b015c710 0x7f36b01f3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:52.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.873+0000 7f36b870e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f36b00cd430 con 0x7f36b00b5790 2024-04-15T15:16:52.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.873+0000 7f36b870e640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f36b00cd150 con 0x7f36b015c710 2024-04-15T15:16:52.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.873+0000 7f36b870e640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f36b00ce960 con 0x7f36b00d15c0 2024-04-15T15:16:52.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.873+0000 7f36b64aa640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36b00b5790 0x7f36b00c88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:52.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.873+0000 7f36b64aa640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36b00b5790 0x7f36b00c88c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37564/0 (socket says 172.21.15.72:37564) 2024-04-15T15:16:52.878 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.873+0000 7f36b64aa640 1 -- 172.21.15.72:0/2758862517 learned_addr learned my addr 172.21.15.72:0/2758862517 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:52.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.873+0000 7f36b6cab640 1 --2- 172.21.15.72:0/2758862517 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f36b015c710 0x7f36b01f3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:52.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.873+0000 7f36b64aa640 1 -- 172.21.15.72:0/2758862517 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f36b00d15c0 msgr2=0x7f36b00cb5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:52.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.873+0000 7f36b5ca9640 1 --2- 172.21.15.72:0/2758862517 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f36b00d15c0 0x7f36b00cb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:52.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.873+0000 7f36b64aa640 1 --2- 172.21.15.72:0/2758862517 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f36b00d15c0 0x7f36b00cb5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.873+0000 7f36b64aa640 1 -- 172.21.15.72:0/2758862517 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f36b015c710 msgr2=0x7f36b01f3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:52.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.873+0000 7f36b64aa640 1 --2- 172.21.15.72:0/2758862517 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f36b015c710 0x7f36b01f3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.873+0000 7f36b64aa640 1 -- 172.21.15.72:0/2758862517 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36b00c8490 con 0x7f36b00b5790 2024-04-15T15:16:52.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.873+0000 7f36b6cab640 1 --2- 172.21.15.72:0/2758862517 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f36b015c710 0x7f36b01f3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:52.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.873+0000 7f36b5ca9640 1 --2- 172.21.15.72:0/2758862517 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f36b00d15c0 0x7f36b00cb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:52.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.873+0000 7f36b64aa640 1 --2- 172.21.15.72:0/2758862517 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36b00b5790 0x7f36b00c88c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f36ac00c5e0 tx=0x7f36ac00caa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:52.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.873+0000 7f36a6ffd640 1 -- 172.21.15.72:0/2758862517 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ac015070 con 0x7f36b00b5790 2024-04-15T15:16:52.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.873+0000 7f36a6ffd640 1 -- 172.21.15.72:0/2758862517 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36ac004260 con 0x7f36b00b5790 2024-04-15T15:16:52.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.873+0000 7f36a6ffd640 1 -- 172.21.15.72:0/2758862517 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ac003610 con 0x7f36b00b5790 2024-04-15T15:16:52.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.873+0000 7f36b870e640 1 -- 172.21.15.72:0/2758862517 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36b00c9ae0 con 0x7f36b00b5790 2024-04-15T15:16:52.879 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.873+0000 7f36b870e640 1 -- 172.21.15.72:0/2758862517 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36b00ca0a0 con 0x7f36b00b5790 2024-04-15T15:16:52.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.874+0000 7f36b870e640 1 -- 172.21.15.72:0/2758862517 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36b00689e0 con 0x7f36b00b5790 2024-04-15T15:16:52.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.875+0000 7f36a6ffd640 1 -- 172.21.15.72:0/2758862517 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f36ac0037b0 con 0x7f36b00b5790 2024-04-15T15:16:52.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.875+0000 7f36a6ffd640 1 --2- 172.21.15.72:0/2758862517 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f369406c980 0x7f369406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:52.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.876+0000 7f36b5ca9640 1 --2- 172.21.15.72:0/2758862517 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f369406c980 0x7f369406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:52.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.876+0000 7f36a6ffd640 1 -- 172.21.15.72:0/2758862517 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7f36ac08bc60 con 0x7f36b00b5790 2024-04-15T15:16:52.881 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.876+0000 7f36b5ca9640 1 --2- 172.21.15.72:0/2758862517 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f369406c980 0x7f369406ee40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f36a0036000 tx=0x7f36a003b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:52.882 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:52.880+0000 7f36a6ffd640 1 -- 172.21.15.72:0/2758862517 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f36ac059e20 con 0x7f36b00b5790 2024-04-15T15:16:53.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.057+0000 7f36b870e640 1 -- 172.21.15.72:0/2758862517 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f36b0070a40 con 0x7f36b00b5790 2024-04-15T15:16:53.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.058+0000 7f36a6ffd640 1 -- 172.21.15.72:0/2758862517 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) ==== 74+0+17120 (secure 0 0 0) 0x7f36ac059720 con 0x7f36b00b5790 2024-04-15T15:16:53.063 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:16:53.063 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":41,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:16:41.714841+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:05.551499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:06.003152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.26","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:6836/4270793613":"2024-04-16T15:16:41.714817+0000","172.21.15.72:6834/4270793613":"2024-04-16T15:16:41.714817+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:53.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.060+0000 7f36b870e640 1 -- 172.21.15.72:0/2758862517 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f369406c980 msgr2=0x7f369406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:53.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.060+0000 7f36b870e640 1 --2- 172.21.15.72:0/2758862517 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f369406c980 0x7f369406ee40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f36a0036000 tx=0x7f36a003b040 comp rx=0 tx=0).stop 2024-04-15T15:16:53.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.060+0000 7f36b870e640 1 -- 172.21.15.72:0/2758862517 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36b00b5790 msgr2=0x7f36b00c88c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:53.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.060+0000 7f36b870e640 1 --2- 172.21.15.72:0/2758862517 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36b00b5790 0x7f36b00c88c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f36ac00c5e0 tx=0x7f36ac00caa0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.060+0000 7f36b870e640 1 -- 172.21.15.72:0/2758862517 shutdown_connections 2024-04-15T15:16:53.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.060+0000 7f36b870e640 1 --2- 172.21.15.72:0/2758862517 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f369406c980 0x7f369406ee40 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.060+0000 7f36b870e640 1 --2- 172.21.15.72:0/2758862517 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f36b015c710 0x7f36b01f3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.060+0000 7f36b870e640 1 --2- 172.21.15.72:0/2758862517 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f36b00d15c0 0x7f36b00cb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.064 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.060+0000 7f36b870e640 1 --2- 172.21.15.72:0/2758862517 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f36b00b5790 0x7f36b00c88c0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.060+0000 7f36b870e640 1 -- 172.21.15.72:0/2758862517 >> 172.21.15.72:0/2758862517 conn(0x7f36b00b3f90 msgr2=0x7f36b0070160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:53.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.060+0000 7f36b870e640 1 -- 172.21.15.72:0/2758862517 shutdown_connections 2024-04-15T15:16:53.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.060+0000 7f36b870e640 1 -- 172.21.15.72:0/2758862517 wait complete. 2024-04-15T15:16:53.074 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:39.359102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:16:05.551499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:16:06.003152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:16:53.075 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:53.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.179+0000 7f0b3ae4a640 1 Processor -- start 2024-04-15T15:16:53.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.179+0000 7f0b3ae4a640 1 -- start start 2024-04-15T15:16:53.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.180+0000 7f0b3ae4a640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f0b340cfe30 0x7f0b340cf040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:53.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.180+0000 7f0b3ae4a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b340ae160 con 0x7f0b340b4dc0 2024-04-15T15:16:53.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.180+0000 7f0b3ae4a640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b340b5310 con 0x7f0b3415e770 2024-04-15T15:16:53.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.180+0000 7f0b393e7640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f0b340b4dc0 0x7f0b340ae600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60042/0 (socket says 172.21.15.72:60042) 2024-04-15T15:16:53.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.180+0000 7f0b3ae4a640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f0b340cf580 con 0x7f0b340cfe30 2024-04-15T15:16:53.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.180+0000 7f0b393e7640 1 -- 172.21.15.72:0/1550697860 learned_addr learned my addr 172.21.15.72:0/1550697860 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:53.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.180+0000 7f0b2bfff640 1 --2- 172.21.15.72:0/1550697860 >> v2:172.21.15.175:3300/0 conn(0x7f0b340cfe30 0x7f0b340cf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:53.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.180+0000 7f0b2b7fe640 1 -- 172.21.15.72:0/1550697860 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4185842987 0 0) 0x7f0b340ae160 con 0x7f0b340b4dc0 2024-04-15T15:16:53.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.180+0000 7f0b2b7fe640 1 -- 172.21.15.72:0/1550697860 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b1c003520 con 0x7f0b340b4dc0 2024-04-15T15:16:53.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.180+0000 7f0b2b7fe640 1 -- 172.21.15.72:0/1550697860 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1246462981 0 0) 0x7f0b340b5310 con 0x7f0b3415e770 2024-04-15T15:16:53.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.180+0000 7f0b2b7fe640 1 -- 172.21.15.72:0/1550697860 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b340ae160 con 0x7f0b3415e770 2024-04-15T15:16:53.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.181+0000 7f0b2bfff640 1 -- 172.21.15.72:0/1550697860 >> v1:172.21.15.175:6789/0 conn(0x7f0b3415e770 legacy=0x7f0b340cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:53.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.181+0000 7f0b2bfff640 1 -- 172.21.15.72:0/1550697860 >> v1:172.21.15.72:6789/0 conn(0x7f0b340b4dc0 legacy=0x7f0b340ae600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:53.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.181+0000 7f0b2bfff640 1 -- 172.21.15.72:0/1550697860 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b340b55f0 con 0x7f0b340cfe30 2024-04-15T15:16:53.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.181+0000 7f0b2bfff640 1 --2- 172.21.15.72:0/1550697860 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0b340cfe30 0x7f0b340cf040 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0b240097a0 tx=0x7f0b2402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76c9b6400e933097 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:53.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.182+0000 7f0b2b7fe640 1 -- 172.21.15.72:0/1550697860 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b24034070 con 0x7f0b340cfe30 2024-04-15T15:16:53.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.182+0000 7f0b2b7fe640 1 -- 172.21.15.72:0/1550697860 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b24030610 con 0x7f0b340cfe30 2024-04-15T15:16:53.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.182+0000 7f0b2b7fe640 1 -- 172.21.15.72:0/1550697860 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b240309c0 con 0x7f0b340cfe30 2024-04-15T15:16:53.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.182+0000 7f0b3ae4a640 1 -- 172.21.15.72:0/1550697860 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0b340cfe30 msgr2=0x7f0b340cf040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:53.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.182+0000 7f0b3ae4a640 1 --2- 172.21.15.72:0/1550697860 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0b340cfe30 0x7f0b340cf040 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0b240097a0 tx=0x7f0b2402b960 comp rx=0 tx=0).stop 2024-04-15T15:16:53.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.182+0000 7f0b3ae4a640 1 -- 172.21.15.72:0/1550697860 shutdown_connections 2024-04-15T15:16:53.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.182+0000 7f0b3ae4a640 1 --2- 172.21.15.72:0/1550697860 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0b340cfe30 0x7f0b340cf040 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.182+0000 7f0b3ae4a640 1 -- 172.21.15.72:0/1550697860 >> 172.21.15.72:0/1550697860 conn(0x7f0b340d13a0 msgr2=0x7f0b340d1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:53.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.182+0000 7f0b3ae4a640 1 -- 172.21.15.72:0/1550697860 shutdown_connections 2024-04-15T15:16:53.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.182+0000 7f0b3ae4a640 1 -- 172.21.15.72:0/1550697860 wait complete. 2024-04-15T15:16:53.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.183+0000 7f0b3ae4a640 1 Processor -- start 2024-04-15T15:16:53.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.184+0000 7f0b3ae4a640 1 -- start start 2024-04-15T15:16:53.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.184+0000 7f0b3ae4a640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0b340b4dc0 0x7f0b341f3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:53.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.184+0000 7f0b3ae4a640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0b340cfe30 0x7f0b340ae490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:53.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.184+0000 7f0b3ae4a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0b3415e770 0x7f0b340b0390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:53.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.184+0000 7f0b3ae4a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3415eb50 con 0x7f0b3415e770 2024-04-15T15:16:53.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.184+0000 7f0b3ae4a640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f0b340b5bb0 con 0x7f0b340b4dc0 2024-04-15T15:16:53.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.184+0000 7f0b3ae4a640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f0b3415ecd0 con 0x7f0b340cfe30 2024-04-15T15:16:53.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.184+0000 7f0b393e7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0b3415e770 0x7f0b340b0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:53.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.185+0000 7f0b393e7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0b3415e770 0x7f0b340b0390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37566/0 (socket says 172.21.15.72:37566) 2024-04-15T15:16:53.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.185+0000 7f0b393e7640 1 -- 172.21.15.72:0/3337133797 learned_addr learned my addr 172.21.15.72:0/3337133797 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:53.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.185+0000 7f0b2bfff640 1 --2- 172.21.15.72:0/3337133797 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0b340cfe30 0x7f0b340ae490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:53.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.185+0000 7f0b38be6640 1 --2- 172.21.15.72:0/3337133797 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0b340b4dc0 0x7f0b341f3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:53.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.185+0000 7f0b393e7640 1 -- 172.21.15.72:0/3337133797 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0b340cfe30 msgr2=0x7f0b340ae490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:53.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.185+0000 7f0b393e7640 1 --2- 172.21.15.72:0/3337133797 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0b340cfe30 0x7f0b340ae490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.185+0000 7f0b393e7640 1 -- 172.21.15.72:0/3337133797 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0b340b4dc0 msgr2=0x7f0b341f3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:53.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.185+0000 7f0b393e7640 1 --2- 172.21.15.72:0/3337133797 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0b340b4dc0 0x7f0b341f3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.185+0000 7f0b393e7640 1 -- 172.21.15.72:0/3337133797 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b340aed50 con 0x7f0b3415e770 2024-04-15T15:16:53.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.185+0000 7f0b393e7640 1 --2- 172.21.15.72:0/3337133797 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0b3415e770 0x7f0b340b0390 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c00c410 tx=0x7f0b2c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:53.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.185+0000 7f0b297fa640 1 -- 172.21.15.72:0/3337133797 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c017070 con 0x7f0b3415e770 2024-04-15T15:16:53.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.185+0000 7f0b297fa640 1 -- 172.21.15.72:0/3337133797 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b2c005ce0 con 0x7f0b3415e770 2024-04-15T15:16:53.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.185+0000 7f0b297fa640 1 -- 172.21.15.72:0/3337133797 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c0072d0 con 0x7f0b3415e770 2024-04-15T15:16:53.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.185+0000 7f0b3ae4a640 1 -- 172.21.15.72:0/3337133797 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b340af030 con 0x7f0b3415e770 2024-04-15T15:16:53.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.185+0000 7f0b3ae4a640 1 -- 172.21.15.72:0/3337133797 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b340af5f0 con 0x7f0b3415e770 2024-04-15T15:16:53.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.186+0000 7f0b3ae4a640 1 -- 172.21.15.72:0/3337133797 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b34075c50 con 0x7f0b3415e770 2024-04-15T15:16:53.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.187+0000 7f0b297fa640 1 -- 172.21.15.72:0/3337133797 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f0b2c01a070 con 0x7f0b3415e770 2024-04-15T15:16:53.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.187+0000 7f0b297fa640 1 --2- 172.21.15.72:0/3337133797 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0b2006c980 0x7f0b2006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:53.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.187+0000 7f0b38be6640 1 --2- 172.21.15.72:0/3337133797 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0b2006c980 0x7f0b2006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:53.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.187+0000 7f0b297fa640 1 -- 172.21.15.72:0/3337133797 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7f0b2c08b010 con 0x7f0b3415e770 2024-04-15T15:16:53.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.188+0000 7f0b38be6640 1 --2- 172.21.15.72:0/3337133797 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0b2006c980 0x7f0b2006ee40 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0b30001560 tx=0x7f0b30009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:53.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.192+0000 7f0b297fa640 1 -- 172.21.15.72:0/3337133797 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0b2c058c90 con 0x7f0b3415e770 2024-04-15T15:16:53.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.366+0000 7f0b3ae4a640 1 -- 172.21.15.72:0/3337133797 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0b340cfc10 con 0x7f0b3415e770 2024-04-15T15:16:53.369 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.366+0000 7f0b297fa640 1 -- 172.21.15.72:0/3337133797 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) ==== 93+0+10 (secure 0 0 0) 0x7f0b2c05cf90 con 0x7f0b3415e770 2024-04-15T15:16:53.369 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-15T15:16:53.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.368+0000 7f0b3ae4a640 1 -- 172.21.15.72:0/3337133797 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0b2006c980 msgr2=0x7f0b2006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:53.371 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.368+0000 7f0b3ae4a640 1 --2- 172.21.15.72:0/3337133797 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0b2006c980 0x7f0b2006ee40 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0b30001560 tx=0x7f0b30009040 comp rx=0 tx=0).stop 2024-04-15T15:16:53.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.368+0000 7f0b3ae4a640 1 -- 172.21.15.72:0/3337133797 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0b3415e770 msgr2=0x7f0b340b0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:53.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.368+0000 7f0b3ae4a640 1 --2- 172.21.15.72:0/3337133797 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0b3415e770 0x7f0b340b0390 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c00c410 tx=0x7f0b2c00c8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.369+0000 7f0b3ae4a640 1 -- 172.21.15.72:0/3337133797 shutdown_connections 2024-04-15T15:16:53.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.369+0000 7f0b3ae4a640 1 --2- 172.21.15.72:0/3337133797 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f0b2006c980 0x7f0b2006ee40 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.369+0000 7f0b3ae4a640 1 --2- 172.21.15.72:0/3337133797 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0b3415e770 0x7f0b340b0390 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.369+0000 7f0b3ae4a640 1 --2- 172.21.15.72:0/3337133797 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f0b340cfe30 0x7f0b340ae490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.369+0000 7f0b3ae4a640 1 --2- 172.21.15.72:0/3337133797 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f0b340b4dc0 0x7f0b341f3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.369+0000 7f0b3ae4a640 1 -- 172.21.15.72:0/3337133797 >> 172.21.15.72:0/3337133797 conn(0x7f0b340d13a0 msgr2=0x7f0b3415f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:53.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.369+0000 7f0b3ae4a640 1 -- 172.21.15.72:0/3337133797 shutdown_connections 2024-04-15T15:16:53.372 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.369+0000 7f0b3ae4a640 1 -- 172.21.15.72:0/3337133797 wait complete. 2024-04-15T15:16:53.383 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:16:53.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.486+0000 7fb681152640 1 Processor -- start 2024-04-15T15:16:53.488 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.486+0000 7fb681152640 1 -- start start 2024-04-15T15:16:53.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.487+0000 7fb681152640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb67c0d1810 0x7fb67c0b6750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:53.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.487+0000 7fb681152640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb67c160ff0 con 0x7fb67c0ce190 2024-04-15T15:16:53.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.487+0000 7fb681152640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb67c0b5840 con 0x7fb67c15c5f0 2024-04-15T15:16:53.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.487+0000 7fb681152640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fb67c160da0 con 0x7fb67c0d1810 2024-04-15T15:16:53.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.487+0000 7fb67b59e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb67c0d1810 0x7fb67c0b6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:53.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.487+0000 7fb67a59c640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fb67c0ce190 0x7fb67c0b6040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60044/0 (socket says 172.21.15.72:60044) 2024-04-15T15:16:53.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.487+0000 7fb67a59c640 1 -- 172.21.15.72:0/3867623858 learned_addr learned my addr 172.21.15.72:0/3867623858 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:53.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.487+0000 7fb67b59e640 1 -- 172.21.15.72:0/3867623858 >> v1:172.21.15.175:6789/0 conn(0x7fb67c15c5f0 legacy=0x7fb67c0d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:53.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.487+0000 7fb67b59e640 1 -- 172.21.15.72:0/3867623858 >> v1:172.21.15.72:6789/0 conn(0x7fb67c0ce190 legacy=0x7fb67c0b6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:53.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.487+0000 7fb67b59e640 1 -- 172.21.15.72:0/3867623858 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb67c0d2150 con 0x7fb67c0d1810 2024-04-15T15:16:53.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.487+0000 7fb679d9b640 1 -- 172.21.15.72:0/3867623858 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 91188952 0 0) 0x7fb67c160ff0 con 0x7fb67c0ce190 2024-04-15T15:16:53.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.488+0000 7fb67b59e640 1 --2- 172.21.15.72:0/3867623858 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb67c0d1810 0x7fb67c0b6750 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fb674009d90 tx=0x7fb67402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf0244dcffe863b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:53.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.488+0000 7fb679d9b640 1 -- 172.21.15.72:0/3867623858 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb674039070 con 0x7fb67c0d1810 2024-04-15T15:16:53.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.488+0000 7fb679d9b640 1 -- 172.21.15.72:0/3867623858 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb67402cbd0 con 0x7fb67c0d1810 2024-04-15T15:16:53.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.488+0000 7fb679d9b640 1 -- 172.21.15.72:0/3867623858 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb674034440 con 0x7fb67c0d1810 2024-04-15T15:16:53.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.488+0000 7fb681152640 1 -- 172.21.15.72:0/3867623858 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb67c0d1810 msgr2=0x7fb67c0b6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:53.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.488+0000 7fb681152640 1 --2- 172.21.15.72:0/3867623858 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb67c0d1810 0x7fb67c0b6750 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fb674009d90 tx=0x7fb67402be20 comp rx=0 tx=0).stop 2024-04-15T15:16:53.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.488+0000 7fb681152640 1 -- 172.21.15.72:0/3867623858 shutdown_connections 2024-04-15T15:16:53.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.488+0000 7fb681152640 1 --2- 172.21.15.72:0/3867623858 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb67c0d1810 0x7fb67c0b6750 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.488+0000 7fb681152640 1 -- 172.21.15.72:0/3867623858 >> 172.21.15.72:0/3867623858 conn(0x7fb67c0b4010 msgr2=0x7fb67c0b4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:53.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.488+0000 7fb681152640 1 -- 172.21.15.72:0/3867623858 shutdown_connections 2024-04-15T15:16:53.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.489+0000 7fb681152640 1 -- 172.21.15.72:0/3867623858 wait complete. 2024-04-15T15:16:53.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.490+0000 7fb681152640 1 Processor -- start 2024-04-15T15:16:53.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.490+0000 7fb681152640 1 -- start start 2024-04-15T15:16:53.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.491+0000 7fb681152640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb67c0d1810 0x7fb67c150b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:53.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.491+0000 7fb681152640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb67c15c5f0 0x7fb67c0ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:53.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.491+0000 7fb681152640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb67c0cc290 0x7fb67c0b0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:53.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.491+0000 7fb681152640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c1612d0 con 0x7fb67c15c5f0 2024-04-15T15:16:53.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.491+0000 7fb681152640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c0b5bb0 con 0x7fb67c0d1810 2024-04-15T15:16:53.495 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.491+0000 7fb681152640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fb67c06f510 con 0x7fb67c0cc290 2024-04-15T15:16:53.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.491+0000 7fb67a59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb67c15c5f0 0x7fb67c0ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:53.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.491+0000 7fb67a59c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb67c15c5f0 0x7fb67c0ad8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37576/0 (socket says 172.21.15.72:37576) 2024-04-15T15:16:53.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.491+0000 7fb67a59c640 1 -- 172.21.15.72:0/185696081 learned_addr learned my addr 172.21.15.72:0/185696081 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:53.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.491+0000 7fb67ad9d640 1 --2- 172.21.15.72:0/185696081 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb67c0d1810 0x7fb67c150b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:53.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.491+0000 7fb67b59e640 1 --2- 172.21.15.72:0/185696081 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb67c0cc290 0x7fb67c0b0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:53.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.491+0000 7fb67a59c640 1 -- 172.21.15.72:0/185696081 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb67c0cc290 msgr2=0x7fb67c0b0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:53.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.491+0000 7fb67a59c640 1 --2- 172.21.15.72:0/185696081 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb67c0cc290 0x7fb67c0b0b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.491+0000 7fb67a59c640 1 -- 172.21.15.72:0/185696081 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb67c0d1810 msgr2=0x7fb67c150b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:53.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.491+0000 7fb67a59c640 1 --2- 172.21.15.72:0/185696081 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb67c0d1810 0x7fb67c150b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.491+0000 7fb67a59c640 1 -- 172.21.15.72:0/185696081 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb67c151060 con 0x7fb67c15c5f0 2024-04-15T15:16:53.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.491+0000 7fb67a59c640 1 --2- 172.21.15.72:0/185696081 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb67c15c5f0 0x7fb67c0ad8d0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fb67000c500 tx=0x7fb67000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:53.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.491+0000 7fb6637fe640 1 -- 172.21.15.72:0/185696081 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb670018070 con 0x7fb67c15c5f0 2024-04-15T15:16:53.496 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.492+0000 7fb6637fe640 1 -- 172.21.15.72:0/185696081 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb670007580 con 0x7fb67c15c5f0 2024-04-15T15:16:53.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.492+0000 7fb681152640 1 -- 172.21.15.72:0/185696081 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb67c1e7310 con 0x7fb67c15c5f0 2024-04-15T15:16:53.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.492+0000 7fb6637fe640 1 -- 172.21.15.72:0/185696081 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb670004a20 con 0x7fb67c15c5f0 2024-04-15T15:16:53.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.492+0000 7fb681152640 1 -- 172.21.15.72:0/185696081 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb67c1e78d0 con 0x7fb67c15c5f0 2024-04-15T15:16:53.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.493+0000 7fb681152640 1 -- 172.21.15.72:0/185696081 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb67c075c50 con 0x7fb67c15c5f0 2024-04-15T15:16:53.497 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.493+0000 7fb6637fe640 1 -- 172.21.15.72:0/185696081 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fb67001b070 con 0x7fb67c15c5f0 2024-04-15T15:16:53.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.493+0000 7fb6637fe640 1 --2- 172.21.15.72:0/185696081 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb66806c980 0x7fb66806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:53.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.494+0000 7fb6637fe640 1 -- 172.21.15.72:0/185696081 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7fb67008b150 con 0x7fb67c15c5f0 2024-04-15T15:16:53.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.494+0000 7fb67ad9d640 1 --2- 172.21.15.72:0/185696081 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb66806c980 0x7fb66806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:53.498 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.494+0000 7fb67ad9d640 1 --2- 172.21.15.72:0/185696081 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb66806c980 0x7fb66806ee40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb66c00a730 tx=0x7fb66c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:53.500 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.498+0000 7fb6637fe640 1 -- 172.21.15.72:0/185696081 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb670059320 con 0x7fb67c15c5f0 2024-04-15T15:16:53.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.670+0000 7fb681152640 1 -- 172.21.15.72:0/185696081 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb67c0d1e50 con 0x7fb67c15c5f0 2024-04-15T15:16:53.673 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.671+0000 7fb6637fe640 1 -- 172.21.15.72:0/185696081 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v41) ==== 104+0+11 (secure 0 0 0) 0x7fb670058c20 con 0x7fb67c15c5f0 2024-04-15T15:16:53.673 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:16:53.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.673+0000 7fb681152640 1 -- 172.21.15.72:0/185696081 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb66806c980 msgr2=0x7fb66806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:53.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.673+0000 7fb681152640 1 --2- 172.21.15.72:0/185696081 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb66806c980 0x7fb66806ee40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb66c00a730 tx=0x7fb66c008040 comp rx=0 tx=0).stop 2024-04-15T15:16:53.675 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.673+0000 7fb681152640 1 -- 172.21.15.72:0/185696081 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb67c15c5f0 msgr2=0x7fb67c0ad8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:53.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.673+0000 7fb681152640 1 --2- 172.21.15.72:0/185696081 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb67c15c5f0 0x7fb67c0ad8d0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fb67000c500 tx=0x7fb67000c9c0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.673+0000 7fb681152640 1 -- 172.21.15.72:0/185696081 shutdown_connections 2024-04-15T15:16:53.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.673+0000 7fb681152640 1 --2- 172.21.15.72:0/185696081 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb66806c980 0x7fb66806ee40 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.673+0000 7fb681152640 1 --2- 172.21.15.72:0/185696081 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb67c0cc290 0x7fb67c0b0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.673+0000 7fb681152640 1 --2- 172.21.15.72:0/185696081 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb67c15c5f0 0x7fb67c0ad8d0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.673+0000 7fb681152640 1 --2- 172.21.15.72:0/185696081 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb67c0d1810 0x7fb67c150b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.673+0000 7fb681152640 1 -- 172.21.15.72:0/185696081 >> 172.21.15.72:0/185696081 conn(0x7fb67c0b4010 msgr2=0x7fb67c070800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:53.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.673+0000 7fb681152640 1 -- 172.21.15.72:0/185696081 shutdown_connections 2024-04-15T15:16:53.676 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.673+0000 7fb681152640 1 -- 172.21.15.72:0/185696081 wait complete. 2024-04-15T15:16:53.687 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:16:53.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.791+0000 7fac19f3d640 1 Processor -- start 2024-04-15T15:16:53.794 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.791+0000 7fac19f3d640 1 -- start start 2024-04-15T15:16:53.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.791+0000 7fac19f3d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fac140b5410 0x7fac140b57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:53.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.792+0000 7fac19f3d640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fac140afae0 0x7fac140cf960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:53.796 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.792+0000 7fac19f3d640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac140d07c0 con 0x7fac140b0050 2024-04-15T15:16:53.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.792+0000 7fac19f3d640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fac140d0330 con 0x7fac140b5410 2024-04-15T15:16:53.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.792+0000 7fac19f3d640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fac140b5d30 con 0x7fac140afae0 2024-04-15T15:16:53.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.792+0000 7fac137fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fac140b5410 0x7fac140b57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:53.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.792+0000 7fac137fe640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fac140b5410 0x7fac140b57f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37584/0 (socket says 172.21.15.72:37584) 2024-04-15T15:16:53.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.792+0000 7fac137fe640 1 -- 172.21.15.72:0/3889895530 learned_addr learned my addr 172.21.15.72:0/3889895530 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:53.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.792+0000 7fac13fff640 1 --2- 172.21.15.72:0/3889895530 >> v2:172.21.15.175:3300/0 conn(0x7fac140afae0 0x7fac140cf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:53.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.792+0000 7fac137fe640 1 -- 172.21.15.72:0/3889895530 >> v2:172.21.15.175:3300/0 conn(0x7fac140afae0 msgr2=0x7fac140cf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:53.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.792+0000 7fac137fe640 1 --2- 172.21.15.72:0/3889895530 >> v2:172.21.15.175:3300/0 conn(0x7fac140afae0 0x7fac140cf960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.792+0000 7fac137fe640 1 -- 172.21.15.72:0/3889895530 >> v1:172.21.15.175:6789/0 conn(0x7fac140b0050 legacy=0x7fac1406c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:53.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.792+0000 7fac137fe640 1 -- 172.21.15.72:0/3889895530 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac140b3aa0 con 0x7fac140b5410 2024-04-15T15:16:53.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.793+0000 7fac137fe640 1 --2- 172.21.15.72:0/3889895530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac140b5410 0x7fac140b57f0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fac14070430 tx=0x7fac0402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4afa72e5d568998 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:53.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.793+0000 7fac127fc640 1 -- 172.21.15.72:0/3889895530 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac04035020 con 0x7fac140b5410 2024-04-15T15:16:53.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.793+0000 7fac127fc640 1 -- 172.21.15.72:0/3889895530 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac04031870 con 0x7fac140b5410 2024-04-15T15:16:53.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.793+0000 7fac127fc640 1 -- 172.21.15.72:0/3889895530 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac04031c40 con 0x7fac140b5410 2024-04-15T15:16:53.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.793+0000 7fac19f3d640 1 -- 172.21.15.72:0/3889895530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac140b5410 msgr2=0x7fac140b57f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:53.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.793+0000 7fac19f3d640 1 --2- 172.21.15.72:0/3889895530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac140b5410 0x7fac140b57f0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fac14070430 tx=0x7fac0402b910 comp rx=0 tx=0).stop 2024-04-15T15:16:53.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.793+0000 7fac19f3d640 1 -- 172.21.15.72:0/3889895530 shutdown_connections 2024-04-15T15:16:53.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.793+0000 7fac19f3d640 1 --2- 172.21.15.72:0/3889895530 >> v2:172.21.15.175:3300/0 conn(0x7fac140afae0 0x7fac140cf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.793+0000 7fac19f3d640 1 --2- 172.21.15.72:0/3889895530 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac140b5410 0x7fac140b57f0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.793+0000 7fac19f3d640 1 -- 172.21.15.72:0/3889895530 >> 172.21.15.72:0/3889895530 conn(0x7fac140d0a50 msgr2=0x7fac140d0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:53.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.794+0000 7fac19f3d640 1 -- 172.21.15.72:0/3889895530 shutdown_connections 2024-04-15T15:16:53.798 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.794+0000 7fac19f3d640 1 -- 172.21.15.72:0/3889895530 wait complete. 2024-04-15T15:16:53.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.795+0000 7fac19f3d640 1 Processor -- start 2024-04-15T15:16:53.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.795+0000 7fac19f3d640 1 -- start start 2024-04-15T15:16:53.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.795+0000 7fac19f3d640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac140afae0 0x7fac1414f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:53.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.795+0000 7fac19f3d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac140b0050 0x7fac1414dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:53.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.795+0000 7fac19f3d640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fac140b5410 0x7fac14151250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:53.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.795+0000 7fac19f3d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fac140d18f0 con 0x7fac140b0050 2024-04-15T15:16:53.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.796+0000 7fac19f3d640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fac140d1610 con 0x7fac140afae0 2024-04-15T15:16:53.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.796+0000 7fac19f3d640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fac140cfea0 con 0x7fac140b5410 2024-04-15T15:16:53.801 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.796+0000 7fac12ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac140b0050 0x7fac1414dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:53.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.796+0000 7fac12ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac140b0050 0x7fac1414dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37590/0 (socket says 172.21.15.72:37590) 2024-04-15T15:16:53.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.796+0000 7fac12ffd640 1 -- 172.21.15.72:0/4212143764 learned_addr learned my addr 172.21.15.72:0/4212143764 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:53.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.796+0000 7fac13fff640 1 --2- 172.21.15.72:0/4212143764 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fac140b5410 0x7fac14151250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:53.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.796+0000 7fac137fe640 1 --2- 172.21.15.72:0/4212143764 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac140afae0 0x7fac1414f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:53.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.796+0000 7fac12ffd640 1 -- 172.21.15.72:0/4212143764 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fac140b5410 msgr2=0x7fac14151250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:53.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.796+0000 7fac12ffd640 1 --2- 172.21.15.72:0/4212143764 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fac140b5410 0x7fac14151250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.796+0000 7fac12ffd640 1 -- 172.21.15.72:0/4212143764 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac140afae0 msgr2=0x7fac1414f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:53.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.796+0000 7fac12ffd640 1 --2- 172.21.15.72:0/4212143764 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac140afae0 0x7fac1414f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.796+0000 7fac12ffd640 1 -- 172.21.15.72:0/4212143764 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac1414e6d0 con 0x7fac140b0050 2024-04-15T15:16:53.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.796+0000 7fac12ffd640 1 --2- 172.21.15.72:0/4212143764 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac140b0050 0x7fac1414dd60 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fac0800c3c0 tx=0x7fac0800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:53.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.796+0000 7fabfbfff640 1 -- 172.21.15.72:0/4212143764 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac080052d0 con 0x7fac140b0050 2024-04-15T15:16:53.802 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.796+0000 7fabfbfff640 1 -- 172.21.15.72:0/4212143764 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac08005490 con 0x7fac140b0050 2024-04-15T15:16:53.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.796+0000 7fabfbfff640 1 -- 172.21.15.72:0/4212143764 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac08007260 con 0x7fac140b0050 2024-04-15T15:16:53.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.796+0000 7fac19f3d640 1 -- 172.21.15.72:0/4212143764 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac1414e9b0 con 0x7fac140b0050 2024-04-15T15:16:53.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.796+0000 7fac19f3d640 1 -- 172.21.15.72:0/4212143764 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac1414ef70 con 0x7fac140b0050 2024-04-15T15:16:53.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.797+0000 7fac19f3d640 1 -- 172.21.15.72:0/4212143764 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac14075c50 con 0x7fac140b0050 2024-04-15T15:16:53.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.798+0000 7fabfbfff640 1 -- 172.21.15.72:0/4212143764 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fac08007400 con 0x7fac140b0050 2024-04-15T15:16:53.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.798+0000 7fabfbfff640 1 --2- 172.21.15.72:0/4212143764 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fac0006c980 0x7fac0006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:53.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.798+0000 7fabfbfff640 1 -- 172.21.15.72:0/4212143764 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7fac0808b280 con 0x7fac140b0050 2024-04-15T15:16:53.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.798+0000 7fac137fe640 1 --2- 172.21.15.72:0/4212143764 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fac0006c980 0x7fac0006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:53.803 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.799+0000 7fac137fe640 1 --2- 172.21.15.72:0/4212143764 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fac0006c980 0x7fac0006ee40 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fac04036000 tx=0x7fac0403c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:53.805 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.802+0000 7fabfbfff640 1 -- 172.21.15.72:0/4212143764 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fac080594c0 con 0x7fac140b0050 2024-04-15T15:16:53.981 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.979+0000 7fac19f3d640 1 -- 172.21.15.72:0/4212143764 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fac1406d610 con 0x7fac140b0050 2024-04-15T15:16:53.982 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.979+0000 7fabfbfff640 1 -- 172.21.15.72:0/4212143764 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v41) ==== 100+0+11 (secure 0 0 0) 0x7fac08058dc0 con 0x7fac140b0050 2024-04-15T15:16:53.982 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-04-15T15:16:53.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.981+0000 7fac19f3d640 1 -- 172.21.15.72:0/4212143764 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fac0006c980 msgr2=0x7fac0006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:53.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.981+0000 7fac19f3d640 1 --2- 172.21.15.72:0/4212143764 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fac0006c980 0x7fac0006ee40 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fac04036000 tx=0x7fac0403c040 comp rx=0 tx=0).stop 2024-04-15T15:16:53.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.981+0000 7fac19f3d640 1 -- 172.21.15.72:0/4212143764 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac140b0050 msgr2=0x7fac1414dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:53.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.981+0000 7fac19f3d640 1 --2- 172.21.15.72:0/4212143764 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac140b0050 0x7fac1414dd60 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fac0800c3c0 tx=0x7fac0800c880 comp rx=0 tx=0).stop 2024-04-15T15:16:53.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.981+0000 7fac19f3d640 1 -- 172.21.15.72:0/4212143764 shutdown_connections 2024-04-15T15:16:53.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.981+0000 7fac19f3d640 1 --2- 172.21.15.72:0/4212143764 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fac0006c980 0x7fac0006ee40 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.981+0000 7fac19f3d640 1 --2- 172.21.15.72:0/4212143764 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fac140b5410 0x7fac14151250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.981+0000 7fac19f3d640 1 --2- 172.21.15.72:0/4212143764 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fac140b0050 0x7fac1414dd60 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.981+0000 7fac19f3d640 1 --2- 172.21.15.72:0/4212143764 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fac140afae0 0x7fac1414f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.981+0000 7fac19f3d640 1 -- 172.21.15.72:0/4212143764 >> 172.21.15.72:0/4212143764 conn(0x7fac140d0a50 msgr2=0x7fac1415e690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:53.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.981+0000 7fac19f3d640 1 -- 172.21.15.72:0/4212143764 shutdown_connections 2024-04-15T15:16:53.985 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:53.981+0000 7fac19f3d640 1 -- 172.21.15.72:0/4212143764 wait complete. 2024-04-15T15:16:53.996 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:16:54.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.097+0000 7fcaf9293640 1 Processor -- start 2024-04-15T15:16:54.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.097+0000 7fcaf9293640 1 -- start start 2024-04-15T15:16:54.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.098+0000 7fcaf9293640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fcaf406c400 0x7fcaf406c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:54.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.098+0000 7fcaf9293640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcaf40d2320 con 0x7fcaf415e9f0 2024-04-15T15:16:54.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.098+0000 7fcaf9293640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcaf40d25c0 con 0x7fcaf415e610 2024-04-15T15:16:54.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.098+0000 7fcaf9293640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fcaf40d31d0 con 0x7fcaf406c400 2024-04-15T15:16:54.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.098+0000 7fcaf27fc640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fcaf415e9f0 0x7fcaf40d3660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60054/0 (socket says 172.21.15.72:60054) 2024-04-15T15:16:54.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.098+0000 7fcaf27fc640 1 -- 172.21.15.72:0/3731564638 learned_addr learned my addr 172.21.15.72:0/3731564638 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:54.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.098+0000 7fcaf37fe640 1 --2- 172.21.15.72:0/3731564638 >> v2:172.21.15.175:3300/0 conn(0x7fcaf406c400 0x7fcaf406c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:54.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.098+0000 7fcaf1ffb640 1 -- 172.21.15.72:0/3731564638 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2429628260 0 0) 0x7fcaf40d2320 con 0x7fcaf415e9f0 2024-04-15T15:16:54.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.098+0000 7fcaf1ffb640 1 -- 172.21.15.72:0/3731564638 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcadc003520 con 0x7fcaf415e9f0 2024-04-15T15:16:54.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.098+0000 7fcaf1ffb640 1 -- 172.21.15.72:0/3731564638 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4165425342 0 0) 0x7fcadc003520 con 0x7fcaf415e9f0 2024-04-15T15:16:54.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.098+0000 7fcaf1ffb640 1 -- 172.21.15.72:0/3731564638 >> v2:172.21.15.175:3300/0 conn(0x7fcaf406c400 msgr2=0x7fcaf406c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:54.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.098+0000 7fcaf1ffb640 1 --2- 172.21.15.72:0/3731564638 >> v2:172.21.15.175:3300/0 conn(0x7fcaf406c400 0x7fcaf406c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.098+0000 7fcaf1ffb640 1 -- 172.21.15.72:0/3731564638 >> v1:172.21.15.175:6789/0 conn(0x7fcaf415e610 legacy=0x7fcaf40d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:54.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.098+0000 7fcaf1ffb640 1 -- 172.21.15.72:0/3731564638 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaf40d28a0 con 0x7fcaf415e9f0 2024-04-15T15:16:54.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.099+0000 7fcaf1ffb640 1 -- 172.21.15.72:0/3731564638 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7fcae8001690 con 0x7fcaf415e9f0 2024-04-15T15:16:54.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.099+0000 7fcaf37fe640 1 --2- 172.21.15.72:0/3731564638 >> v2:172.21.15.175:3300/0 conn(0x7fcaf406c400 0x7fcaf406c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:54.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.099+0000 7fcaf1ffb640 1 -- 172.21.15.72:0/3731564638 >> v1:172.21.15.72:6789/0 conn(0x7fcaf415e9f0 legacy=0x7fcaf40d3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:54.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.099+0000 7fcaf1ffb640 1 --2- 172.21.15.72:0/3731564638 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcadc003a00 0x7fcadc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:54.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.099+0000 7fcaf1ffb640 1 --2- 172.21.15.72:0/3731564638 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcadc004800 0x7fcadc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:54.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.099+0000 7fcaf1ffb640 1 --2- 172.21.15.72:0/3731564638 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcadc0051c0 0x7fcadc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:54.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.099+0000 7fcaf1ffb640 1 -- 172.21.15.72:0/3731564638 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf40d31d0 con 0x7fcadc0051c0 2024-04-15T15:16:54.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.099+0000 7fcaf1ffb640 1 -- 172.21.15.72:0/3731564638 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fcadc00c0a0 con 0x7fcadc003a00 2024-04-15T15:16:54.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.099+0000 7fcaf1ffb640 1 -- 172.21.15.72:0/3731564638 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fcadc00c2a0 con 0x7fcadc004800 2024-04-15T15:16:54.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.099+0000 7fcaf37fe640 1 --2- 172.21.15.72:0/3731564638 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcadc0051c0 0x7fcadc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:54.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.099+0000 7fcaf27fc640 1 --2- 172.21.15.72:0/3731564638 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcadc004800 0x7fcadc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:54.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.099+0000 7fcaf2ffd640 1 --2- 172.21.15.72:0/3731564638 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcadc003a00 0x7fcadc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:54.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.099+0000 7fcaf37fe640 1 -- 172.21.15.72:0/3731564638 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcadc004800 msgr2=0x7fcadc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:54.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.099+0000 7fcaf37fe640 1 --2- 172.21.15.72:0/3731564638 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcadc004800 0x7fcadc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.099+0000 7fcaf37fe640 1 -- 172.21.15.72:0/3731564638 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcadc003a00 msgr2=0x7fcadc003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:54.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.099+0000 7fcaf37fe640 1 --2- 172.21.15.72:0/3731564638 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcadc003a00 0x7fcadc003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.099+0000 7fcaf37fe640 1 -- 172.21.15.72:0/3731564638 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcadc003520 con 0x7fcadc0051c0 2024-04-15T15:16:54.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.099+0000 7fcaf27fc640 1 --2- 172.21.15.72:0/3731564638 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcadc004800 0x7fcadc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:54.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.099+0000 7fcaf2ffd640 1 --2- 172.21.15.72:0/3731564638 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcadc003a00 0x7fcadc003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:54.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.100+0000 7fcaf37fe640 1 -- 172.21.15.72:0/3731564638 reap_dead start 2024-04-15T15:16:54.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.100+0000 7fcaf37fe640 1 --2- 172.21.15.72:0/3731564638 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcadc0051c0 0x7fcadc00b9c0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fcaec00e350 tx=0x7fcadc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbd0b69557c6b9ae server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:54.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.100+0000 7fcaf1ffb640 1 -- 172.21.15.72:0/3731564638 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaec009de0 con 0x7fcadc0051c0 2024-04-15T15:16:54.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.100+0000 7fcaf1ffb640 1 -- 172.21.15.72:0/3731564638 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcaec02ce40 con 0x7fcadc0051c0 2024-04-15T15:16:54.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.100+0000 7fcaf1ffb640 1 -- 172.21.15.72:0/3731564638 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaec003bd0 con 0x7fcadc0051c0 2024-04-15T15:16:54.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.100+0000 7fcaf9293640 1 -- 172.21.15.72:0/3731564638 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcadc0051c0 msgr2=0x7fcadc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:54.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.100+0000 7fcaf9293640 1 --2- 172.21.15.72:0/3731564638 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcadc0051c0 0x7fcadc00b9c0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fcaec00e350 tx=0x7fcadc005830 comp rx=0 tx=0).stop 2024-04-15T15:16:54.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.100+0000 7fcaf9293640 1 -- 172.21.15.72:0/3731564638 shutdown_connections 2024-04-15T15:16:54.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.100+0000 7fcaf9293640 1 --2- 172.21.15.72:0/3731564638 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcadc0051c0 0x7fcadc00b9c0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.100+0000 7fcaf9293640 1 -- 172.21.15.72:0/3731564638 >> 172.21.15.72:0/3731564638 conn(0x7fcaf40d4750 msgr2=0x7fcaf40d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:54.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.100+0000 7fcaf9293640 1 -- 172.21.15.72:0/3731564638 shutdown_connections 2024-04-15T15:16:54.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.100+0000 7fcaf9293640 1 -- 172.21.15.72:0/3731564638 wait complete. 2024-04-15T15:16:54.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.102+0000 7fcaf9293640 1 Processor -- start 2024-04-15T15:16:54.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.102+0000 7fcaf9293640 1 -- start start 2024-04-15T15:16:54.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.102+0000 7fcaf9293640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcadc0051c0 0x7fcaf4148990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:54.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.102+0000 7fcaf9293640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcaf4145480 0x7fcaf4155630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:54.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.102+0000 7fcaf9293640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcaf40cd9b0 0x7fcaf40cde30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:54.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.102+0000 7fcaf9293640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf406d130 con 0x7fcaf40cd9b0 2024-04-15T15:16:54.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.102+0000 7fcaf9293640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fcaf40d2e60 con 0x7fcadc0051c0 2024-04-15T15:16:54.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.102+0000 7fcaf9293640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fcaf406c890 con 0x7fcaf4145480 2024-04-15T15:16:54.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.103+0000 7fcaf37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcaf40cd9b0 0x7fcaf40cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:54.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.103+0000 7fcaf37fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcaf40cd9b0 0x7fcaf40cde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37614/0 (socket says 172.21.15.72:37614) 2024-04-15T15:16:54.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.103+0000 7fcaf37fe640 1 -- 172.21.15.72:0/3361158185 learned_addr learned my addr 172.21.15.72:0/3361158185 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:54.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.103+0000 7fcaf27fc640 1 --2- 172.21.15.72:0/3361158185 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcaf4145480 0x7fcaf4155630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:54.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.103+0000 7fcaf2ffd640 1 --2- 172.21.15.72:0/3361158185 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcadc0051c0 0x7fcaf4148990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:54.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.103+0000 7fcaf37fe640 1 -- 172.21.15.72:0/3361158185 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcaf4145480 msgr2=0x7fcaf4155630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:54.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.103+0000 7fcaf37fe640 1 --2- 172.21.15.72:0/3361158185 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcaf4145480 0x7fcaf4155630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.103+0000 7fcaf37fe640 1 -- 172.21.15.72:0/3361158185 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcadc0051c0 msgr2=0x7fcaf4148990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:54.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.103+0000 7fcaf37fe640 1 --2- 172.21.15.72:0/3361158185 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcadc0051c0 0x7fcaf4148990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.103+0000 7fcaf37fe640 1 -- 172.21.15.72:0/3361158185 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaf4158260 con 0x7fcaf40cd9b0 2024-04-15T15:16:54.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.103+0000 7fcaf27fc640 1 --2- 172.21.15.72:0/3361158185 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcaf4145480 0x7fcaf4155630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:54.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.103+0000 7fcaf2ffd640 1 --2- 172.21.15.72:0/3361158185 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcadc0051c0 0x7fcaf4148990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:54.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.103+0000 7fcaf37fe640 1 --2- 172.21.15.72:0/3361158185 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcaf40cd9b0 0x7fcaf40cde30 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fcaec00e440 tx=0x7fcaec009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:54.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.103+0000 7fcadb7fe640 1 -- 172.21.15.72:0/3361158185 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaec009de0 con 0x7fcaf40cd9b0 2024-04-15T15:16:54.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.103+0000 7fcadb7fe640 1 -- 172.21.15.72:0/3361158185 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcaec046070 con 0x7fcaf40cd9b0 2024-04-15T15:16:54.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.103+0000 7fcaf9293640 1 -- 172.21.15.72:0/3361158185 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcaf4158500 con 0x7fcaf40cd9b0 2024-04-15T15:16:54.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.103+0000 7fcadb7fe640 1 -- 172.21.15.72:0/3361158185 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaec03f640 con 0x7fcaf40cd9b0 2024-04-15T15:16:54.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.103+0000 7fcaf9293640 1 -- 172.21.15.72:0/3361158185 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcaf40cd190 con 0x7fcaf40cd9b0 2024-04-15T15:16:54.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.104+0000 7fcaf9293640 1 -- 172.21.15.72:0/3361158185 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcaf4075c50 con 0x7fcaf40cd9b0 2024-04-15T15:16:54.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.105+0000 7fcadb7fe640 1 -- 172.21.15.72:0/3361158185 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fcaec03d040 con 0x7fcaf40cd9b0 2024-04-15T15:16:54.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.105+0000 7fcadb7fe640 1 --2- 172.21.15.72:0/3361158185 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcae006c980 0x7fcae006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:54.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.105+0000 7fcadb7fe640 1 -- 172.21.15.72:0/3361158185 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7fcaec0ae5a0 con 0x7fcaf40cd9b0 2024-04-15T15:16:54.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.105+0000 7fcaf2ffd640 1 --2- 172.21.15.72:0/3361158185 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcae006c980 0x7fcae006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:54.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.106+0000 7fcaf2ffd640 1 --2- 172.21.15.72:0/3361158185 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcae006c980 0x7fcae006ee40 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fcae4000c00 tx=0x7fcae4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:54.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.109+0000 7fcadb7fe640 1 -- 172.21.15.72:0/3361158185 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcaec07c7e0 con 0x7fcaf40cd9b0 2024-04-15T15:16:54.298 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.296+0000 7fcaf9293640 1 -- 172.21.15.72:0/3361158185 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fcaf40cf740 con 0x7fcaf40cd9b0 2024-04-15T15:16:54.299 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.296+0000 7fcadb7fe640 1 -- 172.21.15.72:0/3361158185 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v41) ==== 80+0+122 (secure 0 0 0) 0x7fcaec07c0e0 con 0x7fcaf40cd9b0 2024-04-15T15:16:54.299 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6836/4270793613 2024-04-16T15:16:41.714817+0000 2024-04-15T15:16:54.299 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72:6834/4270793613 2024-04-16T15:16:41.714817+0000 2024-04-15T15:16:54.299 INFO:teuthology.orchestra.run.smithi072.stderr:listed 2 entries 2024-04-15T15:16:54.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.298+0000 7fcaf9293640 1 -- 172.21.15.72:0/3361158185 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcae006c980 msgr2=0x7fcae006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:54.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.298+0000 7fcaf9293640 1 --2- 172.21.15.72:0/3361158185 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcae006c980 0x7fcae006ee40 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fcae4000c00 tx=0x7fcae4009040 comp rx=0 tx=0).stop 2024-04-15T15:16:54.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.298+0000 7fcaf9293640 1 -- 172.21.15.72:0/3361158185 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcaf40cd9b0 msgr2=0x7fcaf40cde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:54.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.298+0000 7fcaf9293640 1 --2- 172.21.15.72:0/3361158185 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcaf40cd9b0 0x7fcaf40cde30 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fcaec00e440 tx=0x7fcaec009750 comp rx=0 tx=0).stop 2024-04-15T15:16:54.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.298+0000 7fcaf9293640 1 -- 172.21.15.72:0/3361158185 shutdown_connections 2024-04-15T15:16:54.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.298+0000 7fcaf9293640 1 --2- 172.21.15.72:0/3361158185 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcae006c980 0x7fcae006ee40 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.298+0000 7fcaf9293640 1 --2- 172.21.15.72:0/3361158185 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcaf40cd9b0 0x7fcaf40cde30 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.298+0000 7fcaf9293640 1 --2- 172.21.15.72:0/3361158185 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcaf4145480 0x7fcaf4155630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.298+0000 7fcaf9293640 1 --2- 172.21.15.72:0/3361158185 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcadc0051c0 0x7fcaf4148990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.298+0000 7fcaf9293640 1 -- 172.21.15.72:0/3361158185 >> 172.21.15.72:0/3361158185 conn(0x7fcaf40d4750 msgr2=0x7fcaf40b8190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:54.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.298+0000 7fcaf9293640 1 -- 172.21.15.72:0/3361158185 shutdown_connections 2024-04-15T15:16:54.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.298+0000 7fcaf9293640 1 -- 172.21.15.72:0/3361158185 wait complete. 2024-04-15T15:16:54.312 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi175.front.sepia.ceph.com... 2024-04-15T15:16:54.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:54.312 DEBUG:teuthology.orchestra.run.smithi175:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-04-15T15:16:54.345 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi175.stderr:ceph-fuse[45840]: fuse finished with error 0 and tester_r 0 2024-04-15T15:16:54.346 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi175.front.sepia.ceph.com 2024-04-15T15:16:54.346 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:54.347 DEBUG:teuthology.orchestra.run.smithi175:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-15T15:16:54.517 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:16:54.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.619+0000 7f1f5fd00640 1 Processor -- start 2024-04-15T15:16:54.621 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.619+0000 7f1f5fd00640 1 -- start start 2024-04-15T15:16:54.623 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.620+0000 7f1f5fd00640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1f5815e7e0 0x7f1f580b67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:54.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.620+0000 7f1f5fd00640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f1f580ae360 0x7f1f58072b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:54.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.620+0000 7f1f5fd00640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f58162800 con 0x7f1f580730c0 2024-04-15T15:16:54.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.620+0000 7f1f5fd00640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f1f58073530 con 0x7f1f5815e7e0 2024-04-15T15:16:54.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.620+0000 7f1f5fd00640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f1f58162ae0 con 0x7f1f580ae360 2024-04-15T15:16:54.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.620+0000 7f1f5da9c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1f5815e7e0 0x7f1f580b67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:54.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.620+0000 7f1f5da9c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f1f5815e7e0 0x7f1f580b67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37622/0 (socket says 172.21.15.72:37622) 2024-04-15T15:16:54.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.620+0000 7f1f5da9c640 1 -- 172.21.15.72:0/2012955681 learned_addr learned my addr 172.21.15.72:0/2012955681 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:54.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.620+0000 7f1f5e29d640 1 --1- 172.21.15.72:0/2012955681 >> v1:172.21.15.72:6789/0 conn(0x7f1f580730c0 0x7f1f580cf180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60056/0 (socket says 172.21.15.72:60056) 2024-04-15T15:16:54.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.620+0000 7f1f5d29b640 1 --2- 172.21.15.72:0/2012955681 >> v2:172.21.15.175:3300/0 conn(0x7f1f580ae360 0x7f1f58072b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:54.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.620+0000 7f1f5da9c640 1 -- 172.21.15.72:0/2012955681 >> v2:172.21.15.175:3300/0 conn(0x7f1f580ae360 msgr2=0x7f1f58072b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:54.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.620+0000 7f1f5da9c640 1 --2- 172.21.15.72:0/2012955681 >> v2:172.21.15.175:3300/0 conn(0x7f1f580ae360 0x7f1f58072b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.624 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.620+0000 7f1f5da9c640 1 -- 172.21.15.72:0/2012955681 >> v1:172.21.15.72:6789/0 conn(0x7f1f580730c0 legacy=0x7f1f580cf180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:54.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.620+0000 7f1f5da9c640 1 -- 172.21.15.72:0/2012955681 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f58162dc0 con 0x7f1f5815e7e0 2024-04-15T15:16:54.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.620+0000 7f1f5d29b640 1 --2- 172.21.15.72:0/2012955681 >> v2:172.21.15.175:3300/0 conn(0x7f1f580ae360 0x7f1f58072b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:54.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.620+0000 7f1f5ca9a640 1 -- 172.21.15.72:0/2012955681 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2712208566 0 0) 0x7f1f58162800 con 0x7f1f580730c0 2024-04-15T15:16:54.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.621+0000 7f1f5da9c640 1 --2- 172.21.15.72:0/2012955681 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f5815e7e0 0x7f1f580b67b0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f1f580addf0 tx=0x7f1f5402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d88622203a3fed7c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:54.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.621+0000 7f1f5ca9a640 1 -- 172.21.15.72:0/2012955681 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f54034070 con 0x7f1f5815e7e0 2024-04-15T15:16:54.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.621+0000 7f1f5ca9a640 1 -- 172.21.15.72:0/2012955681 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f54030860 con 0x7f1f5815e7e0 2024-04-15T15:16:54.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.621+0000 7f1f5ca9a640 1 -- 172.21.15.72:0/2012955681 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f54030c30 con 0x7f1f5815e7e0 2024-04-15T15:16:54.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.622+0000 7f1f5fd00640 1 -- 172.21.15.72:0/2012955681 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f5815e7e0 msgr2=0x7f1f580b67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:54.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.622+0000 7f1f5fd00640 1 --2- 172.21.15.72:0/2012955681 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f5815e7e0 0x7f1f580b67b0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f1f580addf0 tx=0x7f1f5402b9a0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.622+0000 7f1f5fd00640 1 -- 172.21.15.72:0/2012955681 shutdown_connections 2024-04-15T15:16:54.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.622+0000 7f1f5fd00640 1 --2- 172.21.15.72:0/2012955681 >> v2:172.21.15.175:3300/0 conn(0x7f1f580ae360 0x7f1f58072b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.622+0000 7f1f5fd00640 1 --2- 172.21.15.72:0/2012955681 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f5815e7e0 0x7f1f580b67b0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.622+0000 7f1f5fd00640 1 -- 172.21.15.72:0/2012955681 >> 172.21.15.72:0/2012955681 conn(0x7f1f580b31b0 msgr2=0x7f1f580b3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:54.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.622+0000 7f1f5fd00640 1 -- 172.21.15.72:0/2012955681 shutdown_connections 2024-04-15T15:16:54.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.622+0000 7f1f5fd00640 1 -- 172.21.15.72:0/2012955681 wait complete. 2024-04-15T15:16:54.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.623+0000 7f1f5fd00640 1 Processor -- start 2024-04-15T15:16:54.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.623+0000 7f1f5fd00640 1 -- start start 2024-04-15T15:16:54.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.624+0000 7f1f5fd00640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f580730c0 0x7f1f58155660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:54.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.624+0000 7f1f5fd00640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f580ae360 0x7f1f581f3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:54.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.624+0000 7f1f5fd00640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1f5815e7e0 0x7f1f58154710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:54.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.624+0000 7f1f5fd00640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f1f581623d0 con 0x7f1f580ae360 2024-04-15T15:16:54.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.624+0000 7f1f5fd00640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f1f5815ebc0 con 0x7f1f580730c0 2024-04-15T15:16:54.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.624+0000 7f1f5fd00640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f1f58162550 con 0x7f1f5815e7e0 2024-04-15T15:16:54.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.624+0000 7f1f5d29b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f580ae360 0x7f1f581f3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:54.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.624+0000 7f1f5d29b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f580ae360 0x7f1f581f3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37626/0 (socket says 172.21.15.72:37626) 2024-04-15T15:16:54.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.624+0000 7f1f5d29b640 1 -- 172.21.15.72:0/1228892566 learned_addr learned my addr 172.21.15.72:0/1228892566 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:54.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.624+0000 7f1f5d29b640 1 -- 172.21.15.72:0/1228892566 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1f5815e7e0 msgr2=0x7f1f58154710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:54.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.624+0000 7f1f5d29b640 1 --2- 172.21.15.72:0/1228892566 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1f5815e7e0 0x7f1f58154710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.624+0000 7f1f5d29b640 1 -- 172.21.15.72:0/1228892566 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f580730c0 msgr2=0x7f1f58155660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:54.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.624+0000 7f1f5d29b640 1 --2- 172.21.15.72:0/1228892566 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f580730c0 0x7f1f58155660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.624+0000 7f1f5d29b640 1 -- 172.21.15.72:0/1228892566 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f58155eb0 con 0x7f1f580ae360 2024-04-15T15:16:54.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.624+0000 7f1f5d29b640 1 --2- 172.21.15.72:0/1228892566 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f580ae360 0x7f1f581f3cf0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f1f4800c7f0 tx=0x7f1f4800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:54.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.624+0000 7f1f4e7fc640 1 -- 172.21.15.72:0/1228892566 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f48015070 con 0x7f1f580ae360 2024-04-15T15:16:54.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.624+0000 7f1f4e7fc640 1 -- 172.21.15.72:0/1228892566 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f48007a30 con 0x7f1f580ae360 2024-04-15T15:16:54.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.624+0000 7f1f5fd00640 1 -- 172.21.15.72:0/1228892566 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f581456b0 con 0x7f1f580ae360 2024-04-15T15:16:54.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.624+0000 7f1f4e7fc640 1 -- 172.21.15.72:0/1228892566 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f48007e00 con 0x7f1f580ae360 2024-04-15T15:16:54.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.624+0000 7f1f5fd00640 1 -- 172.21.15.72:0/1228892566 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f58155ba0 con 0x7f1f580ae360 2024-04-15T15:16:54.630 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.625+0000 7f1f5fd00640 1 -- 172.21.15.72:0/1228892566 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f58145880 con 0x7f1f580ae360 2024-04-15T15:16:54.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.626+0000 7f1f4e7fc640 1 -- 172.21.15.72:0/1228892566 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f1f48018070 con 0x7f1f580ae360 2024-04-15T15:16:54.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.626+0000 7f1f4e7fc640 1 --2- 172.21.15.72:0/1228892566 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1f3c06c980 0x7f1f3c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:54.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.626+0000 7f1f4e7fc640 1 -- 172.21.15.72:0/1228892566 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7f1f4808be80 con 0x7f1f580ae360 2024-04-15T15:16:54.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.626+0000 7f1f5da9c640 1 --2- 172.21.15.72:0/1228892566 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1f3c06c980 0x7f1f3c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:54.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.627+0000 7f1f5da9c640 1 --2- 172.21.15.72:0/1228892566 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1f3c06c980 0x7f1f3c06ee40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f1f54036000 tx=0x7f1f54030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:54.633 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.630+0000 7f1f4e7fc640 1 -- 172.21.15.72:0/1228892566 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1f48059fb0 con 0x7f1f580ae360 2024-04-15T15:16:54.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.843+0000 7f1f5fd00640 1 -- 172.21.15.72:0/1228892566 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1f58074310 con 0x7f1f580ae360 2024-04-15T15:16:54.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.844+0000 7f1f4e7fc640 1 -- 172.21.15.72:0/1228892566 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+5185 (secure 0 0 0) 0x7f1f58074310 con 0x7f1f580ae360 2024-04-15T15:16:54.847 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:16:54.847 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":22,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4883,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3363272836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3363272836},{"type":"v1","addr":"172.21.15.175:6834","nonce":3363272836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5134,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6835/1059187505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":1059187505},{"type":"v1","addr":"172.21.15.175:6835","nonce":1059187505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5556,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/690132275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":690132275},{"type":"v1","addr":"172.21.15.72:6837","nonce":690132275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:16:07.050078+0000","modified":"2024-04-15T15:16:45.725971+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.72:6836/3407328263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3407328263},{"type":"v1","addr":"172.21.15.72:6836","nonce":3407328263}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-04-15T15:16:54.847 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 22 2024-04-15T15:16:54.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.846+0000 7f1f5fd00640 1 -- 172.21.15.72:0/1228892566 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1f3c06c980 msgr2=0x7f1f3c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:54.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.846+0000 7f1f5fd00640 1 --2- 172.21.15.72:0/1228892566 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1f3c06c980 0x7f1f3c06ee40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f1f54036000 tx=0x7f1f54030000 comp rx=0 tx=0).stop 2024-04-15T15:16:54.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.846+0000 7f1f5fd00640 1 -- 172.21.15.72:0/1228892566 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f580ae360 msgr2=0x7f1f581f3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:54.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.846+0000 7f1f5fd00640 1 --2- 172.21.15.72:0/1228892566 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f580ae360 0x7f1f581f3cf0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f1f4800c7f0 tx=0x7f1f4800ccb0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.846+0000 7f1f5fd00640 1 -- 172.21.15.72:0/1228892566 shutdown_connections 2024-04-15T15:16:54.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.846+0000 7f1f5fd00640 1 --2- 172.21.15.72:0/1228892566 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f1f3c06c980 0x7f1f3c06ee40 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.846+0000 7f1f5fd00640 1 --2- 172.21.15.72:0/1228892566 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f1f5815e7e0 0x7f1f58154710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.846+0000 7f1f5fd00640 1 --2- 172.21.15.72:0/1228892566 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1f580ae360 0x7f1f581f3cf0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.846+0000 7f1f5fd00640 1 --2- 172.21.15.72:0/1228892566 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f1f580730c0 0x7f1f58155660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.846+0000 7f1f5fd00640 1 -- 172.21.15.72:0/1228892566 >> 172.21.15.72:0/1228892566 conn(0x7f1f580b31b0 msgr2=0x7f1f5806d9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:54.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.846+0000 7f1f5fd00640 1 -- 172.21.15.72:0/1228892566 shutdown_connections 2024-04-15T15:16:54.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.846+0000 7f1f5fd00640 1 -- 172.21.15.72:0/1228892566 wait complete. 2024-04-15T15:16:54.860 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:54.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.960+0000 7fd0d3b0e640 1 Processor -- start 2024-04-15T15:16:54.963 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.961+0000 7fd0d3b0e640 1 -- start start 2024-04-15T15:16:54.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.961+0000 7fd0d3b0e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd0cc161e40 0x7fd0cc0b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:54.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.961+0000 7fd0d3b0e640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0cc162220 con 0x7fd0cc0d45b0 2024-04-15T15:16:54.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.961+0000 7fd0d3b0e640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0cc0d3070 con 0x7fd0cc0d12b0 2024-04-15T15:16:54.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.961+0000 7fd0d3b0e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fd0cc0d21f0 con 0x7fd0cc161e40 2024-04-15T15:16:54.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.962+0000 7fd0d18aa640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fd0cc161e40 0x7fd0cc0b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:54.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.962+0000 7fd0d20ab640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fd0cc0d45b0 0x7fd0cc162490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60060/0 (socket says 172.21.15.72:60060) 2024-04-15T15:16:54.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.962+0000 7fd0d20ab640 1 -- 172.21.15.72:0/1386180692 learned_addr learned my addr 172.21.15.72:0/1386180692 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:54.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.962+0000 7fd0d18aa640 1 -- 172.21.15.72:0/1386180692 >> v1:172.21.15.175:6789/0 conn(0x7fd0cc0d12b0 legacy=0x7fd0cc0d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:54.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.962+0000 7fd0d18aa640 1 -- 172.21.15.72:0/1386180692 >> v1:172.21.15.72:6789/0 conn(0x7fd0cc0d45b0 legacy=0x7fd0cc162490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:54.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.962+0000 7fd0d18aa640 1 -- 172.21.15.72:0/1386180692 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0cc0d1820 con 0x7fd0cc161e40 2024-04-15T15:16:54.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.962+0000 7fd0d08a8640 1 -- 172.21.15.72:0/1386180692 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2779907411 0 0) 0x7fd0cc162220 con 0x7fd0cc0d45b0 2024-04-15T15:16:54.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.963+0000 7fd0d18aa640 1 --2- 172.21.15.72:0/1386180692 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0cc161e40 0x7fd0cc0b37f0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fd0c00097d0 tx=0x7fd0c002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b094e9533e73eef5 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:54.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.963+0000 7fd0d08a8640 1 -- 172.21.15.72:0/1386180692 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c00315a0 con 0x7fd0cc161e40 2024-04-15T15:16:54.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.963+0000 7fd0d08a8640 1 -- 172.21.15.72:0/1386180692 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0c0037070 con 0x7fd0cc161e40 2024-04-15T15:16:54.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.963+0000 7fd0d08a8640 1 -- 172.21.15.72:0/1386180692 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c0031960 con 0x7fd0cc161e40 2024-04-15T15:16:54.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.963+0000 7fd0d3b0e640 1 -- 172.21.15.72:0/1386180692 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0cc161e40 msgr2=0x7fd0cc0b37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:54.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.963+0000 7fd0d3b0e640 1 --2- 172.21.15.72:0/1386180692 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0cc161e40 0x7fd0cc0b37f0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fd0c00097d0 tx=0x7fd0c002b840 comp rx=0 tx=0).stop 2024-04-15T15:16:54.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.963+0000 7fd0d3b0e640 1 -- 172.21.15.72:0/1386180692 shutdown_connections 2024-04-15T15:16:54.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.963+0000 7fd0d3b0e640 1 --2- 172.21.15.72:0/1386180692 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0cc161e40 0x7fd0cc0b37f0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.963+0000 7fd0d3b0e640 1 -- 172.21.15.72:0/1386180692 >> 172.21.15.72:0/1386180692 conn(0x7fd0cc09d970 msgr2=0x7fd0cc0b95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:54.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.963+0000 7fd0d3b0e640 1 -- 172.21.15.72:0/1386180692 shutdown_connections 2024-04-15T15:16:54.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.963+0000 7fd0d3b0e640 1 -- 172.21.15.72:0/1386180692 wait complete. 2024-04-15T15:16:54.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.965+0000 7fd0d3b0e640 1 Processor -- start 2024-04-15T15:16:54.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.965+0000 7fd0d3b0e640 1 -- start start 2024-04-15T15:16:54.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.965+0000 7fd0d3b0e640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0cc0d12b0 0x7fd0cc157d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:54.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.965+0000 7fd0d3b0e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0cc0d45b0 0x7fd0cc159110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:54.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.965+0000 7fd0d3b0e640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd0cc161e40 0x7fd0cc15acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:54.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.965+0000 7fd0d3b0e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd0cc0d2440 con 0x7fd0cc0d45b0 2024-04-15T15:16:54.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.965+0000 7fd0d3b0e640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fd0cc0d1de0 con 0x7fd0cc0d12b0 2024-04-15T15:16:54.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.965+0000 7fd0d3b0e640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fd0cc0b3d30 con 0x7fd0cc161e40 2024-04-15T15:16:54.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.965+0000 7fd0d10a9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0cc0d45b0 0x7fd0cc159110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:54.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.965+0000 7fd0d10a9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0cc0d45b0 0x7fd0cc159110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37650/0 (socket says 172.21.15.72:37650) 2024-04-15T15:16:54.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.965+0000 7fd0d10a9640 1 -- 172.21.15.72:0/2552551984 learned_addr learned my addr 172.21.15.72:0/2552551984 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:54.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.966+0000 7fd0d18aa640 1 --2- 172.21.15.72:0/2552551984 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0cc0d12b0 0x7fd0cc157d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:54.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.966+0000 7fd0d20ab640 1 --2- 172.21.15.72:0/2552551984 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd0cc161e40 0x7fd0cc15acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:54.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.966+0000 7fd0d10a9640 1 -- 172.21.15.72:0/2552551984 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd0cc161e40 msgr2=0x7fd0cc15acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:54.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.966+0000 7fd0d10a9640 1 --2- 172.21.15.72:0/2552551984 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd0cc161e40 0x7fd0cc15acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.966+0000 7fd0d10a9640 1 -- 172.21.15.72:0/2552551984 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0cc0d12b0 msgr2=0x7fd0cc157d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:54.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.966+0000 7fd0d10a9640 1 --2- 172.21.15.72:0/2552551984 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0cc0d12b0 0x7fd0cc157d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.966+0000 7fd0d10a9640 1 -- 172.21.15.72:0/2552551984 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0cc06dd20 con 0x7fd0cc0d45b0 2024-04-15T15:16:54.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.966+0000 7fd0d18aa640 1 --2- 172.21.15.72:0/2552551984 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0cc0d12b0 0x7fd0cc157d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:54.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.966+0000 7fd0d20ab640 1 --2- 172.21.15.72:0/2552551984 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd0cc161e40 0x7fd0cc15acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:54.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.966+0000 7fd0d10a9640 1 --2- 172.21.15.72:0/2552551984 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0cc0d45b0 0x7fd0cc159110 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fd0c800c410 tx=0x7fd0c800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:54.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.966+0000 7fd0be7fc640 1 -- 172.21.15.72:0/2552551984 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c8017070 con 0x7fd0cc0d45b0 2024-04-15T15:16:54.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.966+0000 7fd0be7fc640 1 -- 172.21.15.72:0/2552551984 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0c80035f0 con 0x7fd0cc0d45b0 2024-04-15T15:16:54.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.966+0000 7fd0be7fc640 1 -- 172.21.15.72:0/2552551984 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c80039e0 con 0x7fd0cc0d45b0 2024-04-15T15:16:54.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.966+0000 7fd0d3b0e640 1 -- 172.21.15.72:0/2552551984 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0cc06e000 con 0x7fd0cc0d45b0 2024-04-15T15:16:54.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.966+0000 7fd0d3b0e640 1 -- 172.21.15.72:0/2552551984 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0cc06e5c0 con 0x7fd0cc0d45b0 2024-04-15T15:16:54.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.967+0000 7fd0d3b0e640 1 -- 172.21.15.72:0/2552551984 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0cc0cf130 con 0x7fd0cc0d45b0 2024-04-15T15:16:54.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.968+0000 7fd0be7fc640 1 -- 172.21.15.72:0/2552551984 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fd0c8003b80 con 0x7fd0cc0d45b0 2024-04-15T15:16:54.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.968+0000 7fd0be7fc640 1 --2- 172.21.15.72:0/2552551984 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd0b006c980 0x7fd0b006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:54.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.968+0000 7fd0d18aa640 1 --2- 172.21.15.72:0/2552551984 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd0b006c980 0x7fd0b006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:54.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.968+0000 7fd0be7fc640 1 -- 172.21.15.72:0/2552551984 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7fd0c808afd0 con 0x7fd0cc0d45b0 2024-04-15T15:16:54.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.969+0000 7fd0d18aa640 1 --2- 172.21.15.72:0/2552551984 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd0b006c980 0x7fd0b006ee40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd0c0036000 tx=0x7fd0c0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:54.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:54.973+0000 7fd0be7fc640 1 -- 172.21.15.72:0/2552551984 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd0cc0cf130 con 0x7fd0cc0d45b0 2024-04-15T15:16:55.152 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.149+0000 7fd0d3b0e640 1 -- 172.21.15.72:0/2552551984 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd0cc0689e0 con 0x7fd0cc0d45b0 2024-04-15T15:16:55.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.150+0000 7fd0be7fc640 1 -- 172.21.15.72:0/2552551984 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) ==== 74+0+17120 (secure 0 0 0) 0x7fd0c80592a0 con 0x7fd0cc0d45b0 2024-04-15T15:16:55.153 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:16:55.153 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":41,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:16:41.714841+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:05.551499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:06.003152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.26","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:6836/4270793613":"2024-04-16T15:16:41.714817+0000","172.21.15.72:6834/4270793613":"2024-04-16T15:16:41.714817+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:55.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.152+0000 7fd0d3b0e640 1 -- 172.21.15.72:0/2552551984 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd0b006c980 msgr2=0x7fd0b006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.152+0000 7fd0d3b0e640 1 --2- 172.21.15.72:0/2552551984 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd0b006c980 0x7fd0b006ee40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd0c0036000 tx=0x7fd0c0008030 comp rx=0 tx=0).stop 2024-04-15T15:16:55.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.152+0000 7fd0d3b0e640 1 -- 172.21.15.72:0/2552551984 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0cc0d45b0 msgr2=0x7fd0cc159110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.152+0000 7fd0d3b0e640 1 --2- 172.21.15.72:0/2552551984 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0cc0d45b0 0x7fd0cc159110 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fd0c800c410 tx=0x7fd0c800c8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.152+0000 7fd0d3b0e640 1 -- 172.21.15.72:0/2552551984 shutdown_connections 2024-04-15T15:16:55.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.152+0000 7fd0d3b0e640 1 --2- 172.21.15.72:0/2552551984 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd0b006c980 0x7fd0b006ee40 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.152+0000 7fd0d3b0e640 1 --2- 172.21.15.72:0/2552551984 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd0cc161e40 0x7fd0cc15acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.152+0000 7fd0d3b0e640 1 --2- 172.21.15.72:0/2552551984 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd0cc0d45b0 0x7fd0cc159110 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.155 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.152+0000 7fd0d3b0e640 1 --2- 172.21.15.72:0/2552551984 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd0cc0d12b0 0x7fd0cc157d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.152+0000 7fd0d3b0e640 1 -- 172.21.15.72:0/2552551984 >> 172.21.15.72:0/2552551984 conn(0x7fd0cc09d970 msgr2=0x7fd0cc0b4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:55.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.152+0000 7fd0d3b0e640 1 -- 172.21.15.72:0/2552551984 shutdown_connections 2024-04-15T15:16:55.156 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.152+0000 7fd0d3b0e640 1 -- 172.21.15.72:0/2552551984 wait complete. 2024-04-15T15:16:55.166 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:39.359102+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:16:05.551499+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:16:06.003152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:16:55.167 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:55.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.270+0000 7f85f310e640 1 Processor -- start 2024-04-15T15:16:55.272 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.270+0000 7f85f310e640 1 -- start start 2024-04-15T15:16:55.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.270+0000 7f85f310e640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f85ec155790 0x7f85ec0b5e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.270+0000 7f85f310e640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f85ec1622f0 0x7f85ec162730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.270+0000 7f85f310e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f85ec15ed00 0x7f85ec15f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.270+0000 7f85f310e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f85ec06ecb0 con 0x7f85ec15ed00 2024-04-15T15:16:55.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.270+0000 7f85f310e640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f85ec0b59d0 con 0x7f85ec1622f0 2024-04-15T15:16:55.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.270+0000 7f85f310e640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f85ec162c70 con 0x7f85ec155790 2024-04-15T15:16:55.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.271+0000 7f85f16ab640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f85ec15ed00 0x7f85ec15f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.271+0000 7f85f16ab640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f85ec15ed00 0x7f85ec15f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37652/0 (socket says 172.21.15.72:37652) 2024-04-15T15:16:55.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.271+0000 7f85f16ab640 1 -- 172.21.15.72:0/1715575952 learned_addr learned my addr 172.21.15.72:0/1715575952 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:55.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.271+0000 7f85f16ab640 1 -- 172.21.15.72:0/1715575952 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f85ec155790 msgr2=0x7f85ec0b5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:55.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.271+0000 7f85f16ab640 1 --2- 172.21.15.72:0/1715575952 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f85ec155790 0x7f85ec0b5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.271+0000 7f85f16ab640 1 -- 172.21.15.72:0/1715575952 >> v2:172.21.15.175:3300/0 conn(0x7f85ec1622f0 msgr2=0x7f85ec162730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:55.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.271+0000 7f85f16ab640 1 --2- 172.21.15.72:0/1715575952 >> v2:172.21.15.175:3300/0 conn(0x7f85ec1622f0 0x7f85ec162730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.271+0000 7f85f16ab640 1 -- 172.21.15.72:0/1715575952 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85ec155b70 con 0x7f85ec15ed00 2024-04-15T15:16:55.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.272+0000 7f85f16ab640 1 --2- 172.21.15.72:0/1715575952 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85ec15ed00 0x7f85ec15f170 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f85ec06c9d0 tx=0x7f85e402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b32f7cf198ac774c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:55.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.272+0000 7f85eb7fe640 1 -- 172.21.15.72:0/1715575952 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e402ca70 con 0x7f85ec15ed00 2024-04-15T15:16:55.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.272+0000 7f85eb7fe640 1 -- 172.21.15.72:0/1715575952 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85e402cc10 con 0x7f85ec15ed00 2024-04-15T15:16:55.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.272+0000 7f85eb7fe640 1 -- 172.21.15.72:0/1715575952 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e40343f0 con 0x7f85ec15ed00 2024-04-15T15:16:55.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.272+0000 7f85f310e640 1 -- 172.21.15.72:0/1715575952 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85ec15ed00 msgr2=0x7f85ec15f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.272+0000 7f85f310e640 1 --2- 172.21.15.72:0/1715575952 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85ec15ed00 0x7f85ec15f170 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f85ec06c9d0 tx=0x7f85e402bed0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.272+0000 7f85f310e640 1 -- 172.21.15.72:0/1715575952 shutdown_connections 2024-04-15T15:16:55.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.272+0000 7f85f310e640 1 --2- 172.21.15.72:0/1715575952 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85ec15ed00 0x7f85ec15f170 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.272+0000 7f85f310e640 1 --2- 172.21.15.72:0/1715575952 >> v2:172.21.15.175:3300/0 conn(0x7f85ec1622f0 0x7f85ec162730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.276 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.272+0000 7f85f310e640 1 --2- 172.21.15.72:0/1715575952 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f85ec155790 0x7f85ec0b5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.272+0000 7f85f310e640 1 -- 172.21.15.72:0/1715575952 >> 172.21.15.72:0/1715575952 conn(0x7f85ec0d24d0 msgr2=0x7f85ec0d28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:55.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.272+0000 7f85f310e640 1 -- 172.21.15.72:0/1715575952 shutdown_connections 2024-04-15T15:16:55.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.272+0000 7f85f310e640 1 -- 172.21.15.72:0/1715575952 wait complete. 2024-04-15T15:16:55.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.274+0000 7f85f310e640 1 Processor -- start 2024-04-15T15:16:55.277 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.274+0000 7f85f310e640 1 -- start start 2024-04-15T15:16:55.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.274+0000 7f85f310e640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f85ec155790 0x7f85ec1573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.274+0000 7f85f310e640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f85ec15ed00 0x7f85ec13fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.274+0000 7f85f310e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85ec1622f0 0x7f85ec0b5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.274+0000 7f85f310e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f85ec0b5380 con 0x7f85ec1622f0 2024-04-15T15:16:55.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.274+0000 7f85f310e640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f85ec06f1e0 con 0x7f85ec155790 2024-04-15T15:16:55.280 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.274+0000 7f85f310e640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f85ec0b5660 con 0x7f85ec15ed00 2024-04-15T15:16:55.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.275+0000 7f85f16ab640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85ec1622f0 0x7f85ec0b5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.275+0000 7f85f16ab640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85ec1622f0 0x7f85ec0b5f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37660/0 (socket says 172.21.15.72:37660) 2024-04-15T15:16:55.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.275+0000 7f85f16ab640 1 -- 172.21.15.72:0/1130185572 learned_addr learned my addr 172.21.15.72:0/1130185572 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:55.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.275+0000 7f85f16ab640 1 -- 172.21.15.72:0/1130185572 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f85ec15ed00 msgr2=0x7f85ec13fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.275+0000 7f85f16ab640 1 --2- 172.21.15.72:0/1130185572 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f85ec15ed00 0x7f85ec13fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.275+0000 7f85f16ab640 1 -- 172.21.15.72:0/1130185572 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f85ec155790 msgr2=0x7f85ec1573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.275+0000 7f85f16ab640 1 --2- 172.21.15.72:0/1130185572 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f85ec155790 0x7f85ec1573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.275+0000 7f85f16ab640 1 -- 172.21.15.72:0/1130185572 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85ec13ff70 con 0x7f85ec1622f0 2024-04-15T15:16:55.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.275+0000 7f85f16ab640 1 --2- 172.21.15.72:0/1130185572 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85ec1622f0 0x7f85ec0b5f20 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f85e4036040 tx=0x7f85e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:55.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.275+0000 7f85e97fa640 1 -- 172.21.15.72:0/1130185572 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e4043020 con 0x7f85ec1622f0 2024-04-15T15:16:55.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.275+0000 7f85e97fa640 1 -- 172.21.15.72:0/1130185572 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85e403e450 con 0x7f85ec1622f0 2024-04-15T15:16:55.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.275+0000 7f85f310e640 1 -- 172.21.15.72:0/1130185572 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85ec157a10 con 0x7f85ec1622f0 2024-04-15T15:16:55.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.275+0000 7f85e97fa640 1 -- 172.21.15.72:0/1130185572 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85e403e820 con 0x7f85ec1622f0 2024-04-15T15:16:55.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.275+0000 7f85f310e640 1 -- 172.21.15.72:0/1130185572 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85ec157fd0 con 0x7f85ec1622f0 2024-04-15T15:16:55.281 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.276+0000 7f85f310e640 1 -- 172.21.15.72:0/1130185572 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85ec075c50 con 0x7f85ec1622f0 2024-04-15T15:16:55.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.277+0000 7f85e97fa640 1 -- 172.21.15.72:0/1130185572 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f85e4046020 con 0x7f85ec1622f0 2024-04-15T15:16:55.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.277+0000 7f85e97fa640 1 --2- 172.21.15.72:0/1130185572 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f85d006c980 0x7f85d006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.277+0000 7f85f0eaa640 1 --2- 172.21.15.72:0/1130185572 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f85d006c980 0x7f85d006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.277+0000 7f85e97fa640 1 -- 172.21.15.72:0/1130185572 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7f85e40ae660 con 0x7f85ec1622f0 2024-04-15T15:16:55.282 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.278+0000 7f85f0eaa640 1 --2- 172.21.15.72:0/1130185572 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f85d006c980 0x7f85d006ee40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f85e0009710 tx=0x7f85e0006c90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:55.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.282+0000 7f85e97fa640 1 -- 172.21.15.72:0/1130185572 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f85e404d030 con 0x7f85ec1622f0 2024-04-15T15:16:55.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.457+0000 7f85f310e640 1 -- 172.21.15.72:0/1130185572 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f85ec071230 con 0x7f85ec1622f0 2024-04-15T15:16:55.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.457+0000 7f85e97fa640 1 -- 172.21.15.72:0/1130185572 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) ==== 93+0+10 (secure 0 0 0) 0x7f85e407c930 con 0x7f85ec1622f0 2024-04-15T15:16:55.463 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-04-15T15:16:55.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.459+0000 7f85f310e640 1 -- 172.21.15.72:0/1130185572 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f85d006c980 msgr2=0x7f85d006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.459+0000 7f85f310e640 1 --2- 172.21.15.72:0/1130185572 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f85d006c980 0x7f85d006ee40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f85e0009710 tx=0x7f85e0006c90 comp rx=0 tx=0).stop 2024-04-15T15:16:55.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.459+0000 7f85f310e640 1 -- 172.21.15.72:0/1130185572 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85ec1622f0 msgr2=0x7f85ec0b5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.459+0000 7f85f310e640 1 --2- 172.21.15.72:0/1130185572 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85ec1622f0 0x7f85ec0b5f20 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f85e4036040 tx=0x7f85e4004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:55.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.459+0000 7f85f310e640 1 -- 172.21.15.72:0/1130185572 shutdown_connections 2024-04-15T15:16:55.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.459+0000 7f85f310e640 1 --2- 172.21.15.72:0/1130185572 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f85d006c980 0x7f85d006ee40 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.459+0000 7f85f310e640 1 --2- 172.21.15.72:0/1130185572 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85ec1622f0 0x7f85ec0b5f20 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.459+0000 7f85f310e640 1 --2- 172.21.15.72:0/1130185572 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f85ec15ed00 0x7f85ec13fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.459+0000 7f85f310e640 1 --2- 172.21.15.72:0/1130185572 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f85ec155790 0x7f85ec1573a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.459+0000 7f85f310e640 1 -- 172.21.15.72:0/1130185572 >> 172.21.15.72:0/1130185572 conn(0x7f85ec0d24d0 msgr2=0x7f85ec15ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:55.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.459+0000 7f85f310e640 1 -- 172.21.15.72:0/1130185572 shutdown_connections 2024-04-15T15:16:55.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.459+0000 7f85f310e640 1 -- 172.21.15.72:0/1130185572 wait complete. 2024-04-15T15:16:55.473 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:16:55.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.576+0000 7f43ff959640 1 Processor -- start 2024-04-15T15:16:55.579 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.576+0000 7f43ff959640 1 -- start start 2024-04-15T15:16:55.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.577+0000 7f43ff959640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f43f815c5f0 0x7f43f80d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.577+0000 7f43ff959640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43f80cd990 con 0x7f43f806f510 2024-04-15T15:16:55.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.577+0000 7f43ff959640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43f8160ff0 con 0x7f43f80ce190 2024-04-15T15:16:55.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.577+0000 7f43ff959640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f43f8160da0 con 0x7f43f815c5f0 2024-04-15T15:16:55.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.577+0000 7f43fdef6640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f43f806f510 0x7f43f80cd280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60074/0 (socket says 172.21.15.72:60074) 2024-04-15T15:16:55.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.577+0000 7f43fdef6640 1 -- 172.21.15.72:0/2019521659 learned_addr learned my addr 172.21.15.72:0/2019521659 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:55.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.578+0000 7f43fd6f5640 1 --2- 172.21.15.72:0/2019521659 >> v2:172.21.15.175:3300/0 conn(0x7f43f815c5f0 0x7f43f80d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.578+0000 7f43effff640 1 -- 172.21.15.72:0/2019521659 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1524617705 0 0) 0x7f43f80cd990 con 0x7f43f806f510 2024-04-15T15:16:55.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.578+0000 7f43effff640 1 -- 172.21.15.72:0/2019521659 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43e4003520 con 0x7f43f806f510 2024-04-15T15:16:55.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.578+0000 7f43effff640 1 -- 172.21.15.72:0/2019521659 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4179119996 0 0) 0x7f43e4003520 con 0x7f43f806f510 2024-04-15T15:16:55.582 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.578+0000 7f43effff640 1 -- 172.21.15.72:0/2019521659 >> v2:172.21.15.175:3300/0 conn(0x7f43f815c5f0 msgr2=0x7f43f80d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:55.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.578+0000 7f43effff640 1 --2- 172.21.15.72:0/2019521659 >> v2:172.21.15.175:3300/0 conn(0x7f43f815c5f0 0x7f43f80d1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.578+0000 7f43effff640 1 -- 172.21.15.72:0/2019521659 >> v1:172.21.15.175:6789/0 conn(0x7f43f80ce190 legacy=0x7f43f80ccb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.578+0000 7f43effff640 1 -- 172.21.15.72:0/2019521659 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43f80b5840 con 0x7f43f806f510 2024-04-15T15:16:55.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.578+0000 7f43effff640 1 -- 172.21.15.72:0/2019521659 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f43f0001670 con 0x7f43f806f510 2024-04-15T15:16:55.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.578+0000 7f43effff640 1 -- 172.21.15.72:0/2019521659 >> v1:172.21.15.72:6789/0 conn(0x7f43f806f510 legacy=0x7f43f80cd280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.578+0000 7f43effff640 1 --2- 172.21.15.72:0/2019521659 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43e40039b0 0x7f43e4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.578+0000 7f43effff640 1 --2- 172.21.15.72:0/2019521659 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f43e4004800 0x7f43e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.578+0000 7f43effff640 1 --2- 172.21.15.72:0/2019521659 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f43e40051c0 0x7f43e400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.578+0000 7f43effff640 1 -- 172.21.15.72:0/2019521659 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f43f8160da0 con 0x7f43e40039b0 2024-04-15T15:16:55.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.578+0000 7f43effff640 1 -- 172.21.15.72:0/2019521659 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f43e400c160 con 0x7f43e4004800 2024-04-15T15:16:55.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.578+0000 7f43effff640 1 -- 172.21.15.72:0/2019521659 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f43e400c360 con 0x7f43e40051c0 2024-04-15T15:16:55.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.578+0000 7f43fd6f5640 1 --2- 172.21.15.72:0/2019521659 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43e40039b0 0x7f43e4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.578+0000 7f43fcef4640 1 --2- 172.21.15.72:0/2019521659 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f43e4004800 0x7f43e4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.578+0000 7f43fdef6640 1 --2- 172.21.15.72:0/2019521659 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f43e40051c0 0x7f43e400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.578+0000 7f43fd6f5640 1 -- 172.21.15.72:0/2019521659 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f43e40051c0 msgr2=0x7f43e400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:55.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.578+0000 7f43fd6f5640 1 --2- 172.21.15.72:0/2019521659 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f43e40051c0 0x7f43e400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.578+0000 7f43fd6f5640 1 -- 172.21.15.72:0/2019521659 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f43e4004800 msgr2=0x7f43e4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:55.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.578+0000 7f43fd6f5640 1 --2- 172.21.15.72:0/2019521659 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f43e4004800 0x7f43e4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.578+0000 7f43fdef6640 1 -- 172.21.15.72:0/2019521659 reap_dead start 2024-04-15T15:16:55.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.578+0000 7f43fd6f5640 1 -- 172.21.15.72:0/2019521659 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43e4003520 con 0x7f43e40039b0 2024-04-15T15:16:55.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.579+0000 7f43fd6f5640 1 --2- 172.21.15.72:0/2019521659 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43e40039b0 0x7f43e4003dd0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f43f4009fc0 tx=0x7f43e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8108ff6a674f628a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:55.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.579+0000 7f43effff640 1 -- 172.21.15.72:0/2019521659 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f4039070 con 0x7f43e40039b0 2024-04-15T15:16:55.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.579+0000 7f43effff640 1 -- 172.21.15.72:0/2019521659 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43f402cc20 con 0x7f43e40039b0 2024-04-15T15:16:55.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.579+0000 7f43effff640 1 -- 172.21.15.72:0/2019521659 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f4009c10 con 0x7f43e40039b0 2024-04-15T15:16:55.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.580+0000 7f43ff959640 1 -- 172.21.15.72:0/2019521659 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43e40039b0 msgr2=0x7f43e4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.580+0000 7f43ff959640 1 --2- 172.21.15.72:0/2019521659 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43e40039b0 0x7f43e4003dd0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f43f4009fc0 tx=0x7f43e4005830 comp rx=0 tx=0).stop 2024-04-15T15:16:55.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.580+0000 7f43ff959640 1 -- 172.21.15.72:0/2019521659 shutdown_connections 2024-04-15T15:16:55.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.580+0000 7f43ff959640 1 --2- 172.21.15.72:0/2019521659 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43e40039b0 0x7f43e4003dd0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.580+0000 7f43ff959640 1 -- 172.21.15.72:0/2019521659 >> 172.21.15.72:0/2019521659 conn(0x7f43f80b4010 msgr2=0x7f43f80b4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:55.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.580+0000 7f43ff959640 1 -- 172.21.15.72:0/2019521659 shutdown_connections 2024-04-15T15:16:55.588 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.580+0000 7f43ff959640 1 -- 172.21.15.72:0/2019521659 wait complete. 2024-04-15T15:16:55.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.581+0000 7f43ff959640 1 Processor -- start 2024-04-15T15:16:55.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.581+0000 7f43ff959640 1 -- start start 2024-04-15T15:16:55.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.582+0000 7f43ff959640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f43e40039b0 0x7f43f81507f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.582+0000 7f43ff959640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f43f815c9d0 0x7f43f80b2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.582+0000 7f43ff959640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43f815ce50 0x7f43f80b2c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.582+0000 7f43ff959640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f43f81612d0 con 0x7f43f815ce50 2024-04-15T15:16:55.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.582+0000 7f43ff959640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f43f80d2360 con 0x7f43e40039b0 2024-04-15T15:16:55.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.582+0000 7f43ff959640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f43f80b5bb0 con 0x7f43f815c9d0 2024-04-15T15:16:55.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.582+0000 7f43fdef6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43f815ce50 0x7f43f80b2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.582+0000 7f43fdef6640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43f815ce50 0x7f43f80b2c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37686/0 (socket says 172.21.15.72:37686) 2024-04-15T15:16:55.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.582+0000 7f43fdef6640 1 -- 172.21.15.72:0/623646635 learned_addr learned my addr 172.21.15.72:0/623646635 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:55.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.582+0000 7f43fcef4640 1 --2- 172.21.15.72:0/623646635 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f43f815c9d0 0x7f43f80b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.582+0000 7f43fd6f5640 1 --2- 172.21.15.72:0/623646635 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f43e40039b0 0x7f43f81507f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.582+0000 7f43fdef6640 1 -- 172.21.15.72:0/623646635 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f43f815c9d0 msgr2=0x7f43f80b2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.582+0000 7f43fdef6640 1 --2- 172.21.15.72:0/623646635 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f43f815c9d0 0x7f43f80b2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.582+0000 7f43fdef6640 1 -- 172.21.15.72:0/623646635 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f43e40039b0 msgr2=0x7f43f81507f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.582+0000 7f43fdef6640 1 --2- 172.21.15.72:0/623646635 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f43e40039b0 0x7f43f81507f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.582+0000 7f43fdef6640 1 -- 172.21.15.72:0/623646635 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43f81573e0 con 0x7f43f815ce50 2024-04-15T15:16:55.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.582+0000 7f43fdef6640 1 --2- 172.21.15.72:0/623646635 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43f815ce50 0x7f43f80b2c80 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f43f80696b0 tx=0x7f43f000eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:55.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.582+0000 7f43edffb640 1 -- 172.21.15.72:0/623646635 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f000c610 con 0x7f43f815ce50 2024-04-15T15:16:55.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.582+0000 7f43edffb640 1 -- 172.21.15.72:0/623646635 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43f0018070 con 0x7f43f815ce50 2024-04-15T15:16:55.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.583+0000 7f43ff959640 1 -- 172.21.15.72:0/623646635 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43f81576c0 con 0x7f43f815ce50 2024-04-15T15:16:55.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.583+0000 7f43edffb640 1 -- 172.21.15.72:0/623646635 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f000ca10 con 0x7f43f815ce50 2024-04-15T15:16:55.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.583+0000 7f43ff959640 1 -- 172.21.15.72:0/623646635 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43f8157c80 con 0x7f43f815ce50 2024-04-15T15:16:55.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.584+0000 7f43ff959640 1 -- 172.21.15.72:0/623646635 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43f8157ec0 con 0x7f43f815ce50 2024-04-15T15:16:55.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.584+0000 7f43edffb640 1 -- 172.21.15.72:0/623646635 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f43f000cbb0 con 0x7f43f815ce50 2024-04-15T15:16:55.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.584+0000 7f43edffb640 1 --2- 172.21.15.72:0/623646635 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f43dc06c9b0 0x7f43dc06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.585+0000 7f43edffb640 1 -- 172.21.15.72:0/623646635 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7f43f008b940 con 0x7f43f815ce50 2024-04-15T15:16:55.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.585+0000 7f43fd6f5640 1 --2- 172.21.15.72:0/623646635 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f43dc06c9b0 0x7f43dc06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.585+0000 7f43fd6f5640 1 --2- 172.21.15.72:0/623646635 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f43dc06c9b0 0x7f43dc06ee70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f43f4036000 tx=0x7f43f402e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:55.591 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.589+0000 7f43edffb640 1 -- 172.21.15.72:0/623646635 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f43f8157ec0 con 0x7f43f815ce50 2024-04-15T15:16:55.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.766+0000 7f43ff959640 1 -- 172.21.15.72:0/623646635 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f43f80d1e50 con 0x7f43f815ce50 2024-04-15T15:16:55.769 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.767+0000 7f43edffb640 1 -- 172.21.15.72:0/623646635 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v41) ==== 104+0+11 (secure 0 0 0) 0x7f43f0059c10 con 0x7f43f815ce50 2024-04-15T15:16:55.769 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 32 2024-04-15T15:16:55.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.769+0000 7f43ff959640 1 -- 172.21.15.72:0/623646635 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f43dc06c9b0 msgr2=0x7f43dc06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.769+0000 7f43ff959640 1 --2- 172.21.15.72:0/623646635 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f43dc06c9b0 0x7f43dc06ee70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f43f4036000 tx=0x7f43f402e000 comp rx=0 tx=0).stop 2024-04-15T15:16:55.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.769+0000 7f43ff959640 1 -- 172.21.15.72:0/623646635 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43f815ce50 msgr2=0x7f43f80b2c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.769+0000 7f43ff959640 1 --2- 172.21.15.72:0/623646635 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43f815ce50 0x7f43f80b2c80 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f43f80696b0 tx=0x7f43f000eb90 comp rx=0 tx=0).stop 2024-04-15T15:16:55.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.769+0000 7f43ff959640 1 -- 172.21.15.72:0/623646635 shutdown_connections 2024-04-15T15:16:55.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.769+0000 7f43ff959640 1 --2- 172.21.15.72:0/623646635 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f43dc06c9b0 0x7f43dc06ee70 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.769+0000 7f43ff959640 1 --2- 172.21.15.72:0/623646635 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43f815ce50 0x7f43f80b2c80 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.769+0000 7f43ff959640 1 --2- 172.21.15.72:0/623646635 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f43f815c9d0 0x7f43f80b2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.769+0000 7f43ff959640 1 --2- 172.21.15.72:0/623646635 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f43e40039b0 0x7f43f81507f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.769+0000 7f43ff959640 1 -- 172.21.15.72:0/623646635 >> 172.21.15.72:0/623646635 conn(0x7f43f80b4010 msgr2=0x7f43f80709e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:55.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.769+0000 7f43ff959640 1 -- 172.21.15.72:0/623646635 shutdown_connections 2024-04-15T15:16:55.772 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.769+0000 7f43ff959640 1 -- 172.21.15.72:0/623646635 wait complete. 2024-04-15T15:16:55.783 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:16:55.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.889+0000 7f43db488640 1 Processor -- start 2024-04-15T15:16:55.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.889+0000 7f43db488640 1 -- start start 2024-04-15T15:16:55.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.890+0000 7f43db488640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f43d4161ee0 0x7f43d40c95f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.890+0000 7f43db488640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f43d40b67c0 0x7f43d40d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.890+0000 7f43db488640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43d40ccfe0 con 0x7f43d40b6c20 2024-04-15T15:16:55.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.890+0000 7f43db488640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f43d41622c0 con 0x7f43d40b67c0 2024-04-15T15:16:55.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.890+0000 7f43db488640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f43d40cd2c0 con 0x7f43d4161ee0 2024-04-15T15:16:55.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.890+0000 7f43d8a23640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f43d40b67c0 0x7f43d40d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.890+0000 7f43d9a25640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f43d40b6c20 0x7f43d40ce080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60084/0 (socket says 172.21.15.72:60084) 2024-04-15T15:16:55.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.890+0000 7f43d8a23640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f43d40b67c0 0x7f43d40d1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37694/0 (socket says 172.21.15.72:37694) 2024-04-15T15:16:55.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.890+0000 7f43d9a25640 1 -- 172.21.15.72:0/797105499 learned_addr learned my addr 172.21.15.72:0/797105499 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:55.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.890+0000 7f43d8a23640 1 -- 172.21.15.72:0/797105499 >> v2:172.21.15.175:3300/0 conn(0x7f43d4161ee0 msgr2=0x7f43d40c95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:55.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.890+0000 7f43d9224640 1 --2- 172.21.15.72:0/797105499 >> v2:172.21.15.175:3300/0 conn(0x7f43d4161ee0 0x7f43d40c95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.890+0000 7f43d8a23640 1 --2- 172.21.15.72:0/797105499 >> v2:172.21.15.175:3300/0 conn(0x7f43d4161ee0 0x7f43d40c95f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.890+0000 7f43d8a23640 1 -- 172.21.15.72:0/797105499 >> v1:172.21.15.72:6789/0 conn(0x7f43d40b6c20 legacy=0x7f43d40ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.890+0000 7f43cbfff640 1 -- 172.21.15.72:0/797105499 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2955238989 0 0) 0x7f43d40ccfe0 con 0x7f43d40b6c20 2024-04-15T15:16:55.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.890+0000 7f43d8a23640 1 -- 172.21.15.72:0/797105499 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43d40cd5a0 con 0x7f43d40b67c0 2024-04-15T15:16:55.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.890+0000 7f43d9224640 1 --2- 172.21.15.72:0/797105499 >> v2:172.21.15.175:3300/0 conn(0x7f43d4161ee0 0x7f43d40c95f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:55.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.891+0000 7f43d8a23640 1 --2- 172.21.15.72:0/797105499 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43d40b67c0 0x7f43d40d1100 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f43d4074930 tx=0x7f43c402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e02177e76a0b95cf server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:55.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.891+0000 7f43cbfff640 1 -- 172.21.15.72:0/797105499 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c4035020 con 0x7f43d40b67c0 2024-04-15T15:16:55.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.891+0000 7f43cbfff640 1 -- 172.21.15.72:0/797105499 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43c4031830 con 0x7f43d40b67c0 2024-04-15T15:16:55.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.891+0000 7f43cbfff640 1 -- 172.21.15.72:0/797105499 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c4031c00 con 0x7f43d40b67c0 2024-04-15T15:16:55.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.892+0000 7f43db488640 1 -- 172.21.15.72:0/797105499 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43d40b67c0 msgr2=0x7f43d40d1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.892+0000 7f43db488640 1 --2- 172.21.15.72:0/797105499 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43d40b67c0 0x7f43d40d1100 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f43d4074930 tx=0x7f43c402b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.892+0000 7f43db488640 1 -- 172.21.15.72:0/797105499 shutdown_connections 2024-04-15T15:16:55.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.892+0000 7f43db488640 1 --2- 172.21.15.72:0/797105499 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43d40b67c0 0x7f43d40d1100 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.892+0000 7f43db488640 1 --2- 172.21.15.72:0/797105499 >> v2:172.21.15.175:3300/0 conn(0x7f43d4161ee0 0x7f43d40c95f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.892+0000 7f43db488640 1 -- 172.21.15.72:0/797105499 >> 172.21.15.72:0/797105499 conn(0x7f43d40b4010 msgr2=0x7f43d40b4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:55.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.892+0000 7f43db488640 1 -- 172.21.15.72:0/797105499 shutdown_connections 2024-04-15T15:16:55.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.892+0000 7f43db488640 1 -- 172.21.15.72:0/797105499 wait complete. 2024-04-15T15:16:55.900 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.893+0000 7f43db488640 1 Processor -- start 2024-04-15T15:16:55.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.893+0000 7f43db488640 1 -- start start 2024-04-15T15:16:55.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.894+0000 7f43db488640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f43d40b67c0 0x7f43d40c81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.894+0000 7f43db488640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f43d40b6c20 0x7f43d4155720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.894+0000 7f43db488640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43d4161ee0 0x7f43d4155c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.894+0000 7f43db488640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f43d40ccb70 con 0x7f43d4161ee0 2024-04-15T15:16:55.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.894+0000 7f43db488640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f43d40c9b30 con 0x7f43d40b67c0 2024-04-15T15:16:55.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.894+0000 7f43db488640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f43d40cccf0 con 0x7f43d40b6c20 2024-04-15T15:16:55.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.894+0000 7f43d9a25640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43d4161ee0 0x7f43d4155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.894+0000 7f43d9a25640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43d4161ee0 0x7f43d4155c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37708/0 (socket says 172.21.15.72:37708) 2024-04-15T15:16:55.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.894+0000 7f43d9a25640 1 -- 172.21.15.72:0/1277494734 learned_addr learned my addr 172.21.15.72:0/1277494734 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:55.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.894+0000 7f43d9224640 1 --2- 172.21.15.72:0/1277494734 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f43d40b67c0 0x7f43d40c81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.894+0000 7f43d8a23640 1 --2- 172.21.15.72:0/1277494734 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f43d40b6c20 0x7f43d4155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.901 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.894+0000 7f43d9a25640 1 -- 172.21.15.72:0/1277494734 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f43d40b6c20 msgr2=0x7f43d4155720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.894+0000 7f43d9a25640 1 --2- 172.21.15.72:0/1277494734 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f43d40b6c20 0x7f43d4155720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.894+0000 7f43d9a25640 1 -- 172.21.15.72:0/1277494734 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f43d40b67c0 msgr2=0x7f43d40c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.894+0000 7f43d9a25640 1 --2- 172.21.15.72:0/1277494734 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f43d40b67c0 0x7f43d40c81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.894+0000 7f43d9a25640 1 -- 172.21.15.72:0/1277494734 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43d41580d0 con 0x7f43d4161ee0 2024-04-15T15:16:55.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.894+0000 7f43d8a23640 1 --2- 172.21.15.72:0/1277494734 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f43d40b6c20 0x7f43d4155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:55.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.894+0000 7f43d9a25640 1 --2- 172.21.15.72:0/1277494734 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43d4161ee0 0x7f43d4155c60 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f43cc00c450 tx=0x7f43cc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:55.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.894+0000 7f43c9ffb640 1 -- 172.21.15.72:0/1277494734 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43cc0074e0 con 0x7f43d4161ee0 2024-04-15T15:16:55.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.894+0000 7f43c9ffb640 1 -- 172.21.15.72:0/1277494734 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43cc0047e0 con 0x7f43d4161ee0 2024-04-15T15:16:55.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.894+0000 7f43db488640 1 -- 172.21.15.72:0/1277494734 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43d41583b0 con 0x7f43d4161ee0 2024-04-15T15:16:55.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.894+0000 7f43c9ffb640 1 -- 172.21.15.72:0/1277494734 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43cc004bd0 con 0x7f43d4161ee0 2024-04-15T15:16:55.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.895+0000 7f43db488640 1 -- 172.21.15.72:0/1277494734 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43d4158970 con 0x7f43d4161ee0 2024-04-15T15:16:55.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.896+0000 7f43db488640 1 -- 172.21.15.72:0/1277494734 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43d4075c50 con 0x7f43d4161ee0 2024-04-15T15:16:55.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.896+0000 7f43c9ffb640 1 -- 172.21.15.72:0/1277494734 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f43cc011070 con 0x7f43d4161ee0 2024-04-15T15:16:55.902 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.896+0000 7f43c9ffb640 1 --2- 172.21.15.72:0/1277494734 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f43b806c980 0x7f43b806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.897+0000 7f43d9224640 1 --2- 172.21.15.72:0/1277494734 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f43b806c980 0x7f43b806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.897+0000 7f43c9ffb640 1 -- 172.21.15.72:0/1277494734 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7f43cc08b960 con 0x7f43d4161ee0 2024-04-15T15:16:55.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.897+0000 7f43d9224640 1 --2- 172.21.15.72:0/1277494734 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f43b806c980 0x7f43b806ee40 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f43d00075b0 tx=0x7f43d0007540 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:55.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:55.901+0000 7f43c9ffb640 1 -- 172.21.15.72:0/1277494734 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f43cc01a2b0 con 0x7f43d4161ee0 2024-04-15T15:16:56.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.074+0000 7f43db488640 1 -- 172.21.15.72:0/1277494734 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f43d40d1e50 con 0x7f43d4161ee0 2024-04-15T15:16:56.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.075+0000 7f43c9ffb640 1 -- 172.21.15.72:0/1277494734 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v41) ==== 100+0+11 (secure 0 0 0) 0x7f43cc059c30 con 0x7f43d4161ee0 2024-04-15T15:16:56.078 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 64 2024-04-15T15:16:56.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.077+0000 7f43db488640 1 -- 172.21.15.72:0/1277494734 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f43b806c980 msgr2=0x7f43b806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.077+0000 7f43db488640 1 --2- 172.21.15.72:0/1277494734 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f43b806c980 0x7f43b806ee40 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f43d00075b0 tx=0x7f43d0007540 comp rx=0 tx=0).stop 2024-04-15T15:16:56.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.077+0000 7f43db488640 1 -- 172.21.15.72:0/1277494734 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43d4161ee0 msgr2=0x7f43d4155c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.077+0000 7f43db488640 1 --2- 172.21.15.72:0/1277494734 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43d4161ee0 0x7f43d4155c60 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f43cc00c450 tx=0x7f43cc00c910 comp rx=0 tx=0).stop 2024-04-15T15:16:56.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.077+0000 7f43db488640 1 -- 172.21.15.72:0/1277494734 shutdown_connections 2024-04-15T15:16:56.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.077+0000 7f43db488640 1 --2- 172.21.15.72:0/1277494734 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f43b806c980 0x7f43b806ee40 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.077+0000 7f43db488640 1 --2- 172.21.15.72:0/1277494734 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f43d4161ee0 0x7f43d4155c60 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.077+0000 7f43db488640 1 --2- 172.21.15.72:0/1277494734 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f43d40b6c20 0x7f43d4155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.077+0000 7f43db488640 1 --2- 172.21.15.72:0/1277494734 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f43d40b67c0 0x7f43d40c81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.077+0000 7f43db488640 1 -- 172.21.15.72:0/1277494734 >> 172.21.15.72:0/1277494734 conn(0x7f43d40b4010 msgr2=0x7f43d40cbd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:56.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.077+0000 7f43db488640 1 -- 172.21.15.72:0/1277494734 shutdown_connections 2024-04-15T15:16:56.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.077+0000 7f43db488640 1 -- 172.21.15.72:0/1277494734 wait complete. 2024-04-15T15:16:56.091 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:16:56.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.190+0000 7fbb69206640 1 Processor -- start 2024-04-15T15:16:56.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.190+0000 7fbb69206640 1 -- start start 2024-04-15T15:16:56.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.191+0000 7fbb69206640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbb6413f7a0 0x7fbb640b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.191+0000 7fbb69206640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb6415db60 con 0x7fbb6406fc20 2024-04-15T15:16:56.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.191+0000 7fbb69206640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb64141f60 con 0x7fbb6413f3c0 2024-04-15T15:16:56.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.191+0000 7fbb69206640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fbb64141d10 con 0x7fbb6413f7a0 2024-04-15T15:16:56.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.191+0000 7fbb6359e640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fbb6406fc20 0x7fbb640b4810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60094/0 (socket says 172.21.15.72:60094) 2024-04-15T15:16:56.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.191+0000 7fbb6359e640 1 -- 172.21.15.72:0/1949706713 learned_addr learned my addr 172.21.15.72:0/1949706713 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:56.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.192+0000 7fbb61d9b640 1 -- 172.21.15.72:0/1949706713 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1107667517 0 0) 0x7fbb6415db60 con 0x7fbb6406fc20 2024-04-15T15:16:56.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.192+0000 7fbb61d9b640 1 -- 172.21.15.72:0/1949706713 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb4c003520 con 0x7fbb6406fc20 2024-04-15T15:16:56.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.192+0000 7fbb6259c640 1 --2- 172.21.15.72:0/1949706713 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbb6413f7a0 0x7fbb640b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.192+0000 7fbb61d9b640 1 -- 172.21.15.72:0/1949706713 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1720041065 0 0) 0x7fbb4c003520 con 0x7fbb6406fc20 2024-04-15T15:16:56.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.192+0000 7fbb61d9b640 1 -- 172.21.15.72:0/1949706713 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbb6413f7a0 msgr2=0x7fbb640b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:56.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.192+0000 7fbb61d9b640 1 --2- 172.21.15.72:0/1949706713 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbb6413f7a0 0x7fbb640b42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.192+0000 7fbb61d9b640 1 -- 172.21.15.72:0/1949706713 >> v1:172.21.15.175:6789/0 conn(0x7fbb6413f3c0 legacy=0x7fbb6406f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.192+0000 7fbb61d9b640 1 -- 172.21.15.72:0/1949706713 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb64142240 con 0x7fbb6406fc20 2024-04-15T15:16:56.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.192+0000 7fbb61d9b640 1 -- 172.21.15.72:0/1949706713 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7fbb5c0019f0 con 0x7fbb6406fc20 2024-04-15T15:16:56.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.192+0000 7fbb61d9b640 1 -- 172.21.15.72:0/1949706713 >> v1:172.21.15.72:6789/0 conn(0x7fbb6406fc20 legacy=0x7fbb640b4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.192+0000 7fbb61d9b640 1 --2- 172.21.15.72:0/1949706713 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbb4c0039e0 0x7fbb4c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.192+0000 7fbb61d9b640 1 --2- 172.21.15.72:0/1949706713 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbb6413f7a0 0x7fbb4c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.192+0000 7fbb61d9b640 1 --2- 172.21.15.72:0/1949706713 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbb4c004d40 0x7fbb4c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.192+0000 7fbb61d9b640 1 -- 172.21.15.72:0/1949706713 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fbb64141d10 con 0x7fbb4c0039e0 2024-04-15T15:16:56.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.192+0000 7fbb61d9b640 1 -- 172.21.15.72:0/1949706713 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fbb4c00baf0 con 0x7fbb4c004d40 2024-04-15T15:16:56.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.192+0000 7fbb61d9b640 1 -- 172.21.15.72:0/1949706713 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fbb4c00bcf0 con 0x7fbb6413f7a0 2024-04-15T15:16:56.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.192+0000 7fbb62d9d640 1 --2- 172.21.15.72:0/1949706713 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbb4c0039e0 0x7fbb4c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.192+0000 7fbb6259c640 1 --2- 172.21.15.72:0/1949706713 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbb6413f7a0 0x7fbb4c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.192+0000 7fbb6359e640 1 --2- 172.21.15.72:0/1949706713 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbb4c004d40 0x7fbb4c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.192+0000 7fbb62d9d640 1 -- 172.21.15.72:0/1949706713 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbb6413f7a0 msgr2=0x7fbb4c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:56.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.192+0000 7fbb62d9d640 1 --2- 172.21.15.72:0/1949706713 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbb6413f7a0 0x7fbb4c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.192+0000 7fbb62d9d640 1 -- 172.21.15.72:0/1949706713 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbb4c004d40 msgr2=0x7fbb4c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:56.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.192+0000 7fbb62d9d640 1 --2- 172.21.15.72:0/1949706713 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbb4c004d40 0x7fbb4c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.192+0000 7fbb62d9d640 1 -- 172.21.15.72:0/1949706713 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb4c003520 con 0x7fbb4c0039e0 2024-04-15T15:16:56.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.193+0000 7fbb62d9d640 1 --2- 172.21.15.72:0/1949706713 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbb4c0039e0 0x7fbb4c003e00 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fbb54001750 tx=0x7fbb4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e920819505b87556 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:56.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.193+0000 7fbb61d9b640 1 -- 172.21.15.72:0/1949706713 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb54001370 con 0x7fbb4c0039e0 2024-04-15T15:16:56.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.193+0000 7fbb61d9b640 1 -- 172.21.15.72:0/1949706713 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb540052d0 con 0x7fbb4c0039e0 2024-04-15T15:16:56.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.193+0000 7fbb61d9b640 1 -- 172.21.15.72:0/1949706713 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb54003830 con 0x7fbb4c0039e0 2024-04-15T15:16:56.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.194+0000 7fbb69206640 1 -- 172.21.15.72:0/1949706713 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbb4c0039e0 msgr2=0x7fbb4c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.194+0000 7fbb69206640 1 --2- 172.21.15.72:0/1949706713 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbb4c0039e0 0x7fbb4c003e00 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fbb54001750 tx=0x7fbb4c005830 comp rx=0 tx=0).stop 2024-04-15T15:16:56.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.194+0000 7fbb6359e640 1 -- 172.21.15.72:0/1949706713 reap_dead start 2024-04-15T15:16:56.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.194+0000 7fbb69206640 1 -- 172.21.15.72:0/1949706713 shutdown_connections 2024-04-15T15:16:56.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.194+0000 7fbb69206640 1 -- 172.21.15.72:0/1949706713 >> 172.21.15.72:0/1949706713 conn(0x7fbb640cdd20 msgr2=0x7fbb640ce100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:56.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.194+0000 7fbb69206640 1 -- 172.21.15.72:0/1949706713 shutdown_connections 2024-04-15T15:16:56.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.194+0000 7fbb69206640 1 -- 172.21.15.72:0/1949706713 wait complete. 2024-04-15T15:16:56.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.195+0000 7fbb69206640 1 Processor -- start 2024-04-15T15:16:56.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.195+0000 7fbb69206640 1 -- start start 2024-04-15T15:16:56.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.196+0000 7fbb69206640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbb4c0039e0 0x7fbb641f87e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.196+0000 7fbb69206640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbb641565d0 0x7fbb641f4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.196+0000 7fbb69206640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbb64156a50 0x7fbb6415b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.196+0000 7fbb69206640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fbb640b42d0 con 0x7fbb641565d0 2024-04-15T15:16:56.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.196+0000 7fbb69206640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fbb641601c0 con 0x7fbb4c0039e0 2024-04-15T15:16:56.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.196+0000 7fbb69206640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fbb640b4450 con 0x7fbb64156a50 2024-04-15T15:16:56.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.196+0000 7fbb6259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbb641565d0 0x7fbb641f4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.196+0000 7fbb6259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbb641565d0 0x7fbb641f4840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37734/0 (socket says 172.21.15.72:37734) 2024-04-15T15:16:56.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.196+0000 7fbb6259c640 1 -- 172.21.15.72:0/3711075830 learned_addr learned my addr 172.21.15.72:0/3711075830 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:56.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.196+0000 7fbb6359e640 1 --2- 172.21.15.72:0/3711075830 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbb64156a50 0x7fbb6415b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.196+0000 7fbb62d9d640 1 --2- 172.21.15.72:0/3711075830 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbb4c0039e0 0x7fbb641f87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.196+0000 7fbb6259c640 1 -- 172.21.15.72:0/3711075830 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbb64156a50 msgr2=0x7fbb6415b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.196+0000 7fbb6259c640 1 --2- 172.21.15.72:0/3711075830 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbb64156a50 0x7fbb6415b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.196+0000 7fbb6259c640 1 -- 172.21.15.72:0/3711075830 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbb4c0039e0 msgr2=0x7fbb641f87e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.196+0000 7fbb6259c640 1 --2- 172.21.15.72:0/3711075830 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbb4c0039e0 0x7fbb641f87e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.196+0000 7fbb6259c640 1 -- 172.21.15.72:0/3711075830 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb641f8d20 con 0x7fbb641565d0 2024-04-15T15:16:56.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.196+0000 7fbb62d9d640 1 --2- 172.21.15.72:0/3711075830 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbb4c0039e0 0x7fbb641f87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:56.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.196+0000 7fbb6259c640 1 --2- 172.21.15.72:0/3711075830 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbb641565d0 0x7fbb641f4840 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fbb58011600 tx=0x7fbb58011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:56.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.197+0000 7fbb4b7fe640 1 -- 172.21.15.72:0/3711075830 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb5800f920 con 0x7fbb641565d0 2024-04-15T15:16:56.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.197+0000 7fbb4b7fe640 1 -- 172.21.15.72:0/3711075830 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb5800fac0 con 0x7fbb641565d0 2024-04-15T15:16:56.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.197+0000 7fbb69206640 1 -- 172.21.15.72:0/3711075830 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb641f52f0 con 0x7fbb641565d0 2024-04-15T15:16:56.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.197+0000 7fbb4b7fe640 1 -- 172.21.15.72:0/3711075830 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb5800fe70 con 0x7fbb641565d0 2024-04-15T15:16:56.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.197+0000 7fbb69206640 1 -- 172.21.15.72:0/3711075830 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb641f7fe0 con 0x7fbb641565d0 2024-04-15T15:16:56.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.198+0000 7fbb69206640 1 -- 172.21.15.72:0/3711075830 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb641f7dd0 con 0x7fbb641565d0 2024-04-15T15:16:56.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.198+0000 7fbb4b7fe640 1 -- 172.21.15.72:0/3711075830 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fbb58015460 con 0x7fbb641565d0 2024-04-15T15:16:56.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.199+0000 7fbb4b7fe640 1 --2- 172.21.15.72:0/3711075830 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fbb5006c980 0x7fbb5006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.199+0000 7fbb4b7fe640 1 -- 172.21.15.72:0/3711075830 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7fbb5808c500 con 0x7fbb641565d0 2024-04-15T15:16:56.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.199+0000 7fbb62d9d640 1 --2- 172.21.15.72:0/3711075830 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fbb5006c980 0x7fbb5006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.199+0000 7fbb62d9d640 1 --2- 172.21.15.72:0/3711075830 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fbb5006c980 0x7fbb5006ee40 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fbb54033000 tx=0x7fbb54009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:56.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.203+0000 7fbb4b7fe640 1 -- 172.21.15.72:0/3711075830 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbb641f8d20 con 0x7fbb641565d0 2024-04-15T15:16:56.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.414+0000 7fbb69206640 1 -- 172.21.15.72:0/3711075830 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb640690f0 con 0x7fbb641565d0 2024-04-15T15:16:56.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.415+0000 7fbb4b7fe640 1 -- 172.21.15.72:0/3711075830 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+5185 (secure 0 0 0) 0x7fbb58019030 con 0x7fbb641565d0 2024-04-15T15:16:56.418 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:16:56.418 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":22,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4883,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3363272836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3363272836},{"type":"v1","addr":"172.21.15.175:6834","nonce":3363272836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5134,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6835/1059187505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":1059187505},{"type":"v1","addr":"172.21.15.175:6835","nonce":1059187505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5556,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/690132275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":690132275},{"type":"v1","addr":"172.21.15.72:6837","nonce":690132275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:16:07.050078+0000","modified":"2024-04-15T15:16:45.725971+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.72:6836/3407328263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3407328263},{"type":"v1","addr":"172.21.15.72:6836","nonce":3407328263}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-04-15T15:16:56.418 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 22 2024-04-15T15:16:56.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.417+0000 7fbb69206640 1 -- 172.21.15.72:0/3711075830 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fbb5006c980 msgr2=0x7fbb5006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.417+0000 7fbb69206640 1 --2- 172.21.15.72:0/3711075830 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fbb5006c980 0x7fbb5006ee40 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fbb54033000 tx=0x7fbb54009030 comp rx=0 tx=0).stop 2024-04-15T15:16:56.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.417+0000 7fbb69206640 1 -- 172.21.15.72:0/3711075830 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbb641565d0 msgr2=0x7fbb641f4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.417+0000 7fbb69206640 1 --2- 172.21.15.72:0/3711075830 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbb641565d0 0x7fbb641f4840 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fbb58011600 tx=0x7fbb58011ac0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.417+0000 7fbb69206640 1 -- 172.21.15.72:0/3711075830 shutdown_connections 2024-04-15T15:16:56.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.417+0000 7fbb69206640 1 --2- 172.21.15.72:0/3711075830 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fbb5006c980 0x7fbb5006ee40 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.417+0000 7fbb69206640 1 --2- 172.21.15.72:0/3711075830 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fbb64156a50 0x7fbb6415b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.417+0000 7fbb69206640 1 --2- 172.21.15.72:0/3711075830 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbb641565d0 0x7fbb641f4840 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.417+0000 7fbb69206640 1 --2- 172.21.15.72:0/3711075830 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fbb4c0039e0 0x7fbb641f87e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.417+0000 7fbb69206640 1 -- 172.21.15.72:0/3711075830 >> 172.21.15.72:0/3711075830 conn(0x7fbb640cdd20 msgr2=0x7fbb640b18f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:56.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.417+0000 7fbb69206640 1 -- 172.21.15.72:0/3711075830 shutdown_connections 2024-04-15T15:16:56.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.417+0000 7fbb69206640 1 -- 172.21.15.72:0/3711075830 wait complete. 2024-04-15T15:16:56.431 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:56.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.530+0000 7f37cafcd640 1 Processor -- start 2024-04-15T15:16:56.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.530+0000 7f37cafcd640 1 -- start start 2024-04-15T15:16:56.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.531+0000 7f37cafcd640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f37c406eb30 0x7f37c406c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.531+0000 7f37cafcd640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37c40cffd0 con 0x7f37c40af700 2024-04-15T15:16:56.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.531+0000 7f37cafcd640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37c406de60 con 0x7f37c415c7c0 2024-04-15T15:16:56.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.531+0000 7f37cafcd640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f37c415cba0 con 0x7f37c406eb30 2024-04-15T15:16:56.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.531+0000 7f37bbfff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f37c40af700 0x7f37c406b960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60104/0 (socket says 172.21.15.72:60104) 2024-04-15T15:16:56.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.531+0000 7f37bbfff640 1 -- 172.21.15.72:0/2774302977 learned_addr learned my addr 172.21.15.72:0/2774302977 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:56.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.531+0000 7f37c956a640 1 --2- 172.21.15.72:0/2774302977 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f37c406eb30 0x7f37c406c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.531+0000 7f37bb7fe640 1 -- 172.21.15.72:0/2774302977 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1208555433 0 0) 0x7f37c40cffd0 con 0x7f37c40af700 2024-04-15T15:16:56.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.531+0000 7f37bb7fe640 1 -- 172.21.15.72:0/2774302977 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37ac003520 con 0x7f37c40af700 2024-04-15T15:16:56.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.532+0000 7f37bb7fe640 1 -- 172.21.15.72:0/2774302977 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1879612914 0 0) 0x7f37c406de60 con 0x7f37c415c7c0 2024-04-15T15:16:56.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.532+0000 7f37bb7fe640 1 -- 172.21.15.72:0/2774302977 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37c40cffd0 con 0x7f37c415c7c0 2024-04-15T15:16:56.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.532+0000 7f37c956a640 1 -- 172.21.15.72:0/2774302977 >> v1:172.21.15.175:6789/0 conn(0x7f37c415c7c0 legacy=0x7f37c406e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.532+0000 7f37c956a640 1 -- 172.21.15.72:0/2774302977 >> v1:172.21.15.72:6789/0 conn(0x7f37c40af700 legacy=0x7f37c406b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.532+0000 7f37bb7fe640 1 -- 172.21.15.72:0/2774302977 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2471219211 0 0) 0x7f37ac003520 con 0x7f37c40af700 2024-04-15T15:16:56.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.532+0000 7f37c956a640 1 -- 172.21.15.72:0/2774302977 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37c406e140 con 0x7f37c406eb30 2024-04-15T15:16:56.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.533+0000 7f37c956a640 1 --2- 172.21.15.72:0/2774302977 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f37c406eb30 0x7f37c406c070 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f37bc009dc0 tx=0x7f37bc02bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=916d60da82302f60 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:56.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.533+0000 7f37bb7fe640 1 -- 172.21.15.72:0/2774302977 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37bc039070 con 0x7f37c406eb30 2024-04-15T15:16:56.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.533+0000 7f37bb7fe640 1 -- 172.21.15.72:0/2774302977 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37bc02c910 con 0x7f37c406eb30 2024-04-15T15:16:56.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.533+0000 7f37bb7fe640 1 -- 172.21.15.72:0/2774302977 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37b4001670 con 0x7f37c406eb30 2024-04-15T15:16:56.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.533+0000 7f37cafcd640 1 -- 172.21.15.72:0/2774302977 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f37c406eb30 msgr2=0x7f37c406c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.533+0000 7f37cafcd640 1 --2- 172.21.15.72:0/2774302977 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f37c406eb30 0x7f37c406c070 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f37bc009dc0 tx=0x7f37bc02bde0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.533+0000 7f37cafcd640 1 -- 172.21.15.72:0/2774302977 shutdown_connections 2024-04-15T15:16:56.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.533+0000 7f37cafcd640 1 --2- 172.21.15.72:0/2774302977 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f37c406eb30 0x7f37c406c070 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.537 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.533+0000 7f37cafcd640 1 -- 172.21.15.72:0/2774302977 >> 172.21.15.72:0/2774302977 conn(0x7f37c40b1dd0 msgr2=0x7f37c40b21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:56.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.533+0000 7f37cafcd640 1 -- 172.21.15.72:0/2774302977 shutdown_connections 2024-04-15T15:16:56.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.533+0000 7f37cafcd640 1 -- 172.21.15.72:0/2774302977 wait complete. 2024-04-15T15:16:56.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.534+0000 7f37cafcd640 1 Processor -- start 2024-04-15T15:16:56.538 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.534+0000 7f37cafcd640 1 -- start start 2024-04-15T15:16:56.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.535+0000 7f37cafcd640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f37c406eb30 0x7f37c41f1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.535+0000 7f37cafcd640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37c40af700 0x7f37c4156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.535+0000 7f37cafcd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37c415c7c0 0x7f37c41542c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.535+0000 7f37cafcd640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f37c406efc0 con 0x7f37c415c7c0 2024-04-15T15:16:56.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.535+0000 7f37cafcd640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f37c406dc10 con 0x7f37c40af700 2024-04-15T15:16:56.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.535+0000 7f37cafcd640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f37c40cfab0 con 0x7f37c406eb30 2024-04-15T15:16:56.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.535+0000 7f37c956a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37c415c7c0 0x7f37c41542c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.535+0000 7f37c956a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37c415c7c0 0x7f37c41542c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37738/0 (socket says 172.21.15.72:37738) 2024-04-15T15:16:56.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.536+0000 7f37c956a640 1 -- 172.21.15.72:0/481573960 learned_addr learned my addr 172.21.15.72:0/481573960 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:56.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.536+0000 7f37bbfff640 1 --2- 172.21.15.72:0/481573960 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37c40af700 0x7f37c4156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.536+0000 7f37c8d69640 1 --2- 172.21.15.72:0/481573960 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f37c406eb30 0x7f37c41f1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.536+0000 7f37c956a640 1 -- 172.21.15.72:0/481573960 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f37c406eb30 msgr2=0x7f37c41f1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.536+0000 7f37c956a640 1 --2- 172.21.15.72:0/481573960 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f37c406eb30 0x7f37c41f1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.536+0000 7f37c956a640 1 -- 172.21.15.72:0/481573960 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37c40af700 msgr2=0x7f37c4156110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.536+0000 7f37c956a640 1 --2- 172.21.15.72:0/481573960 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37c40af700 0x7f37c4156110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.536+0000 7f37c956a640 1 -- 172.21.15.72:0/481573960 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37c41f22e0 con 0x7f37c415c7c0 2024-04-15T15:16:56.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.536+0000 7f37c8d69640 1 --2- 172.21.15.72:0/481573960 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f37c406eb30 0x7f37c41f1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:56.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.536+0000 7f37bbfff640 1 --2- 172.21.15.72:0/481573960 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37c40af700 0x7f37c4156110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:56.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.536+0000 7f37c956a640 1 --2- 172.21.15.72:0/481573960 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37c415c7c0 0x7f37c41542c0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f37bc02c3f0 tx=0x7f37bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:56.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.536+0000 7f37b97fa640 1 -- 172.21.15.72:0/481573960 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37bc039050 con 0x7f37c415c7c0 2024-04-15T15:16:56.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.536+0000 7f37b97fa640 1 -- 172.21.15.72:0/481573960 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37bc045070 con 0x7f37c415c7c0 2024-04-15T15:16:56.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.536+0000 7f37cafcd640 1 -- 172.21.15.72:0/481573960 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37c4156b80 con 0x7f37c415c7c0 2024-04-15T15:16:56.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.536+0000 7f37b97fa640 1 -- 172.21.15.72:0/481573960 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37bc009450 con 0x7f37c415c7c0 2024-04-15T15:16:56.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.536+0000 7f37cafcd640 1 -- 172.21.15.72:0/481573960 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37c4157140 con 0x7f37c415c7c0 2024-04-15T15:16:56.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.537+0000 7f37cafcd640 1 -- 172.21.15.72:0/481573960 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37c4067fb0 con 0x7f37c415c7c0 2024-04-15T15:16:56.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.538+0000 7f37b97fa640 1 -- 172.21.15.72:0/481573960 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f37bc040690 con 0x7f37c415c7c0 2024-04-15T15:16:56.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.538+0000 7f37b97fa640 1 --2- 172.21.15.72:0/481573960 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f37b006c980 0x7f37b006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.538+0000 7f37b97fa640 1 -- 172.21.15.72:0/481573960 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7f37bc0ae660 con 0x7f37c415c7c0 2024-04-15T15:16:56.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.538+0000 7f37c8d69640 1 --2- 172.21.15.72:0/481573960 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f37b006c980 0x7f37b006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.538+0000 7f37c8d69640 1 --2- 172.21.15.72:0/481573960 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f37b006c980 0x7f37b006ee40 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f37c00011e0 tx=0x7f37c0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:56.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.542+0000 7f37b97fa640 1 -- 172.21.15.72:0/481573960 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f37bc07ca80 con 0x7f37c415c7c0 2024-04-15T15:16:56.720 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.717+0000 7f37cafcd640 1 -- 172.21.15.72:0/481573960 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f37c406d100 con 0x7f37c415c7c0 2024-04-15T15:16:56.721 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.718+0000 7f37b97fa640 1 -- 172.21.15.72:0/481573960 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) ==== 74+0+17120 (secure 0 0 0) 0x7f37bc07c380 con 0x7f37c415c7c0 2024-04-15T15:16:56.721 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:16:56.721 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":41,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:16:41.714841+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:05.551499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:06.003152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.26","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:6836/4270793613":"2024-04-16T15:16:41.714817+0000","172.21.15.72:6834/4270793613":"2024-04-16T15:16:41.714817+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:56.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.721+0000 7f37cafcd640 1 -- 172.21.15.72:0/481573960 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f37b006c980 msgr2=0x7f37b006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.721+0000 7f37cafcd640 1 --2- 172.21.15.72:0/481573960 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f37b006c980 0x7f37b006ee40 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f37c00011e0 tx=0x7f37c0009040 comp rx=0 tx=0).stop 2024-04-15T15:16:56.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.721+0000 7f37cafcd640 1 -- 172.21.15.72:0/481573960 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37c415c7c0 msgr2=0x7f37c41542c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.721+0000 7f37cafcd640 1 --2- 172.21.15.72:0/481573960 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37c415c7c0 0x7f37c41542c0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f37bc02c3f0 tx=0x7f37bc004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:56.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.721+0000 7f37cafcd640 1 -- 172.21.15.72:0/481573960 shutdown_connections 2024-04-15T15:16:56.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.721+0000 7f37cafcd640 1 --2- 172.21.15.72:0/481573960 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f37b006c980 0x7f37b006ee40 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.721+0000 7f37cafcd640 1 --2- 172.21.15.72:0/481573960 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f37c415c7c0 0x7f37c41542c0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.721+0000 7f37cafcd640 1 --2- 172.21.15.72:0/481573960 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f37c40af700 0x7f37c4156110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.721+0000 7f37cafcd640 1 --2- 172.21.15.72:0/481573960 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f37c406eb30 0x7f37c41f1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.721+0000 7f37cafcd640 1 -- 172.21.15.72:0/481573960 >> 172.21.15.72:0/481573960 conn(0x7f37c40b1dd0 msgr2=0x7f37c406e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:56.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.721+0000 7f37cafcd640 1 -- 172.21.15.72:0/481573960 shutdown_connections 2024-04-15T15:16:56.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.721+0000 7f37cafcd640 1 -- 172.21.15.72:0/481573960 wait complete. 2024-04-15T15:16:56.735 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-15T15:16:56.735 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:16:56.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.836+0000 7f8b6ec0d640 1 Processor -- start 2024-04-15T15:16:56.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.836+0000 7f8b6ec0d640 1 -- start start 2024-04-15T15:16:56.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.837+0000 7f8b6ec0d640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8b6815e7e0 0x7f8b680b67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.837+0000 7f8b6ec0d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8b680ae360 0x7f8b68072b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.837+0000 7f8b6ec0d640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b68162800 con 0x7f8b680730c0 2024-04-15T15:16:56.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.837+0000 7f8b6ec0d640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f8b68162ae0 con 0x7f8b680ae360 2024-04-15T15:16:56.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.837+0000 7f8b6ec0d640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f8b68162dc0 con 0x7f8b6815e7e0 2024-04-15T15:16:56.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.837+0000 7f8b5ffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8b680ae360 0x7f8b68072b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.837+0000 7f8b5ffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8b680ae360 0x7f8b68072b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37740/0 (socket says 172.21.15.72:37740) 2024-04-15T15:16:56.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.837+0000 7f8b5ffff640 1 -- 172.21.15.72:0/3039271121 learned_addr learned my addr 172.21.15.72:0/3039271121 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:56.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.837+0000 7f8b5ffff640 1 -- 172.21.15.72:0/3039271121 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8b6815e7e0 msgr2=0x7f8b680b67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:56.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.837+0000 7f8b5ffff640 1 --2- 172.21.15.72:0/3039271121 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8b6815e7e0 0x7f8b680b67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.837+0000 7f8b5ffff640 1 -- 172.21.15.72:0/3039271121 >> v1:172.21.15.175:6789/0 conn(0x7f8b680730c0 legacy=0x7f8b680cf180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:56.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.837+0000 7f8b5ffff640 1 -- 172.21.15.72:0/3039271121 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b681630a0 con 0x7f8b680ae360 2024-04-15T15:16:56.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.838+0000 7f8b5ffff640 1 --2- 172.21.15.72:0/3039271121 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b680ae360 0x7f8b68072b80 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f8b58009870 tx=0x7f8b5802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eca485ad17f9efe1 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:56.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.838+0000 7f8b5f7fe640 1 -- 172.21.15.72:0/3039271121 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b58035020 con 0x7f8b680ae360 2024-04-15T15:16:56.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.838+0000 7f8b5f7fe640 1 -- 172.21.15.72:0/3039271121 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b58031640 con 0x7f8b680ae360 2024-04-15T15:16:56.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.838+0000 7f8b5f7fe640 1 -- 172.21.15.72:0/3039271121 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b5802d040 con 0x7f8b680ae360 2024-04-15T15:16:56.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.838+0000 7f8b6ec0d640 1 -- 172.21.15.72:0/3039271121 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b680ae360 msgr2=0x7f8b68072b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.838+0000 7f8b6ec0d640 1 --2- 172.21.15.72:0/3039271121 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b680ae360 0x7f8b68072b80 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f8b58009870 tx=0x7f8b5802b8f0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.839+0000 7f8b6ec0d640 1 -- 172.21.15.72:0/3039271121 shutdown_connections 2024-04-15T15:16:56.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.839+0000 7f8b6ec0d640 1 --2- 172.21.15.72:0/3039271121 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b680ae360 0x7f8b68072b80 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.839+0000 7f8b6ec0d640 1 --2- 172.21.15.72:0/3039271121 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8b6815e7e0 0x7f8b680b67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.839+0000 7f8b6ec0d640 1 -- 172.21.15.72:0/3039271121 >> 172.21.15.72:0/3039271121 conn(0x7f8b680b31b0 msgr2=0x7f8b680b3590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:56.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.839+0000 7f8b6ec0d640 1 -- 172.21.15.72:0/3039271121 shutdown_connections 2024-04-15T15:16:56.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.839+0000 7f8b6ec0d640 1 -- 172.21.15.72:0/3039271121 wait complete. 2024-04-15T15:16:56.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.840+0000 7f8b6ec0d640 1 Processor -- start 2024-04-15T15:16:56.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.840+0000 7f8b6ec0d640 1 -- start start 2024-04-15T15:16:56.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.841+0000 7f8b6ec0d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b680730c0 0x7f8b68150670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.841+0000 7f8b6ec0d640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b680ae360 0x7f8b681e8b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.841+0000 7f8b6ec0d640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8b6815e7e0 0x7f8b6806e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.841+0000 7f8b6ec0d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8b68162550 con 0x7f8b680730c0 2024-04-15T15:16:56.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.841+0000 7f8b6ec0d640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f8b681623d0 con 0x7f8b680ae360 2024-04-15T15:16:56.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.841+0000 7f8b6ec0d640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f8b6806fd30 con 0x7f8b6815e7e0 2024-04-15T15:16:56.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.841+0000 7f8b6c9a9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b680730c0 0x7f8b68150670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.841+0000 7f8b6c9a9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b680730c0 0x7f8b68150670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37744/0 (socket says 172.21.15.72:37744) 2024-04-15T15:16:56.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.841+0000 7f8b6c9a9640 1 -- 172.21.15.72:0/664874422 learned_addr learned my addr 172.21.15.72:0/664874422 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:56.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.841+0000 7f8b5ffff640 1 --2- 172.21.15.72:0/664874422 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b680ae360 0x7f8b681e8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.841+0000 7f8b6d1aa640 1 --2- 172.21.15.72:0/664874422 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8b6815e7e0 0x7f8b6806e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.846 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.841+0000 7f8b6c9a9640 1 -- 172.21.15.72:0/664874422 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8b6815e7e0 msgr2=0x7f8b6806e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.841+0000 7f8b6c9a9640 1 --2- 172.21.15.72:0/664874422 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8b6815e7e0 0x7f8b6806e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.841+0000 7f8b6c9a9640 1 -- 172.21.15.72:0/664874422 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b680ae360 msgr2=0x7f8b681e8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.841+0000 7f8b6c9a9640 1 --2- 172.21.15.72:0/664874422 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b680ae360 0x7f8b681e8b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.841+0000 7f8b6c9a9640 1 -- 172.21.15.72:0/664874422 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b6806c270 con 0x7f8b680730c0 2024-04-15T15:16:56.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.841+0000 7f8b5ffff640 1 --2- 172.21.15.72:0/664874422 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b680ae360 0x7f8b681e8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:56.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.841+0000 7f8b6d1aa640 1 --2- 172.21.15.72:0/664874422 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8b6815e7e0 0x7f8b6806e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:56.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.841+0000 7f8b6c9a9640 1 --2- 172.21.15.72:0/664874422 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b680730c0 0x7f8b68150670 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f8b6400b410 tx=0x7f8b6400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:56.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.841+0000 7f8b5d7fa640 1 -- 172.21.15.72:0/664874422 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b64015020 con 0x7f8b680730c0 2024-04-15T15:16:56.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.841+0000 7f8b5d7fa640 1 -- 172.21.15.72:0/664874422 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b64011850 con 0x7f8b680730c0 2024-04-15T15:16:56.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.841+0000 7f8b6ec0d640 1 -- 172.21.15.72:0/664874422 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b68150bb0 con 0x7f8b680730c0 2024-04-15T15:16:56.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.841+0000 7f8b5d7fa640 1 -- 172.21.15.72:0/664874422 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b64011c40 con 0x7f8b680730c0 2024-04-15T15:16:56.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.841+0000 7f8b6ec0d640 1 -- 172.21.15.72:0/664874422 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b6806c7f0 con 0x7f8b680730c0 2024-04-15T15:16:56.847 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.842+0000 7f8b6ec0d640 1 -- 172.21.15.72:0/664874422 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b6806cff0 con 0x7f8b680730c0 2024-04-15T15:16:56.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.843+0000 7f8b5d7fa640 1 -- 172.21.15.72:0/664874422 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f8b64018020 con 0x7f8b680730c0 2024-04-15T15:16:56.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.843+0000 7f8b5d7fa640 1 --2- 172.21.15.72:0/664874422 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8b5406c9b0 0x7f8b5406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.843+0000 7f8b5d7fa640 1 -- 172.21.15.72:0/664874422 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7f8b6408bd30 con 0x7f8b680730c0 2024-04-15T15:16:56.848 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.843+0000 7f8b5ffff640 1 --2- 172.21.15.72:0/664874422 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8b5406c9b0 0x7f8b5406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.844+0000 7f8b5ffff640 1 --2- 172.21.15.72:0/664874422 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8b5406c9b0 0x7f8b5406ee70 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f8b58036000 tx=0x7f8b5803c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:56.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:56.848+0000 7f8b5d7fa640 1 -- 172.21.15.72:0/664874422 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8b6401f020 con 0x7f8b680730c0 2024-04-15T15:16:57.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.062+0000 7f8b6ec0d640 1 -- 172.21.15.72:0/664874422 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b680689e0 con 0x7f8b680730c0 2024-04-15T15:16:57.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.063+0000 7f8b5d7fa640 1 -- 172.21.15.72:0/664874422 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+5185 (secure 0 0 0) 0x7f8b640172c0 con 0x7f8b680730c0 2024-04-15T15:16:57.066 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:16:57.066 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":22,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4883,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3363272836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3363272836},{"type":"v1","addr":"172.21.15.175:6834","nonce":3363272836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5134,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6835/1059187505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":1059187505},{"type":"v1","addr":"172.21.15.175:6835","nonce":1059187505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5556,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/690132275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":690132275},{"type":"v1","addr":"172.21.15.72:6837","nonce":690132275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:16:07.050078+0000","modified":"2024-04-15T15:16:45.725971+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.72:6836/3407328263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3407328263},{"type":"v1","addr":"172.21.15.72:6836","nonce":3407328263}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-04-15T15:16:57.066 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 22 2024-04-15T15:16:57.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.065+0000 7f8b6ec0d640 1 -- 172.21.15.72:0/664874422 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8b5406c9b0 msgr2=0x7f8b5406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:57.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.065+0000 7f8b6ec0d640 1 --2- 172.21.15.72:0/664874422 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8b5406c9b0 0x7f8b5406ee70 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f8b58036000 tx=0x7f8b5803c040 comp rx=0 tx=0).stop 2024-04-15T15:16:57.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.065+0000 7f8b6ec0d640 1 -- 172.21.15.72:0/664874422 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b680730c0 msgr2=0x7f8b68150670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:57.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.065+0000 7f8b6ec0d640 1 --2- 172.21.15.72:0/664874422 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b680730c0 0x7f8b68150670 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f8b6400b410 tx=0x7f8b6400b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.065+0000 7f8b6ec0d640 1 -- 172.21.15.72:0/664874422 shutdown_connections 2024-04-15T15:16:57.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.065+0000 7f8b6ec0d640 1 --2- 172.21.15.72:0/664874422 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8b5406c9b0 0x7f8b5406ee70 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.065+0000 7f8b6ec0d640 1 --2- 172.21.15.72:0/664874422 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8b6815e7e0 0x7f8b6806e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.065+0000 7f8b6ec0d640 1 --2- 172.21.15.72:0/664874422 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8b680ae360 0x7f8b681e8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.065+0000 7f8b6ec0d640 1 --2- 172.21.15.72:0/664874422 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b680730c0 0x7f8b68150670 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.065+0000 7f8b6ec0d640 1 -- 172.21.15.72:0/664874422 >> 172.21.15.72:0/664874422 conn(0x7f8b680b31b0 msgr2=0x7f8b680b17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:57.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.065+0000 7f8b6ec0d640 1 -- 172.21.15.72:0/664874422 shutdown_connections 2024-04-15T15:16:57.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.065+0000 7f8b6ec0d640 1 -- 172.21.15.72:0/664874422 wait complete. 2024-04-15T15:16:57.079 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:16:57.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.179+0000 7f39daa7e640 1 Processor -- start 2024-04-15T15:16:57.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.180+0000 7f39daa7e640 1 -- start start 2024-04-15T15:16:57.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.180+0000 7f39daa7e640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f39d415c480 0x7f39d406ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:57.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.180+0000 7f39daa7e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f39d40ce670 0x7f39d40cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:57.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.180+0000 7f39daa7e640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f39d40cf300 0x7f39d40cff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:57.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.180+0000 7f39daa7e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f39d40d1b30 con 0x7f39d40ce670 2024-04-15T15:16:57.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.180+0000 7f39daa7e640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f39d40cca00 con 0x7f39d415c480 2024-04-15T15:16:57.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.180+0000 7f39daa7e640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f39d40d1d80 con 0x7f39d40cf300 2024-04-15T15:16:57.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.180+0000 7f39d3fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f39d40ce670 0x7f39d40cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:57.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.180+0000 7f39d3fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f39d40ce670 0x7f39d40cedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37752/0 (socket says 172.21.15.72:37752) 2024-04-15T15:16:57.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.180+0000 7f39d3fff640 1 -- 172.21.15.72:0/642924550 learned_addr learned my addr 172.21.15.72:0/642924550 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:57.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.181+0000 7f39d881a640 1 --2- 172.21.15.72:0/642924550 >> v2:172.21.15.175:3300/0 conn(0x7f39d415c480 0x7f39d406ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:57.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.181+0000 7f39d3fff640 1 -- 172.21.15.72:0/642924550 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f39d40cf300 msgr2=0x7f39d40cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:57.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.181+0000 7f39d901b640 1 --2- 172.21.15.72:0/642924550 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f39d40cf300 0x7f39d40cff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:57.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.181+0000 7f39d3fff640 1 --2- 172.21.15.72:0/642924550 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f39d40cf300 0x7f39d40cff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.181+0000 7f39d3fff640 1 -- 172.21.15.72:0/642924550 >> v2:172.21.15.175:3300/0 conn(0x7f39d415c480 msgr2=0x7f39d406ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:57.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.181+0000 7f39d3fff640 1 --2- 172.21.15.72:0/642924550 >> v2:172.21.15.175:3300/0 conn(0x7f39d415c480 0x7f39d406ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.181+0000 7f39d3fff640 1 -- 172.21.15.72:0/642924550 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39d40d2060 con 0x7f39d40ce670 2024-04-15T15:16:57.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.181+0000 7f39d881a640 1 --2- 172.21.15.72:0/642924550 >> v2:172.21.15.175:3300/0 conn(0x7f39d415c480 0x7f39d406ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:57.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.181+0000 7f39d901b640 1 --2- 172.21.15.72:0/642924550 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f39d40cf300 0x7f39d40cff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:57.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.181+0000 7f39d3fff640 1 --2- 172.21.15.72:0/642924550 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39d40ce670 0x7f39d40cedc0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f39d40cff30 tx=0x7f39c802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3256d04cf9d8556 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:57.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.182+0000 7f39d37fe640 1 -- 172.21.15.72:0/642924550 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c8035020 con 0x7f39d40ce670 2024-04-15T15:16:57.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.182+0000 7f39d37fe640 1 -- 172.21.15.72:0/642924550 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39c802d040 con 0x7f39d40ce670 2024-04-15T15:16:57.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.182+0000 7f39d37fe640 1 -- 172.21.15.72:0/642924550 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39c8031a80 con 0x7f39d40ce670 2024-04-15T15:16:57.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.182+0000 7f39daa7e640 1 -- 172.21.15.72:0/642924550 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39d40ce670 msgr2=0x7f39d40cedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:57.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.182+0000 7f39daa7e640 1 --2- 172.21.15.72:0/642924550 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39d40ce670 0x7f39d40cedc0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f39d40cff30 tx=0x7f39c802b900 comp rx=0 tx=0).stop 2024-04-15T15:16:57.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.182+0000 7f39daa7e640 1 -- 172.21.15.72:0/642924550 shutdown_connections 2024-04-15T15:16:57.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.182+0000 7f39daa7e640 1 --2- 172.21.15.72:0/642924550 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f39d40cf300 0x7f39d40cff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.182+0000 7f39daa7e640 1 --2- 172.21.15.72:0/642924550 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39d40ce670 0x7f39d40cedc0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.182+0000 7f39daa7e640 1 --2- 172.21.15.72:0/642924550 >> v2:172.21.15.175:3300/0 conn(0x7f39d415c480 0x7f39d406ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.182+0000 7f39daa7e640 1 -- 172.21.15.72:0/642924550 >> 172.21.15.72:0/642924550 conn(0x7f39d40b3fa0 msgr2=0x7f39d40b43b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:57.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.182+0000 7f39daa7e640 1 -- 172.21.15.72:0/642924550 shutdown_connections 2024-04-15T15:16:57.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.182+0000 7f39daa7e640 1 -- 172.21.15.72:0/642924550 wait complete. 2024-04-15T15:16:57.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.183+0000 7f39daa7e640 1 Processor -- start 2024-04-15T15:16:57.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.183+0000 7f39daa7e640 1 -- start start 2024-04-15T15:16:57.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.184+0000 7f39daa7e640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f39d40ce670 0x7f39d414fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:57.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.184+0000 7f39daa7e640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f39d40cf300 0x7f39d41e8470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:57.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.184+0000 7f39daa7e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39d415c480 0x7f39d41f4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:57.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.184+0000 7f39daa7e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f39d40cd170 con 0x7f39d415c480 2024-04-15T15:16:57.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.184+0000 7f39daa7e640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f39d40cead0 con 0x7f39d40ce670 2024-04-15T15:16:57.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.184+0000 7f39daa7e640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f39d415c860 con 0x7f39d40cf300 2024-04-15T15:16:57.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.184+0000 7f39d901b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39d415c480 0x7f39d41f4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:57.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.184+0000 7f39d901b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39d415c480 0x7f39d41f4020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37758/0 (socket says 172.21.15.72:37758) 2024-04-15T15:16:57.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.184+0000 7f39d901b640 1 -- 172.21.15.72:0/109514312 learned_addr learned my addr 172.21.15.72:0/109514312 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:57.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.185+0000 7f39d3fff640 1 --2- 172.21.15.72:0/109514312 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f39d40cf300 0x7f39d41e8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:57.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.185+0000 7f39d881a640 1 --2- 172.21.15.72:0/109514312 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f39d40ce670 0x7f39d414fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:57.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.185+0000 7f39d901b640 1 -- 172.21.15.72:0/109514312 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f39d40cf300 msgr2=0x7f39d41e8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:57.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.185+0000 7f39d901b640 1 --2- 172.21.15.72:0/109514312 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f39d40cf300 0x7f39d41e8470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.185+0000 7f39d901b640 1 -- 172.21.15.72:0/109514312 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f39d40ce670 msgr2=0x7f39d414fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:57.191 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.185+0000 7f39d901b640 1 --2- 172.21.15.72:0/109514312 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f39d40ce670 0x7f39d414fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.185+0000 7f39d901b640 1 -- 172.21.15.72:0/109514312 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39d41e89b0 con 0x7f39d415c480 2024-04-15T15:16:57.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.185+0000 7f39d901b640 1 --2- 172.21.15.72:0/109514312 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39d415c480 0x7f39d41f4020 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f39cc00cef0 tx=0x7f39cc00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:57.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.185+0000 7f39d17fa640 1 -- 172.21.15.72:0/109514312 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39cc01a070 con 0x7f39d415c480 2024-04-15T15:16:57.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.185+0000 7f39d17fa640 1 -- 172.21.15.72:0/109514312 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39cc00d860 con 0x7f39d415c480 2024-04-15T15:16:57.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.185+0000 7f39daa7e640 1 -- 172.21.15.72:0/109514312 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39d406d840 con 0x7f39d415c480 2024-04-15T15:16:57.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.185+0000 7f39d17fa640 1 -- 172.21.15.72:0/109514312 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39cc00dc50 con 0x7f39d415c480 2024-04-15T15:16:57.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.185+0000 7f39daa7e640 1 -- 172.21.15.72:0/109514312 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39d406de00 con 0x7f39d415c480 2024-04-15T15:16:57.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.186+0000 7f39daa7e640 1 -- 172.21.15.72:0/109514312 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39d40690f0 con 0x7f39d415c480 2024-04-15T15:16:57.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.187+0000 7f39d17fa640 1 -- 172.21.15.72:0/109514312 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f39cc01d070 con 0x7f39d415c480 2024-04-15T15:16:57.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.187+0000 7f39d17fa640 1 --2- 172.21.15.72:0/109514312 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f39c006c980 0x7f39c006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:57.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.187+0000 7f39d17fa640 1 -- 172.21.15.72:0/109514312 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7f39cc08c5c0 con 0x7f39d415c480 2024-04-15T15:16:57.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.187+0000 7f39d881a640 1 --2- 172.21.15.72:0/109514312 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f39c006c980 0x7f39c006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:57.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.187+0000 7f39d881a640 1 --2- 172.21.15.72:0/109514312 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f39c006c980 0x7f39c006ee40 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f39c400aae0 tx=0x7f39c4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:57.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.191+0000 7f39d17fa640 1 -- 172.21.15.72:0/109514312 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f39d41e89b0 con 0x7f39d415c480 2024-04-15T15:16:57.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.405+0000 7f39daa7e640 1 -- 172.21.15.72:0/109514312 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f39d4074e30 con 0x7f39d415c480 2024-04-15T15:16:57.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.406+0000 7f39d17fa640 1 -- 172.21.15.72:0/109514312 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+5185 (secure 0 0 0) 0x7f39cc05a270 con 0x7f39d415c480 2024-04-15T15:16:57.409 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:16:57.409 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":22,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4883,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3363272836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3363272836},{"type":"v1","addr":"172.21.15.175:6834","nonce":3363272836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5134,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6835/1059187505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":1059187505},{"type":"v1","addr":"172.21.15.175:6835","nonce":1059187505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5556,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/690132275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":690132275},{"type":"v1","addr":"172.21.15.72:6837","nonce":690132275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:16:07.050078+0000","modified":"2024-04-15T15:16:45.725971+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":6054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6054":{"gid":6054,"name":"a","rank":0,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.72:6836/3407328263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3407328263},{"type":"v1","addr":"172.21.15.72:6836","nonce":3407328263}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":6054,"qdb_cluster":[6054]},"id":2}]} 2024-04-15T15:16:57.409 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 22 2024-04-15T15:16:57.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.408+0000 7f39daa7e640 1 -- 172.21.15.72:0/109514312 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f39c006c980 msgr2=0x7f39c006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:57.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.408+0000 7f39daa7e640 1 --2- 172.21.15.72:0/109514312 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f39c006c980 0x7f39c006ee40 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f39c400aae0 tx=0x7f39c4008040 comp rx=0 tx=0).stop 2024-04-15T15:16:57.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.408+0000 7f39daa7e640 1 -- 172.21.15.72:0/109514312 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39d415c480 msgr2=0x7f39d41f4020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:57.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.408+0000 7f39daa7e640 1 --2- 172.21.15.72:0/109514312 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39d415c480 0x7f39d41f4020 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f39cc00cef0 tx=0x7f39cc00a710 comp rx=0 tx=0).stop 2024-04-15T15:16:57.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.408+0000 7f39daa7e640 1 -- 172.21.15.72:0/109514312 shutdown_connections 2024-04-15T15:16:57.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.408+0000 7f39daa7e640 1 --2- 172.21.15.72:0/109514312 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f39c006c980 0x7f39c006ee40 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.408+0000 7f39daa7e640 1 --2- 172.21.15.72:0/109514312 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39d415c480 0x7f39d41f4020 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.408+0000 7f39daa7e640 1 --2- 172.21.15.72:0/109514312 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f39d40cf300 0x7f39d41e8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.408+0000 7f39daa7e640 1 --2- 172.21.15.72:0/109514312 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f39d40ce670 0x7f39d414fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.408+0000 7f39daa7e640 1 -- 172.21.15.72:0/109514312 >> 172.21.15.72:0/109514312 conn(0x7f39d40b3fa0 msgr2=0x7f39d40b4ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:57.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.408+0000 7f39daa7e640 1 -- 172.21.15.72:0/109514312 shutdown_connections 2024-04-15T15:16:57.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.408+0000 7f39daa7e640 1 -- 172.21.15.72:0/109514312 wait complete. 2024-04-15T15:16:57.422 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:57.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.522+0000 7f3b5edbd640 1 Processor -- start 2024-04-15T15:16:57.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.522+0000 7f3b5edbd640 1 -- start start 2024-04-15T15:16:57.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.523+0000 7f3b5edbd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3b58156d30 0x7f3b580d11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:57.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.523+0000 7f3b5edbd640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3b580cd160 0x7f3b580cf360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:57.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.523+0000 7f3b5edbd640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b580d0100 con 0x7f3b580cd5a0 2024-04-15T15:16:57.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.523+0000 7f3b5edbd640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f3b580d03e0 con 0x7f3b58156d30 2024-04-15T15:16:57.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.523+0000 7f3b5edbd640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f3b580d06c0 con 0x7f3b580cd160 2024-04-15T15:16:57.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.523+0000 7f3b5cb59640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3b58156d30 0x7f3b580d11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:57.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.523+0000 7f3b5cb59640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3b58156d30 0x7f3b580d11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37772/0 (socket says 172.21.15.72:37772) 2024-04-15T15:16:57.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.523+0000 7f3b5cb59640 1 -- 172.21.15.72:0/3597776808 learned_addr learned my addr 172.21.15.72:0/3597776808 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:57.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.523+0000 7f3b4ffff640 1 --2- 172.21.15.72:0/3597776808 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3b580cd160 0x7f3b580cf360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:57.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.523+0000 7f3b5cb59640 1 -- 172.21.15.72:0/3597776808 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3b580cd160 msgr2=0x7f3b580cf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:57.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.523+0000 7f3b5cb59640 1 --2- 172.21.15.72:0/3597776808 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3b580cd160 0x7f3b580cf360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.523+0000 7f3b5cb59640 1 -- 172.21.15.72:0/3597776808 >> v1:172.21.15.175:6789/0 conn(0x7f3b580cd5a0 legacy=0x7f3b5815dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:57.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.523+0000 7f3b5cb59640 1 -- 172.21.15.72:0/3597776808 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b580d09a0 con 0x7f3b58156d30 2024-04-15T15:16:57.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.524+0000 7f3b5cb59640 1 --2- 172.21.15.72:0/3597776808 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b58156d30 0x7f3b580d11a0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f3b54009e50 tx=0x7f3b5402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d9e06230cd26135 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:57.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.524+0000 7f3b4f7fe640 1 -- 172.21.15.72:0/3597776808 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b5402cb00 con 0x7f3b58156d30 2024-04-15T15:16:57.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.524+0000 7f3b4f7fe640 1 -- 172.21.15.72:0/3597776808 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b5402cca0 con 0x7f3b58156d30 2024-04-15T15:16:57.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.524+0000 7f3b4f7fe640 1 -- 172.21.15.72:0/3597776808 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b54034610 con 0x7f3b58156d30 2024-04-15T15:16:57.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.525+0000 7f3b5edbd640 1 -- 172.21.15.72:0/3597776808 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b58156d30 msgr2=0x7f3b580d11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:57.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.525+0000 7f3b5edbd640 1 --2- 172.21.15.72:0/3597776808 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b58156d30 0x7f3b580d11a0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f3b54009e50 tx=0x7f3b5402bf60 comp rx=0 tx=0).stop 2024-04-15T15:16:57.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.525+0000 7f3b5edbd640 1 -- 172.21.15.72:0/3597776808 shutdown_connections 2024-04-15T15:16:57.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.525+0000 7f3b5edbd640 1 --2- 172.21.15.72:0/3597776808 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3b580cd160 0x7f3b580cf360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.525+0000 7f3b5edbd640 1 --2- 172.21.15.72:0/3597776808 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b58156d30 0x7f3b580d11a0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.525+0000 7f3b5edbd640 1 -- 172.21.15.72:0/3597776808 >> 172.21.15.72:0/3597776808 conn(0x7f3b580b43c0 msgr2=0x7f3b580b47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:57.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.525+0000 7f3b5edbd640 1 -- 172.21.15.72:0/3597776808 shutdown_connections 2024-04-15T15:16:57.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.525+0000 7f3b5edbd640 1 -- 172.21.15.72:0/3597776808 wait complete. 2024-04-15T15:16:57.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.526+0000 7f3b5edbd640 1 Processor -- start 2024-04-15T15:16:57.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.526+0000 7f3b5edbd640 1 -- start start 2024-04-15T15:16:57.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.527+0000 7f3b5edbd640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3b580cd160 0x7f3b581f16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:57.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.527+0000 7f3b5edbd640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3b580cd5a0 0x7f3b5806eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:57.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.527+0000 7f3b5edbd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b58156d30 0x7f3b581f4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:57.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.527+0000 7f3b5edbd640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f3b580cce90 con 0x7f3b58156d30 2024-04-15T15:16:57.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.527+0000 7f3b5edbd640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f3b580ccd10 con 0x7f3b580cd5a0 2024-04-15T15:16:57.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.527+0000 7f3b5edbd640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f3b580cf8a0 con 0x7f3b580cd160 2024-04-15T15:16:57.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.527+0000 7f3b5d35a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b58156d30 0x7f3b581f4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:57.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.527+0000 7f3b5d35a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b58156d30 0x7f3b581f4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37788/0 (socket says 172.21.15.72:37788) 2024-04-15T15:16:57.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.527+0000 7f3b5d35a640 1 -- 172.21.15.72:0/369875946 learned_addr learned my addr 172.21.15.72:0/369875946 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:57.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.527+0000 7f3b4ffff640 1 --2- 172.21.15.72:0/369875946 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3b580cd5a0 0x7f3b5806eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:57.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.527+0000 7f3b5cb59640 1 --2- 172.21.15.72:0/369875946 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3b580cd160 0x7f3b581f16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:57.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.527+0000 7f3b5d35a640 1 -- 172.21.15.72:0/369875946 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3b580cd160 msgr2=0x7f3b581f16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:57.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.527+0000 7f3b5d35a640 1 --2- 172.21.15.72:0/369875946 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3b580cd160 0x7f3b581f16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.527+0000 7f3b5d35a640 1 -- 172.21.15.72:0/369875946 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3b580cd5a0 msgr2=0x7f3b5806eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:57.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.527+0000 7f3b5d35a640 1 --2- 172.21.15.72:0/369875946 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3b580cd5a0 0x7f3b5806eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.527+0000 7f3b5d35a640 1 -- 172.21.15.72:0/369875946 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b581f4ff0 con 0x7f3b58156d30 2024-04-15T15:16:57.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.527+0000 7f3b5cb59640 1 --2- 172.21.15.72:0/369875946 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3b580cd160 0x7f3b581f16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:57.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.527+0000 7f3b5d35a640 1 --2- 172.21.15.72:0/369875946 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b58156d30 0x7f3b581f4050 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f3b5000c400 tx=0x7f3b5000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:57.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.527+0000 7f3b4d7fa640 1 -- 172.21.15.72:0/369875946 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b50017070 con 0x7f3b58156d30 2024-04-15T15:16:57.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.528+0000 7f3b4d7fa640 1 -- 172.21.15.72:0/369875946 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b500052d0 con 0x7f3b58156d30 2024-04-15T15:16:57.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.528+0000 7f3b4d7fa640 1 -- 172.21.15.72:0/369875946 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b50003810 con 0x7f3b58156d30 2024-04-15T15:16:57.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.528+0000 7f3b5edbd640 1 -- 172.21.15.72:0/369875946 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b581f4750 con 0x7f3b58156d30 2024-04-15T15:16:57.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.528+0000 7f3b5edbd640 1 -- 172.21.15.72:0/369875946 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b581f4d10 con 0x7f3b58156d30 2024-04-15T15:16:57.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.529+0000 7f3b5edbd640 1 -- 172.21.15.72:0/369875946 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b580689e0 con 0x7f3b58156d30 2024-04-15T15:16:57.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.529+0000 7f3b4d7fa640 1 -- 172.21.15.72:0/369875946 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f3b5001a070 con 0x7f3b58156d30 2024-04-15T15:16:57.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.530+0000 7f3b4d7fa640 1 --2- 172.21.15.72:0/369875946 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3b4406c980 0x7f3b4406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:57.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.530+0000 7f3b5cb59640 1 --2- 172.21.15.72:0/369875946 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3b4406c980 0x7f3b4406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:57.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.530+0000 7f3b4d7fa640 1 -- 172.21.15.72:0/369875946 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7f3b5008b020 con 0x7f3b58156d30 2024-04-15T15:16:57.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.530+0000 7f3b5cb59640 1 --2- 172.21.15.72:0/369875946 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3b4406c980 0x7f3b4406ee40 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3b54035000 tx=0x7f3b54030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:57.536 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.534+0000 7f3b4d7fa640 1 -- 172.21.15.72:0/369875946 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3b50058ca0 con 0x7f3b58156d30 2024-04-15T15:16:57.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.710+0000 7f3b5edbd640 1 -- 172.21.15.72:0/369875946 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3b581556a0 con 0x7f3b58156d30 2024-04-15T15:16:57.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.711+0000 7f3b4d7fa640 1 -- 172.21.15.72:0/369875946 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) ==== 74+0+17120 (secure 0 0 0) 0x7f3b5005cfa0 con 0x7f3b58156d30 2024-04-15T15:16:57.714 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:16:57.714 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":41,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:16:41.714841+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:05.551499+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:06.003152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.26","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:6836/4270793613":"2024-04-16T15:16:41.714817+0000","172.21.15.72:6834/4270793613":"2024-04-16T15:16:41.714817+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:57.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.713+0000 7f3b5edbd640 1 -- 172.21.15.72:0/369875946 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3b4406c980 msgr2=0x7f3b4406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:57.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.713+0000 7f3b5edbd640 1 --2- 172.21.15.72:0/369875946 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3b4406c980 0x7f3b4406ee40 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3b54035000 tx=0x7f3b54030000 comp rx=0 tx=0).stop 2024-04-15T15:16:57.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.713+0000 7f3b5edbd640 1 -- 172.21.15.72:0/369875946 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b58156d30 msgr2=0x7f3b581f4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:57.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.713+0000 7f3b5edbd640 1 --2- 172.21.15.72:0/369875946 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b58156d30 0x7f3b581f4050 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f3b5000c400 tx=0x7f3b5000c8c0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.713+0000 7f3b5edbd640 1 -- 172.21.15.72:0/369875946 shutdown_connections 2024-04-15T15:16:57.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.713+0000 7f3b5edbd640 1 --2- 172.21.15.72:0/369875946 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3b4406c980 0x7f3b4406ee40 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.713+0000 7f3b5edbd640 1 --2- 172.21.15.72:0/369875946 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3b58156d30 0x7f3b581f4050 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.713+0000 7f3b5edbd640 1 --2- 172.21.15.72:0/369875946 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3b580cd5a0 0x7f3b5806eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.713+0000 7f3b5edbd640 1 --2- 172.21.15.72:0/369875946 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3b580cd160 0x7f3b581f16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.713+0000 7f3b5edbd640 1 -- 172.21.15.72:0/369875946 >> 172.21.15.72:0/369875946 conn(0x7f3b580b43c0 msgr2=0x7f3b580d1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:57.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.714+0000 7f3b5edbd640 1 -- 172.21.15.72:0/369875946 shutdown_connections 2024-04-15T15:16:57.717 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.714+0000 7f3b5edbd640 1 -- 172.21.15.72:0/369875946 wait complete. 2024-04-15T15:16:57.728 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-15T15:16:57.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.829+0000 7faa88dfa640 1 Processor -- start 2024-04-15T15:16:57.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.829+0000 7faa88dfa640 1 -- start start 2024-04-15T15:16:57.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.829+0000 7faa88dfa640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7faa8415eac0 0x7faa840d1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:57.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.829+0000 7faa88dfa640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7faa840d0670 0x7faa840d49f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:57.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.830+0000 7faa88dfa640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa840d0de0 con 0x7faa840d4f30 2024-04-15T15:16:57.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.830+0000 7faa88dfa640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7faa840d2280 con 0x7faa8415eac0 2024-04-15T15:16:57.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.830+0000 7faa88dfa640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7faa840d24d0 con 0x7faa840d0670 2024-04-15T15:16:57.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.830+0000 7faa8259c640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7faa8415eac0 0x7faa840d1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:57.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.830+0000 7faa81d9b640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7faa840d0670 0x7faa840d49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:57.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.830+0000 7faa8259c640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7faa8415eac0 0x7faa840d1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3300/0 says I am v2:172.21.15.72:59052/0 (socket says 172.21.15.72:59052) 2024-04-15T15:16:57.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.830+0000 7faa82d9d640 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7faa840d4f30 0x7faa840b6130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.72:55446/0 (socket says 172.21.15.72:55446) 2024-04-15T15:16:57.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.830+0000 7faa81d9b640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7faa840d0670 0x7faa840d49f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.175:3301/0 says I am v2:172.21.15.72:58444/0 (socket says 172.21.15.72:58444) 2024-04-15T15:16:57.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.830+0000 7faa8259c640 1 -- 172.21.15.72:0/769547691 learned_addr learned my addr 172.21.15.72:0/769547691 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:57.833 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.830+0000 7faa81d9b640 1 -- 172.21.15.72:0/769547691 >> v2:172.21.15.175:3300/0 conn(0x7faa8415eac0 msgr2=0x7faa840d1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:57.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.830+0000 7faa81d9b640 1 --2- 172.21.15.72:0/769547691 >> v2:172.21.15.175:3300/0 conn(0x7faa8415eac0 0x7faa840d1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.830+0000 7faa81d9b640 1 -- 172.21.15.72:0/769547691 >> v1:172.21.15.175:6789/0 conn(0x7faa840d4f30 legacy=0x7faa840b6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:57.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.830+0000 7faa81d9b640 1 -- 172.21.15.72:0/769547691 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa840d27b0 con 0x7faa840d0670 2024-04-15T15:16:57.834 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.830+0000 7faa8259c640 1 --2- 172.21.15.72:0/769547691 >> v2:172.21.15.175:3300/0 conn(0x7faa8415eac0 0x7faa840d1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:57.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.831+0000 7faa81d9b640 1 --2- 172.21.15.72:0/769547691 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7faa840d0670 0x7faa840d49f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faa6c0097a0 tx=0x7faa6c02b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1c81c59ca5d0391 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:57.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.831+0000 7faa8159a640 1 -- 172.21.15.72:0/769547691 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6c034070 con 0x7faa840d0670 2024-04-15T15:16:57.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.831+0000 7faa8159a640 1 -- 172.21.15.72:0/769547691 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa6c030610 con 0x7faa840d0670 2024-04-15T15:16:57.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.831+0000 7faa8159a640 1 -- 172.21.15.72:0/769547691 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa6c0309e0 con 0x7faa840d0670 2024-04-15T15:16:57.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.832+0000 7faa88dfa640 1 -- 172.21.15.72:0/769547691 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7faa840d0670 msgr2=0x7faa840d49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:57.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.832+0000 7faa88dfa640 1 --2- 172.21.15.72:0/769547691 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7faa840d0670 0x7faa840d49f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faa6c0097a0 tx=0x7faa6c02b960 comp rx=0 tx=0).stop 2024-04-15T15:16:57.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.832+0000 7faa88dfa640 1 -- 172.21.15.72:0/769547691 shutdown_connections 2024-04-15T15:16:57.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.832+0000 7faa88dfa640 1 --2- 172.21.15.72:0/769547691 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7faa840d0670 0x7faa840d49f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.835 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.832+0000 7faa88dfa640 1 --2- 172.21.15.72:0/769547691 >> v2:172.21.15.175:3300/0 conn(0x7faa8415eac0 0x7faa840d1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.832+0000 7faa88dfa640 1 -- 172.21.15.72:0/769547691 >> 172.21.15.72:0/769547691 conn(0x7faa840b9670 msgr2=0x7faa840b9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:57.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.832+0000 7faa88dfa640 1 -- 172.21.15.72:0/769547691 shutdown_connections 2024-04-15T15:16:57.836 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.832+0000 7faa88dfa640 1 -- 172.21.15.72:0/769547691 wait complete. 2024-04-15T15:16:57.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.833+0000 7faa88dfa640 1 Processor -- start 2024-04-15T15:16:57.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.833+0000 7faa88dfa640 1 -- start start 2024-04-15T15:16:57.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.834+0000 7faa88dfa640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7faa840d0670 0x7faa840cae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:57.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.834+0000 7faa88dfa640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faa840d4f30 0x7faa840b56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:57.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.834+0000 7faa88dfa640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa840b5bf0 0x7faa8406c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:57.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.834+0000 7faa88dfa640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7faa840d3fc0 con 0x7faa840b5bf0 2024-04-15T15:16:57.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.834+0000 7faa88dfa640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7faa840d0ab0 con 0x7faa840d4f30 2024-04-15T15:16:57.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.834+0000 7faa88dfa640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7faa840b12a0 con 0x7faa840d0670 2024-04-15T15:16:57.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.834+0000 7faa82d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa840b5bf0 0x7faa8406c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:57.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.834+0000 7faa82d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa840b5bf0 0x7faa8406c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37794/0 (socket says 172.21.15.72:37794) 2024-04-15T15:16:57.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.834+0000 7faa82d9d640 1 -- 172.21.15.72:0/3154367733 learned_addr learned my addr 172.21.15.72:0/3154367733 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:57.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.834+0000 7faa8259c640 1 --2- 172.21.15.72:0/3154367733 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7faa840d0670 0x7faa840cae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:57.838 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.834+0000 7faa81d9b640 1 --2- 172.21.15.72:0/3154367733 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faa840d4f30 0x7faa840b56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:57.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.834+0000 7faa82d9d640 1 -- 172.21.15.72:0/3154367733 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7faa840d0670 msgr2=0x7faa840cae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:57.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.834+0000 7faa82d9d640 1 --2- 172.21.15.72:0/3154367733 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7faa840d0670 0x7faa840cae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.834+0000 7faa82d9d640 1 -- 172.21.15.72:0/3154367733 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faa840d4f30 msgr2=0x7faa840b56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:57.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.834+0000 7faa82d9d640 1 --2- 172.21.15.72:0/3154367733 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faa840d4f30 0x7faa840b56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:57.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.834+0000 7faa82d9d640 1 -- 172.21.15.72:0/3154367733 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa840cb360 con 0x7faa840b5bf0 2024-04-15T15:16:57.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.835+0000 7faa82d9d640 1 --2- 172.21.15.72:0/3154367733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa840b5bf0 0x7faa8406c250 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7faa7400c400 tx=0x7faa7400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:57.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.835+0000 7faa7affd640 1 -- 172.21.15.72:0/3154367733 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa740074e0 con 0x7faa840b5bf0 2024-04-15T15:16:57.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.835+0000 7faa7affd640 1 -- 172.21.15.72:0/3154367733 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa74007680 con 0x7faa840b5bf0 2024-04-15T15:16:57.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.835+0000 7faa7affd640 1 -- 172.21.15.72:0/3154367733 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa740049e0 con 0x7faa840b5bf0 2024-04-15T15:16:57.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.835+0000 7faa88dfa640 1 -- 172.21.15.72:0/3154367733 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa840c8280 con 0x7faa840b5bf0 2024-04-15T15:16:57.839 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.835+0000 7faa88dfa640 1 -- 172.21.15.72:0/3154367733 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa840c8800 con 0x7faa840b5bf0 2024-04-15T15:16:57.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.836+0000 7faa88dfa640 1 -- 172.21.15.72:0/3154367733 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa84075c50 con 0x7faa840b5bf0 2024-04-15T15:16:57.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.836+0000 7faa7affd640 1 -- 172.21.15.72:0/3154367733 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7faa7401a020 con 0x7faa840b5bf0 2024-04-15T15:16:57.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.837+0000 7faa7affd640 1 --2- 172.21.15.72:0/3154367733 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7faa6406c9b0 0x7faa6406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:57.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.837+0000 7faa7affd640 1 -- 172.21.15.72:0/3154367733 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6216+0+0 (secure 0 0 0) 0x7faa7408b310 con 0x7faa840b5bf0 2024-04-15T15:16:57.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.837+0000 7faa8259c640 1 --2- 172.21.15.72:0/3154367733 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7faa6406c9b0 0x7faa6406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:57.840 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.837+0000 7faa8259c640 1 --2- 172.21.15.72:0/3154367733 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7faa6406c9b0 0x7faa6406ee70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7faa840b62f0 tx=0x7faa7c00b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:57.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:57.841+0000 7faa7affd640 1 -- 172.21.15.72:0/3154367733 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7faa74058f90 con 0x7faa840b5bf0 2024-04-15T15:16:58.056 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.053+0000 7faa88dfa640 1 -- 172.21.15.72:0/3154367733 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7faa840d0fb0 con 0x7faa840b5bf0 2024-04-15T15:16:58.056 INFO:tasks.ceph.mon.a.smithi072.stderr:2024-04-15T15:16:58.054+0000 7f01d0611640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-15T15:16:58.057 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.055+0000 7faa7affd640 1 -- 172.21.15.72:0/3154367733 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v23) ==== 161+0+0 (secure 0 0 0) 0x7faa7405d290 con 0x7faa840b5bf0 2024-04-15T15:16:58.057 INFO:teuthology.orchestra.run.smithi072.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-15T15:16:58.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.057+0000 7faa88dfa640 1 -- 172.21.15.72:0/3154367733 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7faa6406c9b0 msgr2=0x7faa6406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.057+0000 7faa88dfa640 1 --2- 172.21.15.72:0/3154367733 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7faa6406c9b0 0x7faa6406ee70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7faa840b62f0 tx=0x7faa7c00b040 comp rx=0 tx=0).stop 2024-04-15T15:16:58.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.058+0000 7faa88dfa640 1 -- 172.21.15.72:0/3154367733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa840b5bf0 msgr2=0x7faa8406c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.058+0000 7faa88dfa640 1 --2- 172.21.15.72:0/3154367733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa840b5bf0 0x7faa8406c250 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7faa7400c400 tx=0x7faa7400c8c0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.060 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.058+0000 7faa88dfa640 1 -- 172.21.15.72:0/3154367733 shutdown_connections 2024-04-15T15:16:58.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.058+0000 7faa88dfa640 1 --2- 172.21.15.72:0/3154367733 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7faa6406c9b0 0x7faa6406ee70 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.058+0000 7faa88dfa640 1 --2- 172.21.15.72:0/3154367733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa840b5bf0 0x7faa8406c250 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.058+0000 7faa88dfa640 1 --2- 172.21.15.72:0/3154367733 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7faa840d4f30 0x7faa840b56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.058+0000 7faa88dfa640 1 --2- 172.21.15.72:0/3154367733 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7faa840d0670 0x7faa840cae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.058+0000 7faa88dfa640 1 -- 172.21.15.72:0/3154367733 >> 172.21.15.72:0/3154367733 conn(0x7faa840b9670 msgr2=0x7faa8415ef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:58.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.058+0000 7faa88dfa640 1 -- 172.21.15.72:0/3154367733 shutdown_connections 2024-04-15T15:16:58.061 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.058+0000 7faa88dfa640 1 -- 172.21.15.72:0/3154367733 wait complete. 2024-04-15T15:16:58.064 INFO:tasks.ceph.mds.a.smithi072.stderr: -1349> 2024-04-15T15:16:47.727+0000 7f68bd49d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-15T15:16:58.072 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-15T15:16:58.092 INFO:tasks.ceph.mds.a.smithi072.stdout:starting mds.a at 2024-04-15T15:16:58.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.175+0000 7f150b596640 1 Processor -- start 2024-04-15T15:16:58.178 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.176+0000 7f150b596640 1 -- start start 2024-04-15T15:16:58.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.176+0000 7f150b596640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f150415e610 0x7f15040d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.176+0000 7f150b596640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f150406c400 0x7f150406c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.176+0000 7f150b596640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15040d2320 con 0x7f150415e9f0 2024-04-15T15:16:58.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.176+0000 7f150b596640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f15040d31d0 con 0x7f150406c400 2024-04-15T15:16:58.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.176+0000 7f150b596640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f15040d25c0 con 0x7f150415e610 2024-04-15T15:16:58.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.177+0000 7f1509332640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f150415e610 0x7f15040d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.177+0000 7f1509b33640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f150406c400 0x7f150406c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.177+0000 7f1508b31640 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f150415e9f0 0x7f15040d3660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.72:55466/0 (socket says 172.21.15.72:55466) 2024-04-15T15:16:58.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.177+0000 7f1508b31640 1 -- 172.21.15.72:0/1642991451 learned_addr learned my addr 172.21.15.72:0/1642991451 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:58.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.177+0000 7f14fbfff640 1 -- 172.21.15.72:0/1642991451 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2847711151 0 0) 0x7f15040d2320 con 0x7f150415e9f0 2024-04-15T15:16:58.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.177+0000 7f14fbfff640 1 -- 172.21.15.72:0/1642991451 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f14f0003520 con 0x7f150415e9f0 2024-04-15T15:16:58.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.177+0000 7f1509b33640 1 -- 172.21.15.72:0/1642991451 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f150415e610 msgr2=0x7f15040d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:58.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.177+0000 7f1509b33640 1 --2- 172.21.15.72:0/1642991451 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f150415e610 0x7f15040d0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.177+0000 7f1509b33640 1 -- 172.21.15.72:0/1642991451 >> v1:172.21.15.175:6789/0 conn(0x7f150415e9f0 legacy=0x7f15040d3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.177+0000 7f1509b33640 1 -- 172.21.15.72:0/1642991451 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15040d28a0 con 0x7f150406c400 2024-04-15T15:16:58.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.177+0000 7f1509332640 1 --2- 172.21.15.72:0/1642991451 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f150415e610 0x7f15040d0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:58.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.178+0000 7f1509b33640 1 --2- 172.21.15.72:0/1642991451 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f150406c400 0x7f150406c890 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f14fc009d90 tx=0x7f14fc02bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=583d1167c190a12c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:58.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.178+0000 7f14fbfff640 1 -- 172.21.15.72:0/1642991451 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14fc039070 con 0x7f150406c400 2024-04-15T15:16:58.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.178+0000 7f14fbfff640 1 -- 172.21.15.72:0/1642991451 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14fc02cce0 con 0x7f150406c400 2024-04-15T15:16:58.182 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.178+0000 7f14fbfff640 1 -- 172.21.15.72:0/1642991451 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14fc034640 con 0x7f150406c400 2024-04-15T15:16:58.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.179+0000 7f150b596640 1 -- 172.21.15.72:0/1642991451 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f150406c400 msgr2=0x7f150406c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.179+0000 7f150b596640 1 --2- 172.21.15.72:0/1642991451 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f150406c400 0x7f150406c890 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f14fc009d90 tx=0x7f14fc02bf30 comp rx=0 tx=0).stop 2024-04-15T15:16:58.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.179+0000 7f150b596640 1 -- 172.21.15.72:0/1642991451 shutdown_connections 2024-04-15T15:16:58.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.179+0000 7f150b596640 1 --2- 172.21.15.72:0/1642991451 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f150406c400 0x7f150406c890 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.179+0000 7f150b596640 1 --2- 172.21.15.72:0/1642991451 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f150415e610 0x7f15040d0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.179+0000 7f150b596640 1 -- 172.21.15.72:0/1642991451 >> 172.21.15.72:0/1642991451 conn(0x7f15040d4750 msgr2=0x7f15040d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:58.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.179+0000 7f150b596640 1 -- 172.21.15.72:0/1642991451 shutdown_connections 2024-04-15T15:16:58.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.179+0000 7f150b596640 1 -- 172.21.15.72:0/1642991451 wait complete. 2024-04-15T15:16:58.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.180+0000 7f150b596640 1 Processor -- start 2024-04-15T15:16:58.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.180+0000 7f150b596640 1 -- start start 2024-04-15T15:16:58.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.181+0000 7f150b596640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f150406c400 0x7f150415f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.181+0000 7f150b596640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f150415e610 0x7f150414ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.181+0000 7f150b596640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f150415e9f0 0x7f15040cc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.181+0000 7f150b596640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f150406d130 con 0x7f150415e610 2024-04-15T15:16:58.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.181+0000 7f150b596640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f15040d2e60 con 0x7f150415e9f0 2024-04-15T15:16:58.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.181+0000 7f150b596640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f15040d0bd0 con 0x7f150406c400 2024-04-15T15:16:58.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.181+0000 7f1508b31640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f150415e610 0x7f150414ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.181+0000 7f1508b31640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f150415e610 0x7f150414ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37814/0 (socket says 172.21.15.72:37814) 2024-04-15T15:16:58.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.181+0000 7f1508b31640 1 -- 172.21.15.72:0/4023677755 learned_addr learned my addr 172.21.15.72:0/4023677755 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:58.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.181+0000 7f1509b33640 1 --2- 172.21.15.72:0/4023677755 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f150415e9f0 0x7f15040cc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.181+0000 7f1509332640 1 --2- 172.21.15.72:0/4023677755 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f150406c400 0x7f150415f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.181+0000 7f1508b31640 1 -- 172.21.15.72:0/4023677755 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f150406c400 msgr2=0x7f150415f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.181+0000 7f1508b31640 1 --2- 172.21.15.72:0/4023677755 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f150406c400 0x7f150415f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.181+0000 7f1508b31640 1 -- 172.21.15.72:0/4023677755 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f150415e9f0 msgr2=0x7f15040cc180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.181+0000 7f1508b31640 1 --2- 172.21.15.72:0/4023677755 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f150415e9f0 0x7f15040cc180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.186 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.181+0000 7f1508b31640 1 -- 172.21.15.72:0/4023677755 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f150415ff30 con 0x7f150415e610 2024-04-15T15:16:58.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.181+0000 7f1508b31640 1 --2- 172.21.15.72:0/4023677755 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f150415e610 0x7f150414ce90 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f14f400c390 tx=0x7f14f400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:58.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.181+0000 7f14f9ffb640 1 -- 172.21.15.72:0/4023677755 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14f4017070 con 0x7f150415e610 2024-04-15T15:16:58.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.181+0000 7f14f9ffb640 1 -- 172.21.15.72:0/4023677755 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14f4005ce0 con 0x7f150415e610 2024-04-15T15:16:58.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.181+0000 7f14f9ffb640 1 -- 172.21.15.72:0/4023677755 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14f40072f0 con 0x7f150415e610 2024-04-15T15:16:58.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.181+0000 7f150b596640 1 -- 172.21.15.72:0/4023677755 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1504160210 con 0x7f150415e610 2024-04-15T15:16:58.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.182+0000 7f150b596640 1 -- 172.21.15.72:0/4023677755 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15041607d0 con 0x7f150415e610 2024-04-15T15:16:58.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.182+0000 7f150b596640 1 -- 172.21.15.72:0/4023677755 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1504075c50 con 0x7f150415e610 2024-04-15T15:16:58.187 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.183+0000 7f14f9ffb640 1 -- 172.21.15.72:0/4023677755 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f14f401a070 con 0x7f150415e610 2024-04-15T15:16:58.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.183+0000 7f14f9ffb640 1 --2- 172.21.15.72:0/4023677755 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f14e806c980 0x7f14e806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.183+0000 7f14f9ffb640 1 -- 172.21.15.72:0/4023677755 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6302+0+0 (secure 0 0 0) 0x7f14f408b0d0 con 0x7f150415e610 2024-04-15T15:16:58.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.183+0000 7f1509332640 1 --2- 172.21.15.72:0/4023677755 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f14e806c980 0x7f14e806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.184+0000 7f1509332640 1 --2- 172.21.15.72:0/4023677755 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f14e806c980 0x7f14e806ee40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1500009560 tx=0x7f1500009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:58.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.188+0000 7f14f9ffb640 1 -- 172.21.15.72:0/4023677755 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f14f4058d00 con 0x7f150415e610 2024-04-15T15:16:58.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:58.398+0000 7f150b596640 1 -- 172.21.15.72:0/4023677755 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f15040cf740 con 0x7f150415e610 2024-04-15T15:16:59.059 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.057+0000 7f14f9ffb640 1 -- 172.21.15.72:0/4023677755 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v24) ==== 104+0+0 (secure 0 0 0) 0x7f14f405d000 con 0x7f150415e610 2024-04-15T15:16:59.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.059+0000 7f150b596640 1 -- 172.21.15.72:0/4023677755 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f14e806c980 msgr2=0x7f14e806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:59.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.059+0000 7f150b596640 1 --2- 172.21.15.72:0/4023677755 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f14e806c980 0x7f14e806ee40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f1500009560 tx=0x7f1500009190 comp rx=0 tx=0).stop 2024-04-15T15:16:59.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.059+0000 7f150b596640 1 -- 172.21.15.72:0/4023677755 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f150415e610 msgr2=0x7f150414ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:59.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.059+0000 7f150b596640 1 --2- 172.21.15.72:0/4023677755 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f150415e610 0x7f150414ce90 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f14f400c390 tx=0x7f14f400c850 comp rx=0 tx=0).stop 2024-04-15T15:16:59.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.059+0000 7f150b596640 1 -- 172.21.15.72:0/4023677755 shutdown_connections 2024-04-15T15:16:59.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.059+0000 7f150b596640 1 --2- 172.21.15.72:0/4023677755 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f14e806c980 0x7f14e806ee40 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.062 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.059+0000 7f150b596640 1 --2- 172.21.15.72:0/4023677755 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f150415e9f0 0x7f15040cc180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.059+0000 7f150b596640 1 --2- 172.21.15.72:0/4023677755 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f150415e610 0x7f150414ce90 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.059+0000 7f150b596640 1 --2- 172.21.15.72:0/4023677755 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f150406c400 0x7f150415f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.059+0000 7f150b596640 1 -- 172.21.15.72:0/4023677755 >> 172.21.15.72:0/4023677755 conn(0x7f15040d4750 msgr2=0x7f15040d3660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:59.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.059+0000 7f150b596640 1 -- 172.21.15.72:0/4023677755 shutdown_connections 2024-04-15T15:16:59.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.059+0000 7f150b596640 1 -- 172.21.15.72:0/4023677755 wait complete. 2024-04-15T15:16:59.075 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-15T15:16:59.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.223+0000 7f448f752640 1 Processor -- start 2024-04-15T15:16:59.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.223+0000 7f448f752640 1 -- start start 2024-04-15T15:16:59.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.224+0000 7f448f752640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f44880cd2b0 0x7f44880cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:59.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.224+0000 7f448f752640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f448806ba00 con 0x7f4488159130 2024-04-15T15:16:59.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.224+0000 7f448f752640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f448806bce0 con 0x7f44880ce320 2024-04-15T15:16:59.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.224+0000 7f448f752640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f448806c610 con 0x7f44880cd2b0 2024-04-15T15:16:59.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.224+0000 7f448d4ee640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f4488159130 0x7f44880ccba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60124/0 (socket says 172.21.15.72:60124) 2024-04-15T15:16:59.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.224+0000 7f448d4ee640 1 -- 172.21.15.72:0/2611829585 learned_addr learned my addr 172.21.15.72:0/2611829585 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:16:59.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.225+0000 7f447ffff640 1 -- 172.21.15.72:0/2611829585 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 918693783 0 0) 0x7f448806ba00 con 0x7f4488159130 2024-04-15T15:16:59.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.225+0000 7f447ffff640 1 -- 172.21.15.72:0/2611829585 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4474003520 con 0x7f4488159130 2024-04-15T15:16:59.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.225+0000 7f448cced640 1 --2- 172.21.15.72:0/2611829585 >> v2:172.21.15.175:3300/0 conn(0x7f44880cd2b0 0x7f44880cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:59.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.225+0000 7f447ffff640 1 -- 172.21.15.72:0/2611829585 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2365502262 0 0) 0x7f4474003520 con 0x7f4488159130 2024-04-15T15:16:59.230 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.225+0000 7f447ffff640 1 -- 172.21.15.72:0/2611829585 >> v2:172.21.15.175:3300/0 conn(0x7f44880cd2b0 msgr2=0x7f44880cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:59.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.225+0000 7f447ffff640 1 --2- 172.21.15.72:0/2611829585 >> v2:172.21.15.175:3300/0 conn(0x7f44880cd2b0 0x7f44880cdde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.225+0000 7f447ffff640 1 -- 172.21.15.72:0/2611829585 >> v1:172.21.15.175:6789/0 conn(0x7f44880ce320 legacy=0x7f448806dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:59.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.225+0000 7f447ffff640 1 -- 172.21.15.72:0/2611829585 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f448806bfc0 con 0x7f4488159130 2024-04-15T15:16:59.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.225+0000 7f447ffff640 1 -- 172.21.15.72:0/2611829585 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f44840019f0 con 0x7f4488159130 2024-04-15T15:16:59.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.225+0000 7f447ffff640 1 -- 172.21.15.72:0/2611829585 >> v1:172.21.15.72:6789/0 conn(0x7f4488159130 legacy=0x7f44880ccba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:59.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.225+0000 7f447ffff640 1 --2- 172.21.15.72:0/2611829585 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f44740039b0 0x7f4474003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:59.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.225+0000 7f447ffff640 1 --2- 172.21.15.72:0/2611829585 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4474004800 0x7f4474004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:59.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.225+0000 7f447ffff640 1 --2- 172.21.15.72:0/2611829585 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44740051c0 0x7f447400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:59.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.225+0000 7f447ffff640 1 -- 172.21.15.72:0/2611829585 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f448806c610 con 0x7f44740051c0 2024-04-15T15:16:59.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.225+0000 7f447ffff640 1 -- 172.21.15.72:0/2611829585 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f447400c0f0 con 0x7f44740039b0 2024-04-15T15:16:59.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.225+0000 7f447ffff640 1 -- 172.21.15.72:0/2611829585 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f447400c2f0 con 0x7f4474004800 2024-04-15T15:16:59.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.225+0000 7f448dcef640 1 --2- 172.21.15.72:0/2611829585 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44740051c0 0x7f447400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:59.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.225+0000 7f448d4ee640 1 --2- 172.21.15.72:0/2611829585 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f44740039b0 0x7f4474003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:59.231 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.225+0000 7f448cced640 1 --2- 172.21.15.72:0/2611829585 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4474004800 0x7f4474004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:59.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.225+0000 7f448dcef640 1 -- 172.21.15.72:0/2611829585 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4474004800 msgr2=0x7f4474004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:59.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.225+0000 7f448dcef640 1 --2- 172.21.15.72:0/2611829585 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4474004800 0x7f4474004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.225+0000 7f448dcef640 1 -- 172.21.15.72:0/2611829585 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f44740039b0 msgr2=0x7f4474003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:59.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.225+0000 7f448dcef640 1 --2- 172.21.15.72:0/2611829585 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f44740039b0 0x7f4474003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.225+0000 7f448dcef640 1 -- 172.21.15.72:0/2611829585 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4474003520 con 0x7f44740051c0 2024-04-15T15:16:59.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.226+0000 7f448dcef640 1 -- 172.21.15.72:0/2611829585 reap_dead start 2024-04-15T15:16:59.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.226+0000 7f448dcef640 1 --2- 172.21.15.72:0/2611829585 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44740051c0 0x7f447400b9c0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f4480003d80 tx=0x7f4474005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac9797d37685dbdd server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:59.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.226+0000 7f447ffff640 1 -- 172.21.15.72:0/2611829585 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4480003980 con 0x7f44740051c0 2024-04-15T15:16:59.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.226+0000 7f447ffff640 1 -- 172.21.15.72:0/2611829585 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f448002be50 con 0x7f44740051c0 2024-04-15T15:16:59.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.226+0000 7f447ffff640 1 -- 172.21.15.72:0/2611829585 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4480007480 con 0x7f44740051c0 2024-04-15T15:16:59.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.227+0000 7f448f752640 1 -- 172.21.15.72:0/2611829585 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44740051c0 msgr2=0x7f447400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:59.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.227+0000 7f448f752640 1 --2- 172.21.15.72:0/2611829585 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44740051c0 0x7f447400b9c0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f4480003d80 tx=0x7f4474005830 comp rx=0 tx=0).stop 2024-04-15T15:16:59.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.227+0000 7f448f752640 1 -- 172.21.15.72:0/2611829585 shutdown_connections 2024-04-15T15:16:59.232 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.227+0000 7f448f752640 1 --2- 172.21.15.72:0/2611829585 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44740051c0 0x7f447400b9c0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.227+0000 7f448f752640 1 -- 172.21.15.72:0/2611829585 >> 172.21.15.72:0/2611829585 conn(0x7f44880b1ee0 msgr2=0x7f44880b22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:59.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.227+0000 7f448f752640 1 -- 172.21.15.72:0/2611829585 shutdown_connections 2024-04-15T15:16:59.233 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.227+0000 7f448f752640 1 -- 172.21.15.72:0/2611829585 wait complete. 2024-04-15T15:16:59.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.228+0000 7f448f752640 1 Processor -- start 2024-04-15T15:16:59.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.228+0000 7f448f752640 1 -- start start 2024-04-15T15:16:59.235 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.229+0000 7f448f752640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f44740051c0 0x7f44880999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:59.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.229+0000 7f448f752640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44881577e0 0x7f4488155bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:59.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.229+0000 7f448f752640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4488156130 0x7f44880cb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:59.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.229+0000 7f448f752640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f44880ce910 con 0x7f44881577e0 2024-04-15T15:16:59.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.229+0000 7f448f752640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f44880ce790 con 0x7f44740051c0 2024-04-15T15:16:59.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.229+0000 7f448f752640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f44880cdde0 con 0x7f4488156130 2024-04-15T15:16:59.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.229+0000 7f448cced640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44881577e0 0x7f4488155bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:59.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.229+0000 7f448cced640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44881577e0 0x7f4488155bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37826/0 (socket says 172.21.15.72:37826) 2024-04-15T15:16:59.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.229+0000 7f448cced640 1 -- 172.21.15.72:0/1628576643 learned_addr learned my addr 172.21.15.72:0/1628576643 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:16:59.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.229+0000 7f448dcef640 1 --2- 172.21.15.72:0/1628576643 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4488156130 0x7f44880cb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:59.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.229+0000 7f448d4ee640 1 --2- 172.21.15.72:0/1628576643 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f44740051c0 0x7f44880999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:59.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.229+0000 7f448cced640 1 -- 172.21.15.72:0/1628576643 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4488156130 msgr2=0x7f44880cb890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:59.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.229+0000 7f448cced640 1 --2- 172.21.15.72:0/1628576643 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4488156130 0x7f44880cb890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.229+0000 7f448cced640 1 -- 172.21.15.72:0/1628576643 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f44740051c0 msgr2=0x7f44880999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:59.236 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.229+0000 7f448cced640 1 --2- 172.21.15.72:0/1628576643 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f44740051c0 0x7f44880999a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.229+0000 7f448cced640 1 -- 172.21.15.72:0/1628576643 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4488099ee0 con 0x7f44881577e0 2024-04-15T15:16:59.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.229+0000 7f448cced640 1 --2- 172.21.15.72:0/1628576643 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44881577e0 0x7f4488155bf0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f447800b5c0 tx=0x7f447800ba80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:59.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.229+0000 7f447dffb640 1 -- 172.21.15.72:0/1628576643 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4478018070 con 0x7f44881577e0 2024-04-15T15:16:59.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.230+0000 7f447dffb640 1 -- 172.21.15.72:0/1628576643 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44780106a0 con 0x7f44881577e0 2024-04-15T15:16:59.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.230+0000 7f448f752640 1 -- 172.21.15.72:0/1628576643 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4488157da0 con 0x7f44881577e0 2024-04-15T15:16:59.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.230+0000 7f447dffb640 1 -- 172.21.15.72:0/1628576643 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4478010a70 con 0x7f44881577e0 2024-04-15T15:16:59.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.230+0000 7f448f752640 1 -- 172.21.15.72:0/1628576643 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4488158360 con 0x7f44881577e0 2024-04-15T15:16:59.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.231+0000 7f448f752640 1 -- 172.21.15.72:0/1628576643 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44880689e0 con 0x7f44881577e0 2024-04-15T15:16:59.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.231+0000 7f447dffb640 1 -- 172.21.15.72:0/1628576643 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f447801b070 con 0x7f44881577e0 2024-04-15T15:16:59.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.232+0000 7f447dffb640 1 --2- 172.21.15.72:0/1628576643 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f446c06c980 0x7f446c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:59.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.232+0000 7f447dffb640 1 -- 172.21.15.72:0/1628576643 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6302+0+0 (secure 0 0 0) 0x7f447808c0a0 con 0x7f44881577e0 2024-04-15T15:16:59.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.232+0000 7f448d4ee640 1 --2- 172.21.15.72:0/1628576643 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f446c06c980 0x7f446c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:59.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.232+0000 7f448d4ee640 1 --2- 172.21.15.72:0/1628576643 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f446c06c980 0x7f446c06ee40 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f44840013d0 tx=0x7f4484006040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:59.238 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.236+0000 7f447dffb640 1 -- 172.21.15.72:0/1628576643 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4488099ee0 con 0x7f44881577e0 2024-04-15T15:16:59.412 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:16:59.409+0000 7f448f752640 1 -- 172.21.15.72:0/1628576643 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f448806cde0 con 0x7f44881577e0 2024-04-15T15:17:00.063 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.060+0000 7f447dffb640 1 -- 172.21.15.72:0/1628576643 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v43) ==== 181+0+0 (secure 0 0 0) 0x7f447805a290 con 0x7f44881577e0 2024-04-15T15:17:00.063 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_metadata' removed 2024-04-15T15:17:00.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.064+0000 7f448f752640 1 -- 172.21.15.72:0/1628576643 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f446c06c980 msgr2=0x7f446c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:00.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.064+0000 7f448f752640 1 --2- 172.21.15.72:0/1628576643 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f446c06c980 0x7f446c06ee40 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f44840013d0 tx=0x7f4484006040 comp rx=0 tx=0).stop 2024-04-15T15:17:00.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.064+0000 7f448f752640 1 -- 172.21.15.72:0/1628576643 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44881577e0 msgr2=0x7f4488155bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:00.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.064+0000 7f448f752640 1 --2- 172.21.15.72:0/1628576643 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44881577e0 0x7f4488155bf0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f447800b5c0 tx=0x7f447800ba80 comp rx=0 tx=0).stop 2024-04-15T15:17:00.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.064+0000 7f448f752640 1 -- 172.21.15.72:0/1628576643 shutdown_connections 2024-04-15T15:17:00.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.064+0000 7f448f752640 1 --2- 172.21.15.72:0/1628576643 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f446c06c980 0x7f446c06ee40 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.064+0000 7f448f752640 1 --2- 172.21.15.72:0/1628576643 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f4488156130 0x7f44880cb890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.064+0000 7f448f752640 1 --2- 172.21.15.72:0/1628576643 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f44881577e0 0x7f4488155bf0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.064+0000 7f448f752640 1 --2- 172.21.15.72:0/1628576643 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f44740051c0 0x7f44880999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.064+0000 7f448f752640 1 -- 172.21.15.72:0/1628576643 >> 172.21.15.72:0/1628576643 conn(0x7f44880b1ee0 msgr2=0x7f448806f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:00.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.064+0000 7f448f752640 1 -- 172.21.15.72:0/1628576643 shutdown_connections 2024-04-15T15:17:00.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.064+0000 7f448f752640 1 -- 172.21.15.72:0/1628576643 wait complete. 2024-04-15T15:17:00.080 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-15T15:17:00.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.186+0000 7f5147e88640 1 Processor -- start 2024-04-15T15:17:00.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.186+0000 7f5147e88640 1 -- start start 2024-04-15T15:17:00.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.187+0000 7f5147e88640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f51400ce4f0 0x7f514015bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:00.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.187+0000 7f5147e88640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f514015fe60 con 0x7f514015c0f0 2024-04-15T15:17:00.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.187+0000 7f5147e88640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f514015c8e0 con 0x7f514006f6c0 2024-04-15T15:17:00.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.187+0000 7f5147e88640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f514015f0a0 con 0x7f51400ce4f0 2024-04-15T15:17:00.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.187+0000 7f5146425640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f514015c0f0 0x7f514006dab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60130/0 (socket says 172.21.15.72:60130) 2024-04-15T15:17:00.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.187+0000 7f5146425640 1 -- 172.21.15.72:0/4075055141 learned_addr learned my addr 172.21.15.72:0/4075055141 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:17:00.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.187+0000 7f5145423640 1 --2- 172.21.15.72:0/4075055141 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f51400ce4f0 0x7f514015bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:00.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.187+0000 7f5144c22640 1 -- 172.21.15.72:0/4075055141 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3480998951 0 0) 0x7f514015fe60 con 0x7f514015c0f0 2024-04-15T15:17:00.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.187+0000 7f5144c22640 1 -- 172.21.15.72:0/4075055141 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f512c003520 con 0x7f514015c0f0 2024-04-15T15:17:00.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.187+0000 7f5144c22640 1 -- 172.21.15.72:0/4075055141 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3616186981 0 0) 0x7f512c003520 con 0x7f514015c0f0 2024-04-15T15:17:00.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.187+0000 7f5144c22640 1 -- 172.21.15.72:0/4075055141 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f51400ce4f0 msgr2=0x7f514015bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:00.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.187+0000 7f5144c22640 1 --2- 172.21.15.72:0/4075055141 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f51400ce4f0 0x7f514015bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.187+0000 7f5144c22640 1 -- 172.21.15.72:0/4075055141 >> v1:172.21.15.175:6789/0 conn(0x7f514006f6c0 legacy=0x7f51400cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:00.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.187+0000 7f5144c22640 1 -- 172.21.15.72:0/4075055141 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f514015f380 con 0x7f514015c0f0 2024-04-15T15:17:00.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.187+0000 7f5144c22640 1 -- 172.21.15.72:0/4075055141 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f51380019f0 con 0x7f514015c0f0 2024-04-15T15:17:00.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.187+0000 7f5144c22640 1 -- 172.21.15.72:0/4075055141 >> v1:172.21.15.72:6789/0 conn(0x7f514015c0f0 legacy=0x7f514006dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:00.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.187+0000 7f5144c22640 1 --2- 172.21.15.72:0/4075055141 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f512c0039e0 0x7f512c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:00.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.187+0000 7f5144c22640 1 --2- 172.21.15.72:0/4075055141 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f51400ce4f0 0x7f512c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:00.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.187+0000 7f5144c22640 1 --2- 172.21.15.72:0/4075055141 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f512c004d40 0x7f512c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:00.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.187+0000 7f5144c22640 1 -- 172.21.15.72:0/4075055141 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f514015f0a0 con 0x7f512c004d40 2024-04-15T15:17:00.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.187+0000 7f5144c22640 1 -- 172.21.15.72:0/4075055141 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f512c00bb50 con 0x7f512c0039e0 2024-04-15T15:17:00.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.187+0000 7f5144c22640 1 -- 172.21.15.72:0/4075055141 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f512c00bd20 con 0x7f51400ce4f0 2024-04-15T15:17:00.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.188+0000 7f5146425640 1 --2- 172.21.15.72:0/4075055141 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f512c004d40 0x7f512c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:00.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.188+0000 7f5145c24640 1 --2- 172.21.15.72:0/4075055141 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f512c0039e0 0x7f512c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:00.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.188+0000 7f5145423640 1 --2- 172.21.15.72:0/4075055141 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f51400ce4f0 0x7f512c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:00.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.188+0000 7f5146425640 1 -- 172.21.15.72:0/4075055141 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f51400ce4f0 msgr2=0x7f512c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:00.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.188+0000 7f5146425640 1 --2- 172.21.15.72:0/4075055141 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f51400ce4f0 0x7f512c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.188+0000 7f5146425640 1 -- 172.21.15.72:0/4075055141 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f512c0039e0 msgr2=0x7f512c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:00.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.188+0000 7f5146425640 1 --2- 172.21.15.72:0/4075055141 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f512c0039e0 0x7f512c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.188+0000 7f5146425640 1 -- 172.21.15.72:0/4075055141 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f512c003520 con 0x7f512c004d40 2024-04-15T15:17:00.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.188+0000 7f5145c24640 1 --2- 172.21.15.72:0/4075055141 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f512c0039e0 0x7f512c003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:00.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.188+0000 7f5145423640 1 --2- 172.21.15.72:0/4075055141 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f51400ce4f0 0x7f512c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:00.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.189+0000 7f5146425640 1 --2- 172.21.15.72:0/4075055141 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f512c004d40 0x7f512c0051c0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f513800a460 tx=0x7f512c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b991c3fd3b84f0e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:00.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.189+0000 7f5144c22640 1 -- 172.21.15.72:0/4075055141 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513803a070 con 0x7f512c004d40 2024-04-15T15:17:00.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.189+0000 7f5144c22640 1 -- 172.21.15.72:0/4075055141 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5138035620 con 0x7f512c004d40 2024-04-15T15:17:00.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.189+0000 7f5144c22640 1 -- 172.21.15.72:0/4075055141 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513800a4a0 con 0x7f512c004d40 2024-04-15T15:17:00.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.189+0000 7f5147e88640 1 -- 172.21.15.72:0/4075055141 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f512c004d40 msgr2=0x7f512c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:00.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.189+0000 7f5147e88640 1 --2- 172.21.15.72:0/4075055141 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f512c004d40 0x7f512c0051c0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f513800a460 tx=0x7f512c005830 comp rx=0 tx=0).stop 2024-04-15T15:17:00.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.189+0000 7f5146425640 1 -- 172.21.15.72:0/4075055141 reap_dead start 2024-04-15T15:17:00.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.189+0000 7f5147e88640 1 -- 172.21.15.72:0/4075055141 shutdown_connections 2024-04-15T15:17:00.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.189+0000 7f5147e88640 1 -- 172.21.15.72:0/4075055141 >> 172.21.15.72:0/4075055141 conn(0x7f51400b1ee0 msgr2=0x7f51400b22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:00.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.189+0000 7f5147e88640 1 -- 172.21.15.72:0/4075055141 shutdown_connections 2024-04-15T15:17:00.196 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.189+0000 7f5147e88640 1 -- 172.21.15.72:0/4075055141 wait complete. 2024-04-15T15:17:00.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.190+0000 7f5147e88640 1 Processor -- start 2024-04-15T15:17:00.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.190+0000 7f5147e88640 1 -- start start 2024-04-15T15:17:00.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.191+0000 7f5147e88640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f512c004d40 0x7f514009dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:00.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.191+0000 7f5147e88640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f51401f0550 0x7f51401f3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:00.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.191+0000 7f5147e88640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51401f35c0 0x7f514006b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:00.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.191+0000 7f5147e88640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f514015bbb0 con 0x7f51401f35c0 2024-04-15T15:17:00.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.191+0000 7f5147e88640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f514015c570 con 0x7f512c004d40 2024-04-15T15:17:00.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.191+0000 7f5147e88640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f514015bd30 con 0x7f51401f0550 2024-04-15T15:17:00.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.191+0000 7f5146425640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51401f35c0 0x7f514006b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:00.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.191+0000 7f5146425640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51401f35c0 0x7f514006b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37854/0 (socket says 172.21.15.72:37854) 2024-04-15T15:17:00.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.191+0000 7f5146425640 1 -- 172.21.15.72:0/1001098924 learned_addr learned my addr 172.21.15.72:0/1001098924 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:00.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.191+0000 7f5145423640 1 --2- 172.21.15.72:0/1001098924 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f51401f0550 0x7f51401f3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:00.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.191+0000 7f5145c24640 1 --2- 172.21.15.72:0/1001098924 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f512c004d40 0x7f514009dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:00.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.191+0000 7f5146425640 1 -- 172.21.15.72:0/1001098924 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f51401f0550 msgr2=0x7f51401f3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:00.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.191+0000 7f5146425640 1 --2- 172.21.15.72:0/1001098924 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f51401f0550 0x7f51401f3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.191+0000 7f5146425640 1 -- 172.21.15.72:0/1001098924 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f512c004d40 msgr2=0x7f514009dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:00.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.192+0000 7f5146425640 1 --2- 172.21.15.72:0/1001098924 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f512c004d40 0x7f514009dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.192+0000 7f5146425640 1 -- 172.21.15.72:0/1001098924 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51401f3b00 con 0x7f51401f35c0 2024-04-15T15:17:00.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.192+0000 7f5145423640 1 --2- 172.21.15.72:0/1001098924 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f51401f0550 0x7f51401f3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:00.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.192+0000 7f5145c24640 1 --2- 172.21.15.72:0/1001098924 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f512c004d40 0x7f514009dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:00.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.192+0000 7f5146425640 1 --2- 172.21.15.72:0/1001098924 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51401f35c0 0x7f514006b960 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f51380019e0 tx=0x7f514015b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:00.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.192+0000 7f51367fc640 1 -- 172.21.15.72:0/1001098924 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5138047070 con 0x7f51401f35c0 2024-04-15T15:17:00.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.192+0000 7f51367fc640 1 -- 172.21.15.72:0/1001098924 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5138004d60 con 0x7f51401f35c0 2024-04-15T15:17:00.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.192+0000 7f5147e88640 1 -- 172.21.15.72:0/1001098924 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51401f3de0 con 0x7f51401f35c0 2024-04-15T15:17:00.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.192+0000 7f51367fc640 1 -- 172.21.15.72:0/1001098924 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513803a040 con 0x7f51401f35c0 2024-04-15T15:17:00.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.192+0000 7f5147e88640 1 -- 172.21.15.72:0/1001098924 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51401f09d0 con 0x7f51401f35c0 2024-04-15T15:17:00.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.193+0000 7f5147e88640 1 -- 172.21.15.72:0/1001098924 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51400689e0 con 0x7f51401f35c0 2024-04-15T15:17:00.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.193+0000 7f51367fc640 1 -- 172.21.15.72:0/1001098924 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f513804a070 con 0x7f51401f35c0 2024-04-15T15:17:00.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.194+0000 7f51367fc640 1 --2- 172.21.15.72:0/1001098924 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f512406c9b0 0x7f512406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:00.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.194+0000 7f51367fc640 1 -- 172.21.15.72:0/1001098924 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 5864+0+0 (secure 0 0 0) 0x7f51380af300 con 0x7f51401f35c0 2024-04-15T15:17:00.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.194+0000 7f5145c24640 1 --2- 172.21.15.72:0/1001098924 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f512406c9b0 0x7f512406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:00.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.194+0000 7f5145c24640 1 --2- 172.21.15.72:0/1001098924 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f512406c9b0 0x7f512406ee70 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f51401ed680 tx=0x7f513c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:00.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.198+0000 7f51367fc640 1 -- 172.21.15.72:0/1001098924 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f513807cfd0 con 0x7f51401f35c0 2024-04-15T15:17:00.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:00.371+0000 7f5147e88640 1 -- 172.21.15.72:0/1001098924 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f514006ce00 con 0x7f51401f35c0 2024-04-15T15:17:01.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.062+0000 7f51367fc640 1 -- 172.21.15.72:0/1001098924 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v44) ==== 169+0+0 (secure 0 0 0) 0x7f51380812d0 con 0x7f51401f35c0 2024-04-15T15:17:01.065 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'cephfs_data' removed 2024-04-15T15:17:01.067 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.065+0000 7f5147e88640 1 -- 172.21.15.72:0/1001098924 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f512406c9b0 msgr2=0x7f512406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.065+0000 7f5147e88640 1 --2- 172.21.15.72:0/1001098924 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f512406c9b0 0x7f512406ee70 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f51401ed680 tx=0x7f513c009040 comp rx=0 tx=0).stop 2024-04-15T15:17:01.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.065+0000 7f5147e88640 1 -- 172.21.15.72:0/1001098924 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51401f35c0 msgr2=0x7f514006b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.065+0000 7f5147e88640 1 --2- 172.21.15.72:0/1001098924 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51401f35c0 0x7f514006b960 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f51380019e0 tx=0x7f514015b980 comp rx=0 tx=0).stop 2024-04-15T15:17:01.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.065+0000 7f5147e88640 1 -- 172.21.15.72:0/1001098924 shutdown_connections 2024-04-15T15:17:01.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.065+0000 7f5147e88640 1 --2- 172.21.15.72:0/1001098924 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f512406c9b0 0x7f512406ee70 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.065+0000 7f5147e88640 1 --2- 172.21.15.72:0/1001098924 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f51401f35c0 0x7f514006b960 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.065+0000 7f5147e88640 1 --2- 172.21.15.72:0/1001098924 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f51401f0550 0x7f51401f3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.065+0000 7f5147e88640 1 --2- 172.21.15.72:0/1001098924 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f512c004d40 0x7f514009dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.065+0000 7f5147e88640 1 -- 172.21.15.72:0/1001098924 >> 172.21.15.72:0/1001098924 conn(0x7f51400b1ee0 msgr2=0x7f514006dab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:01.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.065+0000 7f5147e88640 1 -- 172.21.15.72:0/1001098924 shutdown_connections 2024-04-15T15:17:01.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.065+0000 7f5147e88640 1 -- 172.21.15.72:0/1001098924 wait complete. 2024-04-15T15:17:01.083 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds debug_mds 2024-04-15T15:17:01.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.186+0000 7f00fdeb2640 1 Processor -- start 2024-04-15T15:17:01.189 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.186+0000 7f00fdeb2640 1 -- start start 2024-04-15T15:17:01.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.187+0000 7f00fdeb2640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f00f8161ee0 0x7f00f80c95f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.187+0000 7f00fdeb2640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f00f80d1810 0x7f00f80ce080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.187+0000 7f00fdeb2640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00f80ccfe0 con 0x7f00f80b67c0 2024-04-15T15:17:01.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.187+0000 7f00fdeb2640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f00f81622c0 con 0x7f00f8161ee0 2024-04-15T15:17:01.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.187+0000 7f00fdeb2640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f00f80cd2c0 con 0x7f00f80d1810 2024-04-15T15:17:01.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.187+0000 7f00f6ffd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f00f80b67c0 0x7f00f80d1100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:60144/0 (socket says 172.21.15.72:60144) 2024-04-15T15:17:01.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.187+0000 7f00f6ffd640 1 -- 172.21.15.72:0/2986019476 learned_addr learned my addr 172.21.15.72:0/2986019476 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:17:01.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.187+0000 7f00f67fc640 1 -- 172.21.15.72:0/2986019476 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2944540203 0 0) 0x7f00f80ccfe0 con 0x7f00f80b67c0 2024-04-15T15:17:01.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.187+0000 7f00f67fc640 1 -- 172.21.15.72:0/2986019476 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f00e0003520 con 0x7f00f80b67c0 2024-04-15T15:17:01.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.187+0000 7f00f77fe640 1 --2- 172.21.15.72:0/2986019476 >> v2:172.21.15.175:3300/0 conn(0x7f00f8161ee0 0x7f00f80c95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.187+0000 7f00f7fff640 1 --2- 172.21.15.72:0/2986019476 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f00f80d1810 0x7f00f80ce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.187+0000 7f00f67fc640 1 -- 172.21.15.72:0/2986019476 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1139410909 0 0) 0x7f00e0003520 con 0x7f00f80b67c0 2024-04-15T15:17:01.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.187+0000 7f00f67fc640 1 -- 172.21.15.72:0/2986019476 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f00f80d1810 msgr2=0x7f00f80ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:01.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.187+0000 7f00f67fc640 1 --2- 172.21.15.72:0/2986019476 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f00f80d1810 0x7f00f80ce080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.187+0000 7f00f67fc640 1 -- 172.21.15.72:0/2986019476 >> v2:172.21.15.175:3300/0 conn(0x7f00f8161ee0 msgr2=0x7f00f80c95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:01.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.187+0000 7f00f67fc640 1 --2- 172.21.15.72:0/2986019476 >> v2:172.21.15.175:3300/0 conn(0x7f00f8161ee0 0x7f00f80c95f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.187+0000 7f00f67fc640 1 -- 172.21.15.72:0/2986019476 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00f80cd5a0 con 0x7f00f80b67c0 2024-04-15T15:17:01.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.187+0000 7f00f67fc640 1 -- 172.21.15.72:0/2986019476 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f00ec001670 con 0x7f00f80b67c0 2024-04-15T15:17:01.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.187+0000 7f00f67fc640 1 -- 172.21.15.72:0/2986019476 >> v1:172.21.15.72:6789/0 conn(0x7f00f80b67c0 legacy=0x7f00f80d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.188+0000 7f00f67fc640 1 --2- 172.21.15.72:0/2986019476 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f00e00039b0 0x7f00e0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.188+0000 7f00f67fc640 1 --2- 172.21.15.72:0/2986019476 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f00f80d1810 0x7f00e0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.188+0000 7f00f67fc640 1 --2- 172.21.15.72:0/2986019476 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00e0004d40 0x7f00e00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.188+0000 7f00f67fc640 1 -- 172.21.15.72:0/2986019476 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f00f80cd2c0 con 0x7f00e00039b0 2024-04-15T15:17:01.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.188+0000 7f00f67fc640 1 -- 172.21.15.72:0/2986019476 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f00e000b9c0 con 0x7f00e0004d40 2024-04-15T15:17:01.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.188+0000 7f00f67fc640 1 -- 172.21.15.72:0/2986019476 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f00e000bb40 con 0x7f00f80d1810 2024-04-15T15:17:01.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.188+0000 7f00f77fe640 1 --2- 172.21.15.72:0/2986019476 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f00e00039b0 0x7f00e0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.188+0000 7f00f7fff640 1 --2- 172.21.15.72:0/2986019476 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00e0004d40 0x7f00e00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.188+0000 7f00f6ffd640 1 --2- 172.21.15.72:0/2986019476 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f00f80d1810 0x7f00e0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.188+0000 7f00f77fe640 1 -- 172.21.15.72:0/2986019476 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f00f80d1810 msgr2=0x7f00e0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:01.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.188+0000 7f00f77fe640 1 --2- 172.21.15.72:0/2986019476 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f00f80d1810 0x7f00e0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.188+0000 7f00f77fe640 1 -- 172.21.15.72:0/2986019476 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00e0004d40 msgr2=0x7f00e00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:01.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.188+0000 7f00f77fe640 1 --2- 172.21.15.72:0/2986019476 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00e0004d40 0x7f00e00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.188+0000 7f00f77fe640 1 -- 172.21.15.72:0/2986019476 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00e0003520 con 0x7f00e00039b0 2024-04-15T15:17:01.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.189+0000 7f00f77fe640 1 --2- 172.21.15.72:0/2986019476 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f00e00039b0 0x7f00e0003dd0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f00e8009ac0 tx=0x7f00e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14fb1c210ffb5bb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:01.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.189+0000 7f00f67fc640 1 -- 172.21.15.72:0/2986019476 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e8039070 con 0x7f00e00039b0 2024-04-15T15:17:01.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.189+0000 7f00f67fc640 1 -- 172.21.15.72:0/2986019476 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00e802c6d0 con 0x7f00e00039b0 2024-04-15T15:17:01.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.189+0000 7f00f67fc640 1 -- 172.21.15.72:0/2986019476 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00e8009690 con 0x7f00e00039b0 2024-04-15T15:17:01.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.189+0000 7f00fdeb2640 1 -- 172.21.15.72:0/2986019476 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f00e00039b0 msgr2=0x7f00e0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.189+0000 7f00fdeb2640 1 --2- 172.21.15.72:0/2986019476 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f00e00039b0 0x7f00e0003dd0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f00e8009ac0 tx=0x7f00e0005830 comp rx=0 tx=0).stop 2024-04-15T15:17:01.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.189+0000 7f00f7fff640 1 -- 172.21.15.72:0/2986019476 reap_dead start 2024-04-15T15:17:01.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.189+0000 7f00fdeb2640 1 -- 172.21.15.72:0/2986019476 shutdown_connections 2024-04-15T15:17:01.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.189+0000 7f00fdeb2640 1 -- 172.21.15.72:0/2986019476 >> 172.21.15.72:0/2986019476 conn(0x7f00f80b4010 msgr2=0x7f00f80b4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:01.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.189+0000 7f00fdeb2640 1 -- 172.21.15.72:0/2986019476 shutdown_connections 2024-04-15T15:17:01.195 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.189+0000 7f00fdeb2640 1 -- 172.21.15.72:0/2986019476 wait complete. 2024-04-15T15:17:01.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.191+0000 7f00fdeb2640 1 Processor -- start 2024-04-15T15:17:01.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.191+0000 7f00fdeb2640 1 -- start start 2024-04-15T15:17:01.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.191+0000 7f00fdeb2640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f00e00039b0 0x7f00f8155c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.191+0000 7f00fdeb2640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00f81596f0 0x7f00f81f4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.191+0000 7f00fdeb2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f00f8159b70 0x7f00f81f4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.192+0000 7f00fdeb2640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f00f80ccb70 con 0x7f00f8159b70 2024-04-15T15:17:01.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.192+0000 7f00fdeb2640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f00f80c9b30 con 0x7f00f81596f0 2024-04-15T15:17:01.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.192+0000 7f00fdeb2640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f00f80cccf0 con 0x7f00e00039b0 2024-04-15T15:17:01.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.192+0000 7f00f7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f00f8159b70 0x7f00f81f4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.192+0000 7f00f7fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f00f8159b70 0x7f00f81f4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37870/0 (socket says 172.21.15.72:37870) 2024-04-15T15:17:01.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.192+0000 7f00f7fff640 1 -- 172.21.15.72:0/2353523046 learned_addr learned my addr 172.21.15.72:0/2353523046 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:01.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.192+0000 7f00f77fe640 1 --2- 172.21.15.72:0/2353523046 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f00e00039b0 0x7f00f8155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.192+0000 7f00f6ffd640 1 --2- 172.21.15.72:0/2353523046 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00f81596f0 0x7f00f81f4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.192+0000 7f00f7fff640 1 -- 172.21.15.72:0/2353523046 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f00e00039b0 msgr2=0x7f00f8155c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.192+0000 7f00f7fff640 1 --2- 172.21.15.72:0/2353523046 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f00e00039b0 0x7f00f8155c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.192+0000 7f00f7fff640 1 -- 172.21.15.72:0/2353523046 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00f81596f0 msgr2=0x7f00f81f4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.192+0000 7f00f7fff640 1 --2- 172.21.15.72:0/2353523046 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00f81596f0 0x7f00f81f4560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.192+0000 7f00f7fff640 1 -- 172.21.15.72:0/2353523046 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00f81561b0 con 0x7f00f8159b70 2024-04-15T15:17:01.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.192+0000 7f00f7fff640 1 --2- 172.21.15.72:0/2353523046 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f00f8159b70 0x7f00f81f4aa0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f00f8073990 tx=0x7f00f000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:01.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.192+0000 7f00dffff640 1 -- 172.21.15.72:0/2353523046 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f000fde0 con 0x7f00f8159b70 2024-04-15T15:17:01.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.192+0000 7f00dffff640 1 -- 172.21.15.72:0/2353523046 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00f001d070 con 0x7f00f8159b70 2024-04-15T15:17:01.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.192+0000 7f00fdeb2640 1 -- 172.21.15.72:0/2353523046 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00f81570c0 con 0x7f00f8159b70 2024-04-15T15:17:01.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.192+0000 7f00dffff640 1 -- 172.21.15.72:0/2353523046 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f0015670 con 0x7f00f8159b70 2024-04-15T15:17:01.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.192+0000 7f00fdeb2640 1 -- 172.21.15.72:0/2353523046 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00f8158460 con 0x7f00f8159b70 2024-04-15T15:17:01.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.193+0000 7f00fdeb2640 1 -- 172.21.15.72:0/2353523046 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00f8075c50 con 0x7f00f8159b70 2024-04-15T15:17:01.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.194+0000 7f00dffff640 1 -- 172.21.15.72:0/2353523046 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f00f0015810 con 0x7f00f8159b70 2024-04-15T15:17:01.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.194+0000 7f00dffff640 1 --2- 172.21.15.72:0/2353523046 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f00e406c980 0x7f00e406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.194+0000 7f00dffff640 1 -- 172.21.15.72:0/2353523046 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5421+0+0 (secure 0 0 0) 0x7f00f008c4d0 con 0x7f00f8159b70 2024-04-15T15:17:01.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.194+0000 7f00f77fe640 1 --2- 172.21.15.72:0/2353523046 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f00e406c980 0x7f00e406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.195+0000 7f00f77fe640 1 --2- 172.21.15.72:0/2353523046 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f00e406c980 0x7f00e406ee40 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f00e8036000 tx=0x7f00e8030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:01.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.198+0000 7f00dffff640 1 -- 172.21.15.72:0/2353523046 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f00f81561b0 con 0x7f00f8159b70 2024-04-15T15:17:01.388 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.386+0000 7f00fdeb2640 1 -- 172.21.15.72:0/2353523046 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "debug_mds"} v 0) -- 0x7f00f80d1e50 con 0x7f00f8159b70 2024-04-15T15:17:01.391 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.389+0000 7f00dffff640 1 -- 172.21.15.72:0/2353523046 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "debug_mds"}]=0 v13) ==== 92+0+0 (secure 0 0 0) 0x7f00f0010040 con 0x7f00f8159b70 2024-04-15T15:17:01.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.391+0000 7f00fdeb2640 1 -- 172.21.15.72:0/2353523046 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f00e406c980 msgr2=0x7f00e406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.391+0000 7f00fdeb2640 1 --2- 172.21.15.72:0/2353523046 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f00e406c980 0x7f00e406ee40 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f00e8036000 tx=0x7f00e8030000 comp rx=0 tx=0).stop 2024-04-15T15:17:01.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.391+0000 7f00fdeb2640 1 -- 172.21.15.72:0/2353523046 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f00f8159b70 msgr2=0x7f00f81f4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.391+0000 7f00fdeb2640 1 --2- 172.21.15.72:0/2353523046 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f00f8159b70 0x7f00f81f4aa0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f00f8073990 tx=0x7f00f000f580 comp rx=0 tx=0).stop 2024-04-15T15:17:01.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.391+0000 7f00fdeb2640 1 -- 172.21.15.72:0/2353523046 shutdown_connections 2024-04-15T15:17:01.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.391+0000 7f00fdeb2640 1 --2- 172.21.15.72:0/2353523046 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f00e406c980 0x7f00e406ee40 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.391+0000 7f00fdeb2640 1 --2- 172.21.15.72:0/2353523046 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f00f8159b70 0x7f00f81f4aa0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.391+0000 7f00fdeb2640 1 --2- 172.21.15.72:0/2353523046 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f00f81596f0 0x7f00f81f4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.391+0000 7f00fdeb2640 1 --2- 172.21.15.72:0/2353523046 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f00e00039b0 0x7f00f8155c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.394 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.391+0000 7f00fdeb2640 1 -- 172.21.15.72:0/2353523046 >> 172.21.15.72:0/2353523046 conn(0x7f00f80b4010 msgr2=0x7f00f80b5b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:01.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.391+0000 7f00fdeb2640 1 -- 172.21.15.72:0/2353523046 shutdown_connections 2024-04-15T15:17:01.395 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.391+0000 7f00fdeb2640 1 -- 172.21.15.72:0/2353523046 wait complete. 2024-04-15T15:17:01.405 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-15T15:17:01.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.507+0000 7fcb76a9a640 1 Processor -- start 2024-04-15T15:17:01.509 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.507+0000 7fcb76a9a640 1 -- start start 2024-04-15T15:17:01.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.507+0000 7fcb76a9a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcb700b0270 0x7fcb700cea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.507+0000 7fcb76a9a640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fcb700d1570 0x7fcb700cef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.507+0000 7fcb76a9a640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb7015ce00 con 0x7fcb70156cf0 2024-04-15T15:17:01.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.508+0000 7fcb76a9a640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fcb7015d0e0 con 0x7fcb700b0270 2024-04-15T15:17:01.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.508+0000 7fcb76a9a640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fcb7015d3c0 con 0x7fcb700d1570 2024-04-15T15:17:01.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.508+0000 7fcb6ffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcb700b0270 0x7fcb700cea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.508+0000 7fcb6ffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fcb700b0270 0x7fcb700cea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37872/0 (socket says 172.21.15.72:37872) 2024-04-15T15:17:01.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.508+0000 7fcb6ffff640 1 -- 172.21.15.72:0/1718443579 learned_addr learned my addr 172.21.15.72:0/1718443579 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:01.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.508+0000 7fcb6ffff640 1 -- 172.21.15.72:0/1718443579 >> v2:172.21.15.175:3300/0 conn(0x7fcb700d1570 msgr2=0x7fcb700cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:01.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.508+0000 7fcb6ffff640 1 --2- 172.21.15.72:0/1718443579 >> v2:172.21.15.175:3300/0 conn(0x7fcb700d1570 0x7fcb700cef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.508+0000 7fcb6ffff640 1 -- 172.21.15.72:0/1718443579 >> v1:172.21.15.175:6789/0 conn(0x7fcb70156cf0 legacy=0x7fcb7006ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:01.511 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.508+0000 7fcb6ffff640 1 -- 172.21.15.72:0/1718443579 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb7015d6a0 con 0x7fcb700b0270 2024-04-15T15:17:01.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.509+0000 7fcb6ffff640 1 --2- 172.21.15.72:0/1718443579 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb700b0270 0x7fcb700cea50 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fcb58009870 tx=0x7fcb5802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecf9e74061969f5c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:01.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.509+0000 7fcb6f7fe640 1 -- 172.21.15.72:0/1718443579 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5802e070 con 0x7fcb700b0270 2024-04-15T15:17:01.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.509+0000 7fcb6f7fe640 1 -- 172.21.15.72:0/1718443579 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb58037070 con 0x7fcb700b0270 2024-04-15T15:17:01.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.509+0000 7fcb6f7fe640 1 -- 172.21.15.72:0/1718443579 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb58031a00 con 0x7fcb700b0270 2024-04-15T15:17:01.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.509+0000 7fcb76a9a640 1 -- 172.21.15.72:0/1718443579 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb700b0270 msgr2=0x7fcb700cea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.509+0000 7fcb76a9a640 1 --2- 172.21.15.72:0/1718443579 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb700b0270 0x7fcb700cea50 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fcb58009870 tx=0x7fcb5802b880 comp rx=0 tx=0).stop 2024-04-15T15:17:01.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.509+0000 7fcb76a9a640 1 -- 172.21.15.72:0/1718443579 shutdown_connections 2024-04-15T15:17:01.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.509+0000 7fcb76a9a640 1 --2- 172.21.15.72:0/1718443579 >> v2:172.21.15.175:3300/0 conn(0x7fcb700d1570 0x7fcb700cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.509+0000 7fcb76a9a640 1 --2- 172.21.15.72:0/1718443579 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb700b0270 0x7fcb700cea50 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.509+0000 7fcb76a9a640 1 -- 172.21.15.72:0/1718443579 >> 172.21.15.72:0/1718443579 conn(0x7fcb700d0a50 msgr2=0x7fcb700d0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:01.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.510+0000 7fcb76a9a640 1 -- 172.21.15.72:0/1718443579 shutdown_connections 2024-04-15T15:17:01.513 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.510+0000 7fcb76a9a640 1 -- 172.21.15.72:0/1718443579 wait complete. 2024-04-15T15:17:01.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.511+0000 7fcb76a9a640 1 Processor -- start 2024-04-15T15:17:01.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.511+0000 7fcb76a9a640 1 -- start start 2024-04-15T15:17:01.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.512+0000 7fcb76a9a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb700b0270 0x7fcb7015ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.512+0000 7fcb76a9a640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcb700d1570 0x7fcb70157c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.512+0000 7fcb76a9a640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb70156cf0 0x7fcb70159110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.512+0000 7fcb76a9a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fcb7006dbb0 con 0x7fcb700b0270 2024-04-15T15:17:01.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.512+0000 7fcb76a9a640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fcb701570d0 con 0x7fcb70156cf0 2024-04-15T15:17:01.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.512+0000 7fcb76a9a640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fcb7006d540 con 0x7fcb700d1570 2024-04-15T15:17:01.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.512+0000 7fcb74836640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb700b0270 0x7fcb7015ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.512+0000 7fcb74836640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb700b0270 0x7fcb7015ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37874/0 (socket says 172.21.15.72:37874) 2024-04-15T15:17:01.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.512+0000 7fcb74836640 1 -- 172.21.15.72:0/1349108965 learned_addr learned my addr 172.21.15.72:0/1349108965 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:01.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.512+0000 7fcb74836640 1 -- 172.21.15.72:0/1349108965 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcb700d1570 msgr2=0x7fcb70157c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.512+0000 7fcb74836640 1 --2- 172.21.15.72:0/1349108965 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcb700d1570 0x7fcb70157c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.512+0000 7fcb74836640 1 -- 172.21.15.72:0/1349108965 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb70156cf0 msgr2=0x7fcb70159110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.512+0000 7fcb75037640 1 --2- 172.21.15.72:0/1349108965 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb70156cf0 0x7fcb70159110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.512+0000 7fcb74836640 1 --2- 172.21.15.72:0/1349108965 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb70156cf0 0x7fcb70159110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.512+0000 7fcb74836640 1 -- 172.21.15.72:0/1349108965 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb7015b490 con 0x7fcb700b0270 2024-04-15T15:17:01.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.512+0000 7fcb75037640 1 --2- 172.21.15.72:0/1349108965 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb70156cf0 0x7fcb70159110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:01.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.512+0000 7fcb74836640 1 --2- 172.21.15.72:0/1349108965 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb700b0270 0x7fcb7015ae90 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fcb6000c7c0 tx=0x7fcb6000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:01.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.512+0000 7fcb6d7fa640 1 -- 172.21.15.72:0/1349108965 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb60016020 con 0x7fcb700b0270 2024-04-15T15:17:01.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.512+0000 7fcb6d7fa640 1 -- 172.21.15.72:0/1349108965 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb6000aa50 con 0x7fcb700b0270 2024-04-15T15:17:01.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.512+0000 7fcb76a9a640 1 -- 172.21.15.72:0/1349108965 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb7015b770 con 0x7fcb700b0270 2024-04-15T15:17:01.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.512+0000 7fcb6d7fa640 1 -- 172.21.15.72:0/1349108965 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb60012420 con 0x7fcb700b0270 2024-04-15T15:17:01.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.512+0000 7fcb76a9a640 1 -- 172.21.15.72:0/1349108965 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb7015bd30 con 0x7fcb700b0270 2024-04-15T15:17:01.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.513+0000 7fcb76a9a640 1 -- 172.21.15.72:0/1349108965 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb70075c50 con 0x7fcb700b0270 2024-04-15T15:17:01.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.514+0000 7fcb6d7fa640 1 -- 172.21.15.72:0/1349108965 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fcb60019020 con 0x7fcb700b0270 2024-04-15T15:17:01.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.514+0000 7fcb6d7fa640 1 --2- 172.21.15.72:0/1349108965 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcb5006c9b0 0x7fcb5006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.514+0000 7fcb6d7fa640 1 -- 172.21.15.72:0/1349108965 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5421+0+0 (secure 0 0 0) 0x7fcb6008b760 con 0x7fcb700b0270 2024-04-15T15:17:01.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.514+0000 7fcb6ffff640 1 --2- 172.21.15.72:0/1349108965 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcb5006c9b0 0x7fcb5006ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.519 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.515+0000 7fcb6ffff640 1 --2- 172.21.15.72:0/1349108965 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcb5006c9b0 0x7fcb5006ee70 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fcb58036000 tx=0x7fcb58030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:01.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.519+0000 7fcb6d7fa640 1 -- 172.21.15.72:0/1349108965 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcb60020070 con 0x7fcb700b0270 2024-04-15T15:17:01.695 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.693+0000 7fcb76a9a640 1 -- 172.21.15.72:0/1349108965 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7fcb70074e30 con 0x7fcb700b0270 2024-04-15T15:17:01.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.696+0000 7fcb6d7fa640 1 -- 172.21.15.72:0/1349108965 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v14) ==== 104+0+0 (secure 0 0 0) 0x7fcb60059d50 con 0x7fcb700b0270 2024-04-15T15:17:01.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.698+0000 7fcb76a9a640 1 -- 172.21.15.72:0/1349108965 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcb5006c9b0 msgr2=0x7fcb5006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.698+0000 7fcb76a9a640 1 --2- 172.21.15.72:0/1349108965 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcb5006c9b0 0x7fcb5006ee70 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fcb58036000 tx=0x7fcb58030000 comp rx=0 tx=0).stop 2024-04-15T15:17:01.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.698+0000 7fcb76a9a640 1 -- 172.21.15.72:0/1349108965 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb700b0270 msgr2=0x7fcb7015ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.698+0000 7fcb76a9a640 1 --2- 172.21.15.72:0/1349108965 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb700b0270 0x7fcb7015ae90 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fcb6000c7c0 tx=0x7fcb6000cc80 comp rx=0 tx=0).stop 2024-04-15T15:17:01.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.698+0000 7fcb76a9a640 1 -- 172.21.15.72:0/1349108965 shutdown_connections 2024-04-15T15:17:01.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.698+0000 7fcb76a9a640 1 --2- 172.21.15.72:0/1349108965 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fcb5006c9b0 0x7fcb5006ee70 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.698+0000 7fcb76a9a640 1 --2- 172.21.15.72:0/1349108965 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fcb70156cf0 0x7fcb70159110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.698+0000 7fcb76a9a640 1 --2- 172.21.15.72:0/1349108965 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fcb700d1570 0x7fcb70157c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.698+0000 7fcb76a9a640 1 --2- 172.21.15.72:0/1349108965 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcb700b0270 0x7fcb7015ae90 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.698+0000 7fcb76a9a640 1 -- 172.21.15.72:0/1349108965 >> 172.21.15.72:0/1349108965 conn(0x7fcb700d0a50 msgr2=0x7fcb7015dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:01.701 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.698+0000 7fcb76a9a640 1 -- 172.21.15.72:0/1349108965 shutdown_connections 2024-04-15T15:17:01.702 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.698+0000 7fcb76a9a640 1 -- 172.21.15.72:0/1349108965 wait complete. 2024-04-15T15:17:01.711 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_cache_quiesce_splitauth 2024-04-15T15:17:01.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.814+0000 7fb5fa2a7640 1 Processor -- start 2024-04-15T15:17:01.816 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.814+0000 7fb5fa2a7640 1 -- start start 2024-04-15T15:17:01.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.814+0000 7fb5fa2a7640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb5f406ed30 0x7fb5f406cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.815+0000 7fb5fa2a7640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5f406bd50 con 0x7fb5f40cd2a0 2024-04-15T15:17:01.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.815+0000 7fb5fa2a7640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5f406bff0 con 0x7fb5f415c630 2024-04-15T15:17:01.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.815+0000 7fb5fa2a7640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fb5f406d2f0 con 0x7fb5f406ed30 2024-04-15T15:17:01.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.815+0000 7fb5ebfff640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fb5f40cd2a0 0x7fb5f406e620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:33976/0 (socket says 172.21.15.72:33976) 2024-04-15T15:17:01.820 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.815+0000 7fb5ebfff640 1 -- 172.21.15.72:0/1282018103 learned_addr learned my addr 172.21.15.72:0/1282018103 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:17:01.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.815+0000 7fb5f8844640 1 --2- 172.21.15.72:0/1282018103 >> v2:172.21.15.175:3300/0 conn(0x7fb5f406ed30 0x7fb5f406cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.815+0000 7fb5f37fe640 1 -- 172.21.15.72:0/1282018103 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 775580379 0 0) 0x7fb5f406bd50 con 0x7fb5f40cd2a0 2024-04-15T15:17:01.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.815+0000 7fb5f37fe640 1 -- 172.21.15.72:0/1282018103 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb5e0003520 con 0x7fb5f40cd2a0 2024-04-15T15:17:01.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.815+0000 7fb5f37fe640 1 -- 172.21.15.72:0/1282018103 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 65974074 0 0) 0x7fb5e0003520 con 0x7fb5f40cd2a0 2024-04-15T15:17:01.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.815+0000 7fb5f37fe640 1 -- 172.21.15.72:0/1282018103 >> v2:172.21.15.175:3300/0 conn(0x7fb5f406ed30 msgr2=0x7fb5f406cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:01.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.815+0000 7fb5f37fe640 1 --2- 172.21.15.72:0/1282018103 >> v2:172.21.15.175:3300/0 conn(0x7fb5f406ed30 0x7fb5f406cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.815+0000 7fb5f37fe640 1 -- 172.21.15.72:0/1282018103 >> v1:172.21.15.175:6789/0 conn(0x7fb5f415c630 legacy=0x7fb5f40ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.815+0000 7fb5f37fe640 1 -- 172.21.15.72:0/1282018103 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5f406c2d0 con 0x7fb5f40cd2a0 2024-04-15T15:17:01.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.815+0000 7fb5f37fe640 1 -- 172.21.15.72:0/1282018103 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7fb5dc001670 con 0x7fb5f40cd2a0 2024-04-15T15:17:01.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.815+0000 7fb5f37fe640 1 -- 172.21.15.72:0/1282018103 >> v1:172.21.15.72:6789/0 conn(0x7fb5f40cd2a0 legacy=0x7fb5f406e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.815+0000 7fb5f37fe640 1 --2- 172.21.15.72:0/1282018103 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb5e00039b0 0x7fb5e0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.815+0000 7fb5f37fe640 1 --2- 172.21.15.72:0/1282018103 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5e0004800 0x7fb5e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.815+0000 7fb5f37fe640 1 --2- 172.21.15.72:0/1282018103 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb5e00051c0 0x7fb5e000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.815+0000 7fb5f37fe640 1 -- 172.21.15.72:0/1282018103 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f406d2f0 con 0x7fb5e0004800 2024-04-15T15:17:01.821 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.815+0000 7fb5f37fe640 1 -- 172.21.15.72:0/1282018103 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fb5e000c060 con 0x7fb5e00051c0 2024-04-15T15:17:01.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.815+0000 7fb5f37fe640 1 -- 172.21.15.72:0/1282018103 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fb5e000c260 con 0x7fb5e00039b0 2024-04-15T15:17:01.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.815+0000 7fb5ebfff640 1 --2- 172.21.15.72:0/1282018103 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5e0004800 0x7fb5e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.815+0000 7fb5f8844640 1 --2- 172.21.15.72:0/1282018103 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb5e00051c0 0x7fb5e000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.815+0000 7fb5f3fff640 1 --2- 172.21.15.72:0/1282018103 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb5e00039b0 0x7fb5e0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.816+0000 7fb5ebfff640 1 -- 172.21.15.72:0/1282018103 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb5e00039b0 msgr2=0x7fb5e0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:01.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.816+0000 7fb5ebfff640 1 --2- 172.21.15.72:0/1282018103 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb5e00039b0 0x7fb5e0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.816+0000 7fb5ebfff640 1 -- 172.21.15.72:0/1282018103 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb5e00051c0 msgr2=0x7fb5e000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:01.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.816+0000 7fb5ebfff640 1 --2- 172.21.15.72:0/1282018103 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb5e00051c0 0x7fb5e000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.816+0000 7fb5ebfff640 1 -- 172.21.15.72:0/1282018103 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5e0003520 con 0x7fb5e0004800 2024-04-15T15:17:01.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.816+0000 7fb5f8844640 1 -- 172.21.15.72:0/1282018103 reap_dead start 2024-04-15T15:17:01.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.816+0000 7fb5f3fff640 1 --2- 172.21.15.72:0/1282018103 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb5e00039b0 0x7fb5e0003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:01.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.816+0000 7fb5ebfff640 1 --2- 172.21.15.72:0/1282018103 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5e0004800 0x7fb5e0004c80 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc00c240 tx=0x7fb5e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf2910e96589c4fb server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:01.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.817+0000 7fb5f37fe640 1 -- 172.21.15.72:0/1282018103 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc007d10 con 0x7fb5e0004800 2024-04-15T15:17:01.822 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.817+0000 7fb5f37fe640 1 -- 172.21.15.72:0/1282018103 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5dc030070 con 0x7fb5e0004800 2024-04-15T15:17:01.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.817+0000 7fb5f37fe640 1 -- 172.21.15.72:0/1282018103 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5dc005d90 con 0x7fb5e0004800 2024-04-15T15:17:01.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.817+0000 7fb5fa2a7640 1 -- 172.21.15.72:0/1282018103 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5e0004800 msgr2=0x7fb5e0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.817+0000 7fb5fa2a7640 1 --2- 172.21.15.72:0/1282018103 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5e0004800 0x7fb5e0004c80 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc00c240 tx=0x7fb5e0005830 comp rx=0 tx=0).stop 2024-04-15T15:17:01.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.817+0000 7fb5fa2a7640 1 -- 172.21.15.72:0/1282018103 shutdown_connections 2024-04-15T15:17:01.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.817+0000 7fb5fa2a7640 1 --2- 172.21.15.72:0/1282018103 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5e0004800 0x7fb5e0004c80 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.817+0000 7fb5fa2a7640 1 -- 172.21.15.72:0/1282018103 >> 172.21.15.72:0/1282018103 conn(0x7fb5f40b31d0 msgr2=0x7fb5f40b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:01.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.817+0000 7fb5fa2a7640 1 -- 172.21.15.72:0/1282018103 shutdown_connections 2024-04-15T15:17:01.823 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.817+0000 7fb5fa2a7640 1 -- 172.21.15.72:0/1282018103 wait complete. 2024-04-15T15:17:01.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.818+0000 7fb5fa2a7640 1 Processor -- start 2024-04-15T15:17:01.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.819+0000 7fb5fa2a7640 1 -- start start 2024-04-15T15:17:01.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.819+0000 7fb5fa2a7640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb5e0004800 0x7fb5f414c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.819+0000 7fb5fa2a7640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb5f40cbc80 0x7fb5f41e6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.819+0000 7fb5fa2a7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5f41e6940 0x7fb5f40ca000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.819+0000 7fb5fa2a7640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f406f180 con 0x7fb5f41e6940 2024-04-15T15:17:01.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.819+0000 7fb5fa2a7640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fb5f406c920 con 0x7fb5e0004800 2024-04-15T15:17:01.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.819+0000 7fb5fa2a7640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fb5f415ca10 con 0x7fb5f40cbc80 2024-04-15T15:17:01.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.819+0000 7fb5f8844640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5f41e6940 0x7fb5f40ca000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.819+0000 7fb5f8844640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5f41e6940 0x7fb5f40ca000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36666/0 (socket says 172.21.15.72:36666) 2024-04-15T15:17:01.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.819+0000 7fb5f8844640 1 -- 172.21.15.72:0/2718649135 learned_addr learned my addr 172.21.15.72:0/2718649135 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:01.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.820+0000 7fb5f3fff640 1 --2- 172.21.15.72:0/2718649135 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb5e0004800 0x7fb5f414c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.820+0000 7fb5ebfff640 1 --2- 172.21.15.72:0/2718649135 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb5f40cbc80 0x7fb5f41e6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.820+0000 7fb5f8844640 1 -- 172.21.15.72:0/2718649135 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb5f40cbc80 msgr2=0x7fb5f41e6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.820+0000 7fb5f8844640 1 --2- 172.21.15.72:0/2718649135 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb5f40cbc80 0x7fb5f41e6400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.820+0000 7fb5f8844640 1 -- 172.21.15.72:0/2718649135 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb5e0004800 msgr2=0x7fb5f414c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:01.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.820+0000 7fb5f8844640 1 --2- 172.21.15.72:0/2718649135 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb5e0004800 0x7fb5f414c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:01.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.820+0000 7fb5f8844640 1 -- 172.21.15.72:0/2718649135 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5f40cabe0 con 0x7fb5f41e6940 2024-04-15T15:17:01.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.820+0000 7fb5f3fff640 1 --2- 172.21.15.72:0/2718649135 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb5e0004800 0x7fb5f414c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:01.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.820+0000 7fb5f8844640 1 --2- 172.21.15.72:0/2718649135 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5f41e6940 0x7fb5f40ca000 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fb5f4074160 tx=0x7fb5ec00f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:01.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.820+0000 7fb5f17fa640 1 -- 172.21.15.72:0/2718649135 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec00fce0 con 0x7fb5f41e6940 2024-04-15T15:17:01.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.820+0000 7fb5f17fa640 1 -- 172.21.15.72:0/2718649135 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5ec01d070 con 0x7fb5f41e6940 2024-04-15T15:17:01.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.820+0000 7fb5fa2a7640 1 -- 172.21.15.72:0/2718649135 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5f40caec0 con 0x7fb5f41e6940 2024-04-15T15:17:01.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.820+0000 7fb5f17fa640 1 -- 172.21.15.72:0/2718649135 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec015670 con 0x7fb5f41e6940 2024-04-15T15:17:01.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.820+0000 7fb5fa2a7640 1 -- 172.21.15.72:0/2718649135 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5f40cb480 con 0x7fb5f41e6940 2024-04-15T15:17:01.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.821+0000 7fb5fa2a7640 1 -- 172.21.15.72:0/2718649135 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5f4075c50 con 0x7fb5f41e6940 2024-04-15T15:17:01.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.821+0000 7fb5f17fa640 1 -- 172.21.15.72:0/2718649135 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fb5ec015810 con 0x7fb5f41e6940 2024-04-15T15:17:01.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.822+0000 7fb5f17fa640 1 --2- 172.21.15.72:0/2718649135 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb5d406c980 0x7fb5d406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:01.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.822+0000 7fb5f17fa640 1 -- 172.21.15.72:0/2718649135 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5421+0+0 (secure 0 0 0) 0x7fb5ec08c5e0 con 0x7fb5f41e6940 2024-04-15T15:17:01.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.822+0000 7fb5f3fff640 1 --2- 172.21.15.72:0/2718649135 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb5d406c980 0x7fb5d406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:01.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.822+0000 7fb5f3fff640 1 --2- 172.21.15.72:0/2718649135 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb5d406c980 0x7fb5d406ee40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4000c00 tx=0x7fb5e4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:01.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:01.826+0000 7fb5f17fa640 1 -- 172.21.15.72:0/2718649135 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb5ec010040 con 0x7fb5f41e6940 2024-04-15T15:17:02.004 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.002+0000 7fb5fa2a7640 1 -- 172.21.15.72:0/2718649135 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_cache_quiesce_splitauth"} v 0) -- 0x7fb5f40d0f10 con 0x7fb5f41e6940 2024-04-15T15:17:02.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.005+0000 7fb5f17fa640 1 -- 172.21.15.72:0/2718649135 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_cache_quiesce_splitauth"}]=0 v15) ==== 110+0+0 (secure 0 0 0) 0x7fb5ec05a9a0 con 0x7fb5f41e6940 2024-04-15T15:17:02.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.007+0000 7fb5fa2a7640 1 -- 172.21.15.72:0/2718649135 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb5d406c980 msgr2=0x7fb5d406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:02.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.007+0000 7fb5fa2a7640 1 --2- 172.21.15.72:0/2718649135 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb5d406c980 0x7fb5d406ee40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4000c00 tx=0x7fb5e4009040 comp rx=0 tx=0).stop 2024-04-15T15:17:02.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.008+0000 7fb5fa2a7640 1 -- 172.21.15.72:0/2718649135 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5f41e6940 msgr2=0x7fb5f40ca000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:02.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.008+0000 7fb5fa2a7640 1 --2- 172.21.15.72:0/2718649135 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5f41e6940 0x7fb5f40ca000 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fb5f4074160 tx=0x7fb5ec00f1b0 comp rx=0 tx=0).stop 2024-04-15T15:17:02.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.008+0000 7fb5fa2a7640 1 -- 172.21.15.72:0/2718649135 shutdown_connections 2024-04-15T15:17:02.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.008+0000 7fb5fa2a7640 1 --2- 172.21.15.72:0/2718649135 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb5d406c980 0x7fb5d406ee40 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:02.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.008+0000 7fb5fa2a7640 1 --2- 172.21.15.72:0/2718649135 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5f41e6940 0x7fb5f40ca000 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:02.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.008+0000 7fb5fa2a7640 1 --2- 172.21.15.72:0/2718649135 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb5f40cbc80 0x7fb5f41e6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:02.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.008+0000 7fb5fa2a7640 1 --2- 172.21.15.72:0/2718649135 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb5e0004800 0x7fb5f414c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:02.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.008+0000 7fb5fa2a7640 1 -- 172.21.15.72:0/2718649135 >> 172.21.15.72:0/2718649135 conn(0x7fb5f40b31d0 msgr2=0x7fb5f40b0400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:02.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.008+0000 7fb5fa2a7640 1 -- 172.21.15.72:0/2718649135 shutdown_connections 2024-04-15T15:17:02.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.008+0000 7fb5fa2a7640 1 -- 172.21.15.72:0/2718649135 wait complete. 2024-04-15T15:17:02.021 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_blocked' 2024-04-15T15:17:02.125 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.123+0000 7f8e2a51c640 1 Processor -- start 2024-04-15T15:17:02.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.123+0000 7f8e2a51c640 1 -- start start 2024-04-15T15:17:02.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.124+0000 7f8e2a51c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8e24161e40 0x7f8e240b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:02.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.124+0000 7f8e2a51c640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8e240d12b0 0x7f8e240d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:02.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.124+0000 7f8e2a51c640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e24162220 con 0x7f8e240d2ea0 2024-04-15T15:17:02.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.124+0000 7f8e2a51c640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f8e240d21f0 con 0x7f8e24161e40 2024-04-15T15:17:02.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.124+0000 7f8e2a51c640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f8e240d1820 con 0x7f8e240d12b0 2024-04-15T15:17:02.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.124+0000 7f8e23fff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f8e24161e40 0x7f8e240b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:02.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.124+0000 7f8e28ab9640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f8e240d2ea0 0x7f8e240d45b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:33980/0 (socket says 172.21.15.72:33980) 2024-04-15T15:17:02.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.124+0000 7f8e28ab9640 1 -- 172.21.15.72:0/1249681872 learned_addr learned my addr 172.21.15.72:0/1249681872 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:17:02.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.124+0000 7f8e23fff640 1 -- 172.21.15.72:0/1249681872 >> v2:172.21.15.175:3300/0 conn(0x7f8e240d12b0 msgr2=0x7f8e240d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:02.127 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.124+0000 7f8e23fff640 1 --2- 172.21.15.72:0/1249681872 >> v2:172.21.15.175:3300/0 conn(0x7f8e240d12b0 0x7f8e240d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:02.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.124+0000 7f8e23fff640 1 -- 172.21.15.72:0/1249681872 >> v1:172.21.15.72:6789/0 conn(0x7f8e240d2ea0 legacy=0x7f8e240d45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:02.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.124+0000 7f8e23fff640 1 -- 172.21.15.72:0/1249681872 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e240d1b00 con 0x7f8e24161e40 2024-04-15T15:17:02.128 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.124+0000 7f8e237fe640 1 -- 172.21.15.72:0/1249681872 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2326936294 0 0) 0x7f8e24162220 con 0x7f8e240d2ea0 2024-04-15T15:17:02.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.125+0000 7f8e23fff640 1 --2- 172.21.15.72:0/1249681872 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e24161e40 0x7f8e240b37f0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f8e140097a0 tx=0x7f8e1402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a86d92fac1eb4bd server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:02.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.125+0000 7f8e237fe640 1 -- 172.21.15.72:0/1249681872 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e140315a0 con 0x7f8e24161e40 2024-04-15T15:17:02.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.125+0000 7f8e237fe640 1 -- 172.21.15.72:0/1249681872 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e14037070 con 0x7f8e24161e40 2024-04-15T15:17:02.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.125+0000 7f8e237fe640 1 -- 172.21.15.72:0/1249681872 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e14031960 con 0x7f8e24161e40 2024-04-15T15:17:02.129 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.126+0000 7f8e2a51c640 1 -- 172.21.15.72:0/1249681872 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e24161e40 msgr2=0x7f8e240b37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:02.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.126+0000 7f8e2a51c640 1 --2- 172.21.15.72:0/1249681872 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e24161e40 0x7f8e240b37f0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f8e140097a0 tx=0x7f8e1402b840 comp rx=0 tx=0).stop 2024-04-15T15:17:02.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.126+0000 7f8e2a51c640 1 -- 172.21.15.72:0/1249681872 shutdown_connections 2024-04-15T15:17:02.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.126+0000 7f8e2a51c640 1 --2- 172.21.15.72:0/1249681872 >> v2:172.21.15.175:3300/0 conn(0x7f8e240d12b0 0x7f8e240d2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:02.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.126+0000 7f8e2a51c640 1 --2- 172.21.15.72:0/1249681872 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e24161e40 0x7f8e240b37f0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:02.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.126+0000 7f8e2a51c640 1 -- 172.21.15.72:0/1249681872 >> 172.21.15.72:0/1249681872 conn(0x7f8e2409d970 msgr2=0x7f8e240b95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:02.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.126+0000 7f8e2a51c640 1 -- 172.21.15.72:0/1249681872 shutdown_connections 2024-04-15T15:17:02.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.126+0000 7f8e2a51c640 1 -- 172.21.15.72:0/1249681872 wait complete. 2024-04-15T15:17:02.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.127+0000 7f8e2a51c640 1 Processor -- start 2024-04-15T15:17:02.130 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.127+0000 7f8e2a51c640 1 -- start start 2024-04-15T15:17:02.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.128+0000 7f8e2a51c640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8e240d12b0 0x7f8e241f79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:02.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.128+0000 7f8e2a51c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e24161e40 0x7f8e241f7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:02.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.128+0000 7f8e2a51c640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8e241ee640 0x7f8e2414a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:02.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.128+0000 7f8e2a51c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8e240b3d30 con 0x7f8e24161e40 2024-04-15T15:17:02.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.128+0000 7f8e2a51c640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f8e240d2440 con 0x7f8e240d12b0 2024-04-15T15:17:02.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.128+0000 7f8e2a51c640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f8e240b3eb0 con 0x7f8e241ee640 2024-04-15T15:17:02.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.128+0000 7f8e1bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e24161e40 0x7f8e241f7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:02.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.128+0000 7f8e1bfff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e24161e40 0x7f8e241f7ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36678/0 (socket says 172.21.15.72:36678) 2024-04-15T15:17:02.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.128+0000 7f8e1bfff640 1 -- 172.21.15.72:0/2576491709 learned_addr learned my addr 172.21.15.72:0/2576491709 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:02.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.128+0000 7f8e28ab9640 1 --2- 172.21.15.72:0/2576491709 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8e241ee640 0x7f8e2414a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:02.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.128+0000 7f8e23fff640 1 --2- 172.21.15.72:0/2576491709 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8e240d12b0 0x7f8e241f79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:02.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.128+0000 7f8e1bfff640 1 -- 172.21.15.72:0/2576491709 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8e241ee640 msgr2=0x7f8e2414a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:02.133 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.128+0000 7f8e1bfff640 1 --2- 172.21.15.72:0/2576491709 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8e241ee640 0x7f8e2414a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:02.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.128+0000 7f8e1bfff640 1 -- 172.21.15.72:0/2576491709 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8e240d12b0 msgr2=0x7f8e241f79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:02.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.128+0000 7f8e1bfff640 1 --2- 172.21.15.72:0/2576491709 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8e240d12b0 0x7f8e241f79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:02.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.128+0000 7f8e1bfff640 1 -- 172.21.15.72:0/2576491709 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e2415a740 con 0x7f8e24161e40 2024-04-15T15:17:02.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.128+0000 7f8e28ab9640 1 --2- 172.21.15.72:0/2576491709 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8e241ee640 0x7f8e2414a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:02.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.128+0000 7f8e1bfff640 1 --2- 172.21.15.72:0/2576491709 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e24161e40 0x7f8e241f7ee0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c00b4d0 tx=0x7f8e0c00b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:02.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.128+0000 7f8e217fa640 1 -- 172.21.15.72:0/2576491709 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e0c014070 con 0x7f8e24161e40 2024-04-15T15:17:02.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.128+0000 7f8e217fa640 1 -- 172.21.15.72:0/2576491709 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e0c010870 con 0x7f8e24161e40 2024-04-15T15:17:02.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.128+0000 7f8e2a51c640 1 -- 172.21.15.72:0/2576491709 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e2415aa20 con 0x7f8e24161e40 2024-04-15T15:17:02.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.128+0000 7f8e217fa640 1 -- 172.21.15.72:0/2576491709 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e0c010c60 con 0x7f8e24161e40 2024-04-15T15:17:02.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.129+0000 7f8e2a51c640 1 -- 172.21.15.72:0/2576491709 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e2415afe0 con 0x7f8e24161e40 2024-04-15T15:17:02.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.129+0000 7f8e2a51c640 1 -- 172.21.15.72:0/2576491709 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e240689e0 con 0x7f8e24161e40 2024-04-15T15:17:02.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.130+0000 7f8e217fa640 1 -- 172.21.15.72:0/2576491709 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f8e0c017070 con 0x7f8e24161e40 2024-04-15T15:17:02.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.130+0000 7f8e217fa640 1 --2- 172.21.15.72:0/2576491709 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8e0406c980 0x7f8e0406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:02.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.130+0000 7f8e217fa640 1 -- 172.21.15.72:0/2576491709 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5421+0+0 (secure 0 0 0) 0x7f8e0c08b8a0 con 0x7f8e24161e40 2024-04-15T15:17:02.134 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.130+0000 7f8e23fff640 1 --2- 172.21.15.72:0/2576491709 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8e0406c980 0x7f8e0406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:02.135 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.131+0000 7f8e23fff640 1 --2- 172.21.15.72:0/2576491709 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8e0406c980 0x7f8e0406ee40 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f8e14036000 tx=0x7f8e1403d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:02.137 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.135+0000 7f8e217fa640 1 -- 172.21.15.72:0/2576491709 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8e0c059e00 con 0x7f8e24161e40 2024-04-15T15:17:02.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:02.375+0000 7f8e2a51c640 1 -- 172.21.15.72:0/2576491709 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_blocked"]} v 0) -- 0x7f8e2406c220 con 0x7f8e24161e40 2024-04-15T15:17:03.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:03.077+0000 7f8e217fa640 1 -- 172.21.15.72:0/2576491709 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_blocked"]}]=0 v145) ==== 137+0+0 (secure 0 0 0) 0x7f8e0c059700 con 0x7f8e24161e40 2024-04-15T15:17:03.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:03.079+0000 7f8e2a51c640 1 -- 172.21.15.72:0/2576491709 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8e0406c980 msgr2=0x7f8e0406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:03.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:03.079+0000 7f8e2a51c640 1 --2- 172.21.15.72:0/2576491709 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8e0406c980 0x7f8e0406ee40 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f8e14036000 tx=0x7f8e1403d040 comp rx=0 tx=0).stop 2024-04-15T15:17:03.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:03.079+0000 7f8e2a51c640 1 -- 172.21.15.72:0/2576491709 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e24161e40 msgr2=0x7f8e241f7ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:03.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:03.079+0000 7f8e2a51c640 1 --2- 172.21.15.72:0/2576491709 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e24161e40 0x7f8e241f7ee0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c00b4d0 tx=0x7f8e0c00b990 comp rx=0 tx=0).stop 2024-04-15T15:17:03.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:03.079+0000 7f8e2a51c640 1 -- 172.21.15.72:0/2576491709 shutdown_connections 2024-04-15T15:17:03.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:03.079+0000 7f8e2a51c640 1 --2- 172.21.15.72:0/2576491709 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8e0406c980 0x7f8e0406ee40 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:03.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:03.079+0000 7f8e2a51c640 1 --2- 172.21.15.72:0/2576491709 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8e241ee640 0x7f8e2414a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:03.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:03.079+0000 7f8e2a51c640 1 --2- 172.21.15.72:0/2576491709 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e24161e40 0x7f8e241f7ee0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:03.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:03.079+0000 7f8e2a51c640 1 --2- 172.21.15.72:0/2576491709 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8e240d12b0 0x7f8e241f79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:03.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:03.079+0000 7f8e2a51c640 1 -- 172.21.15.72:0/2576491709 >> 172.21.15.72:0/2576491709 conn(0x7f8e2409d970 msgr2=0x7f8e240b46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:03.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:03.079+0000 7f8e2a51c640 1 -- 172.21.15.72:0/2576491709 shutdown_connections 2024-04-15T15:17:03.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:03.079+0000 7f8e2a51c640 1 -- 172.21.15.72:0/2576491709 wait complete. 2024-04-15T15:17:03.095 INFO:tasks.cephfs_test_runner:That a file with ceph.quiesce.block is not quiesced. ... FAIL 2024-04-15T15:17:03.096 INFO:tasks.cephfs_test_runner: 2024-04-15T15:17:03.096 INFO:tasks.cephfs_test_runner:====================================================================== 2024-04-15T15:17:03.096 INFO:tasks.cephfs_test_runner:FAIL: test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce) 2024-04-15T15:17:03.096 INFO:tasks.cephfs_test_runner:That a file with ceph.quiesce.block is not quiesced. 2024-04-15T15:17:03.096 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-15T15:17:03.096 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-04-15T15:17:03.096 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks/cephfs/test_quiesce.py", line 541, in test_quiesce_blocked 2024-04-15T15:17:03.096 INFO:tasks.cephfs_test_runner: self._verify_quiesce(root=self.subvolume) 2024-04-15T15:17:03.096 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks/cephfs/test_quiesce.py", line 138, in _verify_quiesce 2024-04-15T15:17:03.096 INFO:tasks.cephfs_test_runner: return self._verify_quiesce_wrapped(rank, status, root, root_inode, ops, cache, splitauth) 2024-04-15T15:17:03.096 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks/cephfs/test_quiesce.py", line 272, in _verify_quiesce_wrapped 2024-04-15T15:17:03.096 INFO:tasks.cephfs_test_runner: self.assertFalse(lock_type.startswith("i")) 2024-04-15T15:17:03.096 INFO:tasks.cephfs_test_runner:AssertionError: True is not false 2024-04-15T15:17:03.096 INFO:tasks.cephfs_test_runner: 2024-04-15T15:17:03.097 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-15T15:17:03.097 INFO:tasks.cephfs_test_runner:Ran 1 test in 89.099s 2024-04-15T15:17:03.097 INFO:tasks.cephfs_test_runner: 2024-04-15T15:17:03.097 INFO:tasks.cephfs_test_runner:FAILED (failures=1) 2024-04-15T15:17:03.097 INFO:tasks.cephfs_test_runner: 2024-04-15T15:17:03.097 INFO:tasks.cephfs_test_runner:====================================================================== 2024-04-15T15:17:03.097 INFO:tasks.cephfs_test_runner:FAIL: test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce) 2024-04-15T15:17:03.097 INFO:tasks.cephfs_test_runner:That a file with ceph.quiesce.block is not quiesced. 2024-04-15T15:17:03.097 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-15T15:17:03.097 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-04-15T15:17:03.097 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks/cephfs/test_quiesce.py", line 541, in test_quiesce_blocked 2024-04-15T15:17:03.097 INFO:tasks.cephfs_test_runner: self._verify_quiesce(root=self.subvolume) 2024-04-15T15:17:03.097 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks/cephfs/test_quiesce.py", line 138, in _verify_quiesce 2024-04-15T15:17:03.097 INFO:tasks.cephfs_test_runner: return self._verify_quiesce_wrapped(rank, status, root, root_inode, ops, cache, splitauth) 2024-04-15T15:17:03.097 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks/cephfs/test_quiesce.py", line 272, in _verify_quiesce_wrapped 2024-04-15T15:17:03.097 INFO:tasks.cephfs_test_runner: self.assertFalse(lock_type.startswith("i")) 2024-04-15T15:17:03.097 INFO:tasks.cephfs_test_runner:AssertionError: True is not false 2024-04-15T15:17:03.098 INFO:tasks.cephfs_test_runner: 2024-04-15T15:17:03.098 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce) 2024-04-15T15:17:03.393 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=5fffb08842ea475c99a1ecd5351bc571 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce) 2024-04-15T15:17:03.395 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-04-15T15:17:03.406 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-04-15T15:17:03.427 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-04-15T15:17:03.428 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:17:03.428 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:17:03.451 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16951 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248808k,nr_inodes=812202,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-15T15:17:03.452 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-04-15T15:17:03.452 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:17:03.510 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056329,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16951 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248808k,nr_inodes=812202,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497624k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-15T15:17:03.511 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:17:03.511 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:17:03.532 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20056 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-15T15:17:03.532 DEBUG:teuthology.orchestra.run.smithi175:> set -ex 2024-04-15T15:17:03.532 DEBUG:teuthology.orchestra.run.smithi175:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:17:03.590 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20056 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-15T15:17:03.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:03.590 DEBUG:teuthology.orchestra.run.smithi175:> ip netns list 2024-04-15T15:17:03.650 INFO:teuthology.orchestra.run.smithi175.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-15T15:17:03.650 INFO:teuthology.orchestra.run.smithi175.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-15T15:17:03.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:03.651 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:17:03.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:03.725 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:17:03.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:03.796 DEBUG:teuthology.orchestra.run.smithi175:> sudo ip link delete ceph-brx 2024-04-15T15:17:03.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:03.902 DEBUG:teuthology.orchestra.run.smithi072:> ip netns list 2024-04-15T15:17:03.921 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-15T15:17:03.968 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-15T15:17:03.968 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:03.968 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:17:04.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:04.003 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:17:04.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:04.075 DEBUG:teuthology.orchestra.run.smithi072:> sudo ip link delete ceph-brx 2024-04-15T15:17:04.168 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-15T15:17:04.178 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-15T15:17:04.178 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:17:04.283 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.280+0000 7fd548f04640 1 Processor -- start 2024-04-15T15:17:04.284 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.280+0000 7fd548f04640 1 -- start start 2024-04-15T15:17:04.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.281+0000 7fd548f04640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd5440af400 0x7fd5440ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.281+0000 7fd548f04640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd5440af860 0x7fd5440cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.281+0000 7fd548f04640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd54406f8f0 con 0x7fd54415eba0 2024-04-15T15:17:04.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.281+0000 7fd548f04640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fd54415ef80 con 0x7fd5440af860 2024-04-15T15:17:04.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.281+0000 7fd548f04640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fd5440ce2b0 con 0x7fd5440af400 2024-04-15T15:17:04.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.282+0000 7fd542d9d640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fd5440af860 0x7fd5440cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.282+0000 7fd541d9b640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd5440af400 0x7fd5440ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.282+0000 7fd54259c640 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7fd54415eba0 0x7fd54406f1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.72:39126/0 (socket says 172.21.15.72:39126) 2024-04-15T15:17:04.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.282+0000 7fd54259c640 1 -- 172.21.15.72:0/3010974361 learned_addr learned my addr 172.21.15.72:0/3010974361 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:17:04.287 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.282+0000 7fd541d9b640 1 -- 172.21.15.72:0/3010974361 >> v2:172.21.15.175:3300/0 conn(0x7fd5440af860 msgr2=0x7fd5440cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:04.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.282+0000 7fd541d9b640 1 --2- 172.21.15.72:0/3010974361 >> v2:172.21.15.175:3300/0 conn(0x7fd5440af860 0x7fd5440cd0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.282+0000 7fd541d9b640 1 -- 172.21.15.72:0/3010974361 >> v1:172.21.15.175:6789/0 conn(0x7fd54415eba0 legacy=0x7fd54406f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.282+0000 7fd541d9b640 1 -- 172.21.15.72:0/3010974361 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5440ce590 con 0x7fd5440af400 2024-04-15T15:17:04.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.282+0000 7fd542d9d640 1 --2- 172.21.15.72:0/3010974361 >> v2:172.21.15.175:3300/0 conn(0x7fd5440af860 0x7fd5440cd0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:17:04.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.283+0000 7fd541d9b640 1 --2- 172.21.15.72:0/3010974361 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd5440af400 0x7fd5440ccb90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd52c0097a0 tx=0x7fd52c02b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf9711ddcfd32999 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:04.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.283+0000 7fd54159a640 1 -- 172.21.15.72:0/3010974361 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd52c031780 con 0x7fd5440af400 2024-04-15T15:17:04.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.283+0000 7fd54159a640 1 -- 172.21.15.72:0/3010974361 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd52c037070 con 0x7fd5440af400 2024-04-15T15:17:04.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.283+0000 7fd54159a640 1 -- 172.21.15.72:0/3010974361 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd52c031b40 con 0x7fd5440af400 2024-04-15T15:17:04.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.284+0000 7fd548f04640 1 -- 172.21.15.72:0/3010974361 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd5440af400 msgr2=0x7fd5440ccb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.284+0000 7fd548f04640 1 --2- 172.21.15.72:0/3010974361 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd5440af400 0x7fd5440ccb90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd52c0097a0 tx=0x7fd52c02b820 comp rx=0 tx=0).stop 2024-04-15T15:17:04.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.284+0000 7fd548f04640 1 -- 172.21.15.72:0/3010974361 shutdown_connections 2024-04-15T15:17:04.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.284+0000 7fd548f04640 1 --2- 172.21.15.72:0/3010974361 >> v2:172.21.15.175:3300/0 conn(0x7fd5440af860 0x7fd5440cd0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.284+0000 7fd548f04640 1 --2- 172.21.15.72:0/3010974361 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd5440af400 0x7fd5440ccb90 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.284+0000 7fd548f04640 1 -- 172.21.15.72:0/3010974361 >> 172.21.15.72:0/3010974361 conn(0x7fd5440b1dd0 msgr2=0x7fd5440b21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:04.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.284+0000 7fd548f04640 1 -- 172.21.15.72:0/3010974361 shutdown_connections 2024-04-15T15:17:04.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.284+0000 7fd548f04640 1 -- 172.21.15.72:0/3010974361 wait complete. 2024-04-15T15:17:04.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.285+0000 7fd548f04640 1 Processor -- start 2024-04-15T15:17:04.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.285+0000 7fd548f04640 1 -- start start 2024-04-15T15:17:04.292 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.286+0000 7fd548f04640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5440af400 0x7fd5441f17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.286+0000 7fd548f04640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd5440af860 0x7fd5441f54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.286+0000 7fd548f04640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd54415eba0 0x7fd5441f5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.286+0000 7fd548f04640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fd5440b2b80 con 0x7fd54415eba0 2024-04-15T15:17:04.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.286+0000 7fd548f04640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fd54406ecc0 con 0x7fd5440af400 2024-04-15T15:17:04.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.286+0000 7fd548f04640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fd5440ceb50 con 0x7fd5440af860 2024-04-15T15:17:04.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.286+0000 7fd542d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd54415eba0 0x7fd5441f5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.286+0000 7fd542d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd54415eba0 0x7fd5441f5a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36694/0 (socket says 172.21.15.72:36694) 2024-04-15T15:17:04.293 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.286+0000 7fd542d9d640 1 -- 172.21.15.72:0/3978708987 learned_addr learned my addr 172.21.15.72:0/3978708987 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:04.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.286+0000 7fd54259c640 1 --2- 172.21.15.72:0/3978708987 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5440af400 0x7fd5441f17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.286+0000 7fd541d9b640 1 --2- 172.21.15.72:0/3978708987 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd5440af860 0x7fd5441f54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.286+0000 7fd542d9d640 1 -- 172.21.15.72:0/3978708987 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd5440af860 msgr2=0x7fd5441f54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.286+0000 7fd542d9d640 1 --2- 172.21.15.72:0/3978708987 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd5440af860 0x7fd5441f54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.286+0000 7fd542d9d640 1 -- 172.21.15.72:0/3978708987 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5440af400 msgr2=0x7fd5441f17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.286+0000 7fd542d9d640 1 --2- 172.21.15.72:0/3978708987 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5440af400 0x7fd5441f17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.294 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.286+0000 7fd542d9d640 1 -- 172.21.15.72:0/3978708987 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd54413f5b0 con 0x7fd54415eba0 2024-04-15T15:17:04.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.287+0000 7fd542d9d640 1 --2- 172.21.15.72:0/3978708987 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd54415eba0 0x7fd5441f5a20 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fd53c00e880 tx=0x7fd53c00ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:04.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.287+0000 7fd53affd640 1 -- 172.21.15.72:0/3978708987 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53c017070 con 0x7fd54415eba0 2024-04-15T15:17:04.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.287+0000 7fd53affd640 1 -- 172.21.15.72:0/3978708987 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd53c004020 con 0x7fd54415eba0 2024-04-15T15:17:04.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.287+0000 7fd548f04640 1 -- 172.21.15.72:0/3978708987 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd54413f890 con 0x7fd54415eba0 2024-04-15T15:17:04.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.287+0000 7fd53affd640 1 -- 172.21.15.72:0/3978708987 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53c0043f0 con 0x7fd54415eba0 2024-04-15T15:17:04.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.287+0000 7fd548f04640 1 -- 172.21.15.72:0/3978708987 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5441f1d30 con 0x7fd54415eba0 2024-04-15T15:17:04.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.288+0000 7fd548f04640 1 -- 172.21.15.72:0/3978708987 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5440689e0 con 0x7fd54415eba0 2024-04-15T15:17:04.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.288+0000 7fd53affd640 1 -- 172.21.15.72:0/3978708987 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fd53c00c940 con 0x7fd54415eba0 2024-04-15T15:17:04.295 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.289+0000 7fd53affd640 1 --2- 172.21.15.72:0/3978708987 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd52406c9b0 0x7fd52406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.289+0000 7fd53affd640 1 -- 172.21.15.72:0/3978708987 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5421+0+0 (secure 0 0 0) 0x7fd53c08bab0 con 0x7fd54415eba0 2024-04-15T15:17:04.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.289+0000 7fd54259c640 1 --2- 172.21.15.72:0/3978708987 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd52406c9b0 0x7fd52406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.289+0000 7fd54259c640 1 --2- 172.21.15.72:0/3978708987 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd52406c9b0 0x7fd52406ee70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd544151770 tx=0x7fd534001a30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:04.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.294+0000 7fd53affd640 1 -- 172.21.15.72:0/3978708987 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd53c059a50 con 0x7fd54415eba0 2024-04-15T15:17:04.471 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.468+0000 7fd548f04640 1 -- 172.21.15.72:0/3978708987 --> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd544074e30 con 0x7fd52406c9b0 2024-04-15T15:17:04.472 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.469+0000 7fd53affd640 1 -- 172.21.15.72:0/3978708987 <== mgr.4101 v2:172.21.15.72:6832/3503807805 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13384 (secure 0 0 0) 0x7fd544074e30 con 0x7fd52406c9b0 2024-04-15T15:17:04.472 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:17:04.472 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-04-15T15:17:04.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.472+0000 7fd548f04640 1 -- 172.21.15.72:0/3978708987 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd52406c9b0 msgr2=0x7fd52406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.472+0000 7fd548f04640 1 --2- 172.21.15.72:0/3978708987 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd52406c9b0 0x7fd52406ee70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fd544151770 tx=0x7fd534001a30 comp rx=0 tx=0).stop 2024-04-15T15:17:04.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.472+0000 7fd548f04640 1 -- 172.21.15.72:0/3978708987 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd54415eba0 msgr2=0x7fd5441f5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.472+0000 7fd548f04640 1 --2- 172.21.15.72:0/3978708987 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd54415eba0 0x7fd5441f5a20 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fd53c00e880 tx=0x7fd53c00ed40 comp rx=0 tx=0).stop 2024-04-15T15:17:04.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.472+0000 7fd548f04640 1 -- 172.21.15.72:0/3978708987 shutdown_connections 2024-04-15T15:17:04.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.472+0000 7fd548f04640 1 --2- 172.21.15.72:0/3978708987 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fd52406c9b0 0x7fd52406ee70 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.472+0000 7fd548f04640 1 --2- 172.21.15.72:0/3978708987 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd54415eba0 0x7fd5441f5a20 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.472+0000 7fd548f04640 1 --2- 172.21.15.72:0/3978708987 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fd5440af860 0x7fd5441f54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.472+0000 7fd548f04640 1 --2- 172.21.15.72:0/3978708987 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fd5440af400 0x7fd5441f17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.472+0000 7fd548f04640 1 -- 172.21.15.72:0/3978708987 >> 172.21.15.72:0/3978708987 conn(0x7fd5440b1dd0 msgr2=0x7fd54415c960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:04.475 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.472+0000 7fd548f04640 1 -- 172.21.15.72:0/3978708987 shutdown_connections 2024-04-15T15:17:04.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.472+0000 7fd548f04640 1 -- 172.21.15.72:0/3978708987 wait complete. 2024-04-15T15:17:04.485 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":109,"stamp":"2024-04-15T15:17:03.369946+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221272,"kb_used_data":5116,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753448,"statfs":{"total":773094113280,"available":772867530752,"internally_reserved":0,"allocated":5238784,"data_stored":3011972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-62067,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":-79,"num_read_kb":-129,"num_write":-91,"num_write_kb":-95,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000916"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":211,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-15T15:16:58.321347+0000","last_change":"2024-04-15T15:14:40.494279+0000","last_active":"2024-04-15T15:16:58.321347+0000","last_peered":"2024-04-15T15:16:58.321347+0000","last_clean":"2024-04-15T15:16:58.321347+0000","last_became_active":"2024-04-15T15:14:40.493404+0000","last_became_peered":"2024-04-15T15:14:40.493404+0000","last_unstale":"2024-04-15T15:16:58.321347+0000","last_undegraded":"2024-04-15T15:16:58.321347+0000","last_fullsized":"2024-04-15T15:16:58.321347+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:39.418508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:39.418508+0000","last_clean_scrub_stamp":"2024-04-15T15:14:39.418508+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:01:51.789246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574880,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28152,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343688,"statfs":{"total":96636764160,"available":96607936512,"internally_reserved":0,"allocated":1159168,"data_stored":867671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574880,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27504,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344336,"statfs":{"total":96636764160,"available":96608600064,"internally_reserved":0,"allocated":495616,"data_stored":216538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574879,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27504,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344336,"statfs":{"total":96636764160,"available":96608600064,"internally_reserved":0,"allocated":495616,"data_stored":215935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574879,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27492,"kb_used_data":472,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344348,"statfs":{"total":96636764160,"available":96608612352,"internally_reserved":0,"allocated":483328,"data_stored":209839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574880,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27568,"kb_used_data":548,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344272,"statfs":{"total":96636764160,"available":96608534528,"internally_reserved":0,"allocated":561152,"data_stored":277134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574879,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27484,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344356,"statfs":{"total":96636764160,"available":96608620544,"internally_reserved":0,"allocated":479232,"data_stored":209305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574880,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27500,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344340,"statfs":{"total":96636764160,"available":96608604160,"internally_reserved":0,"allocated":491520,"data_stored":215913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574878,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28068,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343772,"statfs":{"total":96636764160,"available":96608022528,"internally_reserved":0,"allocated":1073152,"data_stored":799637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:17:04.485 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:17:04.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.586+0000 7fb1f3d3d640 1 Processor -- start 2024-04-15T15:17:04.589 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.586+0000 7fb1f3d3d640 1 -- start start 2024-04-15T15:17:04.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.587+0000 7fb1f3d3d640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb1ec0d12b0 0x7fb1ec0d2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.587+0000 7fb1f3d3d640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fb1ec0d2ea0 0x7fb1ec0d45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.587+0000 7fb1f3d3d640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1ec162220 con 0x7fb1ec161e40 2024-04-15T15:17:04.594 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.587+0000 7fb1f3d3d640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fb1ec0b3f00 con 0x7fb1ec0d2ea0 2024-04-15T15:17:04.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.587+0000 7fb1f3d3d640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fb1ec0d21f0 con 0x7fb1ec0d12b0 2024-04-15T15:17:04.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.587+0000 7fb1f1ad9640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fb1ec161e40 0x7fb1ec0b37f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:33992/0 (socket says 172.21.15.72:33992) 2024-04-15T15:17:04.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.587+0000 7fb1f1ad9640 1 -- 172.21.15.72:0/748932011 learned_addr learned my addr 172.21.15.72:0/748932011 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:17:04.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.588+0000 7fb1f12d8640 1 --2- 172.21.15.72:0/748932011 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb1ec0d12b0 0x7fb1ec0d2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.588+0000 7fb1f0ad7640 1 -- 172.21.15.72:0/748932011 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 562074214 0 0) 0x7fb1ec162220 con 0x7fb1ec161e40 2024-04-15T15:17:04.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.588+0000 7fb1f22da640 1 --2- 172.21.15.72:0/748932011 >> v2:172.21.15.175:3300/0 conn(0x7fb1ec0d2ea0 0x7fb1ec0d45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.588+0000 7fb1f0ad7640 1 -- 172.21.15.72:0/748932011 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1d4003520 con 0x7fb1ec161e40 2024-04-15T15:17:04.595 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.588+0000 7fb1f0ad7640 1 -- 172.21.15.72:0/748932011 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 620461764 0 0) 0x7fb1d4003520 con 0x7fb1ec161e40 2024-04-15T15:17:04.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.588+0000 7fb1f0ad7640 1 -- 172.21.15.72:0/748932011 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb1ec0d12b0 msgr2=0x7fb1ec0d2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:04.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.588+0000 7fb1f0ad7640 1 --2- 172.21.15.72:0/748932011 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb1ec0d12b0 0x7fb1ec0d2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.588+0000 7fb1f0ad7640 1 -- 172.21.15.72:0/748932011 >> v2:172.21.15.175:3300/0 conn(0x7fb1ec0d2ea0 msgr2=0x7fb1ec0d45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:04.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.588+0000 7fb1f0ad7640 1 --2- 172.21.15.72:0/748932011 >> v2:172.21.15.175:3300/0 conn(0x7fb1ec0d2ea0 0x7fb1ec0d45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.588+0000 7fb1f0ad7640 1 -- 172.21.15.72:0/748932011 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1ec0d1820 con 0x7fb1ec161e40 2024-04-15T15:17:04.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.588+0000 7fb1f0ad7640 1 -- 172.21.15.72:0/748932011 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7fb1dc0019f0 con 0x7fb1ec161e40 2024-04-15T15:17:04.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.588+0000 7fb1f0ad7640 1 -- 172.21.15.72:0/748932011 >> v1:172.21.15.72:6789/0 conn(0x7fb1ec161e40 legacy=0x7fb1ec0b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.596 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.588+0000 7fb1f0ad7640 1 --2- 172.21.15.72:0/748932011 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1d40039b0 0x7fb1d4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.588+0000 7fb1f0ad7640 1 --2- 172.21.15.72:0/748932011 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb1ec0d12b0 0x7fb1d4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.588+0000 7fb1f0ad7640 1 --2- 172.21.15.72:0/748932011 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1d4004d40 0x7fb1d40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.588+0000 7fb1f0ad7640 1 -- 172.21.15.72:0/748932011 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ec0b3f00 con 0x7fb1d40039b0 2024-04-15T15:17:04.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.588+0000 7fb1f0ad7640 1 -- 172.21.15.72:0/748932011 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ec0d21f0 con 0x7fb1d4004d40 2024-04-15T15:17:04.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.588+0000 7fb1f0ad7640 1 -- 172.21.15.72:0/748932011 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fb1d400b9f0 con 0x7fb1ec0d12b0 2024-04-15T15:17:04.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.588+0000 7fb1f1ad9640 1 --2- 172.21.15.72:0/748932011 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1d40039b0 0x7fb1d4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.588+0000 7fb1f12d8640 1 --2- 172.21.15.72:0/748932011 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb1ec0d12b0 0x7fb1d4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.597 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.588+0000 7fb1f22da640 1 --2- 172.21.15.72:0/748932011 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1d4004d40 0x7fb1d40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.588+0000 7fb1f1ad9640 1 -- 172.21.15.72:0/748932011 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb1ec0d12b0 msgr2=0x7fb1d4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:04.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.588+0000 7fb1f1ad9640 1 --2- 172.21.15.72:0/748932011 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb1ec0d12b0 0x7fb1d4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.588+0000 7fb1f1ad9640 1 -- 172.21.15.72:0/748932011 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1d4004d40 msgr2=0x7fb1d40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:04.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.588+0000 7fb1f1ad9640 1 --2- 172.21.15.72:0/748932011 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1d4004d40 0x7fb1d40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.588+0000 7fb1f1ad9640 1 -- 172.21.15.72:0/748932011 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1d4003520 con 0x7fb1d40039b0 2024-04-15T15:17:04.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.589+0000 7fb1f1ad9640 1 --2- 172.21.15.72:0/748932011 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1d40039b0 0x7fb1d4003dd0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc00a8a0 tx=0x7fb1d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7671b7d76df42c88 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:04.598 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.589+0000 7fb1f0ad7640 1 -- 172.21.15.72:0/748932011 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc00a4c0 con 0x7fb1d40039b0 2024-04-15T15:17:04.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.589+0000 7fb1f0ad7640 1 -- 172.21.15.72:0/748932011 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1dc034650 con 0x7fb1d40039b0 2024-04-15T15:17:04.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.589+0000 7fb1f0ad7640 1 -- 172.21.15.72:0/748932011 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc034a40 con 0x7fb1d40039b0 2024-04-15T15:17:04.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.590+0000 7fb1f3d3d640 1 -- 172.21.15.72:0/748932011 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1d40039b0 msgr2=0x7fb1d4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.590+0000 7fb1f3d3d640 1 --2- 172.21.15.72:0/748932011 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1d40039b0 0x7fb1d4003dd0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc00a8a0 tx=0x7fb1d4005830 comp rx=0 tx=0).stop 2024-04-15T15:17:04.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.590+0000 7fb1f22da640 1 -- 172.21.15.72:0/748932011 reap_dead start 2024-04-15T15:17:04.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.590+0000 7fb1f3d3d640 1 -- 172.21.15.72:0/748932011 shutdown_connections 2024-04-15T15:17:04.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.590+0000 7fb1f3d3d640 1 -- 172.21.15.72:0/748932011 >> 172.21.15.72:0/748932011 conn(0x7fb1ec09d970 msgr2=0x7fb1ec0b95e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:04.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.590+0000 7fb1f3d3d640 1 -- 172.21.15.72:0/748932011 shutdown_connections 2024-04-15T15:17:04.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.590+0000 7fb1f3d3d640 1 -- 172.21.15.72:0/748932011 wait complete. 2024-04-15T15:17:04.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.591+0000 7fb1f3d3d640 1 Processor -- start 2024-04-15T15:17:04.600 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.591+0000 7fb1f3d3d640 1 -- start start 2024-04-15T15:17:04.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.592+0000 7fb1f3d3d640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb1d40039b0 0x7fb1ec0ca1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.592+0000 7fb1f3d3d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1ec0b5f50 0x7fb1ec0b63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.592+0000 7fb1f3d3d640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1ec0b4080 0x7fb1ec0b4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.592+0000 7fb1f3d3d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ec0d2440 con 0x7fb1ec0b5f50 2024-04-15T15:17:04.603 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.592+0000 7fb1f3d3d640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fb1ec0d1de0 con 0x7fb1ec0b4080 2024-04-15T15:17:04.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.592+0000 7fb1f3d3d640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fb1ec0d2960 con 0x7fb1d40039b0 2024-04-15T15:17:04.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.592+0000 7fb1f12d8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1ec0b5f50 0x7fb1ec0b63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.592+0000 7fb1f12d8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1ec0b5f50 0x7fb1ec0b63d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36714/0 (socket says 172.21.15.72:36714) 2024-04-15T15:17:04.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.592+0000 7fb1f12d8640 1 -- 172.21.15.72:0/2019058200 learned_addr learned my addr 172.21.15.72:0/2019058200 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:04.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.592+0000 7fb1f22da640 1 --2- 172.21.15.72:0/2019058200 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1ec0b4080 0x7fb1ec0b4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.604 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.592+0000 7fb1f1ad9640 1 --2- 172.21.15.72:0/2019058200 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb1d40039b0 0x7fb1ec0ca1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.592+0000 7fb1f12d8640 1 -- 172.21.15.72:0/2019058200 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb1d40039b0 msgr2=0x7fb1ec0ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.592+0000 7fb1f12d8640 1 --2- 172.21.15.72:0/2019058200 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb1d40039b0 0x7fb1ec0ca1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.592+0000 7fb1f12d8640 1 -- 172.21.15.72:0/2019058200 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1ec0b4080 msgr2=0x7fb1ec0b4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.592+0000 7fb1f12d8640 1 --2- 172.21.15.72:0/2019058200 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1ec0b4080 0x7fb1ec0b4500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.592+0000 7fb1f12d8640 1 -- 172.21.15.72:0/2019058200 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1ec0b6b10 con 0x7fb1ec0b5f50 2024-04-15T15:17:04.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.592+0000 7fb1f12d8640 1 --2- 172.21.15.72:0/2019058200 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1ec0b5f50 0x7fb1ec0b63d0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fb1e00119a0 tx=0x7fb1e0011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:04.605 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.592+0000 7fb1ea7fc640 1 -- 172.21.15.72:0/2019058200 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e000faf0 con 0x7fb1ec0b5f50 2024-04-15T15:17:04.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.592+0000 7fb1ea7fc640 1 -- 172.21.15.72:0/2019058200 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1e000fc90 con 0x7fb1ec0b5f50 2024-04-15T15:17:04.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.593+0000 7fb1f3d3d640 1 -- 172.21.15.72:0/2019058200 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1ec0b6df0 con 0x7fb1ec0b5f50 2024-04-15T15:17:04.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.593+0000 7fb1ea7fc640 1 -- 172.21.15.72:0/2019058200 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1e0014440 con 0x7fb1ec0b5f50 2024-04-15T15:17:04.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.593+0000 7fb1f3d3d640 1 -- 172.21.15.72:0/2019058200 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1ec1e9540 con 0x7fb1ec0b5f50 2024-04-15T15:17:04.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.593+0000 7fb1f3d3d640 1 -- 172.21.15.72:0/2019058200 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1ec0b6fc0 con 0x7fb1ec0b5f50 2024-04-15T15:17:04.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.594+0000 7fb1ea7fc640 1 -- 172.21.15.72:0/2019058200 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fb1e0012040 con 0x7fb1ec0b5f50 2024-04-15T15:17:04.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.594+0000 7fb1ea7fc640 1 --2- 172.21.15.72:0/2019058200 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb1d806c9b0 0x7fb1d806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.606 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.594+0000 7fb1ea7fc640 1 -- 172.21.15.72:0/2019058200 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5421+0+0 (secure 0 0 0) 0x7fb1e008c120 con 0x7fb1ec0b5f50 2024-04-15T15:17:04.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.594+0000 7fb1f1ad9640 1 --2- 172.21.15.72:0/2019058200 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb1d806c9b0 0x7fb1d806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.595+0000 7fb1f1ad9640 1 --2- 172.21.15.72:0/2019058200 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb1d806c9b0 0x7fb1d806ee70 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc034000 tx=0x7fb1dc009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:04.607 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.600+0000 7fb1ea7fc640 1 -- 172.21.15.72:0/2019058200 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb1e0019030 con 0x7fb1ec0b5f50 2024-04-15T15:17:04.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.773+0000 7fb1f3d3d640 1 -- 172.21.15.72:0/2019058200 --> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb1ec1e9050 con 0x7fb1d806c9b0 2024-04-15T15:17:04.776 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.774+0000 7fb1ea7fc640 1 -- 172.21.15.72:0/2019058200 <== mgr.4101 v2:172.21.15.72:6832/3503807805 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13384 (secure 0 0 0) 0x7fb1ec1e9050 con 0x7fb1d806c9b0 2024-04-15T15:17:04.777 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:17:04.777 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-04-15T15:17:04.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.776+0000 7fb1f3d3d640 1 -- 172.21.15.72:0/2019058200 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb1d806c9b0 msgr2=0x7fb1d806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.776+0000 7fb1f3d3d640 1 --2- 172.21.15.72:0/2019058200 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb1d806c9b0 0x7fb1d806ee70 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc034000 tx=0x7fb1dc009030 comp rx=0 tx=0).stop 2024-04-15T15:17:04.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.776+0000 7fb1f3d3d640 1 -- 172.21.15.72:0/2019058200 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1ec0b5f50 msgr2=0x7fb1ec0b63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.776+0000 7fb1f3d3d640 1 --2- 172.21.15.72:0/2019058200 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1ec0b5f50 0x7fb1ec0b63d0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fb1e00119a0 tx=0x7fb1e0011e60 comp rx=0 tx=0).stop 2024-04-15T15:17:04.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.776+0000 7fb1f3d3d640 1 -- 172.21.15.72:0/2019058200 shutdown_connections 2024-04-15T15:17:04.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.776+0000 7fb1f3d3d640 1 --2- 172.21.15.72:0/2019058200 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb1d806c9b0 0x7fb1d806ee70 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.779 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.776+0000 7fb1f3d3d640 1 --2- 172.21.15.72:0/2019058200 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb1ec0b4080 0x7fb1ec0b4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.776+0000 7fb1f3d3d640 1 --2- 172.21.15.72:0/2019058200 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb1ec0b5f50 0x7fb1ec0b63d0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.776+0000 7fb1f3d3d640 1 --2- 172.21.15.72:0/2019058200 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb1d40039b0 0x7fb1ec0ca1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.776+0000 7fb1f3d3d640 1 -- 172.21.15.72:0/2019058200 >> 172.21.15.72:0/2019058200 conn(0x7fb1ec09d970 msgr2=0x7fb1ec0b7dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:04.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.776+0000 7fb1f3d3d640 1 -- 172.21.15.72:0/2019058200 shutdown_connections 2024-04-15T15:17:04.780 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.776+0000 7fb1f3d3d640 1 -- 172.21.15.72:0/2019058200 wait complete. 2024-04-15T15:17:04.790 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":109,"stamp":"2024-04-15T15:17:03.369946+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221272,"kb_used_data":5116,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753448,"statfs":{"total":773094113280,"available":772867530752,"internally_reserved":0,"allocated":5238784,"data_stored":3011972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-62067,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":-79,"num_read_kb":-129,"num_write":-91,"num_write_kb":-95,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000916"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":211,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-15T15:16:58.321347+0000","last_change":"2024-04-15T15:14:40.494279+0000","last_active":"2024-04-15T15:16:58.321347+0000","last_peered":"2024-04-15T15:16:58.321347+0000","last_clean":"2024-04-15T15:16:58.321347+0000","last_became_active":"2024-04-15T15:14:40.493404+0000","last_became_peered":"2024-04-15T15:14:40.493404+0000","last_unstale":"2024-04-15T15:16:58.321347+0000","last_undegraded":"2024-04-15T15:16:58.321347+0000","last_fullsized":"2024-04-15T15:16:58.321347+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:39.418508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:39.418508+0000","last_clean_scrub_stamp":"2024-04-15T15:14:39.418508+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:01:51.789246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574880,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28152,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343688,"statfs":{"total":96636764160,"available":96607936512,"internally_reserved":0,"allocated":1159168,"data_stored":867671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574880,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27504,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344336,"statfs":{"total":96636764160,"available":96608600064,"internally_reserved":0,"allocated":495616,"data_stored":216538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574879,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27504,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344336,"statfs":{"total":96636764160,"available":96608600064,"internally_reserved":0,"allocated":495616,"data_stored":215935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574879,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27492,"kb_used_data":472,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344348,"statfs":{"total":96636764160,"available":96608612352,"internally_reserved":0,"allocated":483328,"data_stored":209839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574880,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27568,"kb_used_data":548,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344272,"statfs":{"total":96636764160,"available":96608534528,"internally_reserved":0,"allocated":561152,"data_stored":277134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574879,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27484,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344356,"statfs":{"total":96636764160,"available":96608620544,"internally_reserved":0,"allocated":479232,"data_stored":209305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574880,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27500,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344340,"statfs":{"total":96636764160,"available":96608604160,"internally_reserved":0,"allocated":491520,"data_stored":215913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574878,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28068,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343772,"statfs":{"total":96636764160,"available":96608022528,"internally_reserved":0,"allocated":1073152,"data_stored":799637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:17:04.791 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-15T15:17:04.791 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:17:04.893 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.891+0000 7f8b0024a640 1 Processor -- start 2024-04-15T15:17:04.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.891+0000 7f8b0024a640 1 -- start start 2024-04-15T15:17:04.903 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.892+0000 7f8b0024a640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f8af815ad10 0x7f8af815b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.892+0000 7f8afd7e5640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f8af80cc5e0 0x7f8af815b630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:34004/0 (socket says 172.21.15.72:34004) 2024-04-15T15:17:04.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.892+0000 7f8b0024a640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8af80cf990 con 0x7f8af80cc5e0 2024-04-15T15:17:04.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.892+0000 7f8afd7e5640 1 -- 172.21.15.72:0/2625898961 learned_addr learned my addr 172.21.15.72:0/2625898961 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:17:04.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.892+0000 7f8b0024a640 1 -- 172.21.15.72:0/2625898961 --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8af80cd4b0 con 0x7f8af80af740 2024-04-15T15:17:04.904 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.892+0000 7f8b0024a640 1 -- 172.21.15.72:0/2625898961 --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f8af80cd790 con 0x7f8af815ad10 2024-04-15T15:17:04.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.892+0000 7f8afdfe6640 1 --2- 172.21.15.72:0/2625898961 >> v2:172.21.15.175:3300/0 conn(0x7f8af815ad10 0x7f8af815b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.892+0000 7f8afcfe4640 1 -- 172.21.15.72:0/2625898961 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2630613982 0 0) 0x7f8af80cf990 con 0x7f8af80cc5e0 2024-04-15T15:17:04.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.892+0000 7f8afcfe4640 1 -- 172.21.15.72:0/2625898961 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ae4003520 con 0x7f8af80cc5e0 2024-04-15T15:17:04.905 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.892+0000 7f8afcfe4640 1 -- 172.21.15.72:0/2625898961 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 118491532 0 0) 0x7f8ae4003520 con 0x7f8af80cc5e0 2024-04-15T15:17:04.906 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.892+0000 7f8afcfe4640 1 -- 172.21.15.72:0/2625898961 >> v2:172.21.15.175:3300/0 conn(0x7f8af815ad10 msgr2=0x7f8af815b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:04.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.893+0000 7f8afcfe4640 1 --2- 172.21.15.72:0/2625898961 >> v2:172.21.15.175:3300/0 conn(0x7f8af815ad10 0x7f8af815b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.893+0000 7f8afcfe4640 1 -- 172.21.15.72:0/2625898961 >> v1:172.21.15.175:6789/0 conn(0x7f8af80af740 legacy=0x7f8af81648a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.893+0000 7f8afcfe4640 1 -- 172.21.15.72:0/2625898961 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8af815bf10 con 0x7f8af80cc5e0 2024-04-15T15:17:04.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.893+0000 7f8afcfe4640 1 -- 172.21.15.72:0/2625898961 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f8ae8001690 con 0x7f8af80cc5e0 2024-04-15T15:17:04.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.893+0000 7f8afdfe6640 1 --2- 172.21.15.72:0/2625898961 >> v2:172.21.15.175:3300/0 conn(0x7f8af815ad10 0x7f8af815b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:17:04.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.893+0000 7f8afcfe4640 1 -- 172.21.15.72:0/2625898961 >> v1:172.21.15.72:6789/0 conn(0x7f8af80cc5e0 legacy=0x7f8af815b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.893+0000 7f8afcfe4640 1 --2- 172.21.15.72:0/2625898961 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8ae40039b0 0x7f8ae4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.907 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.893+0000 7f8afcfe4640 1 --2- 172.21.15.72:0/2625898961 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8ae4004800 0x7f8ae4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.893+0000 7f8afcfe4640 1 --2- 172.21.15.72:0/2625898961 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8ae40051c0 0x7f8ae400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.893+0000 7f8afcfe4640 1 -- 172.21.15.72:0/2625898961 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8af80cd790 con 0x7f8ae4004800 2024-04-15T15:17:04.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.893+0000 7f8afcfe4640 1 -- 172.21.15.72:0/2625898961 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae400c070 con 0x7f8ae40039b0 2024-04-15T15:17:04.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.893+0000 7f8afcfe4640 1 -- 172.21.15.72:0/2625898961 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f8ae400c270 con 0x7f8ae40051c0 2024-04-15T15:17:04.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.893+0000 7f8afd7e5640 1 --2- 172.21.15.72:0/2625898961 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8ae4004800 0x7f8ae4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.893+0000 7f8afdfe6640 1 --2- 172.21.15.72:0/2625898961 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8ae40039b0 0x7f8ae4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.893+0000 7f8afe7e7640 1 --2- 172.21.15.72:0/2625898961 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8ae40051c0 0x7f8ae400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.893+0000 7f8afd7e5640 1 -- 172.21.15.72:0/2625898961 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8ae40051c0 msgr2=0x7f8ae400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:04.908 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.893+0000 7f8afd7e5640 1 --2- 172.21.15.72:0/2625898961 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8ae40051c0 0x7f8ae400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.893+0000 7f8afd7e5640 1 -- 172.21.15.72:0/2625898961 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8ae40039b0 msgr2=0x7f8ae4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:04.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.893+0000 7f8afd7e5640 1 --2- 172.21.15.72:0/2625898961 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8ae40039b0 0x7f8ae4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.893+0000 7f8afd7e5640 1 -- 172.21.15.72:0/2625898961 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ae4003520 con 0x7f8ae4004800 2024-04-15T15:17:04.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.893+0000 7f8afe7e7640 1 -- 172.21.15.72:0/2625898961 reap_dead start 2024-04-15T15:17:04.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.894+0000 7f8afd7e5640 1 --2- 172.21.15.72:0/2625898961 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8ae4004800 0x7f8ae4004c80 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f8ae800c210 tx=0x7f8ae4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f3af96c74af6305 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:04.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.894+0000 7f8afcfe4640 1 -- 172.21.15.72:0/2625898961 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ae8007d10 con 0x7f8ae4004800 2024-04-15T15:17:04.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.894+0000 7f8afcfe4640 1 -- 172.21.15.72:0/2625898961 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ae8030070 con 0x7f8ae4004800 2024-04-15T15:17:04.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.894+0000 7f8afcfe4640 1 -- 172.21.15.72:0/2625898961 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ae8005d70 con 0x7f8ae4004800 2024-04-15T15:17:04.909 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.894+0000 7f8b0024a640 1 -- 172.21.15.72:0/2625898961 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8ae4004800 msgr2=0x7f8ae4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.894+0000 7f8b0024a640 1 --2- 172.21.15.72:0/2625898961 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8ae4004800 0x7f8ae4004c80 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f8ae800c210 tx=0x7f8ae4005830 comp rx=0 tx=0).stop 2024-04-15T15:17:04.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.894+0000 7f8b0024a640 1 -- 172.21.15.72:0/2625898961 shutdown_connections 2024-04-15T15:17:04.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.894+0000 7f8b0024a640 1 --2- 172.21.15.72:0/2625898961 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8ae4004800 0x7f8ae4004c80 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.894+0000 7f8b0024a640 1 -- 172.21.15.72:0/2625898961 >> 172.21.15.72:0/2625898961 conn(0x7f8af80b1dd0 msgr2=0x7f8af80b21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:04.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.895+0000 7f8b0024a640 1 -- 172.21.15.72:0/2625898961 shutdown_connections 2024-04-15T15:17:04.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.895+0000 7f8b0024a640 1 -- 172.21.15.72:0/2625898961 wait complete. 2024-04-15T15:17:04.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.896+0000 7f8b0024a640 1 Processor -- start 2024-04-15T15:17:04.910 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.896+0000 7f8b0024a640 1 -- start start 2024-04-15T15:17:04.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.896+0000 7f8b0024a640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8ae4004800 0x7f8af815a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.896+0000 7f8b0024a640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8af8099780 0x7f8af8072c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.896+0000 7f8b0024a640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8af8099c00 0x7f8af8073140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.896+0000 7f8b0024a640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8af80ce2d0 con 0x7f8af8099c00 2024-04-15T15:17:04.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.896+0000 7f8b0024a640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f8af815c1f0 con 0x7f8af8099780 2024-04-15T15:17:04.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.896+0000 7f8b0024a640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f8af806dc10 con 0x7f8ae4004800 2024-04-15T15:17:04.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.897+0000 7f8afe7e7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8af8099c00 0x7f8af8073140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.897+0000 7f8afe7e7640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8af8099c00 0x7f8af8073140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36730/0 (socket says 172.21.15.72:36730) 2024-04-15T15:17:04.911 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.897+0000 7f8afe7e7640 1 -- 172.21.15.72:0/405382076 learned_addr learned my addr 172.21.15.72:0/405382076 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:04.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.897+0000 7f8afdfe6640 1 --2- 172.21.15.72:0/405382076 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8ae4004800 0x7f8af815a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.897+0000 7f8afe7e7640 1 -- 172.21.15.72:0/405382076 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8ae4004800 msgr2=0x7f8af815a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.897+0000 7f8afe7e7640 1 --2- 172.21.15.72:0/405382076 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8ae4004800 0x7f8af815a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.897+0000 7f8afe7e7640 1 -- 172.21.15.72:0/405382076 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8af8099780 msgr2=0x7f8af8072c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.897+0000 7f8afe7e7640 1 --2- 172.21.15.72:0/405382076 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8af8099780 0x7f8af8072c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.897+0000 7f8afe7e7640 1 -- 172.21.15.72:0/405382076 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8af8098590 con 0x7f8af8099c00 2024-04-15T15:17:04.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.897+0000 7f8afe7e7640 1 --2- 172.21.15.72:0/405382076 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8af8099c00 0x7f8af8073140 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f8af0000c00 tx=0x7f8af000e820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:04.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.897+0000 7f8aee7fc640 1 -- 172.21.15.72:0/405382076 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af0019070 con 0x7f8af8099c00 2024-04-15T15:17:04.912 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.897+0000 7f8aee7fc640 1 -- 172.21.15.72:0/405382076 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8af00052e0 con 0x7f8af8099c00 2024-04-15T15:17:04.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.897+0000 7f8b0024a640 1 -- 172.21.15.72:0/405382076 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8af8098870 con 0x7f8af8099c00 2024-04-15T15:17:04.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.897+0000 7f8aee7fc640 1 -- 172.21.15.72:0/405382076 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8af0003830 con 0x7f8af8099c00 2024-04-15T15:17:04.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.897+0000 7f8b0024a640 1 -- 172.21.15.72:0/405382076 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8af8098e30 con 0x7f8af8099c00 2024-04-15T15:17:04.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.898+0000 7f8b0024a640 1 -- 172.21.15.72:0/405382076 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8af8099160 con 0x7f8af8099c00 2024-04-15T15:17:04.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.898+0000 7f8aee7fc640 1 -- 172.21.15.72:0/405382076 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f8af00039d0 con 0x7f8af8099c00 2024-04-15T15:17:04.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.899+0000 7f8aee7fc640 1 --2- 172.21.15.72:0/405382076 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8adc06c9b0 0x7f8adc06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.899+0000 7f8afdfe6640 1 --2- 172.21.15.72:0/405382076 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8adc06c9b0 0x7f8adc06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.913 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.899+0000 7f8aee7fc640 1 -- 172.21.15.72:0/405382076 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5421+0+0 (secure 0 0 0) 0x7f8af008ae50 con 0x7f8af8099c00 2024-04-15T15:17:04.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.899+0000 7f8afdfe6640 1 --2- 172.21.15.72:0/405382076 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8adc06c9b0 0x7f8adc06ee70 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f8af80aee10 tx=0x7f8af400b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:04.914 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:04.903+0000 7f8aee7fc640 1 -- 172.21.15.72:0/405382076 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8af00593b0 con 0x7f8af8099c00 2024-04-15T15:17:05.076 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.074+0000 7f8b0024a640 1 -- 172.21.15.72:0/405382076 --> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8af8099350 con 0x7f8adc06c9b0 2024-04-15T15:17:05.077 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.074+0000 7f8aee7fc640 1 -- 172.21.15.72:0/405382076 <== mgr.4101 v2:172.21.15.72:6832/3503807805 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13384 (secure 0 0 0) 0x7f8af8099350 con 0x7f8adc06c9b0 2024-04-15T15:17:05.077 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:17:05.078 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-04-15T15:17:05.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.076+0000 7f8b0024a640 1 -- 172.21.15.72:0/405382076 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8adc06c9b0 msgr2=0x7f8adc06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.076+0000 7f8b0024a640 1 --2- 172.21.15.72:0/405382076 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8adc06c9b0 0x7f8adc06ee70 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f8af80aee10 tx=0x7f8af400b040 comp rx=0 tx=0).stop 2024-04-15T15:17:05.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.076+0000 7f8b0024a640 1 -- 172.21.15.72:0/405382076 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8af8099c00 msgr2=0x7f8af8073140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.076+0000 7f8b0024a640 1 --2- 172.21.15.72:0/405382076 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8af8099c00 0x7f8af8073140 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f8af0000c00 tx=0x7f8af000e820 comp rx=0 tx=0).stop 2024-04-15T15:17:05.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.076+0000 7f8b0024a640 1 -- 172.21.15.72:0/405382076 shutdown_connections 2024-04-15T15:17:05.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.076+0000 7f8b0024a640 1 --2- 172.21.15.72:0/405382076 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f8adc06c9b0 0x7f8adc06ee70 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.076+0000 7f8b0024a640 1 --2- 172.21.15.72:0/405382076 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8af8099c00 0x7f8af8073140 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.076+0000 7f8b0024a640 1 --2- 172.21.15.72:0/405382076 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f8af8099780 0x7f8af8072c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.076+0000 7f8b0024a640 1 --2- 172.21.15.72:0/405382076 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f8ae4004800 0x7f8af815a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.076+0000 7f8b0024a640 1 -- 172.21.15.72:0/405382076 >> 172.21.15.72:0/405382076 conn(0x7f8af80b1dd0 msgr2=0x7f8af80b03f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:05.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.076+0000 7f8b0024a640 1 -- 172.21.15.72:0/405382076 shutdown_connections 2024-04-15T15:17:05.081 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.076+0000 7f8b0024a640 1 -- 172.21.15.72:0/405382076 wait complete. 2024-04-15T15:17:05.090 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":109,"stamp":"2024-04-15T15:17:03.369946+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221272,"kb_used_data":5116,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753448,"statfs":{"total":773094113280,"available":772867530752,"internally_reserved":0,"allocated":5238784,"data_stored":3011972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-62067,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":-79,"num_read_kb":-129,"num_write":-91,"num_write_kb":-95,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000916"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":211,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-04-15T15:16:58.321347+0000","last_change":"2024-04-15T15:14:40.494279+0000","last_active":"2024-04-15T15:16:58.321347+0000","last_peered":"2024-04-15T15:16:58.321347+0000","last_clean":"2024-04-15T15:16:58.321347+0000","last_became_active":"2024-04-15T15:14:40.493404+0000","last_became_peered":"2024-04-15T15:14:40.493404+0000","last_unstale":"2024-04-15T15:16:58.321347+0000","last_undegraded":"2024-04-15T15:16:58.321347+0000","last_fullsized":"2024-04-15T15:16:58.321347+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:39.418508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:39.418508+0000","last_clean_scrub_stamp":"2024-04-15T15:14:39.418508+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:01:51.789246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574880,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28152,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343688,"statfs":{"total":96636764160,"available":96607936512,"internally_reserved":0,"allocated":1159168,"data_stored":867671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574880,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27504,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344336,"statfs":{"total":96636764160,"available":96608600064,"internally_reserved":0,"allocated":495616,"data_stored":216538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574879,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27504,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344336,"statfs":{"total":96636764160,"available":96608600064,"internally_reserved":0,"allocated":495616,"data_stored":215935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574879,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27492,"kb_used_data":472,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344348,"statfs":{"total":96636764160,"available":96608612352,"internally_reserved":0,"allocated":483328,"data_stored":209839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574880,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27568,"kb_used_data":548,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344272,"statfs":{"total":96636764160,"available":96608534528,"internally_reserved":0,"allocated":561152,"data_stored":277134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574879,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27484,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344356,"statfs":{"total":96636764160,"available":96608620544,"internally_reserved":0,"allocated":479232,"data_stored":209305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574880,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27500,"kb_used_data":480,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344340,"statfs":{"total":96636764160,"available":96608604160,"internally_reserved":0,"allocated":491520,"data_stored":215913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574878,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28068,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343772,"statfs":{"total":96636764160,"available":96608022528,"internally_reserved":0,"allocated":1073152,"data_stored":799637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:17:05.091 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:17:05.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.191+0000 7f01acd0d640 1 Processor -- start 2024-04-15T15:17:05.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.191+0000 7f01acd0d640 1 -- start start 2024-04-15T15:17:05.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.192+0000 7f01acd0d640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f01a813f3c0 0x7f01a806f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.192+0000 7f01acd0d640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f01a806fa50 0x7f01a80b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.192+0000 7f01acd0d640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f01a80b4810 0x7f01a806c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.192+0000 7f01acd0d640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f01a8142bc0 con 0x7f01a806fa50 2024-04-15T15:17:05.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.192+0000 7f01acd0d640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f01a80b4f20 con 0x7f01a813f3c0 2024-04-15T15:17:05.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.192+0000 7f01acd0d640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f01a8141e40 con 0x7f01a80b4810 2024-04-15T15:17:05.197 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.192+0000 7f01a5d9b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f01a806fa50 0x7f01a80b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.192+0000 7f01a5d9b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f01a806fa50 0x7f01a80b42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36734/0 (socket says 172.21.15.72:36734) 2024-04-15T15:17:05.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.192+0000 7f01a5d9b640 1 -- 172.21.15.72:0/2889252068 learned_addr learned my addr 172.21.15.72:0/2889252068 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:05.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.193+0000 7f01a6d9d640 1 --2- 172.21.15.72:0/2889252068 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f01a80b4810 0x7f01a806c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.193+0000 7f01a659c640 1 --2- 172.21.15.72:0/2889252068 >> v2:172.21.15.175:3300/0 conn(0x7f01a813f3c0 0x7f01a806f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.193+0000 7f01a5d9b640 1 -- 172.21.15.72:0/2889252068 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f01a80b4810 msgr2=0x7f01a806c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:05.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.193+0000 7f01a5d9b640 1 --2- 172.21.15.72:0/2889252068 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f01a80b4810 0x7f01a806c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.193+0000 7f01a5d9b640 1 -- 172.21.15.72:0/2889252068 >> v2:172.21.15.175:3300/0 conn(0x7f01a813f3c0 msgr2=0x7f01a806f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:05.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.193+0000 7f01a5d9b640 1 --2- 172.21.15.72:0/2889252068 >> v2:172.21.15.175:3300/0 conn(0x7f01a813f3c0 0x7f01a806f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.193+0000 7f01a5d9b640 1 -- 172.21.15.72:0/2889252068 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01a81420e0 con 0x7f01a806fa50 2024-04-15T15:17:05.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.193+0000 7f01a659c640 1 --2- 172.21.15.72:0/2889252068 >> v2:172.21.15.175:3300/0 conn(0x7f01a813f3c0 0x7f01a806f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:05.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.193+0000 7f01a6d9d640 1 --2- 172.21.15.72:0/2889252068 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f01a80b4810 0x7f01a806c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:05.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.193+0000 7f01a5d9b640 1 --2- 172.21.15.72:0/2889252068 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01a806fa50 0x7f01a80b42d0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f01a80b5910 tx=0x7f019002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ef596a93f16bb51 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:05.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.194+0000 7f01a559a640 1 -- 172.21.15.72:0/2889252068 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0190035020 con 0x7f01a806fa50 2024-04-15T15:17:05.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.194+0000 7f01a559a640 1 -- 172.21.15.72:0/2889252068 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0190031870 con 0x7f01a806fa50 2024-04-15T15:17:05.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.194+0000 7f01a559a640 1 -- 172.21.15.72:0/2889252068 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0190031c40 con 0x7f01a806fa50 2024-04-15T15:17:05.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.194+0000 7f01acd0d640 1 -- 172.21.15.72:0/2889252068 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01a806fa50 msgr2=0x7f01a80b42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.194+0000 7f01acd0d640 1 --2- 172.21.15.72:0/2889252068 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01a806fa50 0x7f01a80b42d0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f01a80b5910 tx=0x7f019002b910 comp rx=0 tx=0).stop 2024-04-15T15:17:05.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.194+0000 7f01acd0d640 1 -- 172.21.15.72:0/2889252068 shutdown_connections 2024-04-15T15:17:05.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.194+0000 7f01acd0d640 1 --2- 172.21.15.72:0/2889252068 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f01a80b4810 0x7f01a806c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.194+0000 7f01acd0d640 1 --2- 172.21.15.72:0/2889252068 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01a806fa50 0x7f01a80b42d0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.194+0000 7f01acd0d640 1 --2- 172.21.15.72:0/2889252068 >> v2:172.21.15.175:3300/0 conn(0x7f01a813f3c0 0x7f01a806f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.194+0000 7f01acd0d640 1 -- 172.21.15.72:0/2889252068 >> 172.21.15.72:0/2889252068 conn(0x7f01a80cdd20 msgr2=0x7f01a80ce100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:05.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.194+0000 7f01acd0d640 1 -- 172.21.15.72:0/2889252068 shutdown_connections 2024-04-15T15:17:05.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.194+0000 7f01acd0d640 1 -- 172.21.15.72:0/2889252068 wait complete. 2024-04-15T15:17:05.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.196+0000 7f01acd0d640 1 Processor -- start 2024-04-15T15:17:05.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.196+0000 7f01acd0d640 1 -- start start 2024-04-15T15:17:05.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.196+0000 7f01acd0d640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f01a806fa50 0x7f01a81f1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.196+0000 7f01acd0d640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f01a80b4810 0x7f01a813afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.196+0000 7f01acd0d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01a813f3c0 0x7f01a8145710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.196+0000 7f01acd0d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f01a80b4c40 con 0x7f01a813f3c0 2024-04-15T15:17:05.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.196+0000 7f01acd0d640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f01a81426a0 con 0x7f01a806fa50 2024-04-15T15:17:05.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.196+0000 7f01acd0d640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f01a815caf0 con 0x7f01a80b4810 2024-04-15T15:17:05.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.196+0000 7f01a6d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01a813f3c0 0x7f01a8145710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.196+0000 7f01a6d9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01a813f3c0 0x7f01a8145710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36746/0 (socket says 172.21.15.72:36746) 2024-04-15T15:17:05.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.196+0000 7f01a6d9d640 1 -- 172.21.15.72:0/2598829131 learned_addr learned my addr 172.21.15.72:0/2598829131 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:05.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.198+0000 7f01a659c640 1 --2- 172.21.15.72:0/2598829131 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f01a806fa50 0x7f01a81f1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.198+0000 7f01a5d9b640 1 --2- 172.21.15.72:0/2598829131 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f01a80b4810 0x7f01a813afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.198+0000 7f01a6d9d640 1 -- 172.21.15.72:0/2598829131 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f01a80b4810 msgr2=0x7f01a813afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.198+0000 7f01a6d9d640 1 --2- 172.21.15.72:0/2598829131 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f01a80b4810 0x7f01a813afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.198+0000 7f01a6d9d640 1 -- 172.21.15.72:0/2598829131 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f01a806fa50 msgr2=0x7f01a81f1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.198+0000 7f01a6d9d640 1 --2- 172.21.15.72:0/2598829131 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f01a806fa50 0x7f01a81f1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.198+0000 7f01a6d9d640 1 -- 172.21.15.72:0/2598829131 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01a81f1f70 con 0x7f01a813f3c0 2024-04-15T15:17:05.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.198+0000 7f01a6d9d640 1 --2- 172.21.15.72:0/2598829131 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01a813f3c0 0x7f01a8145710 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f01a000c820 tx=0x7f01a000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:05.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.198+0000 7f019effd640 1 -- 172.21.15.72:0/2598829131 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01a0015070 con 0x7f01a813f3c0 2024-04-15T15:17:05.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.198+0000 7f019effd640 1 -- 172.21.15.72:0/2598829131 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01a0007a30 con 0x7f01a813f3c0 2024-04-15T15:17:05.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.198+0000 7f01acd0d640 1 -- 172.21.15.72:0/2598829131 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01a80b18c0 con 0x7f01a813f3c0 2024-04-15T15:17:05.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.198+0000 7f019effd640 1 -- 172.21.15.72:0/2598829131 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01a0007e20 con 0x7f01a813f3c0 2024-04-15T15:17:05.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.198+0000 7f01acd0d640 1 -- 172.21.15.72:0/2598829131 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01a813d2f0 con 0x7f01a813f3c0 2024-04-15T15:17:05.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.199+0000 7f01acd0d640 1 -- 172.21.15.72:0/2598829131 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01a80690f0 con 0x7f01a813f3c0 2024-04-15T15:17:05.207 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.200+0000 7f019effd640 1 -- 172.21.15.72:0/2598829131 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f01a0018070 con 0x7f01a813f3c0 2024-04-15T15:17:05.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.200+0000 7f019effd640 1 --2- 172.21.15.72:0/2598829131 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f018806c980 0x7f018806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.200+0000 7f019effd640 1 -- 172.21.15.72:0/2598829131 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5421+0+0 (secure 0 0 0) 0x7f01a008bad0 con 0x7f01a813f3c0 2024-04-15T15:17:05.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.200+0000 7f01a659c640 1 --2- 172.21.15.72:0/2598829131 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f018806c980 0x7f018806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.200+0000 7f01a659c640 1 --2- 172.21.15.72:0/2598829131 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f018806c980 0x7f018806ee40 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0198007fb0 tx=0x7f0198007990 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:05.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.204+0000 7f019effd640 1 -- 172.21.15.72:0/2598829131 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f01a80b18c0 con 0x7f01a813f3c0 2024-04-15T15:17:05.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.380+0000 7f01acd0d640 1 -- 172.21.15.72:0/2598829131 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f01a8153750 con 0x7f01a813f3c0 2024-04-15T15:17:05.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.380+0000 7f019effd640 1 -- 172.21.15.72:0/2598829131 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+13423 (secure 0 0 0) 0x7f01a0059f20 con 0x7f01a813f3c0 2024-04-15T15:17:05.382 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:17:05.382 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":44,"fsid":"05091e39-b18a-4545-8b68-c0ae4699e116","created":"2024-04-15T15:14:27.103798+0000","modified":"2024-04-15T15:17:01.062525+0000","last_up_change":"2024-04-15T15:14:38.416838+0000","last_in_change":"2024-04-15T15:14:29.462079+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:39.359102+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"38ec1d7c-98f9-412c-bf00-97f8d82a075e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6801","nonce":1858587354}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6803","nonce":1858587354}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6807","nonce":1858587354}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1858587354},{"type":"v1","addr":"172.21.15.72:6805","nonce":1858587354}]},"public_addr":"172.21.15.72:6801/1858587354","cluster_addr":"172.21.15.72:6803/1858587354","heartbeat_back_addr":"172.21.15.72:6807/1858587354","heartbeat_front_addr":"172.21.15.72:6805/1858587354","state":["exists","up"]},{"osd":1,"uuid":"0e798110-9f52-4dfb-9f05-43236921d829","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6824","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6825","nonce":3675605364}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6826","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6827","nonce":3675605364}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6830","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6831","nonce":3675605364}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6828","nonce":3675605364},{"type":"v1","addr":"172.21.15.72:6829","nonce":3675605364}]},"public_addr":"172.21.15.72:6825/3675605364","cluster_addr":"172.21.15.72:6827/3675605364","heartbeat_back_addr":"172.21.15.72:6831/3675605364","heartbeat_front_addr":"172.21.15.72:6829/3675605364","state":["exists","up"]},{"osd":2,"uuid":"28be1d7e-11ab-4c1c-8202-b443bac0bc04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6817","nonce":4041627082}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6818","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6819","nonce":4041627082}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6822","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6823","nonce":4041627082}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6820","nonce":4041627082},{"type":"v1","addr":"172.21.15.72:6821","nonce":4041627082}]},"public_addr":"172.21.15.72:6817/4041627082","cluster_addr":"172.21.15.72:6819/4041627082","heartbeat_back_addr":"172.21.15.72:6823/4041627082","heartbeat_front_addr":"172.21.15.72:6821/4041627082","state":["exists","up"]},{"osd":3,"uuid":"5de0f86b-c336-4ea1-972e-10a267b9678d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6809","nonce":3973649661}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6811","nonce":3973649661}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6815","nonce":3973649661}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3973649661},{"type":"v1","addr":"172.21.15.72:6813","nonce":3973649661}]},"public_addr":"172.21.15.72:6809/3973649661","cluster_addr":"172.21.15.72:6811/3973649661","heartbeat_back_addr":"172.21.15.72:6815/3973649661","heartbeat_front_addr":"172.21.15.72:6813/3973649661","state":["exists","up"]},{"osd":4,"uuid":"c5bece86-901e-49b0-b7f1-05531bc856d1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6823","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6825","nonce":626414407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6826","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6827","nonce":626414407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6830","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6831","nonce":626414407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6828","nonce":626414407},{"type":"v1","addr":"172.21.15.175:6829","nonce":626414407}]},"public_addr":"172.21.15.175:6825/626414407","cluster_addr":"172.21.15.175:6827/626414407","heartbeat_back_addr":"172.21.15.175:6831/626414407","heartbeat_front_addr":"172.21.15.175:6829/626414407","state":["exists","up"]},{"osd":5,"uuid":"e50375c9-553a-46fb-8747-1471841434c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6800","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6801","nonce":1465263431}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6802","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6803","nonce":1465263431}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6806","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6807","nonce":1465263431}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6804","nonce":1465263431},{"type":"v1","addr":"172.21.15.175:6805","nonce":1465263431}]},"public_addr":"172.21.15.175:6801/1465263431","cluster_addr":"172.21.15.175:6803/1465263431","heartbeat_back_addr":"172.21.15.175:6807/1465263431","heartbeat_front_addr":"172.21.15.175:6805/1465263431","state":["exists","up"]},{"osd":6,"uuid":"7a629a23-4ffe-4b27-a2d0-b42070d22084","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6808","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6809","nonce":1973126001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6810","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6811","nonce":1973126001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6814","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6815","nonce":1973126001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6812","nonce":1973126001},{"type":"v1","addr":"172.21.15.175:6813","nonce":1973126001}]},"public_addr":"172.21.15.175:6809/1973126001","cluster_addr":"172.21.15.175:6811/1973126001","heartbeat_back_addr":"172.21.15.175:6815/1973126001","heartbeat_front_addr":"172.21.15.175:6813/1973126001","state":["exists","up"]},{"osd":7,"uuid":"c120037a-a16b-4504-847c-81cb859d31be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6816","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6817","nonce":2672274294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6818","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6819","nonce":2672274294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6822","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6824","nonce":2672274294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6820","nonce":2672274294},{"type":"v1","addr":"172.21.15.175:6821","nonce":2672274294}]},"public_addr":"172.21.15.175:6817/2672274294","cluster_addr":"172.21.15.175:6819/2672274294","heartbeat_back_addr":"172.21.15.175:6824/2672274294","heartbeat_front_addr":"172.21.15.175:6821/2672274294","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.558623+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.631565+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.603101+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.659429+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.638544+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.586953+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.674752+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:34.595101+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:6836/3407328263":"2024-04-16T15:16:58.055563+0000","172.21.15.72:6834/3407328263":"2024-04-16T15:16:58.055563+0000","172.21.15.72:6836/4270793613":"2024-04-16T15:16:41.714817+0000","172.21.15.72:6834/4270793613":"2024-04-16T15:16:41.714817+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:17:05.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.382+0000 7f01acd0d640 1 -- 172.21.15.72:0/2598829131 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f018806c980 msgr2=0x7f018806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.382+0000 7f01acd0d640 1 --2- 172.21.15.72:0/2598829131 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f018806c980 0x7f018806ee40 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0198007fb0 tx=0x7f0198007990 comp rx=0 tx=0).stop 2024-04-15T15:17:05.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.382+0000 7f01acd0d640 1 -- 172.21.15.72:0/2598829131 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01a813f3c0 msgr2=0x7f01a8145710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.382+0000 7f01acd0d640 1 --2- 172.21.15.72:0/2598829131 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01a813f3c0 0x7f01a8145710 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f01a000c820 tx=0x7f01a000cce0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.383+0000 7f01acd0d640 1 -- 172.21.15.72:0/2598829131 shutdown_connections 2024-04-15T15:17:05.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.383+0000 7f01acd0d640 1 --2- 172.21.15.72:0/2598829131 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f018806c980 0x7f018806ee40 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.383+0000 7f01acd0d640 1 --2- 172.21.15.72:0/2598829131 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01a813f3c0 0x7f01a8145710 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.383+0000 7f01acd0d640 1 --2- 172.21.15.72:0/2598829131 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f01a80b4810 0x7f01a813afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.386 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.383+0000 7f01acd0d640 1 --2- 172.21.15.72:0/2598829131 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f01a806fa50 0x7f01a81f1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.383+0000 7f01acd0d640 1 -- 172.21.15.72:0/2598829131 >> 172.21.15.72:0/2598829131 conn(0x7f01a80cdd20 msgr2=0x7f01a80b6380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:05.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.383+0000 7f01acd0d640 1 -- 172.21.15.72:0/2598829131 shutdown_connections 2024-04-15T15:17:05.387 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:05.383+0000 7f01acd0d640 1 -- 172.21.15.72:0/2598829131 wait complete. 2024-04-15T15:17:06.397 INFO:tasks.ceph:Scrubbing osd.0 2024-04-15T15:17:06.398 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-15T15:17:06.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.515+0000 7f83f47a8640 1 Processor -- start 2024-04-15T15:17:06.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.515+0000 7f83f47a8640 1 -- start start 2024-04-15T15:17:06.523 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.516+0000 7f83f47a8640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f83ec06f220 0x7f83ec1673e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.516+0000 7f83f47a8640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83ec15a4b0 con 0x7f83ec15a0d0 2024-04-15T15:17:06.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.516+0000 7f83f47a8640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83ec0b6190 con 0x7f83ec15ca60 2024-04-15T15:17:06.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.516+0000 7f83f47a8640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f83ec06c680 con 0x7f83ec06f220 2024-04-15T15:17:06.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.516+0000 7f83f2544640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f83ec15a0d0 0x7f83ec0b5370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:34018/0 (socket says 172.21.15.72:34018) 2024-04-15T15:17:06.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.516+0000 7f83f2544640 1 -- 172.21.15.72:0/1814236211 learned_addr learned my addr 172.21.15.72:0/1814236211 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:17:06.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.516+0000 7f83f2d45640 1 --2- 172.21.15.72:0/1814236211 >> v2:172.21.15.175:3300/0 conn(0x7f83ec06f220 0x7f83ec1673e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.524 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.516+0000 7f83f1542640 1 -- 172.21.15.72:0/1814236211 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1831277822 0 0) 0x7f83ec15a4b0 con 0x7f83ec15a0d0 2024-04-15T15:17:06.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.516+0000 7f83f1542640 1 -- 172.21.15.72:0/1814236211 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83d8003520 con 0x7f83ec15a0d0 2024-04-15T15:17:06.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.516+0000 7f83f1542640 1 -- 172.21.15.72:0/1814236211 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 686794268 0 0) 0x7f83ec0b6190 con 0x7f83ec15ca60 2024-04-15T15:17:06.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.516+0000 7f83f1542640 1 -- 172.21.15.72:0/1814236211 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83ec15a4b0 con 0x7f83ec15ca60 2024-04-15T15:17:06.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.516+0000 7f83f1542640 1 -- 172.21.15.72:0/1814236211 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3635170286 0 0) 0x7f83d8003520 con 0x7f83ec15a0d0 2024-04-15T15:17:06.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.516+0000 7f83f1542640 1 -- 172.21.15.72:0/1814236211 >> v2:172.21.15.175:3300/0 conn(0x7f83ec06f220 msgr2=0x7f83ec1673e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:06.525 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.516+0000 7f83f1542640 1 --2- 172.21.15.72:0/1814236211 >> v2:172.21.15.175:3300/0 conn(0x7f83ec06f220 0x7f83ec1673e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.516+0000 7f83f1542640 1 -- 172.21.15.72:0/1814236211 >> v1:172.21.15.175:6789/0 conn(0x7f83ec15ca60 legacy=0x7f83ec0b5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.516+0000 7f83f1542640 1 -- 172.21.15.72:0/1814236211 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83ec06c960 con 0x7f83ec15a0d0 2024-04-15T15:17:06.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.516+0000 7f83f1542640 1 -- 172.21.15.72:0/1814236211 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f83e8001670 con 0x7f83ec15a0d0 2024-04-15T15:17:06.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.517+0000 7f83f1542640 1 -- 172.21.15.72:0/1814236211 >> v1:172.21.15.72:6789/0 conn(0x7f83ec15a0d0 legacy=0x7f83ec0b5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.517+0000 7f83f1542640 1 --2- 172.21.15.72:0/1814236211 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f83d8003b90 0x7f83d8003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.517+0000 7f83f1542640 1 --2- 172.21.15.72:0/1814236211 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f83d8004800 0x7f83d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.517+0000 7f83f1542640 1 --2- 172.21.15.72:0/1814236211 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83d80051c0 0x7f83d800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.517+0000 7f83f1542640 1 -- 172.21.15.72:0/1814236211 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f83ec06c680 con 0x7f83d80051c0 2024-04-15T15:17:06.526 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.517+0000 7f83f1542640 1 -- 172.21.15.72:0/1814236211 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f83d800c090 con 0x7f83d8003b90 2024-04-15T15:17:06.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.517+0000 7f83f1542640 1 -- 172.21.15.72:0/1814236211 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f83d800c290 con 0x7f83d8004800 2024-04-15T15:17:06.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.517+0000 7f83f2544640 1 --2- 172.21.15.72:0/1814236211 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f83d8003b90 0x7f83d8003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.517+0000 7f83f2d45640 1 --2- 172.21.15.72:0/1814236211 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83d80051c0 0x7f83d800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.517+0000 7f83f1d43640 1 --2- 172.21.15.72:0/1814236211 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f83d8004800 0x7f83d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.517+0000 7f83f2d45640 1 -- 172.21.15.72:0/1814236211 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f83d8004800 msgr2=0x7f83d8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:06.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.517+0000 7f83f2d45640 1 --2- 172.21.15.72:0/1814236211 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f83d8004800 0x7f83d8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.527 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.517+0000 7f83f2d45640 1 -- 172.21.15.72:0/1814236211 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f83d8003b90 msgr2=0x7f83d8003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:06.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.517+0000 7f83f2d45640 1 --2- 172.21.15.72:0/1814236211 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f83d8003b90 0x7f83d8003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.517+0000 7f83f2d45640 1 -- 172.21.15.72:0/1814236211 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83d8003520 con 0x7f83d80051c0 2024-04-15T15:17:06.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.518+0000 7f83f2d45640 1 -- 172.21.15.72:0/1814236211 reap_dead start 2024-04-15T15:17:06.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.518+0000 7f83f2d45640 1 --2- 172.21.15.72:0/1814236211 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83d80051c0 0x7f83d800b9c0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f83e4009fc0 tx=0x7f83d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c686d702630388b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:06.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.518+0000 7f83f1542640 1 -- 172.21.15.72:0/1814236211 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e4009c10 con 0x7f83d80051c0 2024-04-15T15:17:06.528 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.518+0000 7f83f1542640 1 -- 172.21.15.72:0/1814236211 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83e402cca0 con 0x7f83d80051c0 2024-04-15T15:17:06.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.518+0000 7f83f1542640 1 -- 172.21.15.72:0/1814236211 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83e4003bd0 con 0x7f83d80051c0 2024-04-15T15:17:06.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.518+0000 7f83f47a8640 1 -- 172.21.15.72:0/1814236211 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83d80051c0 msgr2=0x7f83d800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.518+0000 7f83f47a8640 1 --2- 172.21.15.72:0/1814236211 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83d80051c0 0x7f83d800b9c0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f83e4009fc0 tx=0x7f83d8005830 comp rx=0 tx=0).stop 2024-04-15T15:17:06.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.519+0000 7f83f47a8640 1 -- 172.21.15.72:0/1814236211 shutdown_connections 2024-04-15T15:17:06.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.519+0000 7f83f47a8640 1 --2- 172.21.15.72:0/1814236211 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83d80051c0 0x7f83d800b9c0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.519+0000 7f83f47a8640 1 -- 172.21.15.72:0/1814236211 >> 172.21.15.72:0/1814236211 conn(0x7f83ec0b1ee0 msgr2=0x7f83ec0b22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:06.529 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.519+0000 7f83f47a8640 1 -- 172.21.15.72:0/1814236211 shutdown_connections 2024-04-15T15:17:06.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.519+0000 7f83f47a8640 1 -- 172.21.15.72:0/1814236211 wait complete. 2024-04-15T15:17:06.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.520+0000 7f83f47a8640 1 Processor -- start 2024-04-15T15:17:06.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.520+0000 7f83f47a8640 1 -- start start 2024-04-15T15:17:06.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.520+0000 7f83f47a8640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f83d80051c0 0x7f83ec1519f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.520+0000 7f83f47a8640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83ec1476b0 0x7f83ec154d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.520+0000 7f83f47a8640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f83ec0cd2f0 0x7f83ec1585f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.520+0000 7f83f47a8640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f83ec06d200 con 0x7f83ec1476b0 2024-04-15T15:17:06.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.520+0000 7f83f47a8640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f83ec06cf20 con 0x7f83ec0cd2f0 2024-04-15T15:17:06.530 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.521+0000 7f83f47a8640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f83ec06c310 con 0x7f83d80051c0 2024-04-15T15:17:06.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.521+0000 7f83f1d43640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83ec1476b0 0x7f83ec154d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.521+0000 7f83f1d43640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83ec1476b0 0x7f83ec154d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36776/0 (socket says 172.21.15.72:36776) 2024-04-15T15:17:06.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.521+0000 7f83f1d43640 1 -- 172.21.15.72:0/4189714630 learned_addr learned my addr 172.21.15.72:0/4189714630 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:06.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.521+0000 7f83f2d45640 1 --2- 172.21.15.72:0/4189714630 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f83ec0cd2f0 0x7f83ec1585f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.521+0000 7f83f2544640 1 --2- 172.21.15.72:0/4189714630 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f83d80051c0 0x7f83ec1519f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.521+0000 7f83f1d43640 1 -- 172.21.15.72:0/4189714630 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f83d80051c0 msgr2=0x7f83ec1519f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.521+0000 7f83f1d43640 1 --2- 172.21.15.72:0/4189714630 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f83d80051c0 0x7f83ec1519f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.531 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.521+0000 7f83f1d43640 1 -- 172.21.15.72:0/4189714630 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f83ec0cd2f0 msgr2=0x7f83ec1585f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.521+0000 7f83f1d43640 1 --2- 172.21.15.72:0/4189714630 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f83ec0cd2f0 0x7f83ec1585f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.521+0000 7f83f1d43640 1 -- 172.21.15.72:0/4189714630 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83ec0cc060 con 0x7f83ec1476b0 2024-04-15T15:17:06.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.521+0000 7f83f1d43640 1 --2- 172.21.15.72:0/4189714630 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83ec1476b0 0x7f83ec154d30 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f83dc00f7d0 tx=0x7f83dc00fc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:06.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.521+0000 7f83e2ffd640 1 -- 172.21.15.72:0/4189714630 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83dc01a070 con 0x7f83ec1476b0 2024-04-15T15:17:06.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.521+0000 7f83e2ffd640 1 -- 172.21.15.72:0/4189714630 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83dc00fe80 con 0x7f83ec1476b0 2024-04-15T15:17:06.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.521+0000 7f83e2ffd640 1 -- 172.21.15.72:0/4189714630 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83dc007050 con 0x7f83ec1476b0 2024-04-15T15:17:06.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.521+0000 7f83f47a8640 1 -- 172.21.15.72:0/4189714630 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83ec0cc300 con 0x7f83ec1476b0 2024-04-15T15:17:06.532 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.521+0000 7f83f47a8640 1 -- 172.21.15.72:0/4189714630 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83ec0ccdd0 con 0x7f83ec1476b0 2024-04-15T15:17:06.533 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.522+0000 7f83f47a8640 1 -- 172.21.15.72:0/4189714630 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f83ec0689e0 con 0x7f83ec1476b0 2024-04-15T15:17:06.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.523+0000 7f83e2ffd640 1 -- 172.21.15.72:0/4189714630 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f83dc01d070 con 0x7f83ec1476b0 2024-04-15T15:17:06.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.523+0000 7f83e2ffd640 1 --2- 172.21.15.72:0/4189714630 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f83d006c9b0 0x7f83d006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.534 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.523+0000 7f83e2ffd640 1 -- 172.21.15.72:0/4189714630 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5421+0+0 (secure 0 0 0) 0x7f83dc08b360 con 0x7f83ec1476b0 2024-04-15T15:17:06.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.523+0000 7f83f2544640 1 --2- 172.21.15.72:0/4189714630 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f83d006c9b0 0x7f83d006ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.523+0000 7f83e2ffd640 1 --2- 172.21.15.72:0/4189714630 >> [v2:172.21.15.72:6800/1858587354,v1:172.21.15.72:6801/1858587354] conn(0x7f83d0072ef0 0x7f83d0075330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.523+0000 7f83e2ffd640 1 -- 172.21.15.72:0/4189714630 --> [v2:172.21.15.72:6800/1858587354,v1:172.21.15.72:6801/1858587354] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f83d0075a00 con 0x7f83d0072ef0 2024-04-15T15:17:06.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.523+0000 7f83e2ffd640 1 -- 172.21.15.72:0/4189714630 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f83dc092050 con 0x7f83ec1476b0 2024-04-15T15:17:06.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.524+0000 7f83f2d45640 1 --2- 172.21.15.72:0/4189714630 >> [v2:172.21.15.72:6800/1858587354,v1:172.21.15.72:6801/1858587354] conn(0x7f83d0072ef0 0x7f83d0075330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.524+0000 7f83f2544640 1 --2- 172.21.15.72:0/4189714630 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f83d006c9b0 0x7f83d006ee70 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f83e8001050 tx=0x7f83e8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:06.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.524+0000 7f83f2d45640 1 --2- 172.21.15.72:0/4189714630 >> [v2:172.21.15.72:6800/1858587354,v1:172.21.15.72:6801/1858587354] conn(0x7f83d0072ef0 0x7f83d0075330 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:06.535 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.525+0000 7f83e2ffd640 1 -- 172.21.15.72:0/4189714630 <== osd.0 v2:172.21.15.72:6800/1858587354 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f83d0075a00 con 0x7f83d0072ef0 2024-04-15T15:17:06.539 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.538+0000 7f83f47a8640 1 -- 172.21.15.72:0/4189714630 --> [v2:172.21.15.72:6800/1858587354,v1:172.21.15.72:6801/1858587354] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f83ec06bbe0 con 0x7f83d0072ef0 2024-04-15T15:17:06.540 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.538+0000 7f83e2ffd640 1 -- 172.21.15.72:0/4189714630 <== osd.0 v2:172.21.15.72:6800/1858587354 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f83ec06bbe0 con 0x7f83d0072ef0 2024-04-15T15:17:06.540 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-04-15T15:17:06.540 INFO:teuthology.orchestra.run.smithi072.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-15T15:17:06.540 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-04-15T15:17:06.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.538+0000 7f83f47a8640 1 -- 172.21.15.72:0/4189714630 >> [v2:172.21.15.72:6800/1858587354,v1:172.21.15.72:6801/1858587354] conn(0x7f83d0072ef0 msgr2=0x7f83d0075330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.541 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.538+0000 7f83f47a8640 1 --2- 172.21.15.72:0/4189714630 >> [v2:172.21.15.72:6800/1858587354,v1:172.21.15.72:6801/1858587354] conn(0x7f83d0072ef0 0x7f83d0075330 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.538+0000 7f83f47a8640 1 -- 172.21.15.72:0/4189714630 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f83d006c9b0 msgr2=0x7f83d006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.538+0000 7f83f47a8640 1 --2- 172.21.15.72:0/4189714630 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f83d006c9b0 0x7f83d006ee70 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f83e8001050 tx=0x7f83e8009040 comp rx=0 tx=0).stop 2024-04-15T15:17:06.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.538+0000 7f83f47a8640 1 -- 172.21.15.72:0/4189714630 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83ec1476b0 msgr2=0x7f83ec154d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.538+0000 7f83f47a8640 1 --2- 172.21.15.72:0/4189714630 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f83ec1476b0 0x7f83ec154d30 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f83dc00f7d0 tx=0x7f83dc00fc90 comp rx=0 tx=0).stop 2024-04-15T15:17:06.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.538+0000 7f83f2d45640 1 -- 172.21.15.72:0/4189714630 reap_dead start 2024-04-15T15:17:06.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.538+0000 7f83f47a8640 1 -- 172.21.15.72:0/4189714630 shutdown_connections 2024-04-15T15:17:06.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.538+0000 7f83f47a8640 1 -- 172.21.15.72:0/4189714630 >> 172.21.15.72:0/4189714630 conn(0x7f83ec0b1ee0 msgr2=0x7f83ec15ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:06.542 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.538+0000 7f83f47a8640 1 -- 172.21.15.72:0/4189714630 shutdown_connections 2024-04-15T15:17:06.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.538+0000 7f83f47a8640 1 -- 172.21.15.72:0/4189714630 wait complete. 2024-04-15T15:17:06.550 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-15T15:17:06.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.653+0000 7fe6a62b3640 1 Processor -- start 2024-04-15T15:17:06.654 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.653+0000 7fe6a62b3640 1 -- start start 2024-04-15T15:17:06.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.654+0000 7fe6a62b3640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fe6a015c630 0x7fe6a006cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.656 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.654+0000 7fe6a62b3640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe6a006ed30 0x7fe6a00ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.654+0000 7fe6a62b3640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6a006bd50 con 0x7fe6a00ccb90 2024-04-15T15:17:06.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.654+0000 7fe6a62b3640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fe6a006d2f0 con 0x7fe6a006ed30 2024-04-15T15:17:06.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.654+0000 7fe6a62b3640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fe6a006f1a0 con 0x7fe6a015c630 2024-04-15T15:17:06.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.654+0000 7fe6a4850640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe6a006ed30 0x7fe6a00ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.654+0000 7fe6a4850640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fe6a006ed30 0x7fe6a00ccff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36792/0 (socket says 172.21.15.72:36792) 2024-04-15T15:17:06.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.654+0000 7fe69f7fe640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fe6a00ccb90 0x7fe6a006e620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:34020/0 (socket says 172.21.15.72:34020) 2024-04-15T15:17:06.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.654+0000 7fe6a4850640 1 -- 172.21.15.72:0/1162619628 learned_addr learned my addr 172.21.15.72:0/1162619628 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:06.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.654+0000 7fe69ffff640 1 --2- 172.21.15.72:0/1162619628 >> v2:172.21.15.175:3300/0 conn(0x7fe6a015c630 0x7fe6a006cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.654+0000 7fe6a4850640 1 -- 172.21.15.72:0/1162619628 >> v2:172.21.15.175:3300/0 conn(0x7fe6a015c630 msgr2=0x7fe6a006cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:06.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.654+0000 7fe6a4850640 1 --2- 172.21.15.72:0/1162619628 >> v2:172.21.15.175:3300/0 conn(0x7fe6a015c630 0x7fe6a006cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.654+0000 7fe6a4850640 1 -- 172.21.15.72:0/1162619628 >> v1:172.21.15.72:6789/0 conn(0x7fe6a00ccb90 legacy=0x7fe6a006e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.654+0000 7fe69effd640 1 -- 172.21.15.72:0/1162619628 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2031699295 0 0) 0x7fe6a006bd50 con 0x7fe6a00ccb90 2024-04-15T15:17:06.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.654+0000 7fe6a4850640 1 -- 172.21.15.72:0/1162619628 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6a006bff0 con 0x7fe6a006ed30 2024-04-15T15:17:06.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.655+0000 7fe6a4850640 1 --2- 172.21.15.72:0/1162619628 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6a006ed30 0x7fe6a00ccff0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fe6a006fc40 tx=0x7fe69802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32a3a70161c4c22d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:06.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.655+0000 7fe69effd640 1 -- 172.21.15.72:0/1162619628 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe69802cb20 con 0x7fe6a006ed30 2024-04-15T15:17:06.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.655+0000 7fe69effd640 1 -- 172.21.15.72:0/1162619628 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe69802ccc0 con 0x7fe6a006ed30 2024-04-15T15:17:06.659 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.655+0000 7fe69effd640 1 -- 172.21.15.72:0/1162619628 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe698034610 con 0x7fe6a006ed30 2024-04-15T15:17:06.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.655+0000 7fe6a62b3640 1 -- 172.21.15.72:0/1162619628 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6a006ed30 msgr2=0x7fe6a00ccff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.655+0000 7fe6a62b3640 1 --2- 172.21.15.72:0/1162619628 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6a006ed30 0x7fe6a00ccff0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fe6a006fc40 tx=0x7fe69802bf80 comp rx=0 tx=0).stop 2024-04-15T15:17:06.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.656+0000 7fe6a62b3640 1 -- 172.21.15.72:0/1162619628 shutdown_connections 2024-04-15T15:17:06.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.656+0000 7fe6a62b3640 1 --2- 172.21.15.72:0/1162619628 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6a006ed30 0x7fe6a00ccff0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.656+0000 7fe6a62b3640 1 --2- 172.21.15.72:0/1162619628 >> v2:172.21.15.175:3300/0 conn(0x7fe6a015c630 0x7fe6a006cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.656+0000 7fe6a62b3640 1 -- 172.21.15.72:0/1162619628 >> 172.21.15.72:0/1162619628 conn(0x7fe6a00b31d0 msgr2=0x7fe6a00b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:06.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.656+0000 7fe6a62b3640 1 -- 172.21.15.72:0/1162619628 shutdown_connections 2024-04-15T15:17:06.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.656+0000 7fe6a62b3640 1 -- 172.21.15.72:0/1162619628 wait complete. 2024-04-15T15:17:06.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.657+0000 7fe6a62b3640 1 Processor -- start 2024-04-15T15:17:06.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.657+0000 7fe6a62b3640 1 -- start start 2024-04-15T15:17:06.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.658+0000 7fe6a62b3640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe6a006ed30 0x7fe6a0099870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.658+0000 7fe6a62b3640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe6a00ccb90 0x7fe6a00b1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.658+0000 7fe6a62b3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6a015c630 0x7fe6a00b1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.658+0000 7fe6a62b3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a006c920 con 0x7fe6a015c630 2024-04-15T15:17:06.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.658+0000 7fe6a62b3640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fe6a006c5b0 con 0x7fe6a006ed30 2024-04-15T15:17:06.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.658+0000 7fe6a62b3640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fe6a015ca10 con 0x7fe6a00ccb90 2024-04-15T15:17:06.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.658+0000 7fe6a4850640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6a015c630 0x7fe6a00b1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.658+0000 7fe6a4850640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6a015c630 0x7fe6a00b1db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36796/0 (socket says 172.21.15.72:36796) 2024-04-15T15:17:06.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.658+0000 7fe6a4850640 1 -- 172.21.15.72:0/2301195678 learned_addr learned my addr 172.21.15.72:0/2301195678 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:06.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.658+0000 7fe69f7fe640 1 --2- 172.21.15.72:0/2301195678 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe6a00ccb90 0x7fe6a00b1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.666 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.658+0000 7fe69ffff640 1 --2- 172.21.15.72:0/2301195678 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe6a006ed30 0x7fe6a0099870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.658+0000 7fe6a4850640 1 -- 172.21.15.72:0/2301195678 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe6a00ccb90 msgr2=0x7fe6a00b1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.658+0000 7fe6a4850640 1 --2- 172.21.15.72:0/2301195678 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe6a00ccb90 0x7fe6a00b1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.658+0000 7fe6a4850640 1 -- 172.21.15.72:0/2301195678 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe6a006ed30 msgr2=0x7fe6a0099870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.667 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.658+0000 7fe6a4850640 1 --2- 172.21.15.72:0/2301195678 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe6a006ed30 0x7fe6a0099870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.658+0000 7fe6a4850640 1 -- 172.21.15.72:0/2301195678 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6a01f5940 con 0x7fe6a015c630 2024-04-15T15:17:06.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.658+0000 7fe69ffff640 1 --2- 172.21.15.72:0/2301195678 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe6a006ed30 0x7fe6a0099870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:06.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.658+0000 7fe69f7fe640 1 --2- 172.21.15.72:0/2301195678 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe6a00ccb90 0x7fe6a00b1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:06.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.658+0000 7fe6a4850640 1 --2- 172.21.15.72:0/2301195678 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6a015c630 0x7fe6a00b1db0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fe698009f40 tx=0x7fe698004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:06.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.658+0000 7fe69cff9640 1 -- 172.21.15.72:0/2301195678 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe698043020 con 0x7fe6a015c630 2024-04-15T15:17:06.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.658+0000 7fe69cff9640 1 -- 172.21.15.72:0/2301195678 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe69803e450 con 0x7fe6a015c630 2024-04-15T15:17:06.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.658+0000 7fe6a62b3640 1 -- 172.21.15.72:0/2301195678 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6a01f5c20 con 0x7fe6a015c630 2024-04-15T15:17:06.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.658+0000 7fe69cff9640 1 -- 172.21.15.72:0/2301195678 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe698009510 con 0x7fe6a015c630 2024-04-15T15:17:06.669 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.658+0000 7fe6a62b3640 1 -- 172.21.15.72:0/2301195678 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6a01f61e0 con 0x7fe6a015c630 2024-04-15T15:17:06.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.659+0000 7fe6a62b3640 1 -- 172.21.15.72:0/2301195678 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6a0075c50 con 0x7fe6a015c630 2024-04-15T15:17:06.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.660+0000 7fe69cff9640 1 -- 172.21.15.72:0/2301195678 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fe698046020 con 0x7fe6a015c630 2024-04-15T15:17:06.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.660+0000 7fe69cff9640 1 --2- 172.21.15.72:0/2301195678 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe68c06c980 0x7fe68c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.660+0000 7fe69cff9640 1 -- 172.21.15.72:0/2301195678 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5421+0+0 (secure 0 0 0) 0x7fe6980ae220 con 0x7fe6a015c630 2024-04-15T15:17:06.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.660+0000 7fe69ffff640 1 --2- 172.21.15.72:0/2301195678 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe68c06c980 0x7fe68c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.670 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.661+0000 7fe69ffff640 1 --2- 172.21.15.72:0/2301195678 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe68c06c980 0x7fe68c06ee40 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe690006fd0 tx=0x7fe690008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:06.671 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.665+0000 7fe69cff9640 1 -- 172.21.15.72:0/2301195678 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe69804d030 con 0x7fe6a015c630 2024-04-15T15:17:06.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.839+0000 7fe6a62b3640 1 -- 172.21.15.72:0/2301195678 --> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7fe6a00d0f10 con 0x7fe68c06c980 2024-04-15T15:17:06.841 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.839+0000 7fe69cff9640 1 -- 172.21.15.72:0/2301195678 <== mgr.4101 v2:172.21.15.72:6832/3503807805 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fe6a00d0f10 con 0x7fe68c06c980 2024-04-15T15:17:06.841 INFO:teuthology.orchestra.run.smithi072.stderr:instructed osd(s) 0 to deep-scrub 2024-04-15T15:17:06.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.841+0000 7fe6a62b3640 1 -- 172.21.15.72:0/2301195678 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe68c06c980 msgr2=0x7fe68c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.841+0000 7fe6a62b3640 1 --2- 172.21.15.72:0/2301195678 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe68c06c980 0x7fe68c06ee40 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe690006fd0 tx=0x7fe690008040 comp rx=0 tx=0).stop 2024-04-15T15:17:06.843 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.841+0000 7fe6a62b3640 1 -- 172.21.15.72:0/2301195678 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6a015c630 msgr2=0x7fe6a00b1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.841+0000 7fe6a62b3640 1 --2- 172.21.15.72:0/2301195678 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6a015c630 0x7fe6a00b1db0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fe698009f40 tx=0x7fe698004a40 comp rx=0 tx=0).stop 2024-04-15T15:17:06.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.842+0000 7fe6a62b3640 1 -- 172.21.15.72:0/2301195678 shutdown_connections 2024-04-15T15:17:06.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.842+0000 7fe6a62b3640 1 --2- 172.21.15.72:0/2301195678 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fe68c06c980 0x7fe68c06ee40 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.842+0000 7fe6a62b3640 1 --2- 172.21.15.72:0/2301195678 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe6a015c630 0x7fe6a00b1db0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.842+0000 7fe6a62b3640 1 --2- 172.21.15.72:0/2301195678 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fe6a00ccb90 0x7fe6a00b1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.842+0000 7fe6a62b3640 1 --2- 172.21.15.72:0/2301195678 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fe6a006ed30 0x7fe6a0099870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.842+0000 7fe6a62b3640 1 -- 172.21.15.72:0/2301195678 >> 172.21.15.72:0/2301195678 conn(0x7fe6a00b31d0 msgr2=0x7fe6a00cfc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:06.844 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.842+0000 7fe6a62b3640 1 -- 172.21.15.72:0/2301195678 shutdown_connections 2024-04-15T15:17:06.845 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.842+0000 7fe6a62b3640 1 -- 172.21.15.72:0/2301195678 wait complete. 2024-04-15T15:17:06.854 INFO:tasks.ceph:Scrubbing osd.1 2024-04-15T15:17:06.855 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-15T15:17:06.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.960+0000 7fb3dac63640 1 Processor -- start 2024-04-15T15:17:06.962 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.960+0000 7fb3dac63640 1 -- start start 2024-04-15T15:17:06.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.961+0000 7fb3dac63640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb3d415ead0 0x7fb3d40b74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.961+0000 7fb3dac63640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb3d40cd140 0x7fb3d40b5230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.961+0000 7fb3dac63640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3d415eeb0 con 0x7fb3d40d13c0 2024-04-15T15:17:06.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.961+0000 7fb3dac63640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fb3d40cd5d0 con 0x7fb3d415ead0 2024-04-15T15:17:06.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.961+0000 7fb3dac63640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fb3d40d0f30 con 0x7fb3d40cd140 2024-04-15T15:17:06.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.961+0000 7fb3d89ff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb3d415ead0 0x7fb3d40b74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.962+0000 7fb3d89ff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb3d415ead0 0x7fb3d40b74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36806/0 (socket says 172.21.15.72:36806) 2024-04-15T15:17:06.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.962+0000 7fb3d89ff640 1 -- 172.21.15.72:0/3419340509 learned_addr learned my addr 172.21.15.72:0/3419340509 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:06.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.962+0000 7fb3d9200640 1 --2- 172.21.15.72:0/3419340509 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb3d40cd140 0x7fb3d40b5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.962+0000 7fb3d89ff640 1 -- 172.21.15.72:0/3419340509 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb3d40cd140 msgr2=0x7fb3d40b5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:06.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.962+0000 7fb3d89ff640 1 --2- 172.21.15.72:0/3419340509 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb3d40cd140 0x7fb3d40b5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.962+0000 7fb3d89ff640 1 -- 172.21.15.72:0/3419340509 >> v1:172.21.15.175:6789/0 conn(0x7fb3d40d13c0 legacy=0x7fb3d40cca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.962+0000 7fb3d89ff640 1 -- 172.21.15.72:0/3419340509 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3d40d0320 con 0x7fb3d415ead0 2024-04-15T15:17:06.965 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.963+0000 7fb3d89ff640 1 --2- 172.21.15.72:0/3419340509 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3d415ead0 0x7fb3d40b74a0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4009870 tx=0x7fb3c402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=863b3d501d81fe5b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:06.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.963+0000 7fb3d37fe640 1 -- 172.21.15.72:0/3419340509 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c4034070 con 0x7fb3d415ead0 2024-04-15T15:17:06.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.963+0000 7fb3d37fe640 1 -- 172.21.15.72:0/3419340509 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3c4030840 con 0x7fb3d415ead0 2024-04-15T15:17:06.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.963+0000 7fb3d37fe640 1 -- 172.21.15.72:0/3419340509 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c4030c10 con 0x7fb3d415ead0 2024-04-15T15:17:06.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.963+0000 7fb3dac63640 1 -- 172.21.15.72:0/3419340509 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3d415ead0 msgr2=0x7fb3d40b74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.963+0000 7fb3dac63640 1 --2- 172.21.15.72:0/3419340509 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3d415ead0 0x7fb3d40b74a0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4009870 tx=0x7fb3c402b980 comp rx=0 tx=0).stop 2024-04-15T15:17:06.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.963+0000 7fb3dac63640 1 -- 172.21.15.72:0/3419340509 shutdown_connections 2024-04-15T15:17:06.969 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.963+0000 7fb3dac63640 1 --2- 172.21.15.72:0/3419340509 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb3d40cd140 0x7fb3d40b5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.963+0000 7fb3dac63640 1 --2- 172.21.15.72:0/3419340509 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3d415ead0 0x7fb3d40b74a0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.963+0000 7fb3dac63640 1 -- 172.21.15.72:0/3419340509 >> 172.21.15.72:0/3419340509 conn(0x7fb3d40d23e0 msgr2=0x7fb3d40d27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:06.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.964+0000 7fb3dac63640 1 -- 172.21.15.72:0/3419340509 shutdown_connections 2024-04-15T15:17:06.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.964+0000 7fb3dac63640 1 -- 172.21.15.72:0/3419340509 wait complete. 2024-04-15T15:17:06.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.965+0000 7fb3dac63640 1 Processor -- start 2024-04-15T15:17:06.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.965+0000 7fb3dac63640 1 -- start start 2024-04-15T15:17:06.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.965+0000 7fb3dac63640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb3d40cd140 0x7fb3d414f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.970 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.965+0000 7fb3dac63640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb3d40d13c0 0x7fb3d40b1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.965+0000 7fb3dac63640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3d415ead0 0x7fb3d40ae1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.965+0000 7fb3dac63640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d40d0bc0 con 0x7fb3d415ead0 2024-04-15T15:17:06.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.965+0000 7fb3dac63640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fb3d40d08e0 con 0x7fb3d40cd140 2024-04-15T15:17:06.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.966+0000 7fb3dac63640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fb3d40cffb0 con 0x7fb3d40d13c0 2024-04-15T15:17:06.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.966+0000 7fb3d9200640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3d415ead0 0x7fb3d40ae1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.966+0000 7fb3d9200640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3d415ead0 0x7fb3d40ae1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36820/0 (socket says 172.21.15.72:36820) 2024-04-15T15:17:06.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.966+0000 7fb3d9200640 1 -- 172.21.15.72:0/1569075997 learned_addr learned my addr 172.21.15.72:0/1569075997 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:06.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.966+0000 7fb3d3fff640 1 --2- 172.21.15.72:0/1569075997 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb3d40d13c0 0x7fb3d40b1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.971 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.966+0000 7fb3d89ff640 1 --2- 172.21.15.72:0/1569075997 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb3d40cd140 0x7fb3d414f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.966+0000 7fb3d9200640 1 -- 172.21.15.72:0/1569075997 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb3d40d13c0 msgr2=0x7fb3d40b1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.966+0000 7fb3d9200640 1 --2- 172.21.15.72:0/1569075997 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb3d40d13c0 0x7fb3d40b1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.966+0000 7fb3d9200640 1 -- 172.21.15.72:0/1569075997 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb3d40cd140 msgr2=0x7fb3d414f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.966+0000 7fb3d9200640 1 --2- 172.21.15.72:0/1569075997 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb3d40cd140 0x7fb3d414f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.966+0000 7fb3d9200640 1 -- 172.21.15.72:0/1569075997 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3d414fe90 con 0x7fb3d415ead0 2024-04-15T15:17:06.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.966+0000 7fb3d3fff640 1 --2- 172.21.15.72:0/1569075997 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb3d40d13c0 0x7fb3d40b1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:06.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.966+0000 7fb3d9200640 1 --2- 172.21.15.72:0/1569075997 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3d415ead0 0x7fb3d40ae1e0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc007e60 tx=0x7fb3cc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:06.972 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.966+0000 7fb3d17fa640 1 -- 172.21.15.72:0/1569075997 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc014420 con 0x7fb3d415ead0 2024-04-15T15:17:06.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.966+0000 7fb3d17fa640 1 -- 172.21.15.72:0/1569075997 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3cc019070 con 0x7fb3d415ead0 2024-04-15T15:17:06.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.966+0000 7fb3dac63640 1 -- 172.21.15.72:0/1569075997 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3d40caa00 con 0x7fb3d415ead0 2024-04-15T15:17:06.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.966+0000 7fb3d17fa640 1 -- 172.21.15.72:0/1569075997 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc014820 con 0x7fb3d415ead0 2024-04-15T15:17:06.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.966+0000 7fb3dac63640 1 -- 172.21.15.72:0/1569075997 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3d40cafc0 con 0x7fb3d415ead0 2024-04-15T15:17:06.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.967+0000 7fb3dac63640 1 -- 172.21.15.72:0/1569075997 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb3d40689e0 con 0x7fb3d415ead0 2024-04-15T15:17:06.973 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.968+0000 7fb3d17fa640 1 -- 172.21.15.72:0/1569075997 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fb3cc0149c0 con 0x7fb3d415ead0 2024-04-15T15:17:06.974 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.968+0000 7fb3d17fa640 1 --2- 172.21.15.72:0/1569075997 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb3c006c980 0x7fb3c006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.968+0000 7fb3d17fa640 1 -- 172.21.15.72:0/1569075997 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5421+0+0 (secure 0 0 0) 0x7fb3cc08d100 con 0x7fb3d415ead0 2024-04-15T15:17:06.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.968+0000 7fb3d89ff640 1 --2- 172.21.15.72:0/1569075997 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb3c006c980 0x7fb3c006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.968+0000 7fb3d17fa640 1 --2- 172.21.15.72:0/1569075997 >> [v2:172.21.15.72:6824/3675605364,v1:172.21.15.72:6825/3675605364] conn(0x7fb3c0072ec0 0x7fb3c0075300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.968+0000 7fb3d17fa640 1 -- 172.21.15.72:0/1569075997 --> [v2:172.21.15.72:6824/3675605364,v1:172.21.15.72:6825/3675605364] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb3cc014e50 con 0x7fb3c0072ec0 2024-04-15T15:17:06.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.968+0000 7fb3d17fa640 1 -- 172.21.15.72:0/1569075997 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fb3cc08d5e0 con 0x7fb3d415ead0 2024-04-15T15:17:06.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.969+0000 7fb3d3fff640 1 --2- 172.21.15.72:0/1569075997 >> [v2:172.21.15.72:6824/3675605364,v1:172.21.15.72:6825/3675605364] conn(0x7fb3c0072ec0 0x7fb3c0075300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.969+0000 7fb3d89ff640 1 --2- 172.21.15.72:0/1569075997 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb3c006c980 0x7fb3c006ee40 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4036000 tx=0x7fb3c403b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:06.975 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.969+0000 7fb3d3fff640 1 --2- 172.21.15.72:0/1569075997 >> [v2:172.21.15.72:6824/3675605364,v1:172.21.15.72:6825/3675605364] conn(0x7fb3c0072ec0 0x7fb3c0075300 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:06.976 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.969+0000 7fb3d17fa640 1 -- 172.21.15.72:0/1569075997 <== osd.1 v2:172.21.15.72:6824/3675605364 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb3cc014e50 con 0x7fb3c0072ec0 2024-04-15T15:17:06.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.982+0000 7fb3dac63640 1 -- 172.21.15.72:0/1569075997 --> [v2:172.21.15.72:6824/3675605364,v1:172.21.15.72:6825/3675605364] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fb3d40d1ff0 con 0x7fb3c0072ec0 2024-04-15T15:17:06.984 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.983+0000 7fb3d17fa640 1 -- 172.21.15.72:0/1569075997 <== osd.1 v2:172.21.15.72:6824/3675605364 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fb3d40d1ff0 con 0x7fb3c0072ec0 2024-04-15T15:17:06.984 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-04-15T15:17:06.985 INFO:teuthology.orchestra.run.smithi072.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-15T15:17:06.985 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-04-15T15:17:06.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.983+0000 7fb3dac63640 1 -- 172.21.15.72:0/1569075997 >> [v2:172.21.15.72:6824/3675605364,v1:172.21.15.72:6825/3675605364] conn(0x7fb3c0072ec0 msgr2=0x7fb3c0075300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.983+0000 7fb3dac63640 1 --2- 172.21.15.72:0/1569075997 >> [v2:172.21.15.72:6824/3675605364,v1:172.21.15.72:6825/3675605364] conn(0x7fb3c0072ec0 0x7fb3c0075300 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.983+0000 7fb3dac63640 1 -- 172.21.15.72:0/1569075997 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb3c006c980 msgr2=0x7fb3c006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.983+0000 7fb3dac63640 1 --2- 172.21.15.72:0/1569075997 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb3c006c980 0x7fb3c006ee40 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4036000 tx=0x7fb3c403b040 comp rx=0 tx=0).stop 2024-04-15T15:17:06.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.983+0000 7fb3dac63640 1 -- 172.21.15.72:0/1569075997 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3d415ead0 msgr2=0x7fb3d40ae1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.983+0000 7fb3dac63640 1 --2- 172.21.15.72:0/1569075997 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb3d415ead0 0x7fb3d40ae1e0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc007e60 tx=0x7fb3cc00e4b0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.983+0000 7fb3d9200640 1 -- 172.21.15.72:0/1569075997 reap_dead start 2024-04-15T15:17:06.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.983+0000 7fb3dac63640 1 -- 172.21.15.72:0/1569075997 shutdown_connections 2024-04-15T15:17:06.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.983+0000 7fb3dac63640 1 -- 172.21.15.72:0/1569075997 >> 172.21.15.72:0/1569075997 conn(0x7fb3d40d23e0 msgr2=0x7fb3d40cbfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:06.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.983+0000 7fb3dac63640 1 -- 172.21.15.72:0/1569075997 shutdown_connections 2024-04-15T15:17:06.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:06.983+0000 7fb3dac63640 1 -- 172.21.15.72:0/1569075997 wait complete. 2024-04-15T15:17:06.995 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-15T15:17:07.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.099+0000 7f8727567640 1 Processor -- start 2024-04-15T15:17:07.100 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.099+0000 7f8727567640 1 -- start start 2024-04-15T15:17:07.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.100+0000 7f8727567640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f87200ce2f0 0x7f872015c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.102 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.100+0000 7f8727567640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f87200d08f0 0x7f87200cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.100+0000 7f8727567640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f872006bbb0 con 0x7f87200ce6d0 2024-04-15T15:17:07.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.100+0000 7f8727567640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f872006b960 con 0x7f87200ce2f0 2024-04-15T15:17:07.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.100+0000 7f8727567640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f872006be90 con 0x7f87200d08f0 2024-04-15T15:17:07.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.100+0000 7f8725303640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f87200ce2f0 0x7f872015c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.100+0000 7f8725303640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f87200ce2f0 0x7f872015c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36836/0 (socket says 172.21.15.72:36836) 2024-04-15T15:17:07.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.100+0000 7f8725b04640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f87200ce6d0 0x7f8720155600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:34032/0 (socket says 172.21.15.72:34032) 2024-04-15T15:17:07.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.100+0000 7f8725303640 1 -- 172.21.15.72:0/3993372017 learned_addr learned my addr 172.21.15.72:0/3993372017 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:07.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.100+0000 7f8725303640 1 -- 172.21.15.72:0/3993372017 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f87200d08f0 msgr2=0x7f87200cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:07.103 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.100+0000 7f8724b02640 1 --2- 172.21.15.72:0/3993372017 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f87200d08f0 0x7f87200cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.100+0000 7f8717fff640 1 -- 172.21.15.72:0/3993372017 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1668771329 0 0) 0x7f872006bbb0 con 0x7f87200ce6d0 2024-04-15T15:17:07.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.100+0000 7f8725303640 1 --2- 172.21.15.72:0/3993372017 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f87200d08f0 0x7f87200cc0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.100+0000 7f8725303640 1 -- 172.21.15.72:0/3993372017 >> v1:172.21.15.72:6789/0 conn(0x7f87200ce6d0 legacy=0x7f8720155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.100+0000 7f8725303640 1 -- 172.21.15.72:0/3993372017 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f872006c170 con 0x7f87200ce2f0 2024-04-15T15:17:07.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.100+0000 7f8724b02640 1 --2- 172.21.15.72:0/3993372017 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f87200d08f0 0x7f87200cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:07.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.101+0000 7f8725303640 1 --2- 172.21.15.72:0/3993372017 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87200ce2f0 0x7f872015c530 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f871c009870 tx=0x7f871c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94cdd420d70e98b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:07.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.101+0000 7f8717fff640 1 -- 172.21.15.72:0/3993372017 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871c034070 con 0x7f87200ce2f0 2024-04-15T15:17:07.104 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.101+0000 7f8717fff640 1 -- 172.21.15.72:0/3993372017 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f871c030840 con 0x7f87200ce2f0 2024-04-15T15:17:07.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.101+0000 7f8717fff640 1 -- 172.21.15.72:0/3993372017 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871c030c10 con 0x7f87200ce2f0 2024-04-15T15:17:07.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.101+0000 7f8727567640 1 -- 172.21.15.72:0/3993372017 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87200ce2f0 msgr2=0x7f872015c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.101+0000 7f8727567640 1 --2- 172.21.15.72:0/3993372017 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87200ce2f0 0x7f872015c530 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f871c009870 tx=0x7f871c02b980 comp rx=0 tx=0).stop 2024-04-15T15:17:07.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.101+0000 7f8727567640 1 -- 172.21.15.72:0/3993372017 shutdown_connections 2024-04-15T15:17:07.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.101+0000 7f8727567640 1 --2- 172.21.15.72:0/3993372017 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f87200d08f0 0x7f87200cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.101+0000 7f8727567640 1 --2- 172.21.15.72:0/3993372017 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87200ce2f0 0x7f872015c530 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.101+0000 7f8727567640 1 -- 172.21.15.72:0/3993372017 >> 172.21.15.72:0/3993372017 conn(0x7f87200b31d0 msgr2=0x7f87200b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:07.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.102+0000 7f8727567640 1 -- 172.21.15.72:0/3993372017 shutdown_connections 2024-04-15T15:17:07.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.102+0000 7f8727567640 1 -- 172.21.15.72:0/3993372017 wait complete. 2024-04-15T15:17:07.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.103+0000 7f8727567640 1 Processor -- start 2024-04-15T15:17:07.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.103+0000 7f8727567640 1 -- start start 2024-04-15T15:17:07.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.103+0000 7f8727567640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f87200ce2f0 0x7f872009c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.103+0000 7f8727567640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87200ce6d0 0x7f8720145350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.103+0000 7f8727567640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87200d08f0 0x7f8720145890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.103+0000 7f8727567640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8720155d10 con 0x7f87200ce6d0 2024-04-15T15:17:07.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.103+0000 7f8727567640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f872015ca70 con 0x7f87200d08f0 2024-04-15T15:17:07.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.103+0000 7f8727567640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f8720155e90 con 0x7f87200ce2f0 2024-04-15T15:17:07.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.103+0000 7f8724b02640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87200ce6d0 0x7f8720145350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.103+0000 7f8724b02640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87200ce6d0 0x7f8720145350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36840/0 (socket says 172.21.15.72:36840) 2024-04-15T15:17:07.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.103+0000 7f8724b02640 1 -- 172.21.15.72:0/2956865887 learned_addr learned my addr 172.21.15.72:0/2956865887 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:07.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.104+0000 7f8725b04640 1 --2- 172.21.15.72:0/2956865887 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87200d08f0 0x7f8720145890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.104+0000 7f8725303640 1 --2- 172.21.15.72:0/2956865887 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f87200ce2f0 0x7f872009c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.104+0000 7f8724b02640 1 -- 172.21.15.72:0/2956865887 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f87200ce2f0 msgr2=0x7f872009c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.104+0000 7f8724b02640 1 --2- 172.21.15.72:0/2956865887 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f87200ce2f0 0x7f872009c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.104+0000 7f8724b02640 1 -- 172.21.15.72:0/2956865887 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87200d08f0 msgr2=0x7f8720145890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.104+0000 7f8724b02640 1 --2- 172.21.15.72:0/2956865887 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87200d08f0 0x7f8720145890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.104+0000 7f8724b02640 1 -- 172.21.15.72:0/2956865887 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f872009cd00 con 0x7f87200ce6d0 2024-04-15T15:17:07.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.104+0000 7f8725b04640 1 --2- 172.21.15.72:0/2956865887 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87200d08f0 0x7f8720145890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:07.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.104+0000 7f8724b02640 1 --2- 172.21.15.72:0/2956865887 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87200ce6d0 0x7f8720145350 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f871000c7f0 tx=0x7f871000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:07.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.104+0000 7f8715ffb640 1 -- 172.21.15.72:0/2956865887 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8710015070 con 0x7f87200ce6d0 2024-04-15T15:17:07.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.104+0000 7f8715ffb640 1 -- 172.21.15.72:0/2956865887 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8710007820 con 0x7f87200ce6d0 2024-04-15T15:17:07.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.104+0000 7f8727567640 1 -- 172.21.15.72:0/2956865887 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f872009cfe0 con 0x7f87200ce6d0 2024-04-15T15:17:07.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.104+0000 7f8715ffb640 1 -- 172.21.15.72:0/2956865887 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8710007bd0 con 0x7f87200ce6d0 2024-04-15T15:17:07.114 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.104+0000 7f8727567640 1 -- 172.21.15.72:0/2956865887 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f872009d5a0 con 0x7f87200ce6d0 2024-04-15T15:17:07.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.105+0000 7f8727567640 1 -- 172.21.15.72:0/2956865887 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8720075c50 con 0x7f87200ce6d0 2024-04-15T15:17:07.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.106+0000 7f8715ffb640 1 -- 172.21.15.72:0/2956865887 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f8710018070 con 0x7f87200ce6d0 2024-04-15T15:17:07.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.106+0000 7f8715ffb640 1 --2- 172.21.15.72:0/2956865887 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f870406c980 0x7f870406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.106+0000 7f8715ffb640 1 -- 172.21.15.72:0/2956865887 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5421+0+0 (secure 0 0 0) 0x7f871008bc80 con 0x7f87200ce6d0 2024-04-15T15:17:07.115 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.106+0000 7f8725303640 1 --2- 172.21.15.72:0/2956865887 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f870406c980 0x7f870406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.107+0000 7f8725303640 1 --2- 172.21.15.72:0/2956865887 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f870406c980 0x7f870406ee40 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f871c036000 tx=0x7f871c03b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:07.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.111+0000 7f8715ffb640 1 -- 172.21.15.72:0/2956865887 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f871005a0d0 con 0x7f87200ce6d0 2024-04-15T15:17:07.285 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.283+0000 7f8727567640 1 -- 172.21.15.72:0/2956865887 --> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f8720156560 con 0x7f870406c980 2024-04-15T15:17:07.286 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.284+0000 7f8715ffb640 1 -- 172.21.15.72:0/2956865887 <== mgr.4101 v2:172.21.15.72:6832/3503807805 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f8720156560 con 0x7f870406c980 2024-04-15T15:17:07.286 INFO:teuthology.orchestra.run.smithi072.stderr:instructed osd(s) 1 to deep-scrub 2024-04-15T15:17:07.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.286+0000 7f8727567640 1 -- 172.21.15.72:0/2956865887 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f870406c980 msgr2=0x7f870406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.286+0000 7f8727567640 1 --2- 172.21.15.72:0/2956865887 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f870406c980 0x7f870406ee40 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f871c036000 tx=0x7f871c03b040 comp rx=0 tx=0).stop 2024-04-15T15:17:07.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.286+0000 7f8727567640 1 -- 172.21.15.72:0/2956865887 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87200ce6d0 msgr2=0x7f8720145350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.286+0000 7f8727567640 1 --2- 172.21.15.72:0/2956865887 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87200ce6d0 0x7f8720145350 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f871000c7f0 tx=0x7f871000ccb0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.286+0000 7f8727567640 1 -- 172.21.15.72:0/2956865887 shutdown_connections 2024-04-15T15:17:07.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.286+0000 7f8727567640 1 --2- 172.21.15.72:0/2956865887 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f870406c980 0x7f870406ee40 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.288 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.286+0000 7f8727567640 1 --2- 172.21.15.72:0/2956865887 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f87200d08f0 0x7f8720145890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.286+0000 7f8727567640 1 --2- 172.21.15.72:0/2956865887 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f87200ce6d0 0x7f8720145350 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.286+0000 7f8727567640 1 --2- 172.21.15.72:0/2956865887 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f87200ce2f0 0x7f872009c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.286+0000 7f8727567640 1 -- 172.21.15.72:0/2956865887 >> 172.21.15.72:0/2956865887 conn(0x7f87200b31d0 msgr2=0x7f872015e2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:07.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.286+0000 7f8727567640 1 -- 172.21.15.72:0/2956865887 shutdown_connections 2024-04-15T15:17:07.289 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.286+0000 7f8727567640 1 -- 172.21.15.72:0/2956865887 wait complete. 2024-04-15T15:17:07.298 INFO:tasks.ceph:Scrubbing osd.2 2024-04-15T15:17:07.299 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-15T15:17:07.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.405+0000 7f762b559640 1 Processor -- start 2024-04-15T15:17:07.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.405+0000 7f762b559640 1 -- start start 2024-04-15T15:17:07.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.406+0000 7f762b559640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f762406edc0 0x7f762406f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.406+0000 7f762b559640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76240cf020 con 0x7f76240d0a80 2024-04-15T15:17:07.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.406+0000 7f762b559640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76240cf300 con 0x7f762415e9a0 2024-04-15T15:17:07.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.406+0000 7f762b559640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f76240cedf0 con 0x7f762406edc0 2024-04-15T15:17:07.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.406+0000 7f7629af6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f762406edc0 0x7f762406f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.406+0000 7f7628af4640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f76240d0a80 0x7f76240b42d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:34046/0 (socket says 172.21.15.72:34046) 2024-04-15T15:17:07.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.406+0000 7f7629af6640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f762406edc0 0x7f762406f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36854/0 (socket says 172.21.15.72:36854) 2024-04-15T15:17:07.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.406+0000 7f7628af4640 1 -- 172.21.15.72:0/2779839371 learned_addr learned my addr 172.21.15.72:0/2779839371 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:17:07.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.407+0000 7f7629af6640 1 -- 172.21.15.72:0/2779839371 >> v1:172.21.15.175:6789/0 conn(0x7f762415e9a0 legacy=0x7f762406dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:07.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.407+0000 7f7629af6640 1 -- 172.21.15.72:0/2779839371 >> v1:172.21.15.72:6789/0 conn(0x7f76240d0a80 legacy=0x7f76240b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.407+0000 7f7629af6640 1 -- 172.21.15.72:0/2779839371 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76240cf5e0 con 0x7f762406edc0 2024-04-15T15:17:07.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.407+0000 7f7623fff640 1 -- 172.21.15.72:0/2779839371 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1094737871 0 0) 0x7f76240cf020 con 0x7f76240d0a80 2024-04-15T15:17:07.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.407+0000 7f7629af6640 1 --2- 172.21.15.72:0/2779839371 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f762406edc0 0x7f762406f250 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f761c0097a0 tx=0x7f761c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2231efcdf0262046 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:07.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.408+0000 7f7623fff640 1 -- 172.21.15.72:0/2779839371 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f761c035020 con 0x7f762406edc0 2024-04-15T15:17:07.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.408+0000 7f7623fff640 1 -- 172.21.15.72:0/2779839371 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f761c031830 con 0x7f762406edc0 2024-04-15T15:17:07.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.408+0000 7f7623fff640 1 -- 172.21.15.72:0/2779839371 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f761c031c00 con 0x7f762406edc0 2024-04-15T15:17:07.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.408+0000 7f762b559640 1 -- 172.21.15.72:0/2779839371 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f762406edc0 msgr2=0x7f762406f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.408+0000 7f762b559640 1 --2- 172.21.15.72:0/2779839371 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f762406edc0 0x7f762406f250 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f761c0097a0 tx=0x7f761c02b8d0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.408+0000 7f762b559640 1 -- 172.21.15.72:0/2779839371 shutdown_connections 2024-04-15T15:17:07.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.408+0000 7f762b559640 1 --2- 172.21.15.72:0/2779839371 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f762406edc0 0x7f762406f250 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.408+0000 7f762b559640 1 -- 172.21.15.72:0/2779839371 >> 172.21.15.72:0/2779839371 conn(0x7f76240703a0 msgr2=0x7f7624070780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:07.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.408+0000 7f762b559640 1 -- 172.21.15.72:0/2779839371 shutdown_connections 2024-04-15T15:17:07.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.408+0000 7f762b559640 1 -- 172.21.15.72:0/2779839371 wait complete. 2024-04-15T15:17:07.415 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.409+0000 7f762b559640 1 Processor -- start 2024-04-15T15:17:07.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.409+0000 7f762b559640 1 -- start start 2024-04-15T15:17:07.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.410+0000 7f762b559640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f762406edc0 0x7f76240cc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.410+0000 7f762b559640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f76240d0a80 0x7f7624159420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.410+0000 7f762b559640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f762415e9a0 0x7f76240cb010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.410+0000 7f762b559640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f762415ed80 con 0x7f762406edc0 2024-04-15T15:17:07.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.410+0000 7f762b559640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f762406e480 con 0x7f762415e9a0 2024-04-15T15:17:07.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.410+0000 7f762b559640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f76240cfde0 con 0x7f76240d0a80 2024-04-15T15:17:07.416 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.410+0000 7f76292f5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f762406edc0 0x7f76240cc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.410+0000 7f76292f5640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f762406edc0 0x7f76240cc9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36868/0 (socket says 172.21.15.72:36868) 2024-04-15T15:17:07.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.410+0000 7f76292f5640 1 -- 172.21.15.72:0/2236092210 learned_addr learned my addr 172.21.15.72:0/2236092210 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:07.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.410+0000 7f7628af4640 1 --2- 172.21.15.72:0/2236092210 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f76240d0a80 0x7f7624159420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.410+0000 7f7629af6640 1 --2- 172.21.15.72:0/2236092210 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f762415e9a0 0x7f76240cb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.410+0000 7f76292f5640 1 -- 172.21.15.72:0/2236092210 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f76240d0a80 msgr2=0x7f7624159420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.410+0000 7f76292f5640 1 --2- 172.21.15.72:0/2236092210 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f76240d0a80 0x7f7624159420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.410+0000 7f76292f5640 1 -- 172.21.15.72:0/2236092210 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f762415e9a0 msgr2=0x7f76240cb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.410+0000 7f76292f5640 1 --2- 172.21.15.72:0/2236092210 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f762415e9a0 0x7f76240cb010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.417 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.410+0000 7f76292f5640 1 -- 172.21.15.72:0/2236092210 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7624159e70 con 0x7f762406edc0 2024-04-15T15:17:07.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.411+0000 7f76292f5640 1 --2- 172.21.15.72:0/2236092210 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f762406edc0 0x7f76240cc9c0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f761400b420 tx=0x7f761400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:07.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.411+0000 7f7621ffb640 1 -- 172.21.15.72:0/2236092210 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7614011830 con 0x7f762406edc0 2024-04-15T15:17:07.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.411+0000 7f7621ffb640 1 -- 172.21.15.72:0/2236092210 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7614016070 con 0x7f762406edc0 2024-04-15T15:17:07.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.411+0000 7f762b559640 1 -- 172.21.15.72:0/2236092210 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f762415a150 con 0x7f762406edc0 2024-04-15T15:17:07.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.411+0000 7f7621ffb640 1 -- 172.21.15.72:0/2236092210 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7614011c10 con 0x7f762406edc0 2024-04-15T15:17:07.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.411+0000 7f762b559640 1 -- 172.21.15.72:0/2236092210 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f762415b400 con 0x7f762406edc0 2024-04-15T15:17:07.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.412+0000 7f762b559640 1 -- 172.21.15.72:0/2236092210 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f762415a390 con 0x7f762406edc0 2024-04-15T15:17:07.418 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.412+0000 7f7621ffb640 1 -- 172.21.15.72:0/2236092210 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f761401b460 con 0x7f762406edc0 2024-04-15T15:17:07.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.413+0000 7f7621ffb640 1 --2- 172.21.15.72:0/2236092210 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f761006c9b0 0x7f761006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.413+0000 7f7621ffb640 1 -- 172.21.15.72:0/2236092210 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5421+0+0 (secure 0 0 0) 0x7f761408b4b0 con 0x7f762406edc0 2024-04-15T15:17:07.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.413+0000 7f7628af4640 1 --2- 172.21.15.72:0/2236092210 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f761006c9b0 0x7f761006ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.413+0000 7f7621ffb640 1 --2- 172.21.15.72:0/2236092210 >> [v2:172.21.15.72:6816/4041627082,v1:172.21.15.72:6817/4041627082] conn(0x7f7610072ef0 0x7f7610075330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.413+0000 7f7621ffb640 1 -- 172.21.15.72:0/2236092210 --> [v2:172.21.15.72:6816/4041627082,v1:172.21.15.72:6817/4041627082] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7610075a00 con 0x7f7610072ef0 2024-04-15T15:17:07.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.413+0000 7f7621ffb640 1 -- 172.21.15.72:0/2236092210 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f7614092050 con 0x7f762406edc0 2024-04-15T15:17:07.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.413+0000 7f7629af6640 1 --2- 172.21.15.72:0/2236092210 >> [v2:172.21.15.72:6816/4041627082,v1:172.21.15.72:6817/4041627082] conn(0x7f7610072ef0 0x7f7610075330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.419 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.413+0000 7f7628af4640 1 --2- 172.21.15.72:0/2236092210 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f761006c9b0 0x7f761006ee70 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f762406c6c0 tx=0x7f7618007ed0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:07.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.413+0000 7f7629af6640 1 --2- 172.21.15.72:0/2236092210 >> [v2:172.21.15.72:6816/4041627082,v1:172.21.15.72:6817/4041627082] conn(0x7f7610072ef0 0x7f7610075330 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:07.420 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.414+0000 7f7621ffb640 1 -- 172.21.15.72:0/2236092210 <== osd.2 v2:172.21.15.72:6816/4041627082 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7610075a00 con 0x7f7610072ef0 2024-04-15T15:17:07.428 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.427+0000 7f762b559640 1 -- 172.21.15.72:0/2236092210 --> [v2:172.21.15.72:6816/4041627082,v1:172.21.15.72:6817/4041627082] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f76240ca650 con 0x7f7610072ef0 2024-04-15T15:17:07.429 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.427+0000 7f7621ffb640 1 -- 172.21.15.72:0/2236092210 <== osd.2 v2:172.21.15.72:6816/4041627082 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f761c004040 con 0x7f7610072ef0 2024-04-15T15:17:07.429 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-04-15T15:17:07.429 INFO:teuthology.orchestra.run.smithi072.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-15T15:17:07.429 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-04-15T15:17:07.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.427+0000 7f762b559640 1 -- 172.21.15.72:0/2236092210 >> [v2:172.21.15.72:6816/4041627082,v1:172.21.15.72:6817/4041627082] conn(0x7f7610072ef0 msgr2=0x7f7610075330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.427+0000 7f762b559640 1 --2- 172.21.15.72:0/2236092210 >> [v2:172.21.15.72:6816/4041627082,v1:172.21.15.72:6817/4041627082] conn(0x7f7610072ef0 0x7f7610075330 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.430 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.427+0000 7f762b559640 1 -- 172.21.15.72:0/2236092210 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f761006c9b0 msgr2=0x7f761006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.427+0000 7f762b559640 1 --2- 172.21.15.72:0/2236092210 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f761006c9b0 0x7f761006ee70 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f762406c6c0 tx=0x7f7618007ed0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.427+0000 7f762b559640 1 -- 172.21.15.72:0/2236092210 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f762406edc0 msgr2=0x7f76240cc9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.427+0000 7f762b559640 1 --2- 172.21.15.72:0/2236092210 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f762406edc0 0x7f76240cc9c0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f761400b420 tx=0x7f761400b8e0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.427+0000 7f7629af6640 1 -- 172.21.15.72:0/2236092210 reap_dead start 2024-04-15T15:17:07.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.427+0000 7f762b559640 1 -- 172.21.15.72:0/2236092210 shutdown_connections 2024-04-15T15:17:07.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.427+0000 7f762b559640 1 -- 172.21.15.72:0/2236092210 >> 172.21.15.72:0/2236092210 conn(0x7f76240703a0 msgr2=0x7f7624158200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:07.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.428+0000 7f762b559640 1 -- 172.21.15.72:0/2236092210 shutdown_connections 2024-04-15T15:17:07.431 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.428+0000 7f762b559640 1 -- 172.21.15.72:0/2236092210 wait complete. 2024-04-15T15:17:07.440 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-15T15:17:07.543 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.542+0000 7f9d94288640 1 Processor -- start 2024-04-15T15:17:07.544 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.542+0000 7f9d94288640 1 -- start start 2024-04-15T15:17:07.545 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.543+0000 7f9d94288640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9d8c15c650 0x7f9d8c0b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.543+0000 7f9d94288640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f9d8c0d0a60 0x7f9d8c0d0e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.543+0000 7f9d94288640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d8c0d02d0 con 0x7f9d8c0b36f0 2024-04-15T15:17:07.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.543+0000 7f9d94288640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f9d8c15ca30 con 0x7f9d8c15c650 2024-04-15T15:17:07.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.543+0000 7f9d94288640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f9d8c0b26b0 con 0x7f9d8c0d0a60 2024-04-15T15:17:07.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.543+0000 7f9d92024640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9d8c15c650 0x7f9d8c0b1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.543+0000 7f9d92024640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f9d8c15c650 0x7f9d8c0b1d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36872/0 (socket says 172.21.15.72:36872) 2024-04-15T15:17:07.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.543+0000 7f9d92024640 1 -- 172.21.15.72:0/1745603151 learned_addr learned my addr 172.21.15.72:0/1745603151 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:07.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.543+0000 7f9d92825640 1 --2- 172.21.15.72:0/1745603151 >> v2:172.21.15.175:3300/0 conn(0x7f9d8c0d0a60 0x7f9d8c0d0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.543+0000 7f9d92024640 1 -- 172.21.15.72:0/1745603151 >> v2:172.21.15.175:3300/0 conn(0x7f9d8c0d0a60 msgr2=0x7f9d8c0d0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:07.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.543+0000 7f9d92024640 1 --2- 172.21.15.72:0/1745603151 >> v2:172.21.15.175:3300/0 conn(0x7f9d8c0d0a60 0x7f9d8c0d0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.543+0000 7f9d92024640 1 -- 172.21.15.72:0/1745603151 >> v1:172.21.15.175:6789/0 conn(0x7f9d8c0b36f0 legacy=0x7f9d8c0b5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:07.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.543+0000 7f9d92024640 1 -- 172.21.15.72:0/1745603151 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d8c0b2990 con 0x7f9d8c15c650 2024-04-15T15:17:07.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.544+0000 7f9d92024640 1 --2- 172.21.15.72:0/1745603151 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d8c15c650 0x7f9d8c0b1d40 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c0b1bd0 tx=0x7f9d8802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7448719f2f173621 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:07.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.544+0000 7f9d91022640 1 -- 172.21.15.72:0/1745603151 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d8802ca80 con 0x7f9d8c15c650 2024-04-15T15:17:07.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.544+0000 7f9d91022640 1 -- 172.21.15.72:0/1745603151 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d8802cc20 con 0x7f9d8c15c650 2024-04-15T15:17:07.547 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.544+0000 7f9d91022640 1 -- 172.21.15.72:0/1745603151 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d880343f0 con 0x7f9d8c15c650 2024-04-15T15:17:07.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.545+0000 7f9d94288640 1 -- 172.21.15.72:0/1745603151 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d8c15c650 msgr2=0x7f9d8c0b1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.545+0000 7f9d94288640 1 --2- 172.21.15.72:0/1745603151 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d8c15c650 0x7f9d8c0b1d40 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c0b1bd0 tx=0x7f9d8802bee0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.545+0000 7f9d94288640 1 -- 172.21.15.72:0/1745603151 shutdown_connections 2024-04-15T15:17:07.551 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.545+0000 7f9d94288640 1 --2- 172.21.15.72:0/1745603151 >> v2:172.21.15.175:3300/0 conn(0x7f9d8c0d0a60 0x7f9d8c0d0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.545+0000 7f9d94288640 1 --2- 172.21.15.72:0/1745603151 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d8c15c650 0x7f9d8c0b1d40 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.545+0000 7f9d94288640 1 -- 172.21.15.72:0/1745603151 >> 172.21.15.72:0/1745603151 conn(0x7f9d8c0ceec0 msgr2=0x7f9d8c0cf2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:07.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.545+0000 7f9d94288640 1 -- 172.21.15.72:0/1745603151 shutdown_connections 2024-04-15T15:17:07.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.545+0000 7f9d94288640 1 -- 172.21.15.72:0/1745603151 wait complete. 2024-04-15T15:17:07.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.546+0000 7f9d94288640 1 Processor -- start 2024-04-15T15:17:07.552 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.546+0000 7f9d94288640 1 -- start start 2024-04-15T15:17:07.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.547+0000 7f9d94288640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d8c0b36f0 0x7f9d8c1596b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.547+0000 7f9d94288640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9d8c0d0a60 0x7f9d8c15a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.547+0000 7f9d94288640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9d8c15c650 0x7f9d8c15a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.547+0000 7f9d94288640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f9d8c0b59a0 con 0x7f9d8c0b36f0 2024-04-15T15:17:07.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.547+0000 7f9d94288640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f9d8c0b2c70 con 0x7f9d8c15c650 2024-04-15T15:17:07.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.547+0000 7f9d94288640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f9d8c0b2280 con 0x7f9d8c0d0a60 2024-04-15T15:17:07.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.547+0000 7f9d92024640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d8c0b36f0 0x7f9d8c1596b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.547+0000 7f9d92024640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d8c0b36f0 0x7f9d8c1596b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36882/0 (socket says 172.21.15.72:36882) 2024-04-15T15:17:07.553 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.547+0000 7f9d92024640 1 -- 172.21.15.72:0/3870599422 learned_addr learned my addr 172.21.15.72:0/3870599422 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:07.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.547+0000 7f9d92825640 1 --2- 172.21.15.72:0/3870599422 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9d8c15c650 0x7f9d8c15a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.547+0000 7f9d91823640 1 --2- 172.21.15.72:0/3870599422 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9d8c0d0a60 0x7f9d8c15a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.547+0000 7f9d92024640 1 -- 172.21.15.72:0/3870599422 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9d8c0d0a60 msgr2=0x7f9d8c15a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.547+0000 7f9d92024640 1 --2- 172.21.15.72:0/3870599422 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9d8c0d0a60 0x7f9d8c15a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.554 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.547+0000 7f9d92024640 1 -- 172.21.15.72:0/3870599422 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9d8c15c650 msgr2=0x7f9d8c15a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.547+0000 7f9d92024640 1 --2- 172.21.15.72:0/3870599422 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9d8c15c650 0x7f9d8c15a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.547+0000 7f9d92024640 1 -- 172.21.15.72:0/3870599422 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d8c1e61b0 con 0x7f9d8c0b36f0 2024-04-15T15:17:07.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.547+0000 7f9d92024640 1 --2- 172.21.15.72:0/3870599422 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d8c0b36f0 0x7f9d8c1596b0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f9d88000f80 tx=0x7f9d88004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:07.555 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.547+0000 7f9d82ffd640 1 -- 172.21.15.72:0/3870599422 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d88043020 con 0x7f9d8c0b36f0 2024-04-15T15:17:07.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.547+0000 7f9d82ffd640 1 -- 172.21.15.72:0/3870599422 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d8803e450 con 0x7f9d8c0b36f0 2024-04-15T15:17:07.557 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.547+0000 7f9d94288640 1 -- 172.21.15.72:0/3870599422 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d8c1e6490 con 0x7f9d8c0b36f0 2024-04-15T15:17:07.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.547+0000 7f9d82ffd640 1 -- 172.21.15.72:0/3870599422 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d88009510 con 0x7f9d8c0b36f0 2024-04-15T15:17:07.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.547+0000 7f9d94288640 1 -- 172.21.15.72:0/3870599422 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d8c1e6a50 con 0x7f9d8c0b36f0 2024-04-15T15:17:07.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.548+0000 7f9d94288640 1 -- 172.21.15.72:0/3870599422 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d8c075c50 con 0x7f9d8c0b36f0 2024-04-15T15:17:07.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.549+0000 7f9d82ffd640 1 -- 172.21.15.72:0/3870599422 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f9d88046020 con 0x7f9d8c0b36f0 2024-04-15T15:17:07.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.549+0000 7f9d82ffd640 1 --2- 172.21.15.72:0/3870599422 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f9d7006c980 0x7f9d7006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.549+0000 7f9d82ffd640 1 -- 172.21.15.72:0/3870599422 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5421+0+0 (secure 0 0 0) 0x7f9d880ae2f0 con 0x7f9d8c0b36f0 2024-04-15T15:17:07.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.549+0000 7f9d91823640 1 --2- 172.21.15.72:0/3870599422 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f9d7006c980 0x7f9d7006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.558 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.550+0000 7f9d91823640 1 --2- 172.21.15.72:0/3870599422 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f9d7006c980 0x7f9d7006ee40 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c0cedb0 tx=0x7f9d7c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:07.559 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.554+0000 7f9d82ffd640 1 -- 172.21.15.72:0/3870599422 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9d8804d030 con 0x7f9d8c0b36f0 2024-04-15T15:17:07.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.727+0000 7f9d94288640 1 -- 172.21.15.72:0/3870599422 --> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f9d8c0d0010 con 0x7f9d7006c980 2024-04-15T15:17:07.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.727+0000 7f9d82ffd640 1 -- 172.21.15.72:0/3870599422 <== mgr.4101 v2:172.21.15.72:6832/3503807805 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f9d8c0d0010 con 0x7f9d7006c980 2024-04-15T15:17:07.729 INFO:teuthology.orchestra.run.smithi072.stderr:instructed osd(s) 2 to deep-scrub 2024-04-15T15:17:07.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.729+0000 7f9d94288640 1 -- 172.21.15.72:0/3870599422 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f9d7006c980 msgr2=0x7f9d7006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.729+0000 7f9d94288640 1 --2- 172.21.15.72:0/3870599422 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f9d7006c980 0x7f9d7006ee40 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c0cedb0 tx=0x7f9d7c009040 comp rx=0 tx=0).stop 2024-04-15T15:17:07.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.729+0000 7f9d94288640 1 -- 172.21.15.72:0/3870599422 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d8c0b36f0 msgr2=0x7f9d8c1596b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.729+0000 7f9d94288640 1 --2- 172.21.15.72:0/3870599422 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d8c0b36f0 0x7f9d8c1596b0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f9d88000f80 tx=0x7f9d88004a40 comp rx=0 tx=0).stop 2024-04-15T15:17:07.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.729+0000 7f9d94288640 1 -- 172.21.15.72:0/3870599422 shutdown_connections 2024-04-15T15:17:07.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.729+0000 7f9d94288640 1 --2- 172.21.15.72:0/3870599422 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f9d7006c980 0x7f9d7006ee40 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.731 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.729+0000 7f9d94288640 1 --2- 172.21.15.72:0/3870599422 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f9d8c15c650 0x7f9d8c15a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.729+0000 7f9d94288640 1 --2- 172.21.15.72:0/3870599422 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f9d8c0d0a60 0x7f9d8c15a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.729+0000 7f9d94288640 1 --2- 172.21.15.72:0/3870599422 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9d8c0b36f0 0x7f9d8c1596b0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.729+0000 7f9d94288640 1 -- 172.21.15.72:0/3870599422 >> 172.21.15.72:0/3870599422 conn(0x7f9d8c0ceec0 msgr2=0x7f9d8c06fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:07.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.729+0000 7f9d94288640 1 -- 172.21.15.72:0/3870599422 shutdown_connections 2024-04-15T15:17:07.732 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.729+0000 7f9d94288640 1 -- 172.21.15.72:0/3870599422 wait complete. 2024-04-15T15:17:07.742 INFO:tasks.ceph:Scrubbing osd.3 2024-04-15T15:17:07.742 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-15T15:17:07.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.847+0000 7fb9a4e47640 1 Processor -- start 2024-04-15T15:17:07.849 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.847+0000 7fb9a4e47640 1 -- start start 2024-04-15T15:17:07.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.848+0000 7fb9a4e47640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb9a015c630 0x7fb9a00703b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.848+0000 7fb9a4e47640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb9a00708f0 0x7fb9a006d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.850 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.848+0000 7fb9a4e47640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9a00cf970 con 0x7fb9a006d740 2024-04-15T15:17:07.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.848+0000 7fb9a4e47640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fb9a00d0580 con 0x7fb9a00708f0 2024-04-15T15:17:07.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.848+0000 7fb9a4e47640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fb9a0070d50 con 0x7fb9a015c630 2024-04-15T15:17:07.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.848+0000 7fb99dd9b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb9a00708f0 0x7fb9a006d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.848+0000 7fb99dd9b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fb9a00708f0 0x7fb9a006d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36888/0 (socket says 172.21.15.72:36888) 2024-04-15T15:17:07.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.848+0000 7fb99dd9b640 1 -- 172.21.15.72:0/2120955049 learned_addr learned my addr 172.21.15.72:0/2120955049 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:07.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.848+0000 7fb99dd9b640 1 -- 172.21.15.72:0/2120955049 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb9a015c630 msgr2=0x7fb9a00703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:07.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.849+0000 7fb99e59c640 1 --2- 172.21.15.72:0/2120955049 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb9a015c630 0x7fb9a00703b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.849+0000 7fb99dd9b640 1 --2- 172.21.15.72:0/2120955049 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb9a015c630 0x7fb9a00703b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.851 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.849+0000 7fb99dd9b640 1 -- 172.21.15.72:0/2120955049 >> v1:172.21.15.175:6789/0 conn(0x7fb9a006d740 legacy=0x7fb9a0160200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:07.852 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.849+0000 7fb99dd9b640 1 -- 172.21.15.72:0/2120955049 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9a00cfc50 con 0x7fb9a00708f0 2024-04-15T15:17:07.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.849+0000 7fb99dd9b640 1 --2- 172.21.15.72:0/2120955049 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9a00708f0 0x7fb9a006d200 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fb994009870 tx=0x7fb99402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33906bdccef1fd13 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:07.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.850+0000 7fb99d59a640 1 -- 172.21.15.72:0/2120955049 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb994035020 con 0x7fb9a00708f0 2024-04-15T15:17:07.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.850+0000 7fb99d59a640 1 -- 172.21.15.72:0/2120955049 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb994031640 con 0x7fb9a00708f0 2024-04-15T15:17:07.855 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.850+0000 7fb99d59a640 1 -- 172.21.15.72:0/2120955049 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99402d040 con 0x7fb9a00708f0 2024-04-15T15:17:07.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.850+0000 7fb9a4e47640 1 -- 172.21.15.72:0/2120955049 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9a00708f0 msgr2=0x7fb9a006d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.850+0000 7fb9a4e47640 1 --2- 172.21.15.72:0/2120955049 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9a00708f0 0x7fb9a006d200 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fb994009870 tx=0x7fb99402b8f0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.850+0000 7fb9a4e47640 1 -- 172.21.15.72:0/2120955049 shutdown_connections 2024-04-15T15:17:07.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.850+0000 7fb9a4e47640 1 --2- 172.21.15.72:0/2120955049 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9a00708f0 0x7fb9a006d200 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.850+0000 7fb9a4e47640 1 --2- 172.21.15.72:0/2120955049 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb9a015c630 0x7fb9a00703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.856 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.850+0000 7fb9a4e47640 1 -- 172.21.15.72:0/2120955049 >> 172.21.15.72:0/2120955049 conn(0x7fb9a0072420 msgr2=0x7fb9a0072800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:07.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.850+0000 7fb9a4e47640 1 -- 172.21.15.72:0/2120955049 shutdown_connections 2024-04-15T15:17:07.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.850+0000 7fb9a4e47640 1 -- 172.21.15.72:0/2120955049 wait complete. 2024-04-15T15:17:07.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.851+0000 7fb9a4e47640 1 Processor -- start 2024-04-15T15:17:07.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.851+0000 7fb9a4e47640 1 -- start start 2024-04-15T15:17:07.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.852+0000 7fb9a4e47640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb9a006d740 0x7fb9a01f4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.852+0000 7fb9a4e47640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb9a00708f0 0x7fb9a0151c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.852+0000 7fb9a4e47640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9a015c630 0x7fb9a0156b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.857 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.852+0000 7fb9a4e47640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a015ca10 con 0x7fb9a015c630 2024-04-15T15:17:07.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.852+0000 7fb9a4e47640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a00d0210 con 0x7fb9a00708f0 2024-04-15T15:17:07.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.852+0000 7fb9a4e47640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fb9a00ce2d0 con 0x7fb9a006d740 2024-04-15T15:17:07.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.852+0000 7fb99ed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9a015c630 0x7fb9a0156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.852+0000 7fb99ed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9a015c630 0x7fb9a0156b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36892/0 (socket says 172.21.15.72:36892) 2024-04-15T15:17:07.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.852+0000 7fb99ed9d640 1 -- 172.21.15.72:0/1144222873 learned_addr learned my addr 172.21.15.72:0/1144222873 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:07.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.852+0000 7fb99dd9b640 1 --2- 172.21.15.72:0/1144222873 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb9a00708f0 0x7fb9a0151c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.858 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.852+0000 7fb99e59c640 1 --2- 172.21.15.72:0/1144222873 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb9a006d740 0x7fb9a01f4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.853+0000 7fb99ed9d640 1 -- 172.21.15.72:0/1144222873 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb9a006d740 msgr2=0x7fb9a01f4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.853+0000 7fb99ed9d640 1 --2- 172.21.15.72:0/1144222873 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fb9a006d740 0x7fb9a01f4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.853+0000 7fb99ed9d640 1 -- 172.21.15.72:0/1144222873 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb9a00708f0 msgr2=0x7fb9a0151c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.853+0000 7fb99ed9d640 1 --2- 172.21.15.72:0/1144222873 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fb9a00708f0 0x7fb9a0151c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.853+0000 7fb99ed9d640 1 -- 172.21.15.72:0/1144222873 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9a01f5430 con 0x7fb9a015c630 2024-04-15T15:17:07.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.853+0000 7fb99ed9d640 1 --2- 172.21.15.72:0/1144222873 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9a015c630 0x7fb9a0156b20 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fb99800b970 tx=0x7fb99800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:07.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.853+0000 7fb98effd640 1 -- 172.21.15.72:0/1144222873 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb998019070 con 0x7fb9a015c630 2024-04-15T15:17:07.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.853+0000 7fb98effd640 1 -- 172.21.15.72:0/1144222873 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb99800cb40 con 0x7fb9a015c630 2024-04-15T15:17:07.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.853+0000 7fb9a4e47640 1 -- 172.21.15.72:0/1144222873 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9a01f2430 con 0x7fb9a015c630 2024-04-15T15:17:07.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.853+0000 7fb98effd640 1 -- 172.21.15.72:0/1144222873 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb998014440 con 0x7fb9a015c630 2024-04-15T15:17:07.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.853+0000 7fb9a4e47640 1 -- 172.21.15.72:0/1144222873 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9a01f29f0 con 0x7fb9a015c630 2024-04-15T15:17:07.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.854+0000 7fb9a4e47640 1 -- 172.21.15.72:0/1144222873 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb9a00689e0 con 0x7fb9a015c630 2024-04-15T15:17:07.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.854+0000 7fb98effd640 1 -- 172.21.15.72:0/1144222873 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fb99801c070 con 0x7fb9a015c630 2024-04-15T15:17:07.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.855+0000 7fb98effd640 1 --2- 172.21.15.72:0/1144222873 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb98006c980 0x7fb98006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.855+0000 7fb98effd640 1 -- 172.21.15.72:0/1144222873 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5421+0+0 (secure 0 0 0) 0x7fb99808b800 con 0x7fb9a015c630 2024-04-15T15:17:07.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.855+0000 7fb99e59c640 1 --2- 172.21.15.72:0/1144222873 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb98006c980 0x7fb98006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.855+0000 7fb98effd640 1 --2- 172.21.15.72:0/1144222873 >> [v2:172.21.15.72:6808/3973649661,v1:172.21.15.72:6809/3973649661] conn(0x7fb980072ec0 0x7fb980075300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.855+0000 7fb98effd640 1 -- 172.21.15.72:0/1144222873 --> [v2:172.21.15.72:6808/3973649661,v1:172.21.15.72:6809/3973649661] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb9800759d0 con 0x7fb980072ec0 2024-04-15T15:17:07.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.855+0000 7fb98effd640 1 -- 172.21.15.72:0/1144222873 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fb9980902c0 con 0x7fb9a015c630 2024-04-15T15:17:07.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.855+0000 7fb99dd9b640 1 --2- 172.21.15.72:0/1144222873 >> [v2:172.21.15.72:6808/3973649661,v1:172.21.15.72:6809/3973649661] conn(0x7fb980072ec0 0x7fb980075300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.855+0000 7fb99e59c640 1 --2- 172.21.15.72:0/1144222873 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb98006c980 0x7fb98006ee40 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fb9a01543d0 tx=0x7fb990008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:07.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.855+0000 7fb99dd9b640 1 --2- 172.21.15.72:0/1144222873 >> [v2:172.21.15.72:6808/3973649661,v1:172.21.15.72:6809/3973649661] conn(0x7fb980072ec0 0x7fb980075300 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:07.863 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.856+0000 7fb98effd640 1 -- 172.21.15.72:0/1144222873 <== osd.3 v2:172.21.15.72:6808/3973649661 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb9800759d0 con 0x7fb980072ec0 2024-04-15T15:17:07.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.869+0000 7fb9a4e47640 1 -- 172.21.15.72:0/1144222873 --> [v2:172.21.15.72:6808/3973649661,v1:172.21.15.72:6809/3973649661] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fb9a0161080 con 0x7fb980072ec0 2024-04-15T15:17:07.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.869+0000 7fb98effd640 1 -- 172.21.15.72:0/1144222873 <== osd.3 v2:172.21.15.72:6808/3973649661 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fb9a0161080 con 0x7fb980072ec0 2024-04-15T15:17:07.873 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-04-15T15:17:07.873 INFO:teuthology.orchestra.run.smithi072.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-15T15:17:07.873 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-04-15T15:17:07.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.870+0000 7fb9a4e47640 1 -- 172.21.15.72:0/1144222873 >> [v2:172.21.15.72:6808/3973649661,v1:172.21.15.72:6809/3973649661] conn(0x7fb980072ec0 msgr2=0x7fb980075300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.870+0000 7fb9a4e47640 1 --2- 172.21.15.72:0/1144222873 >> [v2:172.21.15.72:6808/3973649661,v1:172.21.15.72:6809/3973649661] conn(0x7fb980072ec0 0x7fb980075300 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.870+0000 7fb9a4e47640 1 -- 172.21.15.72:0/1144222873 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb98006c980 msgr2=0x7fb98006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.870+0000 7fb9a4e47640 1 --2- 172.21.15.72:0/1144222873 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fb98006c980 0x7fb98006ee40 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fb9a01543d0 tx=0x7fb990008040 comp rx=0 tx=0).stop 2024-04-15T15:17:07.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.870+0000 7fb9a4e47640 1 -- 172.21.15.72:0/1144222873 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9a015c630 msgr2=0x7fb9a0156b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.870+0000 7fb9a4e47640 1 --2- 172.21.15.72:0/1144222873 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb9a015c630 0x7fb9a0156b20 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fb99800b970 tx=0x7fb99800be30 comp rx=0 tx=0).stop 2024-04-15T15:17:07.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.870+0000 7fb99ed9d640 1 -- 172.21.15.72:0/1144222873 reap_dead start 2024-04-15T15:17:07.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.870+0000 7fb9a4e47640 1 -- 172.21.15.72:0/1144222873 shutdown_connections 2024-04-15T15:17:07.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.870+0000 7fb9a4e47640 1 -- 172.21.15.72:0/1144222873 >> 172.21.15.72:0/1144222873 conn(0x7fb9a0072420 msgr2=0x7fb9a00b4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:07.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.870+0000 7fb9a4e47640 1 -- 172.21.15.72:0/1144222873 shutdown_connections 2024-04-15T15:17:07.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.870+0000 7fb9a4e47640 1 -- 172.21.15.72:0/1144222873 wait complete. 2024-04-15T15:17:07.882 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-15T15:17:07.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.985+0000 7f3d46738640 1 Processor -- start 2024-04-15T15:17:07.987 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.985+0000 7f3d46738640 1 -- start start 2024-04-15T15:17:07.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.985+0000 7f3d46738640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3d4015e610 0x7f3d400d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.985+0000 7f3d46738640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d400d2320 con 0x7f3d4006c400 2024-04-15T15:17:07.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.985+0000 7f3d46738640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d400d25c0 con 0x7f3d4015e9f0 2024-04-15T15:17:07.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.985+0000 7f3d46738640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f3d400d31d0 con 0x7f3d4015e610 2024-04-15T15:17:07.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.985+0000 7f3d3ffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3d4015e610 0x7f3d400d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.986+0000 7f3d3ffff640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f3d4015e610 0x7f3d400d0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36896/0 (socket says 172.21.15.72:36896) 2024-04-15T15:17:07.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.986+0000 7f3d44cd5640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f3d4006c400 0x7f3d400b7940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:34054/0 (socket says 172.21.15.72:34054) 2024-04-15T15:17:07.988 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.986+0000 7f3d3ffff640 1 -- 172.21.15.72:0/1050775553 learned_addr learned my addr 172.21.15.72:0/1050775553 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:07.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.986+0000 7f3d3ffff640 1 -- 172.21.15.72:0/1050775553 >> v1:172.21.15.175:6789/0 conn(0x7f3d4015e9f0 legacy=0x7f3d400d3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:07.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.986+0000 7f3d3ffff640 1 -- 172.21.15.72:0/1050775553 >> v1:172.21.15.72:6789/0 conn(0x7f3d4006c400 legacy=0x7f3d400b7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.986+0000 7f3d3ffff640 1 -- 172.21.15.72:0/1050775553 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d400d28a0 con 0x7f3d4015e610 2024-04-15T15:17:07.989 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.986+0000 7f3d3effd640 1 -- 172.21.15.72:0/1050775553 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 330402064 0 0) 0x7f3d400d2320 con 0x7f3d4006c400 2024-04-15T15:17:07.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.987+0000 7f3d3ffff640 1 --2- 172.21.15.72:0/1050775553 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d4015e610 0x7f3d400d0690 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f3d30009870 tx=0x7f3d3002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d341d749c24c906 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:07.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.987+0000 7f3d3effd640 1 -- 172.21.15.72:0/1050775553 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d3002f840 con 0x7f3d4015e610 2024-04-15T15:17:07.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.987+0000 7f3d3effd640 1 -- 172.21.15.72:0/1050775553 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d3002f9e0 con 0x7f3d4015e610 2024-04-15T15:17:07.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.987+0000 7f3d3effd640 1 -- 172.21.15.72:0/1050775553 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d3002fdb0 con 0x7f3d4015e610 2024-04-15T15:17:07.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.987+0000 7f3d46738640 1 -- 172.21.15.72:0/1050775553 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d4015e610 msgr2=0x7f3d400d0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.992 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.987+0000 7f3d46738640 1 --2- 172.21.15.72:0/1050775553 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d4015e610 0x7f3d400d0690 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f3d30009870 tx=0x7f3d3002ba20 comp rx=0 tx=0).stop 2024-04-15T15:17:07.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.987+0000 7f3d46738640 1 -- 172.21.15.72:0/1050775553 shutdown_connections 2024-04-15T15:17:07.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.987+0000 7f3d46738640 1 --2- 172.21.15.72:0/1050775553 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d4015e610 0x7f3d400d0690 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.987+0000 7f3d46738640 1 -- 172.21.15.72:0/1050775553 >> 172.21.15.72:0/1050775553 conn(0x7f3d400d4750 msgr2=0x7f3d400d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:07.993 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.987+0000 7f3d46738640 1 -- 172.21.15.72:0/1050775553 shutdown_connections 2024-04-15T15:17:07.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.987+0000 7f3d46738640 1 -- 172.21.15.72:0/1050775553 wait complete. 2024-04-15T15:17:07.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.988+0000 7f3d46738640 1 Processor -- start 2024-04-15T15:17:07.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.989+0000 7f3d46738640 1 -- start start 2024-04-15T15:17:07.994 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.989+0000 7f3d46738640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3d4006c400 0x7f3d400cacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.989+0000 7f3d46738640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d4015e610 0x7f3d40154460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.989+0000 7f3d46738640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3d4015e9f0 0x7f3d40156bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.989+0000 7f3d46738640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f3d4006d130 con 0x7f3d4015e610 2024-04-15T15:17:07.995 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.989+0000 7f3d46738640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f3d400d2e60 con 0x7f3d4015e9f0 2024-04-15T15:17:07.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.989+0000 7f3d46738640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f3d400d0bd0 con 0x7f3d4006c400 2024-04-15T15:17:07.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.989+0000 7f3d3f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d4015e610 0x7f3d40154460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.989+0000 7f3d3f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d4015e610 0x7f3d40154460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36898/0 (socket says 172.21.15.72:36898) 2024-04-15T15:17:07.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.989+0000 7f3d3f7fe640 1 -- 172.21.15.72:0/3700492274 learned_addr learned my addr 172.21.15.72:0/3700492274 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:07.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.990+0000 7f3d44cd5640 1 --2- 172.21.15.72:0/3700492274 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3d4015e9f0 0x7f3d40156bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.990+0000 7f3d3ffff640 1 --2- 172.21.15.72:0/3700492274 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3d4006c400 0x7f3d400cacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.990+0000 7f3d3f7fe640 1 -- 172.21.15.72:0/3700492274 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3d4006c400 msgr2=0x7f3d400cacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.996 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.990+0000 7f3d3f7fe640 1 --2- 172.21.15.72:0/3700492274 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3d4006c400 0x7f3d400cacc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.990+0000 7f3d3f7fe640 1 -- 172.21.15.72:0/3700492274 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3d4015e9f0 msgr2=0x7f3d40156bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.990+0000 7f3d3f7fe640 1 --2- 172.21.15.72:0/3700492274 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3d4015e9f0 0x7f3d40156bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.990+0000 7f3d3f7fe640 1 -- 172.21.15.72:0/3700492274 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d40154eb0 con 0x7f3d4015e610 2024-04-15T15:17:07.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.990+0000 7f3d3ffff640 1 --2- 172.21.15.72:0/3700492274 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3d4006c400 0x7f3d400cacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:07.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.990+0000 7f3d3f7fe640 1 --2- 172.21.15.72:0/3700492274 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d4015e610 0x7f3d40154460 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f3d3400b3f0 tx=0x7f3d3400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:07.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.990+0000 7f3d3cff9640 1 -- 172.21.15.72:0/3700492274 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d34015020 con 0x7f3d4015e610 2024-04-15T15:17:07.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.990+0000 7f3d3cff9640 1 -- 172.21.15.72:0/3700492274 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d34011830 con 0x7f3d4015e610 2024-04-15T15:17:07.997 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.990+0000 7f3d46738640 1 -- 172.21.15.72:0/3700492274 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d40155190 con 0x7f3d4015e610 2024-04-15T15:17:07.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.990+0000 7f3d3cff9640 1 -- 172.21.15.72:0/3700492274 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d34011c20 con 0x7f3d4015e610 2024-04-15T15:17:07.998 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.990+0000 7f3d46738640 1 -- 172.21.15.72:0/3700492274 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d40155750 con 0x7f3d4015e610 2024-04-15T15:17:07.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.991+0000 7f3d46738640 1 -- 172.21.15.72:0/3700492274 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d40075c50 con 0x7f3d4015e610 2024-04-15T15:17:07.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.992+0000 7f3d3cff9640 1 -- 172.21.15.72:0/3700492274 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f3d34018020 con 0x7f3d4015e610 2024-04-15T15:17:07.999 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.992+0000 7f3d3cff9640 1 --2- 172.21.15.72:0/3700492274 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3d2c06c980 0x7f3d2c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.992+0000 7f3d3ffff640 1 --2- 172.21.15.72:0/3700492274 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3d2c06c980 0x7f3d2c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.992+0000 7f3d3cff9640 1 -- 172.21.15.72:0/3700492274 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5421+0+0 (secure 0 0 0) 0x7f3d3408b700 con 0x7f3d4015e610 2024-04-15T15:17:08.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.992+0000 7f3d3ffff640 1 --2- 172.21.15.72:0/3700492274 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3d2c06c980 0x7f3d2c06ee40 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3d30036000 tx=0x7f3d30039040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:08.000 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:07.996+0000 7f3d3cff9640 1 -- 172.21.15.72:0/3700492274 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3d3401f030 con 0x7f3d4015e610 2024-04-15T15:17:08.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.175+0000 7f3d46738640 1 -- 172.21.15.72:0/3700492274 --> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f3d400cf740 con 0x7f3d2c06c980 2024-04-15T15:17:08.177 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.175+0000 7f3d3cff9640 1 -- 172.21.15.72:0/3700492274 <== mgr.4101 v2:172.21.15.72:6832/3503807805 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f3d400cf740 con 0x7f3d2c06c980 2024-04-15T15:17:08.177 INFO:teuthology.orchestra.run.smithi072.stderr:instructed osd(s) 3 to deep-scrub 2024-04-15T15:17:08.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.177+0000 7f3d46738640 1 -- 172.21.15.72:0/3700492274 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3d2c06c980 msgr2=0x7f3d2c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.177+0000 7f3d46738640 1 --2- 172.21.15.72:0/3700492274 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3d2c06c980 0x7f3d2c06ee40 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f3d30036000 tx=0x7f3d30039040 comp rx=0 tx=0).stop 2024-04-15T15:17:08.179 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.177+0000 7f3d46738640 1 -- 172.21.15.72:0/3700492274 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d4015e610 msgr2=0x7f3d40154460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.177+0000 7f3d46738640 1 --2- 172.21.15.72:0/3700492274 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d4015e610 0x7f3d40154460 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f3d3400b3f0 tx=0x7f3d3400b8b0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.177+0000 7f3d46738640 1 -- 172.21.15.72:0/3700492274 shutdown_connections 2024-04-15T15:17:08.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.177+0000 7f3d46738640 1 --2- 172.21.15.72:0/3700492274 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f3d2c06c980 0x7f3d2c06ee40 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.177+0000 7f3d46738640 1 --2- 172.21.15.72:0/3700492274 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f3d4015e9f0 0x7f3d40156bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.177+0000 7f3d46738640 1 --2- 172.21.15.72:0/3700492274 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3d4015e610 0x7f3d40154460 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.177+0000 7f3d46738640 1 --2- 172.21.15.72:0/3700492274 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f3d4006c400 0x7f3d400cacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.177+0000 7f3d46738640 1 -- 172.21.15.72:0/3700492274 >> 172.21.15.72:0/3700492274 conn(0x7f3d400d4750 msgr2=0x7f3d400b4e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:08.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.178+0000 7f3d46738640 1 -- 172.21.15.72:0/3700492274 shutdown_connections 2024-04-15T15:17:08.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.178+0000 7f3d46738640 1 -- 172.21.15.72:0/3700492274 wait complete. 2024-04-15T15:17:08.190 INFO:tasks.ceph:Scrubbing osd.4 2024-04-15T15:17:08.190 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-15T15:17:08.296 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.295+0000 7f7e7b893640 1 Processor -- start 2024-04-15T15:17:08.297 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.295+0000 7f7e7b893640 1 -- start start 2024-04-15T15:17:08.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.296+0000 7f7e7b893640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7e7415a0d0 0x7f7e7415ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.301 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.296+0000 7f7e7b893640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f7e740b5ee0 0x7f7e7406e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.296+0000 7f7e7b893640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e7415a4b0 con 0x7f7e740b5370 2024-04-15T15:17:08.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.296+0000 7f7e7b893640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f7e7406c680 con 0x7f7e740b5ee0 2024-04-15T15:17:08.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.296+0000 7f7e7b893640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f7e7406c960 con 0x7f7e7415a0d0 2024-04-15T15:17:08.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.296+0000 7f7e78e2e640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f7e740b5370 0x7f7e740b57d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:34060/0 (socket says 172.21.15.72:34060) 2024-04-15T15:17:08.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.296+0000 7f7e78e2e640 1 -- 172.21.15.72:0/132683306 learned_addr learned my addr 172.21.15.72:0/132683306 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:17:08.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.296+0000 7f7e6bfff640 1 -- 172.21.15.72:0/132683306 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3017021834 0 0) 0x7f7e7415a4b0 con 0x7f7e740b5370 2024-04-15T15:17:08.302 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.296+0000 7f7e7962f640 1 --2- 172.21.15.72:0/132683306 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7e7415a0d0 0x7f7e7415ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.296+0000 7f7e6bfff640 1 -- 172.21.15.72:0/132683306 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e60003520 con 0x7f7e740b5370 2024-04-15T15:17:08.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.296+0000 7f7e79e30640 1 --2- 172.21.15.72:0/132683306 >> v2:172.21.15.175:3300/0 conn(0x7f7e740b5ee0 0x7f7e7406e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.297+0000 7f7e6bfff640 1 -- 172.21.15.72:0/132683306 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3590187748 0 0) 0x7f7e60003520 con 0x7f7e740b5370 2024-04-15T15:17:08.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.297+0000 7f7e6bfff640 1 -- 172.21.15.72:0/132683306 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7e7415a0d0 msgr2=0x7f7e7415ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:08.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.297+0000 7f7e6bfff640 1 --2- 172.21.15.72:0/132683306 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7e7415a0d0 0x7f7e7415ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.297+0000 7f7e6bfff640 1 -- 172.21.15.72:0/132683306 >> v2:172.21.15.175:3300/0 conn(0x7f7e740b5ee0 msgr2=0x7f7e7406e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:08.303 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.297+0000 7f7e6bfff640 1 --2- 172.21.15.72:0/132683306 >> v2:172.21.15.175:3300/0 conn(0x7f7e740b5ee0 0x7f7e7406e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.297+0000 7f7e6bfff640 1 -- 172.21.15.72:0/132683306 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e7406cc40 con 0x7f7e740b5370 2024-04-15T15:17:08.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.297+0000 7f7e6bfff640 1 -- 172.21.15.72:0/132683306 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f7e64001670 con 0x7f7e740b5370 2024-04-15T15:17:08.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.297+0000 7f7e6bfff640 1 -- 172.21.15.72:0/132683306 >> v1:172.21.15.72:6789/0 conn(0x7f7e740b5370 legacy=0x7f7e740b57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.297+0000 7f7e6bfff640 1 --2- 172.21.15.72:0/132683306 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7e600039b0 0x7f7e60003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.297+0000 7f7e6bfff640 1 --2- 172.21.15.72:0/132683306 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e60004800 0x7f7e60004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.297+0000 7f7e6bfff640 1 --2- 172.21.15.72:0/132683306 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7e7415a0d0 0x7f7e600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.297+0000 7f7e6bfff640 1 -- 172.21.15.72:0/132683306 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f7e7406c680 con 0x7f7e60004800 2024-04-15T15:17:08.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.297+0000 7f7e6bfff640 1 -- 172.21.15.72:0/132683306 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f7e7406c960 con 0x7f7e600039b0 2024-04-15T15:17:08.304 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.297+0000 7f7e6bfff640 1 -- 172.21.15.72:0/132683306 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f7e6000b9c0 con 0x7f7e7415a0d0 2024-04-15T15:17:08.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.297+0000 7f7e78e2e640 1 --2- 172.21.15.72:0/132683306 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e60004800 0x7f7e60004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.297+0000 7f7e7962f640 1 --2- 172.21.15.72:0/132683306 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7e600039b0 0x7f7e60003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.297+0000 7f7e79e30640 1 --2- 172.21.15.72:0/132683306 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7e7415a0d0 0x7f7e600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.297+0000 7f7e78e2e640 1 -- 172.21.15.72:0/132683306 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7e7415a0d0 msgr2=0x7f7e600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:08.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.297+0000 7f7e78e2e640 1 --2- 172.21.15.72:0/132683306 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7e7415a0d0 0x7f7e600051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.297+0000 7f7e78e2e640 1 -- 172.21.15.72:0/132683306 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7e600039b0 msgr2=0x7f7e60003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:08.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.297+0000 7f7e78e2e640 1 --2- 172.21.15.72:0/132683306 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7e600039b0 0x7f7e60003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.305 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.297+0000 7f7e78e2e640 1 -- 172.21.15.72:0/132683306 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e60003520 con 0x7f7e60004800 2024-04-15T15:17:08.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.297+0000 7f7e79e30640 1 --2- 172.21.15.72:0/132683306 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7e7415a0d0 0x7f7e600051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:08.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.298+0000 7f7e78e2e640 1 --2- 172.21.15.72:0/132683306 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e60004800 0x7f7e60004c80 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400c210 tx=0x7f7e60005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db3e63bccd15fd03 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:08.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.298+0000 7f7e6bfff640 1 -- 172.21.15.72:0/132683306 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e64007d70 con 0x7f7e60004800 2024-04-15T15:17:08.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.298+0000 7f7e6bfff640 1 -- 172.21.15.72:0/132683306 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e64030070 con 0x7f7e60004800 2024-04-15T15:17:08.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.298+0000 7f7e6bfff640 1 -- 172.21.15.72:0/132683306 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6402cd60 con 0x7f7e60004800 2024-04-15T15:17:08.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.298+0000 7f7e7b893640 1 -- 172.21.15.72:0/132683306 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e60004800 msgr2=0x7f7e60004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.306 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.298+0000 7f7e7b893640 1 --2- 172.21.15.72:0/132683306 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e60004800 0x7f7e60004c80 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f7e6400c210 tx=0x7f7e60005830 comp rx=0 tx=0).stop 2024-04-15T15:17:08.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.298+0000 7f7e79e30640 1 -- 172.21.15.72:0/132683306 reap_dead start 2024-04-15T15:17:08.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.299+0000 7f7e7b893640 1 -- 172.21.15.72:0/132683306 shutdown_connections 2024-04-15T15:17:08.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.299+0000 7f7e7b893640 1 -- 172.21.15.72:0/132683306 >> 172.21.15.72:0/132683306 conn(0x7f7e740b1ee0 msgr2=0x7f7e740b22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:08.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.299+0000 7f7e7b893640 1 -- 172.21.15.72:0/132683306 shutdown_connections 2024-04-15T15:17:08.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.299+0000 7f7e7b893640 1 -- 172.21.15.72:0/132683306 wait complete. 2024-04-15T15:17:08.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.300+0000 7f7e7b893640 1 Processor -- start 2024-04-15T15:17:08.307 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.300+0000 7f7e7b893640 1 -- start start 2024-04-15T15:17:08.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.301+0000 7f7e7b893640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e60004800 0x7f7e741585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.310 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.301+0000 7f7e7b893640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7e7409db50 0x7f7e740b1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.301+0000 7f7e7b893640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7e740b16a0 0x7f7e740adf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.301+0000 7f7e7b893640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f7e7406c310 con 0x7f7e60004800 2024-04-15T15:17:08.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.301+0000 7f7e7b893640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f7e7406d200 con 0x7f7e7409db50 2024-04-15T15:17:08.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.301+0000 7f7e7b893640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f7e7415ca60 con 0x7f7e740b16a0 2024-04-15T15:17:08.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.301+0000 7f7e7962f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e60004800 0x7f7e741585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.301+0000 7f7e7962f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e60004800 0x7f7e741585e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36930/0 (socket says 172.21.15.72:36930) 2024-04-15T15:17:08.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.301+0000 7f7e7962f640 1 -- 172.21.15.72:0/1837314554 learned_addr learned my addr 172.21.15.72:0/1837314554 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:08.311 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.301+0000 7f7e78e2e640 1 --2- 172.21.15.72:0/1837314554 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7e7409db50 0x7f7e740b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.301+0000 7f7e79e30640 1 --2- 172.21.15.72:0/1837314554 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7e740b16a0 0x7f7e740adf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.301+0000 7f7e7962f640 1 -- 172.21.15.72:0/1837314554 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7e740b16a0 msgr2=0x7f7e740adf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.301+0000 7f7e7962f640 1 --2- 172.21.15.72:0/1837314554 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7e740b16a0 0x7f7e740adf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.301+0000 7f7e7962f640 1 -- 172.21.15.72:0/1837314554 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7e7409db50 msgr2=0x7f7e740b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.301+0000 7f7e7962f640 1 --2- 172.21.15.72:0/1837314554 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7e7409db50 0x7f7e740b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.301+0000 7f7e7962f640 1 -- 172.21.15.72:0/1837314554 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e74158b20 con 0x7f7e60004800 2024-04-15T15:17:08.312 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.301+0000 7f7e78e2e640 1 --2- 172.21.15.72:0/1837314554 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7e7409db50 0x7f7e740b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:08.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.301+0000 7f7e7962f640 1 --2- 172.21.15.72:0/1837314554 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e60004800 0x7f7e741585e0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c011910 tx=0x7f7e6c011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:08.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.301+0000 7f7e69ffb640 1 -- 172.21.15.72:0/1837314554 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c00fa90 con 0x7f7e60004800 2024-04-15T15:17:08.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.301+0000 7f7e69ffb640 1 -- 172.21.15.72:0/1837314554 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e6c00fc30 con 0x7f7e60004800 2024-04-15T15:17:08.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.301+0000 7f7e7b893640 1 -- 172.21.15.72:0/1837314554 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e7409cab0 con 0x7f7e60004800 2024-04-15T15:17:08.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.301+0000 7f7e69ffb640 1 -- 172.21.15.72:0/1837314554 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c014440 con 0x7f7e60004800 2024-04-15T15:17:08.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.301+0000 7f7e7b893640 1 -- 172.21.15.72:0/1837314554 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e7409d070 con 0x7f7e60004800 2024-04-15T15:17:08.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.302+0000 7f7e7b893640 1 -- 172.21.15.72:0/1837314554 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7e740689e0 con 0x7f7e60004800 2024-04-15T15:17:08.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.303+0000 7f7e69ffb640 1 -- 172.21.15.72:0/1837314554 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f7e6c012040 con 0x7f7e60004800 2024-04-15T15:17:08.313 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.303+0000 7f7e69ffb640 1 --2- 172.21.15.72:0/1837314554 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7e5806c980 0x7f7e5806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.303+0000 7f7e69ffb640 1 -- 172.21.15.72:0/1837314554 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5421+0+0 (secure 0 0 0) 0x7f7e6c08bf80 con 0x7f7e60004800 2024-04-15T15:17:08.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.303+0000 7f7e78e2e640 1 --2- 172.21.15.72:0/1837314554 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7e5806c980 0x7f7e5806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.303+0000 7f7e69ffb640 1 --2- 172.21.15.72:0/1837314554 >> [v2:172.21.15.175:6823/626414407,v1:172.21.15.175:6825/626414407] conn(0x7f7e58072ec0 0x7f7e58075300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.303+0000 7f7e69ffb640 1 -- 172.21.15.72:0/1837314554 --> [v2:172.21.15.175:6823/626414407,v1:172.21.15.175:6825/626414407] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7e6c015e50 con 0x7f7e58072ec0 2024-04-15T15:17:08.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.303+0000 7f7e69ffb640 1 -- 172.21.15.72:0/1837314554 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f7e6c08e610 con 0x7f7e60004800 2024-04-15T15:17:08.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.304+0000 7f7e79e30640 1 --2- 172.21.15.72:0/1837314554 >> [v2:172.21.15.175:6823/626414407,v1:172.21.15.175:6825/626414407] conn(0x7f7e58072ec0 0x7f7e58075300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.304+0000 7f7e78e2e640 1 --2- 172.21.15.72:0/1837314554 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7e5806c980 0x7f7e5806ee40 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f7e64034000 tx=0x7f7e64007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:08.314 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.304+0000 7f7e79e30640 1 --2- 172.21.15.72:0/1837314554 >> [v2:172.21.15.175:6823/626414407,v1:172.21.15.175:6825/626414407] conn(0x7f7e58072ec0 0x7f7e58075300 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:08.315 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.305+0000 7f7e69ffb640 1 -- 172.21.15.72:0/1837314554 <== osd.4 v2:172.21.15.175:6823/626414407 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7e6c015e50 con 0x7f7e58072ec0 2024-04-15T15:17:08.319 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.318+0000 7f7e7b893640 1 -- 172.21.15.72:0/1837314554 --> [v2:172.21.15.175:6823/626414407,v1:172.21.15.175:6825/626414407] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f7e7406bbe0 con 0x7f7e58072ec0 2024-04-15T15:17:08.320 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.318+0000 7f7e69ffb640 1 -- 172.21.15.72:0/1837314554 <== osd.4 v2:172.21.15.175:6823/626414407 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f7e7406bbe0 con 0x7f7e58072ec0 2024-04-15T15:17:08.320 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-04-15T15:17:08.320 INFO:teuthology.orchestra.run.smithi072.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-15T15:17:08.320 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-04-15T15:17:08.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.319+0000 7f7e7b893640 1 -- 172.21.15.72:0/1837314554 >> [v2:172.21.15.175:6823/626414407,v1:172.21.15.175:6825/626414407] conn(0x7f7e58072ec0 msgr2=0x7f7e58075300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.319+0000 7f7e7b893640 1 --2- 172.21.15.72:0/1837314554 >> [v2:172.21.15.175:6823/626414407,v1:172.21.15.175:6825/626414407] conn(0x7f7e58072ec0 0x7f7e58075300 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.319+0000 7f7e7b893640 1 -- 172.21.15.72:0/1837314554 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7e5806c980 msgr2=0x7f7e5806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.319+0000 7f7e7b893640 1 --2- 172.21.15.72:0/1837314554 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7e5806c980 0x7f7e5806ee40 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f7e64034000 tx=0x7f7e64007000 comp rx=0 tx=0).stop 2024-04-15T15:17:08.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.319+0000 7f7e7b893640 1 -- 172.21.15.72:0/1837314554 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e60004800 msgr2=0x7f7e741585e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.319+0000 7f7e7b893640 1 --2- 172.21.15.72:0/1837314554 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7e60004800 0x7f7e741585e0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c011910 tx=0x7f7e6c011dd0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.319+0000 7f7e79e30640 1 -- 172.21.15.72:0/1837314554 reap_dead start 2024-04-15T15:17:08.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.319+0000 7f7e7b893640 1 -- 172.21.15.72:0/1837314554 shutdown_connections 2024-04-15T15:17:08.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.319+0000 7f7e7b893640 1 -- 172.21.15.72:0/1837314554 >> 172.21.15.72:0/1837314554 conn(0x7f7e740b1ee0 msgr2=0x7f7e740cccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:08.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.319+0000 7f7e7b893640 1 -- 172.21.15.72:0/1837314554 shutdown_connections 2024-04-15T15:17:08.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.319+0000 7f7e7b893640 1 -- 172.21.15.72:0/1837314554 wait complete. 2024-04-15T15:17:08.331 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-15T15:17:08.435 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.434+0000 7f8665063640 1 Processor -- start 2024-04-15T15:17:08.436 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.434+0000 7f8665063640 1 -- start start 2024-04-15T15:17:08.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.435+0000 7f8665063640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f86600d08f0 0x7f86600cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.435+0000 7f8665063640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86600ce6d0 0x7f86600cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.435+0000 7f8665063640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f866006bbb0 con 0x7f86600ce2f0 2024-04-15T15:17:08.438 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.435+0000 7f8665063640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f866006b960 con 0x7f86600d08f0 2024-04-15T15:17:08.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.435+0000 7f8665063640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f866006be90 con 0x7f86600ce6d0 2024-04-15T15:17:08.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.435+0000 7f865659c640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f86600d08f0 0x7f86600cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.435+0000 7f865ed9d640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f86600ce2f0 0x7f866015c530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:34074/0 (socket says 172.21.15.72:34074) 2024-04-15T15:17:08.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.435+0000 7f865ed9d640 1 -- 172.21.15.72:0/3009917373 learned_addr learned my addr 172.21.15.72:0/3009917373 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:17:08.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.435+0000 7f865659c640 1 -- 172.21.15.72:0/3009917373 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86600ce6d0 msgr2=0x7f86600cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:08.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.435+0000 7f865f59e640 1 --2- 172.21.15.72:0/3009917373 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86600ce6d0 0x7f86600cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.435+0000 7f865659c640 1 --2- 172.21.15.72:0/3009917373 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86600ce6d0 0x7f86600cc5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.435+0000 7f865659c640 1 -- 172.21.15.72:0/3009917373 >> v1:172.21.15.72:6789/0 conn(0x7f86600ce2f0 legacy=0x7f866015c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.439 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.435+0000 7f865e59c640 1 -- 172.21.15.72:0/3009917373 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2753981517 0 0) 0x7f866006bbb0 con 0x7f86600ce2f0 2024-04-15T15:17:08.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.435+0000 7f865659c640 1 -- 172.21.15.72:0/3009917373 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f866006c170 con 0x7f86600d08f0 2024-04-15T15:17:08.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.435+0000 7f865f59e640 1 --2- 172.21.15.72:0/3009917373 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86600ce6d0 0x7f86600cc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:08.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.436+0000 7f865659c640 1 --2- 172.21.15.72:0/3009917373 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86600d08f0 0x7f86600cc0b0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f86480097a0 tx=0x7f864802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31057a787941c5ea server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:08.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.436+0000 7f865e59c640 1 -- 172.21.15.72:0/3009917373 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8648035020 con 0x7f86600d08f0 2024-04-15T15:17:08.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.436+0000 7f865e59c640 1 -- 172.21.15.72:0/3009917373 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8648031830 con 0x7f86600d08f0 2024-04-15T15:17:08.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.436+0000 7f865e59c640 1 -- 172.21.15.72:0/3009917373 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8648031c00 con 0x7f86600d08f0 2024-04-15T15:17:08.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.436+0000 7f8665063640 1 -- 172.21.15.72:0/3009917373 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86600d08f0 msgr2=0x7f86600cc0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.436+0000 7f8665063640 1 --2- 172.21.15.72:0/3009917373 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86600d08f0 0x7f86600cc0b0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f86480097a0 tx=0x7f864802b8d0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.440 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.437+0000 7f8665063640 1 -- 172.21.15.72:0/3009917373 shutdown_connections 2024-04-15T15:17:08.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.437+0000 7f8665063640 1 --2- 172.21.15.72:0/3009917373 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86600ce6d0 0x7f86600cc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.437+0000 7f8665063640 1 --2- 172.21.15.72:0/3009917373 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86600d08f0 0x7f86600cc0b0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.437+0000 7f8665063640 1 -- 172.21.15.72:0/3009917373 >> 172.21.15.72:0/3009917373 conn(0x7f86600b31d0 msgr2=0x7f86600b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:08.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.437+0000 7f8665063640 1 -- 172.21.15.72:0/3009917373 shutdown_connections 2024-04-15T15:17:08.441 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.437+0000 7f8665063640 1 -- 172.21.15.72:0/3009917373 wait complete. 2024-04-15T15:17:08.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.438+0000 7f8665063640 1 Processor -- start 2024-04-15T15:17:08.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.438+0000 7f8665063640 1 -- start start 2024-04-15T15:17:08.445 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.439+0000 7f8665063640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86600ce2f0 0x7f8660099800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.439+0000 7f8665063640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86600ce6d0 0x7f8660157f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.439+0000 7f8665063640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f86600d08f0 0x7f86601584b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.439+0000 7f8665063640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f8660155780 con 0x7f86600ce6d0 2024-04-15T15:17:08.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.439+0000 7f8665063640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f8660155600 con 0x7f86600d08f0 2024-04-15T15:17:08.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.439+0000 7f8665063640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f8660155900 con 0x7f86600ce2f0 2024-04-15T15:17:08.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.439+0000 7f865659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86600ce6d0 0x7f8660157f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.446 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.439+0000 7f865659c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86600ce6d0 0x7f8660157f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36958/0 (socket says 172.21.15.72:36958) 2024-04-15T15:17:08.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.439+0000 7f865659c640 1 -- 172.21.15.72:0/3972930196 learned_addr learned my addr 172.21.15.72:0/3972930196 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:08.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.439+0000 7f865ed9d640 1 --2- 172.21.15.72:0/3972930196 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86600ce2f0 0x7f8660099800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.439+0000 7f865f59e640 1 --2- 172.21.15.72:0/3972930196 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f86600d08f0 0x7f86601584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.439+0000 7f865659c640 1 -- 172.21.15.72:0/3972930196 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86600ce2f0 msgr2=0x7f8660099800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.439+0000 7f865659c640 1 --2- 172.21.15.72:0/3972930196 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86600ce2f0 0x7f8660099800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.439+0000 7f865659c640 1 -- 172.21.15.72:0/3972930196 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f86600d08f0 msgr2=0x7f86601584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.439+0000 7f865659c640 1 --2- 172.21.15.72:0/3972930196 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f86600d08f0 0x7f86601584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.439+0000 7f865659c640 1 -- 172.21.15.72:0/3972930196 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8660158ce0 con 0x7f86600ce6d0 2024-04-15T15:17:08.447 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.439+0000 7f865659c640 1 --2- 172.21.15.72:0/3972930196 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86600ce6d0 0x7f8660157f70 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f8648034040 tx=0x7f86480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:08.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.439+0000 7f8657fff640 1 -- 172.21.15.72:0/3972930196 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8648041070 con 0x7f86600ce6d0 2024-04-15T15:17:08.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.440+0000 7f8657fff640 1 -- 172.21.15.72:0/3972930196 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8648004250 con 0x7f86600ce6d0 2024-04-15T15:17:08.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.440+0000 7f8665063640 1 -- 172.21.15.72:0/3972930196 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86600aff90 con 0x7f86600ce6d0 2024-04-15T15:17:08.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.440+0000 7f8657fff640 1 -- 172.21.15.72:0/3972930196 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8648035040 con 0x7f86600ce6d0 2024-04-15T15:17:08.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.440+0000 7f8665063640 1 -- 172.21.15.72:0/3972930196 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86600b0550 con 0x7f86600ce6d0 2024-04-15T15:17:08.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.440+0000 7f8665063640 1 -- 172.21.15.72:0/3972930196 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86600afd90 con 0x7f86600ce6d0 2024-04-15T15:17:08.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.441+0000 7f8657fff640 1 -- 172.21.15.72:0/3972930196 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f8648038020 con 0x7f86600ce6d0 2024-04-15T15:17:08.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.441+0000 7f8657fff640 1 --2- 172.21.15.72:0/3972930196 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f864006c980 0x7f864006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.448 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.441+0000 7f8657fff640 1 -- 172.21.15.72:0/3972930196 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5421+0+0 (secure 0 0 0) 0x7f86480ad4e0 con 0x7f86600ce6d0 2024-04-15T15:17:08.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.441+0000 7f865ed9d640 1 --2- 172.21.15.72:0/3972930196 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f864006c980 0x7f864006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.442+0000 7f865ed9d640 1 --2- 172.21.15.72:0/3972930196 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f864006c980 0x7f864006ee40 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f8650000f60 tx=0x7f8650007b50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:08.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.446+0000 7f8657fff640 1 -- 172.21.15.72:0/3972930196 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f864807bad0 con 0x7f86600ce6d0 2024-04-15T15:17:08.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.620+0000 7f8665063640 1 -- 172.21.15.72:0/3972930196 --> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f8660075c50 con 0x7f864006c980 2024-04-15T15:17:08.622 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.621+0000 7f8657fff640 1 -- 172.21.15.72:0/3972930196 <== mgr.4101 v2:172.21.15.72:6832/3503807805 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f8660075c50 con 0x7f864006c980 2024-04-15T15:17:08.623 INFO:teuthology.orchestra.run.smithi072.stderr:instructed osd(s) 4 to deep-scrub 2024-04-15T15:17:08.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.622+0000 7f8665063640 1 -- 172.21.15.72:0/3972930196 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f864006c980 msgr2=0x7f864006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.622+0000 7f8665063640 1 --2- 172.21.15.72:0/3972930196 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f864006c980 0x7f864006ee40 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f8650000f60 tx=0x7f8650007b50 comp rx=0 tx=0).stop 2024-04-15T15:17:08.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.622+0000 7f8665063640 1 -- 172.21.15.72:0/3972930196 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86600ce6d0 msgr2=0x7f8660157f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.622+0000 7f8665063640 1 --2- 172.21.15.72:0/3972930196 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86600ce6d0 0x7f8660157f70 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f8648034040 tx=0x7f86480046c0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.623+0000 7f8665063640 1 -- 172.21.15.72:0/3972930196 shutdown_connections 2024-04-15T15:17:08.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.623+0000 7f8665063640 1 --2- 172.21.15.72:0/3972930196 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f864006c980 0x7f864006ee40 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.623+0000 7f8665063640 1 --2- 172.21.15.72:0/3972930196 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f86600d08f0 0x7f86601584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.623+0000 7f8665063640 1 --2- 172.21.15.72:0/3972930196 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f86600ce6d0 0x7f8660157f70 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.623+0000 7f8665063640 1 --2- 172.21.15.72:0/3972930196 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f86600ce2f0 0x7f8660099800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.623+0000 7f8665063640 1 -- 172.21.15.72:0/3972930196 >> 172.21.15.72:0/3972930196 conn(0x7f86600b31d0 msgr2=0x7f866015d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:08.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.623+0000 7f8665063640 1 -- 172.21.15.72:0/3972930196 shutdown_connections 2024-04-15T15:17:08.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.623+0000 7f8665063640 1 -- 172.21.15.72:0/3972930196 wait complete. 2024-04-15T15:17:08.635 INFO:tasks.ceph:Scrubbing osd.5 2024-04-15T15:17:08.636 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-15T15:17:08.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.740+0000 7f60803ba640 1 Processor -- start 2024-04-15T15:17:08.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.741+0000 7f60803ba640 1 -- start start 2024-04-15T15:17:08.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.741+0000 7f60803ba640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f60780d4a40 0x7f60780b3a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.742+0000 7f60803ba640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f60780d1760 0x7f60780d2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.744 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.742+0000 7f60803ba640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60780d50c0 con 0x7f607815e9d0 2024-04-15T15:17:08.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.742+0000 7f60803ba640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f60780d22c0 con 0x7f60780d1760 2024-04-15T15:17:08.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.742+0000 7f60803ba640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f60780d0e80 con 0x7f60780d4a40 2024-04-15T15:17:08.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.742+0000 7f607e957640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f60780d1760 0x7f60780d2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.742+0000 7f607e957640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f60780d1760 0x7f60780d2750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36972/0 (socket says 172.21.15.72:36972) 2024-04-15T15:17:08.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.742+0000 7f607e957640 1 -- 172.21.15.72:0/2127818856 learned_addr learned my addr 172.21.15.72:0/2127818856 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:08.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.742+0000 7f607e957640 1 -- 172.21.15.72:0/2127818856 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f60780d4a40 msgr2=0x7f60780b3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:08.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.742+0000 7f607d955640 1 --2- 172.21.15.72:0/2127818856 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f60780d4a40 0x7f60780b3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.745 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.742+0000 7f607e957640 1 --2- 172.21.15.72:0/2127818856 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f60780d4a40 0x7f60780b3a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.742+0000 7f607e957640 1 -- 172.21.15.72:0/2127818856 >> v1:172.21.15.175:6789/0 conn(0x7f607815e9d0 legacy=0x7f60780d4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:08.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.742+0000 7f607e957640 1 -- 172.21.15.72:0/2127818856 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60780d05e0 con 0x7f60780d1760 2024-04-15T15:17:08.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.743+0000 7f607e957640 1 --2- 172.21.15.72:0/2127818856 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60780d1760 0x7f60780d2750 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f6070009e50 tx=0x7f607002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef26aa51ea58f926 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:08.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.743+0000 7f607d154640 1 -- 172.21.15.72:0/2127818856 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f607002ca70 con 0x7f60780d1760 2024-04-15T15:17:08.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.743+0000 7f607d154640 1 -- 172.21.15.72:0/2127818856 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f607002cc10 con 0x7f60780d1760 2024-04-15T15:17:08.746 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.743+0000 7f607d154640 1 -- 172.21.15.72:0/2127818856 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60700343f0 con 0x7f60780d1760 2024-04-15T15:17:08.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.743+0000 7f60803ba640 1 -- 172.21.15.72:0/2127818856 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60780d1760 msgr2=0x7f60780d2750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.750 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.743+0000 7f60803ba640 1 --2- 172.21.15.72:0/2127818856 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60780d1760 0x7f60780d2750 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f6070009e50 tx=0x7f607002bed0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.744+0000 7f60803ba640 1 -- 172.21.15.72:0/2127818856 shutdown_connections 2024-04-15T15:17:08.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.744+0000 7f60803ba640 1 --2- 172.21.15.72:0/2127818856 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60780d1760 0x7f60780d2750 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.744+0000 7f60803ba640 1 --2- 172.21.15.72:0/2127818856 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f60780d4a40 0x7f60780b3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.744+0000 7f60803ba640 1 -- 172.21.15.72:0/2127818856 >> 172.21.15.72:0/2127818856 conn(0x7f607809d970 msgr2=0x7f60780b9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:08.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.744+0000 7f60803ba640 1 -- 172.21.15.72:0/2127818856 shutdown_connections 2024-04-15T15:17:08.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.744+0000 7f60803ba640 1 -- 172.21.15.72:0/2127818856 wait complete. 2024-04-15T15:17:08.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.745+0000 7f60803ba640 1 Processor -- start 2024-04-15T15:17:08.751 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.745+0000 7f60803ba640 1 -- start start 2024-04-15T15:17:08.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.745+0000 7f60803ba640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60780d1760 0x7f6078158070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.745+0000 7f60803ba640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f60780d4a40 0x7f60781619e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.745+0000 7f60803ba640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f607815e9d0 0x7f607806d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.745+0000 7f60803ba640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f607815edb0 con 0x7f607815e9d0 2024-04-15T15:17:08.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.745+0000 7f60803ba640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f60780d0ba0 con 0x7f60780d1760 2024-04-15T15:17:08.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.745+0000 7f60803ba640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f60780d2c90 con 0x7f60780d4a40 2024-04-15T15:17:08.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.746+0000 7f607e957640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f607815e9d0 0x7f607806d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.752 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.746+0000 7f607e957640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f607815e9d0 0x7f607806d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36974/0 (socket says 172.21.15.72:36974) 2024-04-15T15:17:08.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.746+0000 7f607e957640 1 -- 172.21.15.72:0/1464739548 learned_addr learned my addr 172.21.15.72:0/1464739548 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:08.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.746+0000 7f607e156640 1 --2- 172.21.15.72:0/1464739548 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60780d1760 0x7f6078158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.746+0000 7f607e957640 1 -- 172.21.15.72:0/1464739548 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f60780d4a40 msgr2=0x7f60781619e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.746+0000 7f607d955640 1 --2- 172.21.15.72:0/1464739548 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f60780d4a40 0x7f60781619e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.746+0000 7f607e957640 1 --2- 172.21.15.72:0/1464739548 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f60780d4a40 0x7f60781619e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.746+0000 7f607e957640 1 -- 172.21.15.72:0/1464739548 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60780d1760 msgr2=0x7f6078158070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.746+0000 7f607e957640 1 --2- 172.21.15.72:0/1464739548 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60780d1760 0x7f6078158070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.753 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.746+0000 7f607e957640 1 -- 172.21.15.72:0/1464739548 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60781585b0 con 0x7f607815e9d0 2024-04-15T15:17:08.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.746+0000 7f607e156640 1 --2- 172.21.15.72:0/1464739548 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f60780d1760 0x7f6078158070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:08.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.746+0000 7f607e957640 1 --2- 172.21.15.72:0/1464739548 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f607815e9d0 0x7f607806d000 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f6070009f70 tx=0x7f6070004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:08.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.746+0000 7f606effd640 1 -- 172.21.15.72:0/1464739548 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6070043020 con 0x7f607815e9d0 2024-04-15T15:17:08.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.746+0000 7f606effd640 1 -- 172.21.15.72:0/1464739548 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f607003e450 con 0x7f607815e9d0 2024-04-15T15:17:08.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.746+0000 7f60803ba640 1 -- 172.21.15.72:0/1464739548 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60781f7b70 con 0x7f607815e9d0 2024-04-15T15:17:08.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.746+0000 7f606effd640 1 -- 172.21.15.72:0/1464739548 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6070009510 con 0x7f607815e9d0 2024-04-15T15:17:08.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.746+0000 7f60803ba640 1 -- 172.21.15.72:0/1464739548 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60781f8130 con 0x7f607815e9d0 2024-04-15T15:17:08.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.747+0000 7f60803ba640 1 -- 172.21.15.72:0/1464739548 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f60780689e0 con 0x7f607815e9d0 2024-04-15T15:17:08.754 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.748+0000 7f606effd640 1 -- 172.21.15.72:0/1464739548 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f6070046020 con 0x7f607815e9d0 2024-04-15T15:17:08.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.748+0000 7f606effd640 1 --2- 172.21.15.72:0/1464739548 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f605c06c980 0x7f605c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.748+0000 7f606effd640 1 -- 172.21.15.72:0/1464739548 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5421+0+0 (secure 0 0 0) 0x7f60700ae430 con 0x7f607815e9d0 2024-04-15T15:17:08.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.748+0000 7f607e156640 1 --2- 172.21.15.72:0/1464739548 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f605c06c980 0x7f605c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.748+0000 7f606effd640 1 --2- 172.21.15.72:0/1464739548 >> [v2:172.21.15.175:6800/1465263431,v1:172.21.15.175:6801/1465263431] conn(0x7f605c072ec0 0x7f605c075300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.748+0000 7f606effd640 1 -- 172.21.15.72:0/1464739548 --> [v2:172.21.15.175:6800/1465263431,v1:172.21.15.175:6801/1465263431] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f607003ae50 con 0x7f605c072ec0 2024-04-15T15:17:08.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.748+0000 7f606effd640 1 -- 172.21.15.72:0/1464739548 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f60700b5050 con 0x7f607815e9d0 2024-04-15T15:17:08.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.748+0000 7f607e156640 1 --2- 172.21.15.72:0/1464739548 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f605c06c980 0x7f605c06ee40 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f607806e890 tx=0x7f6074008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:08.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.748+0000 7f607d955640 1 --2- 172.21.15.72:0/1464739548 >> [v2:172.21.15.175:6800/1465263431,v1:172.21.15.175:6801/1465263431] conn(0x7f605c072ec0 0x7f605c075300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.755 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.749+0000 7f607d955640 1 --2- 172.21.15.72:0/1464739548 >> [v2:172.21.15.175:6800/1465263431,v1:172.21.15.175:6801/1465263431] conn(0x7f605c072ec0 0x7f605c075300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:08.756 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.750+0000 7f606effd640 1 -- 172.21.15.72:0/1464739548 <== osd.5 v2:172.21.15.175:6800/1465263431 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f607003ae50 con 0x7f605c072ec0 2024-04-15T15:17:08.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.762+0000 7f60803ba640 1 -- 172.21.15.72:0/1464739548 --> [v2:172.21.15.175:6800/1465263431,v1:172.21.15.175:6801/1465263431] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f6078162170 con 0x7f605c072ec0 2024-04-15T15:17:08.765 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.763+0000 7f606effd640 1 -- 172.21.15.72:0/1464739548 <== osd.5 v2:172.21.15.175:6800/1465263431 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f6078162170 con 0x7f605c072ec0 2024-04-15T15:17:08.765 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-04-15T15:17:08.765 INFO:teuthology.orchestra.run.smithi072.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-15T15:17:08.765 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-04-15T15:17:08.766 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.764+0000 7f60803ba640 1 -- 172.21.15.72:0/1464739548 >> [v2:172.21.15.175:6800/1465263431,v1:172.21.15.175:6801/1465263431] conn(0x7f605c072ec0 msgr2=0x7f605c075300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.764+0000 7f60803ba640 1 --2- 172.21.15.72:0/1464739548 >> [v2:172.21.15.175:6800/1465263431,v1:172.21.15.175:6801/1465263431] conn(0x7f605c072ec0 0x7f605c075300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.764+0000 7f60803ba640 1 -- 172.21.15.72:0/1464739548 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f605c06c980 msgr2=0x7f605c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.764+0000 7f60803ba640 1 --2- 172.21.15.72:0/1464739548 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f605c06c980 0x7f605c06ee40 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f607806e890 tx=0x7f6074008040 comp rx=0 tx=0).stop 2024-04-15T15:17:08.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.764+0000 7f60803ba640 1 -- 172.21.15.72:0/1464739548 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f607815e9d0 msgr2=0x7f607806d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.764+0000 7f60803ba640 1 --2- 172.21.15.72:0/1464739548 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f607815e9d0 0x7f607806d000 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f6070009f70 tx=0x7f6070004a40 comp rx=0 tx=0).stop 2024-04-15T15:17:08.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.764+0000 7f607e957640 1 -- 172.21.15.72:0/1464739548 reap_dead start 2024-04-15T15:17:08.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.764+0000 7f60803ba640 1 -- 172.21.15.72:0/1464739548 shutdown_connections 2024-04-15T15:17:08.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.764+0000 7f60803ba640 1 -- 172.21.15.72:0/1464739548 >> 172.21.15.72:0/1464739548 conn(0x7f607809d970 msgr2=0x7f60780b2c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:08.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.764+0000 7f60803ba640 1 -- 172.21.15.72:0/1464739548 shutdown_connections 2024-04-15T15:17:08.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.764+0000 7f60803ba640 1 -- 172.21.15.72:0/1464739548 wait complete. 2024-04-15T15:17:08.776 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-15T15:17:08.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.878+0000 7f2b752eb640 1 Processor -- start 2024-04-15T15:17:08.880 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.878+0000 7f2b752eb640 1 -- start start 2024-04-15T15:17:08.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.879+0000 7f2b752eb640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f2b7015f9f0 0x7f2b700cca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.879+0000 7f2b752eb640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2b700d0010 0x7f2b700d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.879+0000 7f2b752eb640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b7006df30 con 0x7f2b7015c250 2024-04-15T15:17:08.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.879+0000 7f2b752eb640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f2b700d09b0 con 0x7f2b7015f9f0 2024-04-15T15:17:08.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.879+0000 7f2b752eb640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f2b7006e210 con 0x7f2b700d0010 2024-04-15T15:17:08.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.879+0000 7f2b6effd640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f2b7015c250 0x7f2b7006d6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:34082/0 (socket says 172.21.15.72:34082) 2024-04-15T15:17:08.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.879+0000 7f2b6effd640 1 -- 172.21.15.72:0/468094746 learned_addr learned my addr 172.21.15.72:0/468094746 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:17:08.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.879+0000 7f2b6f7fe640 1 --2- 172.21.15.72:0/468094746 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2b700d0010 0x7f2b700d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.879+0000 7f2b6e7fc640 1 --2- 172.21.15.72:0/468094746 >> v2:172.21.15.175:3300/0 conn(0x7f2b7015f9f0 0x7f2b700cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.879+0000 7f2b6dffb640 1 -- 172.21.15.72:0/468094746 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1837188939 0 0) 0x7f2b7006df30 con 0x7f2b7015c250 2024-04-15T15:17:08.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.879+0000 7f2b6dffb640 1 -- 172.21.15.72:0/468094746 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b58003520 con 0x7f2b7015c250 2024-04-15T15:17:08.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.880+0000 7f2b6dffb640 1 -- 172.21.15.72:0/468094746 <== mon.0 v1:172.21.15.72:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2546459353 0 0) 0x7f2b58003520 con 0x7f2b7015c250 2024-04-15T15:17:08.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.880+0000 7f2b6dffb640 1 -- 172.21.15.72:0/468094746 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2b700d0010 msgr2=0x7f2b700d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:08.885 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.880+0000 7f2b6dffb640 1 --2- 172.21.15.72:0/468094746 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2b700d0010 0x7f2b700d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.880+0000 7f2b6dffb640 1 -- 172.21.15.72:0/468094746 >> v2:172.21.15.175:3300/0 conn(0x7f2b7015f9f0 msgr2=0x7f2b700cca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:08.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.880+0000 7f2b6dffb640 1 --2- 172.21.15.72:0/468094746 >> v2:172.21.15.175:3300/0 conn(0x7f2b7015f9f0 0x7f2b700cca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.880+0000 7f2b6dffb640 1 -- 172.21.15.72:0/468094746 --> v1:172.21.15.72:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b7006e4f0 con 0x7f2b7015c250 2024-04-15T15:17:08.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.880+0000 7f2b6dffb640 1 -- 172.21.15.72:0/468094746 <== mon.0 v1:172.21.15.72:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2834765487 0 0) 0x7f2b600019f0 con 0x7f2b7015c250 2024-04-15T15:17:08.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.880+0000 7f2b6f7fe640 1 --2- 172.21.15.72:0/468094746 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2b700d0010 0x7f2b700d0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:17:08.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.880+0000 7f2b6e7fc640 1 --2- 172.21.15.72:0/468094746 >> v2:172.21.15.175:3300/0 conn(0x7f2b7015f9f0 0x7f2b700cca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:17:08.886 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.880+0000 7f2b6dffb640 1 -- 172.21.15.72:0/468094746 >> v1:172.21.15.72:6789/0 conn(0x7f2b7015c250 legacy=0x7f2b7006d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.880+0000 7f2b6dffb640 1 --2- 172.21.15.72:0/468094746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b580039b0 0x7f2b58003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.880+0000 7f2b6dffb640 1 --2- 172.21.15.72:0/468094746 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2b700d0010 0x7f2b58004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.880+0000 7f2b6dffb640 1 --2- 172.21.15.72:0/468094746 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2b58004d40 0x7f2b580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.880+0000 7f2b6dffb640 1 -- 172.21.15.72:0/468094746 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7006e210 con 0x7f2b580039b0 2024-04-15T15:17:08.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.880+0000 7f2b6dffb640 1 -- 172.21.15.72:0/468094746 --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f2b5800b9c0 con 0x7f2b58004d40 2024-04-15T15:17:08.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.880+0000 7f2b6dffb640 1 -- 172.21.15.72:0/468094746 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f2b5800bb40 con 0x7f2b700d0010 2024-04-15T15:17:08.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.880+0000 7f2b6effd640 1 --2- 172.21.15.72:0/468094746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b580039b0 0x7f2b58003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.880+0000 7f2b6e7fc640 1 --2- 172.21.15.72:0/468094746 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2b700d0010 0x7f2b58004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.880+0000 7f2b6f7fe640 1 --2- 172.21.15.72:0/468094746 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2b58004d40 0x7f2b580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.880+0000 7f2b6effd640 1 -- 172.21.15.72:0/468094746 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2b700d0010 msgr2=0x7f2b58004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:08.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.880+0000 7f2b6effd640 1 --2- 172.21.15.72:0/468094746 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2b700d0010 0x7f2b58004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.880+0000 7f2b6effd640 1 -- 172.21.15.72:0/468094746 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2b58004d40 msgr2=0x7f2b580051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:08.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.880+0000 7f2b6effd640 1 --2- 172.21.15.72:0/468094746 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2b58004d40 0x7f2b580051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.880+0000 7f2b6effd640 1 -- 172.21.15.72:0/468094746 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b58003520 con 0x7f2b580039b0 2024-04-15T15:17:08.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.880+0000 7f2b6f7fe640 1 --2- 172.21.15.72:0/468094746 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2b58004d40 0x7f2b580051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:08.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.881+0000 7f2b6effd640 1 --2- 172.21.15.72:0/468094746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b580039b0 0x7f2b58003dd0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f2b6000a880 tx=0x7f2b58005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5750a8d783b874f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:08.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.881+0000 7f2b6dffb640 1 -- 172.21.15.72:0/468094746 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6000a4a0 con 0x7f2b580039b0 2024-04-15T15:17:08.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.881+0000 7f2b6dffb640 1 -- 172.21.15.72:0/468094746 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b60034650 con 0x7f2b580039b0 2024-04-15T15:17:08.889 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.881+0000 7f2b6dffb640 1 -- 172.21.15.72:0/468094746 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b60034a40 con 0x7f2b580039b0 2024-04-15T15:17:08.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.881+0000 7f2b752eb640 1 -- 172.21.15.72:0/468094746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b580039b0 msgr2=0x7f2b58003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.881+0000 7f2b752eb640 1 --2- 172.21.15.72:0/468094746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b580039b0 0x7f2b58003dd0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f2b6000a880 tx=0x7f2b58005830 comp rx=0 tx=0).stop 2024-04-15T15:17:08.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.882+0000 7f2b6f7fe640 1 -- 172.21.15.72:0/468094746 reap_dead start 2024-04-15T15:17:08.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.882+0000 7f2b752eb640 1 -- 172.21.15.72:0/468094746 shutdown_connections 2024-04-15T15:17:08.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.882+0000 7f2b752eb640 1 -- 172.21.15.72:0/468094746 >> 172.21.15.72:0/468094746 conn(0x7f2b700ceec0 msgr2=0x7f2b700cf2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:08.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.882+0000 7f2b752eb640 1 -- 172.21.15.72:0/468094746 shutdown_connections 2024-04-15T15:17:08.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.882+0000 7f2b752eb640 1 -- 172.21.15.72:0/468094746 wait complete. 2024-04-15T15:17:08.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.883+0000 7f2b752eb640 1 Processor -- start 2024-04-15T15:17:08.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.883+0000 7f2b752eb640 1 -- start start 2024-04-15T15:17:08.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.884+0000 7f2b752eb640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2b580039b0 0x7f2b700b7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.884+0000 7f2b752eb640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2b7015ed00 0x7f2b701f7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.884+0000 7f2b752eb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b701f7f90 0x7f2b701490d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.894 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.884+0000 7f2b752eb640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7015c630 con 0x7f2b701f7f90 2024-04-15T15:17:08.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.884+0000 7f2b752eb640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f2b700b38f0 con 0x7f2b7015ed00 2024-04-15T15:17:08.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.884+0000 7f2b752eb640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f2b7015c7b0 con 0x7f2b580039b0 2024-04-15T15:17:08.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.884+0000 7f2b6f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b701f7f90 0x7f2b701490d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.884+0000 7f2b6f7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b701f7f90 0x7f2b701490d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36988/0 (socket says 172.21.15.72:36988) 2024-04-15T15:17:08.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.884+0000 7f2b6f7fe640 1 -- 172.21.15.72:0/1605452265 learned_addr learned my addr 172.21.15.72:0/1605452265 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:08.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.884+0000 7f2b6effd640 1 --2- 172.21.15.72:0/1605452265 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2b580039b0 0x7f2b700b7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.895 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.884+0000 7f2b6e7fc640 1 --2- 172.21.15.72:0/1605452265 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2b7015ed00 0x7f2b701f7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.884+0000 7f2b6f7fe640 1 -- 172.21.15.72:0/1605452265 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2b580039b0 msgr2=0x7f2b700b7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.884+0000 7f2b6f7fe640 1 --2- 172.21.15.72:0/1605452265 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2b580039b0 0x7f2b700b7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.884+0000 7f2b6f7fe640 1 -- 172.21.15.72:0/1605452265 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2b7015ed00 msgr2=0x7f2b701f7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.884+0000 7f2b6f7fe640 1 --2- 172.21.15.72:0/1605452265 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2b7015ed00 0x7f2b701f7a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.884+0000 7f2b6f7fe640 1 -- 172.21.15.72:0/1605452265 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b70149e00 con 0x7f2b701f7f90 2024-04-15T15:17:08.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.884+0000 7f2b6e7fc640 1 --2- 172.21.15.72:0/1605452265 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2b7015ed00 0x7f2b701f7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:08.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.884+0000 7f2b6f7fe640 1 --2- 172.21.15.72:0/1605452265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b701f7f90 0x7f2b701490d0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f2b700b5000 tx=0x7f2b68011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:08.896 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.884+0000 7f2b577fe640 1 -- 172.21.15.72:0/1605452265 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b68004440 con 0x7f2b701f7f90 2024-04-15T15:17:08.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.884+0000 7f2b577fe640 1 -- 172.21.15.72:0/1605452265 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b6801b070 con 0x7f2b701f7f90 2024-04-15T15:17:08.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.884+0000 7f2b752eb640 1 -- 172.21.15.72:0/1605452265 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b700b7070 con 0x7f2b701f7f90 2024-04-15T15:17:08.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.884+0000 7f2b577fe640 1 -- 172.21.15.72:0/1605452265 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b68003830 con 0x7f2b701f7f90 2024-04-15T15:17:08.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.884+0000 7f2b752eb640 1 -- 172.21.15.72:0/1605452265 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b700b7630 con 0x7f2b701f7f90 2024-04-15T15:17:08.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.885+0000 7f2b752eb640 1 -- 172.21.15.72:0/1605452265 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b70075c50 con 0x7f2b701f7f90 2024-04-15T15:17:08.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.886+0000 7f2b577fe640 1 -- 172.21.15.72:0/1605452265 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f2b680039d0 con 0x7f2b701f7f90 2024-04-15T15:17:08.897 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.886+0000 7f2b577fe640 1 --2- 172.21.15.72:0/1605452265 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2b5c06c980 0x7f2b5c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.886+0000 7f2b577fe640 1 -- 172.21.15.72:0/1605452265 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5421+0+0 (secure 0 0 0) 0x7f2b6808bb80 con 0x7f2b701f7f90 2024-04-15T15:17:08.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.886+0000 7f2b6effd640 1 --2- 172.21.15.72:0/1605452265 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2b5c06c980 0x7f2b5c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.887+0000 7f2b6effd640 1 --2- 172.21.15.72:0/1605452265 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2b5c06c980 0x7f2b5c06ee40 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f2b60034000 tx=0x7f2b60009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:08.898 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:08.891+0000 7f2b577fe640 1 -- 172.21.15.72:0/1605452265 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2b68025020 con 0x7f2b701f7f90 2024-04-15T15:17:09.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.066+0000 7f2b752eb640 1 -- 172.21.15.72:0/1605452265 --> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f2b700b7800 con 0x7f2b5c06c980 2024-04-15T15:17:09.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.066+0000 7f2b577fe640 1 -- 172.21.15.72:0/1605452265 <== mgr.4101 v2:172.21.15.72:6832/3503807805 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f2b700b7800 con 0x7f2b5c06c980 2024-04-15T15:17:09.068 INFO:teuthology.orchestra.run.smithi072.stderr:instructed osd(s) 5 to deep-scrub 2024-04-15T15:17:09.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.068+0000 7f2b752eb640 1 -- 172.21.15.72:0/1605452265 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2b5c06c980 msgr2=0x7f2b5c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.068+0000 7f2b752eb640 1 --2- 172.21.15.72:0/1605452265 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2b5c06c980 0x7f2b5c06ee40 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f2b60034000 tx=0x7f2b60009030 comp rx=0 tx=0).stop 2024-04-15T15:17:09.070 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.068+0000 7f2b752eb640 1 -- 172.21.15.72:0/1605452265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b701f7f90 msgr2=0x7f2b701490d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.068+0000 7f2b752eb640 1 --2- 172.21.15.72:0/1605452265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b701f7f90 0x7f2b701490d0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f2b700b5000 tx=0x7f2b68011920 comp rx=0 tx=0).stop 2024-04-15T15:17:09.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.068+0000 7f2b752eb640 1 -- 172.21.15.72:0/1605452265 shutdown_connections 2024-04-15T15:17:09.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.068+0000 7f2b752eb640 1 --2- 172.21.15.72:0/1605452265 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f2b5c06c980 0x7f2b5c06ee40 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.068+0000 7f2b752eb640 1 --2- 172.21.15.72:0/1605452265 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f2b701f7f90 0x7f2b701490d0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.068+0000 7f2b752eb640 1 --2- 172.21.15.72:0/1605452265 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f2b7015ed00 0x7f2b701f7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.068+0000 7f2b752eb640 1 --2- 172.21.15.72:0/1605452265 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f2b580039b0 0x7f2b700b7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.068+0000 7f2b752eb640 1 -- 172.21.15.72:0/1605452265 >> 172.21.15.72:0/1605452265 conn(0x7f2b700ceec0 msgr2=0x7f2b70158630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:09.071 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.068+0000 7f2b752eb640 1 -- 172.21.15.72:0/1605452265 shutdown_connections 2024-04-15T15:17:09.072 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.069+0000 7f2b752eb640 1 -- 172.21.15.72:0/1605452265 wait complete. 2024-04-15T15:17:09.081 INFO:tasks.ceph:Scrubbing osd.6 2024-04-15T15:17:09.081 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-15T15:17:09.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.188+0000 7f13f34d9640 1 Processor -- start 2024-04-15T15:17:09.190 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.189+0000 7f13f34d9640 1 -- start start 2024-04-15T15:17:09.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.189+0000 7f13f34d9640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f13ec0af8a0 0x7f13ec06f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.189+0000 7f13f34d9640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f13ec0cc5a0 0x7f13ec0cf550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.189+0000 7f13f34d9640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13ec0afc80 con 0x7f13ec0cfa90 2024-04-15T15:17:09.192 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.189+0000 7f13f34d9640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7f13ec15c030 con 0x7f13ec0cc5a0 2024-04-15T15:17:09.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.189+0000 7f13f34d9640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f13ec06f970 con 0x7f13ec0af8a0 2024-04-15T15:17:09.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.190+0000 7f13f0a74640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f13ec0cc5a0 0x7f13ec0cf550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.190+0000 7f13f1a76640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7f13ec0cfa90 0x7f13ec157e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:34094/0 (socket says 172.21.15.72:34094) 2024-04-15T15:17:09.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.190+0000 7f13f1a76640 1 -- 172.21.15.72:0/1344555118 learned_addr learned my addr 172.21.15.72:0/1344555118 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:17:09.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.190+0000 7f13f0a74640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7f13ec0cc5a0 0x7f13ec0cf550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:36998/0 (socket says 172.21.15.72:36998) 2024-04-15T15:17:09.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.190+0000 7f13f1275640 1 --2- 172.21.15.72:0/1344555118 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f13ec0af8a0 0x7f13ec06f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.190+0000 7f13f0a74640 1 -- 172.21.15.72:0/1344555118 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f13ec0af8a0 msgr2=0x7f13ec06f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:09.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.190+0000 7f13f0a74640 1 --2- 172.21.15.72:0/1344555118 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f13ec0af8a0 0x7f13ec06f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.193 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.190+0000 7f13f0a74640 1 -- 172.21.15.72:0/1344555118 >> v1:172.21.15.72:6789/0 conn(0x7f13ec0cfa90 legacy=0x7f13ec157e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.190+0000 7f13dffff640 1 -- 172.21.15.72:0/1344555118 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 900035070 0 0) 0x7f13ec0afc80 con 0x7f13ec0cfa90 2024-04-15T15:17:09.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.190+0000 7f13f0a74640 1 -- 172.21.15.72:0/1344555118 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13ec06f6d0 con 0x7f13ec0cc5a0 2024-04-15T15:17:09.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.190+0000 7f13f1275640 1 --2- 172.21.15.72:0/1344555118 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f13ec0af8a0 0x7f13ec06f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:09.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.191+0000 7f13f0a74640 1 --2- 172.21.15.72:0/1344555118 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13ec0cc5a0 0x7f13ec0cf550 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f13e00097a0 tx=0x7f13e002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4ad3439dbcd8d49 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:09.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.191+0000 7f13dffff640 1 -- 172.21.15.72:0/1344555118 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e00315a0 con 0x7f13ec0cc5a0 2024-04-15T15:17:09.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.191+0000 7f13dffff640 1 -- 172.21.15.72:0/1344555118 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13e0037070 con 0x7f13ec0cc5a0 2024-04-15T15:17:09.194 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.191+0000 7f13dffff640 1 -- 172.21.15.72:0/1344555118 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e0031960 con 0x7f13ec0cc5a0 2024-04-15T15:17:09.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.191+0000 7f13f34d9640 1 -- 172.21.15.72:0/1344555118 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13ec0cc5a0 msgr2=0x7f13ec0cf550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.191+0000 7f13f34d9640 1 --2- 172.21.15.72:0/1344555118 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13ec0cc5a0 0x7f13ec0cf550 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f13e00097a0 tx=0x7f13e002b840 comp rx=0 tx=0).stop 2024-04-15T15:17:09.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.191+0000 7f13f34d9640 1 -- 172.21.15.72:0/1344555118 shutdown_connections 2024-04-15T15:17:09.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.191+0000 7f13f34d9640 1 --2- 172.21.15.72:0/1344555118 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13ec0cc5a0 0x7f13ec0cf550 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.191+0000 7f13f34d9640 1 --2- 172.21.15.72:0/1344555118 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f13ec0af8a0 0x7f13ec06f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.191+0000 7f13f34d9640 1 -- 172.21.15.72:0/1344555118 >> 172.21.15.72:0/1344555118 conn(0x7f13ec0b1ee0 msgr2=0x7f13ec0b22c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:09.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.191+0000 7f13f34d9640 1 -- 172.21.15.72:0/1344555118 shutdown_connections 2024-04-15T15:17:09.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.192+0000 7f13f34d9640 1 -- 172.21.15.72:0/1344555118 wait complete. 2024-04-15T15:17:09.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.193+0000 7f13f34d9640 1 Processor -- start 2024-04-15T15:17:09.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.193+0000 7f13f34d9640 1 -- start start 2024-04-15T15:17:09.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.193+0000 7f13f34d9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13ec0af8a0 0x7f13ec14b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.193+0000 7f13f34d9640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f13ec0cc5a0 0x7f13ec0c97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.193+0000 7f13f34d9640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f13ec0cfa90 0x7f13ec14a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.193+0000 7f13f34d9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f13ec15b550 con 0x7f13ec0af8a0 2024-04-15T15:17:09.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.193+0000 7f13f34d9640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f13ec15bba0 con 0x7f13ec0cc5a0 2024-04-15T15:17:09.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.193+0000 7f13f34d9640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f13ec15b830 con 0x7f13ec0cfa90 2024-04-15T15:17:09.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.193+0000 7f13f1275640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13ec0af8a0 0x7f13ec14b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.193+0000 7f13f1275640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13ec0af8a0 0x7f13ec14b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37000/0 (socket says 172.21.15.72:37000) 2024-04-15T15:17:09.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.193+0000 7f13f1275640 1 -- 172.21.15.72:0/2506009250 learned_addr learned my addr 172.21.15.72:0/2506009250 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:09.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.194+0000 7f13f1a76640 1 --2- 172.21.15.72:0/2506009250 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f13ec0cfa90 0x7f13ec14a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.194+0000 7f13f0a74640 1 --2- 172.21.15.72:0/2506009250 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f13ec0cc5a0 0x7f13ec0c97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.194+0000 7f13f1275640 1 -- 172.21.15.72:0/2506009250 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f13ec0cfa90 msgr2=0x7f13ec14a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.194+0000 7f13f1275640 1 --2- 172.21.15.72:0/2506009250 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f13ec0cfa90 0x7f13ec14a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.194+0000 7f13f1275640 1 -- 172.21.15.72:0/2506009250 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f13ec0cc5a0 msgr2=0x7f13ec0c97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.194+0000 7f13f1275640 1 --2- 172.21.15.72:0/2506009250 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f13ec0cc5a0 0x7f13ec0c97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.194+0000 7f13f1275640 1 -- 172.21.15.72:0/2506009250 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13ec0ae850 con 0x7f13ec0af8a0 2024-04-15T15:17:09.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.194+0000 7f13f1275640 1 --2- 172.21.15.72:0/2506009250 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13ec0af8a0 0x7f13ec14b280 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f13e800c820 tx=0x7f13e800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:09.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.194+0000 7f13ddffb640 1 -- 172.21.15.72:0/2506009250 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e8015070 con 0x7f13ec0af8a0 2024-04-15T15:17:09.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.194+0000 7f13ddffb640 1 -- 172.21.15.72:0/2506009250 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13e8007a30 con 0x7f13ec0af8a0 2024-04-15T15:17:09.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.194+0000 7f13f34d9640 1 -- 172.21.15.72:0/2506009250 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13ec0adab0 con 0x7f13ec0af8a0 2024-04-15T15:17:09.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.194+0000 7f13ddffb640 1 -- 172.21.15.72:0/2506009250 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e8007de0 con 0x7f13ec0af8a0 2024-04-15T15:17:09.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.194+0000 7f13f34d9640 1 -- 172.21.15.72:0/2506009250 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13ec0add70 con 0x7f13ec0af8a0 2024-04-15T15:17:09.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.195+0000 7f13f34d9640 1 -- 172.21.15.72:0/2506009250 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f13ec0689e0 con 0x7f13ec0af8a0 2024-04-15T15:17:09.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.196+0000 7f13ddffb640 1 -- 172.21.15.72:0/2506009250 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f13e8018070 con 0x7f13ec0af8a0 2024-04-15T15:17:09.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.196+0000 7f13ddffb640 1 --2- 172.21.15.72:0/2506009250 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f13d006c980 0x7f13d006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.196+0000 7f13ddffb640 1 -- 172.21.15.72:0/2506009250 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5421+0+0 (secure 0 0 0) 0x7f13e808ba20 con 0x7f13ec0af8a0 2024-04-15T15:17:09.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.196+0000 7f13f0a74640 1 --2- 172.21.15.72:0/2506009250 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f13d006c980 0x7f13d006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.196+0000 7f13ddffb640 1 --2- 172.21.15.72:0/2506009250 >> [v2:172.21.15.175:6808/1973126001,v1:172.21.15.175:6809/1973126001] conn(0x7f13d0072ec0 0x7f13d0075300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.196+0000 7f13ddffb640 1 -- 172.21.15.72:0/2506009250 --> [v2:172.21.15.175:6808/1973126001,v1:172.21.15.175:6809/1973126001] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f13d00759d0 con 0x7f13d0072ec0 2024-04-15T15:17:09.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.196+0000 7f13ddffb640 1 -- 172.21.15.72:0/2506009250 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f13e80902c0 con 0x7f13ec0af8a0 2024-04-15T15:17:09.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.197+0000 7f13f1a76640 1 --2- 172.21.15.72:0/2506009250 >> [v2:172.21.15.175:6808/1973126001,v1:172.21.15.175:6809/1973126001] conn(0x7f13d0072ec0 0x7f13d0075300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.197+0000 7f13f0a74640 1 --2- 172.21.15.72:0/2506009250 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f13d006c980 0x7f13d006ee40 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f13e0036000 tx=0x7f13e003d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:09.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.197+0000 7f13f1a76640 1 --2- 172.21.15.72:0/2506009250 >> [v2:172.21.15.175:6808/1973126001,v1:172.21.15.175:6809/1973126001] conn(0x7f13d0072ec0 0x7f13d0075300 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:09.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.198+0000 7f13ddffb640 1 -- 172.21.15.72:0/2506009250 <== osd.6 v2:172.21.15.175:6808/1973126001 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f13d00759d0 con 0x7f13d0072ec0 2024-04-15T15:17:09.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.210+0000 7f13f34d9640 1 -- 172.21.15.72:0/2506009250 --> [v2:172.21.15.175:6808/1973126001,v1:172.21.15.175:6809/1973126001] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f13ec0cd990 con 0x7f13d0072ec0 2024-04-15T15:17:09.212 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.211+0000 7f13ddffb640 1 -- 172.21.15.72:0/2506009250 <== osd.6 v2:172.21.15.175:6808/1973126001 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f13ec0cd990 con 0x7f13d0072ec0 2024-04-15T15:17:09.213 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-04-15T15:17:09.213 INFO:teuthology.orchestra.run.smithi072.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-15T15:17:09.214 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-04-15T15:17:09.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.212+0000 7f13f34d9640 1 -- 172.21.15.72:0/2506009250 >> [v2:172.21.15.175:6808/1973126001,v1:172.21.15.175:6809/1973126001] conn(0x7f13d0072ec0 msgr2=0x7f13d0075300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.212+0000 7f13f34d9640 1 --2- 172.21.15.72:0/2506009250 >> [v2:172.21.15.175:6808/1973126001,v1:172.21.15.175:6809/1973126001] conn(0x7f13d0072ec0 0x7f13d0075300 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.212+0000 7f13f34d9640 1 -- 172.21.15.72:0/2506009250 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f13d006c980 msgr2=0x7f13d006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.212+0000 7f13f34d9640 1 --2- 172.21.15.72:0/2506009250 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f13d006c980 0x7f13d006ee40 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f13e0036000 tx=0x7f13e003d040 comp rx=0 tx=0).stop 2024-04-15T15:17:09.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.212+0000 7f13f34d9640 1 -- 172.21.15.72:0/2506009250 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13ec0af8a0 msgr2=0x7f13ec14b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.212+0000 7f13f34d9640 1 --2- 172.21.15.72:0/2506009250 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f13ec0af8a0 0x7f13ec14b280 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f13e800c820 tx=0x7f13e800cce0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.212+0000 7f13f1a76640 1 -- 172.21.15.72:0/2506009250 reap_dead start 2024-04-15T15:17:09.215 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.212+0000 7f13f34d9640 1 -- 172.21.15.72:0/2506009250 shutdown_connections 2024-04-15T15:17:09.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.212+0000 7f13f34d9640 1 -- 172.21.15.72:0/2506009250 >> 172.21.15.72:0/2506009250 conn(0x7f13ec0b1ee0 msgr2=0x7f13ec06b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:09.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.212+0000 7f13f34d9640 1 -- 172.21.15.72:0/2506009250 shutdown_connections 2024-04-15T15:17:09.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.212+0000 7f13f34d9640 1 -- 172.21.15.72:0/2506009250 wait complete. 2024-04-15T15:17:09.224 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-15T15:17:09.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.326+0000 7fec956d3640 1 Processor -- start 2024-04-15T15:17:09.328 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.326+0000 7fec956d3640 1 -- start start 2024-04-15T15:17:09.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.327+0000 7fec956d3640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fec9006c400 0x7fec9006c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.327+0000 7fec956d3640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec900d2320 con 0x7fec9015e9f0 2024-04-15T15:17:09.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.327+0000 7fec956d3640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec900d25c0 con 0x7fec9015e610 2024-04-15T15:17:09.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.327+0000 7fec956d3640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fec900d31d0 con 0x7fec9006c400 2024-04-15T15:17:09.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.327+0000 7fec867fc640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fec9015e9f0 0x7fec900d3660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:34102/0 (socket says 172.21.15.72:34102) 2024-04-15T15:17:09.330 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.327+0000 7fec867fc640 1 -- 172.21.15.72:0/4106314984 learned_addr learned my addr 172.21.15.72:0/4106314984 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:17:09.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.327+0000 7fec8f7fe640 1 --2- 172.21.15.72:0/4106314984 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fec9006c400 0x7fec9006c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.327+0000 7fec8e7fc640 1 -- 172.21.15.72:0/4106314984 <== mon.0 v1:172.21.15.72:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1047529679 0 0) 0x7fec900d2320 con 0x7fec9015e9f0 2024-04-15T15:17:09.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.327+0000 7fec8e7fc640 1 -- 172.21.15.72:0/4106314984 --> v1:172.21.15.72:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec7c003520 con 0x7fec9015e9f0 2024-04-15T15:17:09.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.327+0000 7fec8e7fc640 1 -- 172.21.15.72:0/4106314984 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3216244859 0 0) 0x7fec900d25c0 con 0x7fec9015e610 2024-04-15T15:17:09.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.327+0000 7fec8e7fc640 1 -- 172.21.15.72:0/4106314984 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec900d2320 con 0x7fec9015e610 2024-04-15T15:17:09.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.328+0000 7fec8f7fe640 1 -- 172.21.15.72:0/4106314984 >> v1:172.21.15.175:6789/0 conn(0x7fec9015e610 legacy=0x7fec900d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.328+0000 7fec8f7fe640 1 -- 172.21.15.72:0/4106314984 >> v1:172.21.15.72:6789/0 conn(0x7fec9015e9f0 legacy=0x7fec900d3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.331 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.328+0000 7fec8f7fe640 1 -- 172.21.15.72:0/4106314984 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec900d28a0 con 0x7fec9006c400 2024-04-15T15:17:09.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.328+0000 7fec8f7fe640 1 --2- 172.21.15.72:0/4106314984 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fec9006c400 0x7fec9006c890 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fec900d3350 tx=0x7fec8802bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e01c19cea957d935 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:09.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.329+0000 7fec8e7fc640 1 -- 172.21.15.72:0/4106314984 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec88039070 con 0x7fec9006c400 2024-04-15T15:17:09.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.329+0000 7fec8e7fc640 1 -- 172.21.15.72:0/4106314984 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec8802cd50 con 0x7fec9006c400 2024-04-15T15:17:09.332 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.329+0000 7fec8e7fc640 1 -- 172.21.15.72:0/4106314984 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec88034660 con 0x7fec9006c400 2024-04-15T15:17:09.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.329+0000 7fec956d3640 1 -- 172.21.15.72:0/4106314984 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fec9006c400 msgr2=0x7fec9006c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.329+0000 7fec956d3640 1 --2- 172.21.15.72:0/4106314984 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fec9006c400 0x7fec9006c890 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fec900d3350 tx=0x7fec8802bfa0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.329+0000 7fec956d3640 1 -- 172.21.15.72:0/4106314984 shutdown_connections 2024-04-15T15:17:09.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.329+0000 7fec956d3640 1 --2- 172.21.15.72:0/4106314984 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fec9006c400 0x7fec9006c890 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.329+0000 7fec956d3640 1 -- 172.21.15.72:0/4106314984 >> 172.21.15.72:0/4106314984 conn(0x7fec900d4750 msgr2=0x7fec900d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:09.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.329+0000 7fec956d3640 1 -- 172.21.15.72:0/4106314984 shutdown_connections 2024-04-15T15:17:09.336 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.329+0000 7fec956d3640 1 -- 172.21.15.72:0/4106314984 wait complete. 2024-04-15T15:17:09.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.330+0000 7fec956d3640 1 Processor -- start 2024-04-15T15:17:09.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.330+0000 7fec956d3640 1 -- start start 2024-04-15T15:17:09.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.331+0000 7fec956d3640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fec9006c400 0x7fec90156a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.331+0000 7fec956d3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec9015e610 0x7fec90152210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.331+0000 7fec956d3640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fec9015e9f0 0x7fec901585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.331+0000 7fec956d3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fec9006d130 con 0x7fec9015e610 2024-04-15T15:17:09.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.331+0000 7fec956d3640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fec900d2e60 con 0x7fec9015e9f0 2024-04-15T15:17:09.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.331+0000 7fec956d3640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fec900b7940 con 0x7fec9006c400 2024-04-15T15:17:09.337 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.331+0000 7fec867fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec9015e610 0x7fec90152210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.331+0000 7fec867fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec9015e610 0x7fec90152210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37008/0 (socket says 172.21.15.72:37008) 2024-04-15T15:17:09.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.331+0000 7fec867fc640 1 -- 172.21.15.72:0/3070746058 learned_addr learned my addr 172.21.15.72:0/3070746058 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:09.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.331+0000 7fec8f7fe640 1 --2- 172.21.15.72:0/3070746058 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fec9015e9f0 0x7fec901585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.331+0000 7fec8effd640 1 --2- 172.21.15.72:0/3070746058 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fec9006c400 0x7fec90156a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.331+0000 7fec867fc640 1 -- 172.21.15.72:0/3070746058 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fec9006c400 msgr2=0x7fec90156a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.331+0000 7fec867fc640 1 --2- 172.21.15.72:0/3070746058 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fec9006c400 0x7fec90156a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.332+0000 7fec867fc640 1 -- 172.21.15.72:0/3070746058 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fec9015e9f0 msgr2=0x7fec901585e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.338 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.332+0000 7fec867fc640 1 --2- 172.21.15.72:0/3070746058 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fec9015e9f0 0x7fec901585e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.332+0000 7fec867fc640 1 -- 172.21.15.72:0/3070746058 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec90152750 con 0x7fec9015e610 2024-04-15T15:17:09.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.332+0000 7fec8f7fe640 1 --2- 172.21.15.72:0/3070746058 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fec9015e9f0 0x7fec901585e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:09.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.332+0000 7fec867fc640 1 --2- 172.21.15.72:0/3070746058 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec9015e610 0x7fec90152210 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fec7800c440 tx=0x7fec7800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:09.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.332+0000 7fec87fff640 1 -- 172.21.15.72:0/3070746058 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec78017070 con 0x7fec9015e610 2024-04-15T15:17:09.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.332+0000 7fec87fff640 1 -- 172.21.15.72:0/3070746058 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec78005ce0 con 0x7fec9015e610 2024-04-15T15:17:09.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.332+0000 7fec87fff640 1 -- 172.21.15.72:0/3070746058 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec78007270 con 0x7fec9015e610 2024-04-15T15:17:09.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.332+0000 7fec956d3640 1 -- 172.21.15.72:0/3070746058 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec90157120 con 0x7fec9015e610 2024-04-15T15:17:09.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.332+0000 7fec956d3640 1 -- 172.21.15.72:0/3070746058 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec90159670 con 0x7fec9015e610 2024-04-15T15:17:09.339 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.333+0000 7fec956d3640 1 -- 172.21.15.72:0/3070746058 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec90075c50 con 0x7fec9015e610 2024-04-15T15:17:09.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.333+0000 7fec87fff640 1 -- 172.21.15.72:0/3070746058 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fec7801a070 con 0x7fec9015e610 2024-04-15T15:17:09.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.334+0000 7fec87fff640 1 --2- 172.21.15.72:0/3070746058 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fec7006c980 0x7fec7006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.334+0000 7fec8effd640 1 --2- 172.21.15.72:0/3070746058 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fec7006c980 0x7fec7006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.334+0000 7fec87fff640 1 -- 172.21.15.72:0/3070746058 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5421+0+0 (secure 0 0 0) 0x7fec7808abf0 con 0x7fec9015e610 2024-04-15T15:17:09.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.334+0000 7fec8effd640 1 --2- 172.21.15.72:0/3070746058 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fec7006c980 0x7fec7006ee40 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fec80001650 tx=0x7fec80009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:09.340 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.338+0000 7fec87fff640 1 -- 172.21.15.72:0/3070746058 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fec78058b90 con 0x7fec9015e610 2024-04-15T15:17:09.517 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.515+0000 7fec956d3640 1 -- 172.21.15.72:0/3070746058 --> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7fec9015bfb0 con 0x7fec7006c980 2024-04-15T15:17:09.518 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.516+0000 7fec87fff640 1 -- 172.21.15.72:0/3070746058 <== mgr.4101 v2:172.21.15.72:6832/3503807805 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fec9015bfb0 con 0x7fec7006c980 2024-04-15T15:17:09.518 INFO:teuthology.orchestra.run.smithi072.stderr:instructed osd(s) 6 to deep-scrub 2024-04-15T15:17:09.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.517+0000 7fec956d3640 1 -- 172.21.15.72:0/3070746058 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fec7006c980 msgr2=0x7fec7006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.518+0000 7fec956d3640 1 --2- 172.21.15.72:0/3070746058 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fec7006c980 0x7fec7006ee40 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fec80001650 tx=0x7fec80009040 comp rx=0 tx=0).stop 2024-04-15T15:17:09.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.518+0000 7fec956d3640 1 -- 172.21.15.72:0/3070746058 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec9015e610 msgr2=0x7fec90152210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.518+0000 7fec956d3640 1 --2- 172.21.15.72:0/3070746058 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec9015e610 0x7fec90152210 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fec7800c440 tx=0x7fec7800c900 comp rx=0 tx=0).stop 2024-04-15T15:17:09.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.518+0000 7fec956d3640 1 -- 172.21.15.72:0/3070746058 shutdown_connections 2024-04-15T15:17:09.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.518+0000 7fec956d3640 1 --2- 172.21.15.72:0/3070746058 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fec7006c980 0x7fec7006ee40 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.520 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.518+0000 7fec956d3640 1 --2- 172.21.15.72:0/3070746058 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fec9015e9f0 0x7fec901585e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.518+0000 7fec956d3640 1 --2- 172.21.15.72:0/3070746058 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fec9015e610 0x7fec90152210 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.518+0000 7fec956d3640 1 --2- 172.21.15.72:0/3070746058 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fec9006c400 0x7fec90156a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.518+0000 7fec956d3640 1 -- 172.21.15.72:0/3070746058 >> 172.21.15.72:0/3070746058 conn(0x7fec900d4750 msgr2=0x7fec900b91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:09.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.518+0000 7fec956d3640 1 -- 172.21.15.72:0/3070746058 shutdown_connections 2024-04-15T15:17:09.521 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.518+0000 7fec956d3640 1 -- 172.21.15.72:0/3070746058 wait complete. 2024-04-15T15:17:09.530 INFO:tasks.ceph:Scrubbing osd.7 2024-04-15T15:17:09.531 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-15T15:17:09.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.635+0000 7f94b6352640 1 Processor -- start 2024-04-15T15:17:09.637 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.636+0000 7f94b6352640 1 -- start start 2024-04-15T15:17:09.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.636+0000 7f94b6352640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f94b0145f30 0x7f94b006eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.638 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.636+0000 7f94b6352640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f94b006d080 0x7f94b00cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.636+0000 7f94b6352640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94b00cfed0 con 0x7f94b00cfa70 2024-04-15T15:17:09.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.636+0000 7f94b6352640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f94b00d0170 con 0x7f94b006d080 2024-04-15T15:17:09.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.636+0000 7f94b6352640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f94b00d0450 con 0x7f94b0145f30 2024-04-15T15:17:09.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.637+0000 7f94affff640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f94b0145f30 0x7f94b006eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.637+0000 7f94b48ef640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f94b006d080 0x7f94b00cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.637+0000 7f94af7fe640 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f94b00cfa70 0x7f94b006c970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.72:39192/0 (socket says 172.21.15.72:39192) 2024-04-15T15:17:09.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.637+0000 7f94af7fe640 1 -- 172.21.15.72:0/2745878921 learned_addr learned my addr 172.21.15.72:0/2745878921 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:17:09.639 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.637+0000 7f94affff640 1 -- 172.21.15.72:0/2745878921 >> v2:172.21.15.175:3300/0 conn(0x7f94b006d080 msgr2=0x7f94b00cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:09.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.637+0000 7f94affff640 1 --2- 172.21.15.72:0/2745878921 >> v2:172.21.15.175:3300/0 conn(0x7f94b006d080 0x7f94b00cdf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.637+0000 7f94affff640 1 -- 172.21.15.72:0/2745878921 >> v1:172.21.15.175:6789/0 conn(0x7f94b00cfa70 legacy=0x7f94b006c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.640 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.637+0000 7f94affff640 1 -- 172.21.15.72:0/2745878921 --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94b00d0730 con 0x7f94b0145f30 2024-04-15T15:17:09.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.638+0000 7f94affff640 1 --2- 172.21.15.72:0/2745878921 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f94b0145f30 0x7f94b006eb40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f94a00097a0 tx=0x7f94a002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=907767335c558f1e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:09.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.638+0000 7f94aeffd640 1 -- 172.21.15.72:0/2745878921 <== mon.2 v2:172.21.15.175:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a0035020 con 0x7f94b0145f30 2024-04-15T15:17:09.641 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.638+0000 7f94aeffd640 1 -- 172.21.15.72:0/2745878921 <== mon.2 v2:172.21.15.175:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94a0031830 con 0x7f94b0145f30 2024-04-15T15:17:09.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.638+0000 7f94aeffd640 1 -- 172.21.15.72:0/2745878921 <== mon.2 v2:172.21.15.175:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a0031c00 con 0x7f94b0145f30 2024-04-15T15:17:09.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.639+0000 7f94b6352640 1 -- 172.21.15.72:0/2745878921 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f94b0145f30 msgr2=0x7f94b006eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.639+0000 7f94b6352640 1 --2- 172.21.15.72:0/2745878921 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f94b0145f30 0x7f94b006eb40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f94a00097a0 tx=0x7f94a002b8d0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.639+0000 7f94b6352640 1 -- 172.21.15.72:0/2745878921 shutdown_connections 2024-04-15T15:17:09.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.639+0000 7f94b6352640 1 --2- 172.21.15.72:0/2745878921 >> v2:172.21.15.175:3300/0 conn(0x7f94b006d080 0x7f94b00cdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.639+0000 7f94b6352640 1 --2- 172.21.15.72:0/2745878921 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f94b0145f30 0x7f94b006eb40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.639+0000 7f94b6352640 1 -- 172.21.15.72:0/2745878921 >> 172.21.15.72:0/2745878921 conn(0x7f94b00703a0 msgr2=0x7f94b0070780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:09.642 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.639+0000 7f94b6352640 1 -- 172.21.15.72:0/2745878921 shutdown_connections 2024-04-15T15:17:09.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.639+0000 7f94b6352640 1 -- 172.21.15.72:0/2745878921 wait complete. 2024-04-15T15:17:09.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.640+0000 7f94b6352640 1 Processor -- start 2024-04-15T15:17:09.643 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.640+0000 7f94b6352640 1 -- start start 2024-04-15T15:17:09.645 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.641+0000 7f94b6352640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f94b006d080 0x7f94b0132d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.641+0000 7f94b6352640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94b00cfa70 0x7f94b0134c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.641+0000 7f94b6352640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94b0145f30 0x7f94b015f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.641+0000 7f94b6352640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f94b006d670 con 0x7f94b0145f30 2024-04-15T15:17:09.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.641+0000 7f94b6352640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f94b00d0cf0 con 0x7f94b00cfa70 2024-04-15T15:17:09.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.641+0000 7f94b6352640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f94b00ce4d0 con 0x7f94b006d080 2024-04-15T15:17:09.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.641+0000 7f94b48ef640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94b0145f30 0x7f94b015f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.641+0000 7f94b48ef640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94b0145f30 0x7f94b015f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37022/0 (socket says 172.21.15.72:37022) 2024-04-15T15:17:09.646 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.641+0000 7f94b48ef640 1 -- 172.21.15.72:0/546968320 learned_addr learned my addr 172.21.15.72:0/546968320 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:09.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.641+0000 7f94affff640 1 --2- 172.21.15.72:0/546968320 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f94b006d080 0x7f94b0132d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.641+0000 7f94af7fe640 1 --2- 172.21.15.72:0/546968320 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94b00cfa70 0x7f94b0134c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.641+0000 7f94b48ef640 1 -- 172.21.15.72:0/546968320 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f94b006d080 msgr2=0x7f94b0132d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.641+0000 7f94b48ef640 1 --2- 172.21.15.72:0/546968320 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f94b006d080 0x7f94b0132d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.641+0000 7f94b48ef640 1 -- 172.21.15.72:0/546968320 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94b00cfa70 msgr2=0x7f94b0134c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.641+0000 7f94b48ef640 1 --2- 172.21.15.72:0/546968320 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f94b00cfa70 0x7f94b0134c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.641+0000 7f94b48ef640 1 -- 172.21.15.72:0/546968320 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94b012f9b0 con 0x7f94b0145f30 2024-04-15T15:17:09.647 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.641+0000 7f94b48ef640 1 --2- 172.21.15.72:0/546968320 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94b0145f30 0x7f94b015f0e0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f94a800efc0 tx=0x7f94a800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:09.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.641+0000 7f94acff9640 1 -- 172.21.15.72:0/546968320 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a8018020 con 0x7f94b0145f30 2024-04-15T15:17:09.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.642+0000 7f94b6352640 1 -- 172.21.15.72:0/546968320 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94b012fc90 con 0x7f94b0145f30 2024-04-15T15:17:09.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.642+0000 7f94acff9640 1 -- 172.21.15.72:0/546968320 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94a8014480 con 0x7f94b0145f30 2024-04-15T15:17:09.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.642+0000 7f94acff9640 1 -- 172.21.15.72:0/546968320 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a8014870 con 0x7f94b0145f30 2024-04-15T15:17:09.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.642+0000 7f94b6352640 1 -- 172.21.15.72:0/546968320 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94b0130250 con 0x7f94b0145f30 2024-04-15T15:17:09.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.642+0000 7f94b6352640 1 -- 172.21.15.72:0/546968320 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f94b0074310 con 0x7f94b0145f30 2024-04-15T15:17:09.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.643+0000 7f94acff9640 1 -- 172.21.15.72:0/546968320 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f94a8005d10 con 0x7f94b0145f30 2024-04-15T15:17:09.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.643+0000 7f94acff9640 1 --2- 172.21.15.72:0/546968320 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f949c06c980 0x7f949c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.648 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.644+0000 7f94acff9640 1 -- 172.21.15.72:0/546968320 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5421+0+0 (secure 0 0 0) 0x7f94a808ca10 con 0x7f94b0145f30 2024-04-15T15:17:09.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.644+0000 7f94affff640 1 --2- 172.21.15.72:0/546968320 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f949c06c980 0x7f949c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.644+0000 7f94acff9640 1 --2- 172.21.15.72:0/546968320 >> [v2:172.21.15.175:6816/2672274294,v1:172.21.15.175:6817/2672274294] conn(0x7f949c072ec0 0x7f949c075300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.644+0000 7f94acff9640 1 -- 172.21.15.72:0/546968320 --> [v2:172.21.15.175:6816/2672274294,v1:172.21.15.175:6817/2672274294] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f94a8012e50 con 0x7f949c072ec0 2024-04-15T15:17:09.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.644+0000 7f94acff9640 1 -- 172.21.15.72:0/546968320 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f94a80912c0 con 0x7f94b0145f30 2024-04-15T15:17:09.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.644+0000 7f94affff640 1 --2- 172.21.15.72:0/546968320 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f949c06c980 0x7f949c06ee40 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f94a0036000 tx=0x7f94a0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:09.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.644+0000 7f94af7fe640 1 --2- 172.21.15.72:0/546968320 >> [v2:172.21.15.175:6816/2672274294,v1:172.21.15.175:6817/2672274294] conn(0x7f949c072ec0 0x7f949c075300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.649 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.644+0000 7f94af7fe640 1 --2- 172.21.15.72:0/546968320 >> [v2:172.21.15.175:6816/2672274294,v1:172.21.15.175:6817/2672274294] conn(0x7f949c072ec0 0x7f949c075300 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:09.650 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.645+0000 7f94acff9640 1 -- 172.21.15.72:0/546968320 <== osd.7 v2:172.21.15.175:6816/2672274294 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f94a8012e50 con 0x7f949c072ec0 2024-04-15T15:17:09.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.658+0000 7f94b6352640 1 -- 172.21.15.72:0/546968320 --> [v2:172.21.15.175:6816/2672274294,v1:172.21.15.175:6817/2672274294] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f94b00690f0 con 0x7f949c072ec0 2024-04-15T15:17:09.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.659+0000 7f94acff9640 1 -- 172.21.15.72:0/546968320 <== osd.7 v2:172.21.15.175:6816/2672274294 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f94b00690f0 con 0x7f949c072ec0 2024-04-15T15:17:09.661 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-04-15T15:17:09.661 INFO:teuthology.orchestra.run.smithi072.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-15T15:17:09.661 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-04-15T15:17:09.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.659+0000 7f94b6352640 1 -- 172.21.15.72:0/546968320 >> [v2:172.21.15.175:6816/2672274294,v1:172.21.15.175:6817/2672274294] conn(0x7f949c072ec0 msgr2=0x7f949c075300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.659+0000 7f94b6352640 1 --2- 172.21.15.72:0/546968320 >> [v2:172.21.15.175:6816/2672274294,v1:172.21.15.175:6817/2672274294] conn(0x7f949c072ec0 0x7f949c075300 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.659+0000 7f94b6352640 1 -- 172.21.15.72:0/546968320 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f949c06c980 msgr2=0x7f949c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.659+0000 7f94b6352640 1 --2- 172.21.15.72:0/546968320 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f949c06c980 0x7f949c06ee40 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f94a0036000 tx=0x7f94a0008030 comp rx=0 tx=0).stop 2024-04-15T15:17:09.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.659+0000 7f94b6352640 1 -- 172.21.15.72:0/546968320 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94b0145f30 msgr2=0x7f94b015f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.659+0000 7f94b6352640 1 --2- 172.21.15.72:0/546968320 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f94b0145f30 0x7f94b015f0e0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f94a800efc0 tx=0x7f94a800c490 comp rx=0 tx=0).stop 2024-04-15T15:17:09.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.659+0000 7f94b48ef640 1 -- 172.21.15.72:0/546968320 reap_dead start 2024-04-15T15:17:09.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.659+0000 7f94b6352640 1 -- 172.21.15.72:0/546968320 shutdown_connections 2024-04-15T15:17:09.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.659+0000 7f94b6352640 1 -- 172.21.15.72:0/546968320 >> 172.21.15.72:0/546968320 conn(0x7f94b00703a0 msgr2=0x7f94b006c600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:09.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.659+0000 7f94b6352640 1 -- 172.21.15.72:0/546968320 shutdown_connections 2024-04-15T15:17:09.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.659+0000 7f94b6352640 1 -- 172.21.15.72:0/546968320 wait complete. 2024-04-15T15:17:09.671 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-15T15:17:09.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.774+0000 7f069937c640 1 Processor -- start 2024-04-15T15:17:09.775 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.774+0000 7f069937c640 1 -- start start 2024-04-15T15:17:09.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.774+0000 7f069937c640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f069415e9f0 0x7f06940d3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.774+0000 7f069937c640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f069406c400 0x7f069406c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.774+0000 7f069937c640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06940d2320 con 0x7f069415e610 2024-04-15T15:17:09.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.774+0000 7f069937c640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f06940d31d0 con 0x7f069415e9f0 2024-04-15T15:17:09.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.774+0000 7f069937c640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f06940d25c0 con 0x7f069406c400 2024-04-15T15:17:09.777 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.775+0000 7f0692ffd640 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f069415e610 0x7f06940d0690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.72:39204/0 (socket says 172.21.15.72:39204) 2024-04-15T15:17:09.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.775+0000 7f06937fe640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f069406c400 0x7f069406c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.775+0000 7f0692ffd640 1 -- 172.21.15.72:0/1699438004 learned_addr learned my addr 172.21.15.72:0/1699438004 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:17:09.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.775+0000 7f06927fc640 1 --2- 172.21.15.72:0/1699438004 >> v2:172.21.15.175:3300/0 conn(0x7f069415e9f0 0x7f06940d3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.775+0000 7f0691ffb640 1 -- 172.21.15.72:0/1699438004 <== mon.1 v1:172.21.15.175:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4152963889 0 0) 0x7f06940d2320 con 0x7f069415e610 2024-04-15T15:17:09.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.775+0000 7f0691ffb640 1 -- 172.21.15.72:0/1699438004 --> v1:172.21.15.175:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f067c003520 con 0x7f069415e610 2024-04-15T15:17:09.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.775+0000 7f06927fc640 1 -- 172.21.15.72:0/1699438004 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f069406c400 msgr2=0x7f069406c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:09.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.775+0000 7f06927fc640 1 --2- 172.21.15.72:0/1699438004 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f069406c400 0x7f069406c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.775+0000 7f06927fc640 1 -- 172.21.15.72:0/1699438004 >> v1:172.21.15.175:6789/0 conn(0x7f069415e610 legacy=0x7f06940d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.778 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.775+0000 7f06927fc640 1 -- 172.21.15.72:0/1699438004 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06940d28a0 con 0x7f069415e9f0 2024-04-15T15:17:09.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.776+0000 7f06927fc640 1 --2- 172.21.15.72:0/1699438004 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f069415e9f0 0x7f06940d3660 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f06880097a0 tx=0x7f068802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f98da60f9f28b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:09.781 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.776+0000 7f0691ffb640 1 -- 172.21.15.72:0/1699438004 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0688034070 con 0x7f069415e9f0 2024-04-15T15:17:09.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.776+0000 7f0691ffb640 1 -- 172.21.15.72:0/1699438004 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0688030610 con 0x7f069415e9f0 2024-04-15T15:17:09.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.776+0000 7f0691ffb640 1 -- 172.21.15.72:0/1699438004 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06880309c0 con 0x7f069415e9f0 2024-04-15T15:17:09.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.777+0000 7f069937c640 1 -- 172.21.15.72:0/1699438004 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f069415e9f0 msgr2=0x7f06940d3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.777+0000 7f069937c640 1 --2- 172.21.15.72:0/1699438004 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f069415e9f0 0x7f06940d3660 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f06880097a0 tx=0x7f068802b960 comp rx=0 tx=0).stop 2024-04-15T15:17:09.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.777+0000 7f069937c640 1 -- 172.21.15.72:0/1699438004 shutdown_connections 2024-04-15T15:17:09.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.777+0000 7f069937c640 1 --2- 172.21.15.72:0/1699438004 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f069406c400 0x7f069406c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.777+0000 7f069937c640 1 --2- 172.21.15.72:0/1699438004 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f069415e9f0 0x7f06940d3660 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.777+0000 7f069937c640 1 -- 172.21.15.72:0/1699438004 >> 172.21.15.72:0/1699438004 conn(0x7f06940d4750 msgr2=0x7f06940d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:09.782 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.777+0000 7f069937c640 1 -- 172.21.15.72:0/1699438004 shutdown_connections 2024-04-15T15:17:09.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.777+0000 7f069937c640 1 -- 172.21.15.72:0/1699438004 wait complete. 2024-04-15T15:17:09.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.778+0000 7f069937c640 1 Processor -- start 2024-04-15T15:17:09.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.778+0000 7f069937c640 1 -- start start 2024-04-15T15:17:09.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.779+0000 7f069937c640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f069406c400 0x7f069415f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.779+0000 7f069937c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f069415e610 0x7f06940ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.779+0000 7f069937c640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f069415e9f0 0x7f06940cab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.779+0000 7f069937c640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f069406d130 con 0x7f069415e610 2024-04-15T15:17:09.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.779+0000 7f069937c640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f06940d2e60 con 0x7f069406c400 2024-04-15T15:17:09.783 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.779+0000 7f069937c640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f06940d3ba0 con 0x7f069415e9f0 2024-04-15T15:17:09.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.779+0000 7f06927fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f069415e610 0x7f06940ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.779+0000 7f06927fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f069415e610 0x7f06940ca600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37024/0 (socket says 172.21.15.72:37024) 2024-04-15T15:17:09.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.779+0000 7f06927fc640 1 -- 172.21.15.72:0/4261530245 learned_addr learned my addr 172.21.15.72:0/4261530245 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:09.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.779+0000 7f0692ffd640 1 --2- 172.21.15.72:0/4261530245 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f069406c400 0x7f069415f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.779+0000 7f06937fe640 1 --2- 172.21.15.72:0/4261530245 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f069415e9f0 0x7f06940cab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.779+0000 7f06927fc640 1 -- 172.21.15.72:0/4261530245 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f069415e9f0 msgr2=0x7f06940cab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.779+0000 7f06927fc640 1 --2- 172.21.15.72:0/4261530245 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f069415e9f0 0x7f06940cab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.784 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.779+0000 7f06927fc640 1 -- 172.21.15.72:0/4261530245 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f069406c400 msgr2=0x7f069415f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.779+0000 7f06927fc640 1 --2- 172.21.15.72:0/4261530245 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f069406c400 0x7f069415f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.779+0000 7f06927fc640 1 -- 172.21.15.72:0/4261530245 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0694153b60 con 0x7f069415e610 2024-04-15T15:17:09.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.779+0000 7f06927fc640 1 --2- 172.21.15.72:0/4261530245 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f069415e610 0x7f06940ca600 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f0688000c00 tx=0x7f06880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:09.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.780+0000 7f067b7fe640 1 -- 172.21.15.72:0/4261530245 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0688034050 con 0x7f069415e610 2024-04-15T15:17:09.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.780+0000 7f067b7fe640 1 -- 172.21.15.72:0/4261530245 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06880044c0 con 0x7f069415e610 2024-04-15T15:17:09.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.780+0000 7f069937c640 1 -- 172.21.15.72:0/4261530245 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0694153dd0 con 0x7f069415e610 2024-04-15T15:17:09.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.780+0000 7f067b7fe640 1 -- 172.21.15.72:0/4261530245 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0688003a40 con 0x7f069415e610 2024-04-15T15:17:09.785 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.780+0000 7f069937c640 1 -- 172.21.15.72:0/4261530245 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f069415c960 con 0x7f069415e610 2024-04-15T15:17:09.786 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.781+0000 7f069937c640 1 -- 172.21.15.72:0/4261530245 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06941541c0 con 0x7f069415e610 2024-04-15T15:17:09.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.781+0000 7f067b7fe640 1 -- 172.21.15.72:0/4261530245 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f0688037070 con 0x7f069415e610 2024-04-15T15:17:09.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.782+0000 7f067b7fe640 1 --2- 172.21.15.72:0/4261530245 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f068006c980 0x7f068006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.782+0000 7f067b7fe640 1 -- 172.21.15.72:0/4261530245 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5421+0+0 (secure 0 0 0) 0x7f06880ad190 con 0x7f069415e610 2024-04-15T15:17:09.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.782+0000 7f0692ffd640 1 --2- 172.21.15.72:0/4261530245 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f068006c980 0x7f068006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.782+0000 7f0692ffd640 1 --2- 172.21.15.72:0/4261530245 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f068006c980 0x7f068006ee40 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f0684005880 tx=0x7f0684009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:09.787 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.786+0000 7f067b7fe640 1 -- 172.21.15.72:0/4261530245 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f068807af10 con 0x7f069415e610 2024-04-15T15:17:09.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.962+0000 7f069937c640 1 -- 172.21.15.72:0/4261530245 --> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f0694153fa0 con 0x7f068006c980 2024-04-15T15:17:09.964 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.963+0000 7f067b7fe640 1 -- 172.21.15.72:0/4261530245 <== mgr.4101 v2:172.21.15.72:6832/3503807805 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f0694153fa0 con 0x7f068006c980 2024-04-15T15:17:09.964 INFO:teuthology.orchestra.run.smithi072.stderr:instructed osd(s) 7 to deep-scrub 2024-04-15T15:17:09.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.964+0000 7f069937c640 1 -- 172.21.15.72:0/4261530245 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f068006c980 msgr2=0x7f068006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.966 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.964+0000 7f069937c640 1 --2- 172.21.15.72:0/4261530245 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f068006c980 0x7f068006ee40 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f0684005880 tx=0x7f0684009040 comp rx=0 tx=0).stop 2024-04-15T15:17:09.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.965+0000 7f069937c640 1 -- 172.21.15.72:0/4261530245 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f069415e610 msgr2=0x7f06940ca600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.965+0000 7f069937c640 1 --2- 172.21.15.72:0/4261530245 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f069415e610 0x7f06940ca600 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f0688000c00 tx=0x7f06880046c0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.965+0000 7f069937c640 1 -- 172.21.15.72:0/4261530245 shutdown_connections 2024-04-15T15:17:09.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.965+0000 7f069937c640 1 --2- 172.21.15.72:0/4261530245 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f068006c980 0x7f068006ee40 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.965+0000 7f069937c640 1 --2- 172.21.15.72:0/4261530245 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f069415e9f0 0x7f06940cab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.965+0000 7f069937c640 1 --2- 172.21.15.72:0/4261530245 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f069415e610 0x7f06940ca600 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.965+0000 7f069937c640 1 --2- 172.21.15.72:0/4261530245 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f069406c400 0x7f069415f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.967 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.965+0000 7f069937c640 1 -- 172.21.15.72:0/4261530245 >> 172.21.15.72:0/4261530245 conn(0x7f06940d4750 msgr2=0x7f06940b41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:09.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.965+0000 7f069937c640 1 -- 172.21.15.72:0/4261530245 shutdown_connections 2024-04-15T15:17:09.968 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:09.965+0000 7f069937c640 1 -- 172.21.15.72:0/4261530245 wait complete. 2024-04-15T15:17:09.977 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:17:10.079 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.078+0000 7fc2e315f640 1 Processor -- start 2024-04-15T15:17:10.080 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.078+0000 7fc2e315f640 1 -- start start 2024-04-15T15:17:10.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.079+0000 7fc2e315f640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc2dc156d30 0x7fc2dc0d11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:10.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.079+0000 7fc2e315f640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fc2dc0cd160 0x7fc2dc0cf360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:10.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.079+0000 7fc2e315f640 1 -- --> v1:172.21.15.72:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2dc0d0100 con 0x7fc2dc0cd5d0 2024-04-15T15:17:10.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.079+0000 7fc2e315f640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fc2dc0d03e0 con 0x7fc2dc156d30 2024-04-15T15:17:10.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.079+0000 7fc2e315f640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fc2dc0d06c0 con 0x7fc2dc0cd160 2024-04-15T15:17:10.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.079+0000 7fc2e0efb640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fc2dc156d30 0x7fc2dc0d11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:10.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.079+0000 7fc2e16fc640 1 --1- >> v1:172.21.15.72:6789/0 conn(0x7fc2dc0cd5d0 0x7fc2dc15dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.72:6789/0 says I am v1:172.21.15.72:34104/0 (socket says 172.21.15.72:34104) 2024-04-15T15:17:10.082 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.079+0000 7fc2e16fc640 1 -- 172.21.15.72:0/356993455 learned_addr learned my addr 172.21.15.72:0/356993455 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:17:10.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.079+0000 7fc2d3fff640 1 --2- 172.21.15.72:0/356993455 >> v2:172.21.15.175:3300/0 conn(0x7fc2dc0cd160 0x7fc2dc0cf360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:10.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.079+0000 7fc2e0efb640 1 -- 172.21.15.72:0/356993455 >> v2:172.21.15.175:3300/0 conn(0x7fc2dc0cd160 msgr2=0x7fc2dc0cf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:10.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.079+0000 7fc2e0efb640 1 --2- 172.21.15.72:0/356993455 >> v2:172.21.15.175:3300/0 conn(0x7fc2dc0cd160 0x7fc2dc0cf360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.079+0000 7fc2e0efb640 1 -- 172.21.15.72:0/356993455 >> v1:172.21.15.72:6789/0 conn(0x7fc2dc0cd5d0 legacy=0x7fc2dc15dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:10.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.079+0000 7fc2e0efb640 1 -- 172.21.15.72:0/356993455 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2dc0d09a0 con 0x7fc2dc156d30 2024-04-15T15:17:10.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.079+0000 7fc2d3fff640 1 --2- 172.21.15.72:0/356993455 >> v2:172.21.15.175:3300/0 conn(0x7fc2dc0cd160 0x7fc2dc0cf360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:10.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.080+0000 7fc2e0efb640 1 --2- 172.21.15.72:0/356993455 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc2dc156d30 0x7fc2dc0d11a0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc0c93f0 tx=0x7fc2d802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2631e7f714211ce0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:10.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.080+0000 7fc2d37fe640 1 -- 172.21.15.72:0/356993455 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d8035020 con 0x7fc2dc156d30 2024-04-15T15:17:10.083 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.080+0000 7fc2d37fe640 1 -- 172.21.15.72:0/356993455 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2d8031830 con 0x7fc2dc156d30 2024-04-15T15:17:10.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.080+0000 7fc2d37fe640 1 -- 172.21.15.72:0/356993455 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d8031c00 con 0x7fc2dc156d30 2024-04-15T15:17:10.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.081+0000 7fc2e315f640 1 -- 172.21.15.72:0/356993455 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc2dc156d30 msgr2=0x7fc2dc0d11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:10.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.081+0000 7fc2e315f640 1 --2- 172.21.15.72:0/356993455 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc2dc156d30 0x7fc2dc0d11a0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc0c93f0 tx=0x7fc2d802b8d0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.081+0000 7fc2e315f640 1 -- 172.21.15.72:0/356993455 shutdown_connections 2024-04-15T15:17:10.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.081+0000 7fc2e315f640 1 --2- 172.21.15.72:0/356993455 >> v2:172.21.15.175:3300/0 conn(0x7fc2dc0cd160 0x7fc2dc0cf360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.081+0000 7fc2e315f640 1 --2- 172.21.15.72:0/356993455 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc2dc156d30 0x7fc2dc0d11a0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.081+0000 7fc2e315f640 1 -- 172.21.15.72:0/356993455 >> 172.21.15.72:0/356993455 conn(0x7fc2dc0b43c0 msgr2=0x7fc2dc0b47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:10.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.081+0000 7fc2e315f640 1 -- 172.21.15.72:0/356993455 shutdown_connections 2024-04-15T15:17:10.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.081+0000 7fc2e315f640 1 -- 172.21.15.72:0/356993455 wait complete. 2024-04-15T15:17:10.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.082+0000 7fc2e315f640 1 Processor -- start 2024-04-15T15:17:10.088 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.082+0000 7fc2e315f640 1 -- start start 2024-04-15T15:17:10.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.083+0000 7fc2e315f640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc2dc0cd160 0x7fc2dc0b7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:10.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.083+0000 7fc2e315f640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc2dc0cd5d0 0x7fc2dc1f1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:10.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.083+0000 7fc2e315f640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc2dc156d30 0x7fc2dc1f2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:10.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.083+0000 7fc2e315f640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fc2dc0cce90 con 0x7fc2dc156d30 2024-04-15T15:17:10.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.083+0000 7fc2e315f640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fc2dc0ccd10 con 0x7fc2dc0cd160 2024-04-15T15:17:10.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.083+0000 7fc2e315f640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fc2dc0cf8a0 con 0x7fc2dc0cd5d0 2024-04-15T15:17:10.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.083+0000 7fc2e16fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc2dc156d30 0x7fc2dc1f2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:10.089 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.083+0000 7fc2e16fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc2dc156d30 0x7fc2dc1f2420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37034/0 (socket says 172.21.15.72:37034) 2024-04-15T15:17:10.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.083+0000 7fc2e16fc640 1 -- 172.21.15.72:0/2261646377 learned_addr learned my addr 172.21.15.72:0/2261646377 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:10.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.083+0000 7fc2e0efb640 1 --2- 172.21.15.72:0/2261646377 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc2dc0cd160 0x7fc2dc0b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:10.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.083+0000 7fc2d3fff640 1 --2- 172.21.15.72:0/2261646377 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc2dc0cd5d0 0x7fc2dc1f1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:10.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.083+0000 7fc2e16fc640 1 -- 172.21.15.72:0/2261646377 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc2dc0cd5d0 msgr2=0x7fc2dc1f1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:10.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.083+0000 7fc2e16fc640 1 --2- 172.21.15.72:0/2261646377 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc2dc0cd5d0 0x7fc2dc1f1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.083+0000 7fc2e16fc640 1 -- 172.21.15.72:0/2261646377 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc2dc0cd160 msgr2=0x7fc2dc0b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:10.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.083+0000 7fc2e16fc640 1 --2- 172.21.15.72:0/2261646377 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc2dc0cd160 0x7fc2dc0b7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.090 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.083+0000 7fc2e16fc640 1 -- 172.21.15.72:0/2261646377 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2dc158ae0 con 0x7fc2dc156d30 2024-04-15T15:17:10.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.083+0000 7fc2e16fc640 1 --2- 172.21.15.72:0/2261646377 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc2dc156d30 0x7fc2dc1f2420 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fc2d400cbc0 tx=0x7fc2d400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:10.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.083+0000 7fc2d17fa640 1 -- 172.21.15.72:0/2261646377 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4017070 con 0x7fc2dc156d30 2024-04-15T15:17:10.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.083+0000 7fc2d17fa640 1 -- 172.21.15.72:0/2261646377 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2d400ace0 con 0x7fc2dc156d30 2024-04-15T15:17:10.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.083+0000 7fc2e315f640 1 -- 172.21.15.72:0/2261646377 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2dc070f70 con 0x7fc2dc156d30 2024-04-15T15:17:10.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.083+0000 7fc2d17fa640 1 -- 172.21.15.72:0/2261646377 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4012600 con 0x7fc2dc156d30 2024-04-15T15:17:10.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.084+0000 7fc2e315f640 1 -- 172.21.15.72:0/2261646377 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2dc0714f0 con 0x7fc2dc156d30 2024-04-15T15:17:10.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.084+0000 7fc2e315f640 1 -- 172.21.15.72:0/2261646377 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2dc0689e0 con 0x7fc2dc156d30 2024-04-15T15:17:10.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.085+0000 7fc2d17fa640 1 -- 172.21.15.72:0/2261646377 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fc2d401a070 con 0x7fc2dc156d30 2024-04-15T15:17:10.091 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.085+0000 7fc2d17fa640 1 --2- 172.21.15.72:0/2261646377 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc2c806c9b0 0x7fc2c806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:10.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.085+0000 7fc2d17fa640 1 -- 172.21.15.72:0/2261646377 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5421+0+0 (secure 0 0 0) 0x7fc2d408b660 con 0x7fc2dc156d30 2024-04-15T15:17:10.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.085+0000 7fc2e0efb640 1 --2- 172.21.15.72:0/2261646377 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc2c806c9b0 0x7fc2c806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:10.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.086+0000 7fc2e0efb640 1 --2- 172.21.15.72:0/2261646377 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc2c806c9b0 0x7fc2c806ee70 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc2d8036000 tx=0x7fc2d803c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:10.092 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.090+0000 7fc2d17fa640 1 -- 172.21.15.72:0/2261646377 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc2dc158ae0 con 0x7fc2dc156d30 2024-04-15T15:17:10.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.263+0000 7fc2e315f640 1 -- 172.21.15.72:0/2261646377 --> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc2dc0b1ab0 con 0x7fc2c806c9b0 2024-04-15T15:17:10.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.264+0000 7fc2d17fa640 1 -- 172.21.15.72:0/2261646377 <== mgr.4101 v2:172.21.15.72:6832/3503807805 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13472 (secure 0 0 0) 0x7fc2dc0b1ab0 con 0x7fc2c806c9b0 2024-04-15T15:17:10.266 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:17:10.267 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-04-15T15:17:10.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.266+0000 7fc2e315f640 1 -- 172.21.15.72:0/2261646377 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc2c806c9b0 msgr2=0x7fc2c806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:10.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.266+0000 7fc2e315f640 1 --2- 172.21.15.72:0/2261646377 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc2c806c9b0 0x7fc2c806ee70 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc2d8036000 tx=0x7fc2d803c040 comp rx=0 tx=0).stop 2024-04-15T15:17:10.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.266+0000 7fc2e315f640 1 -- 172.21.15.72:0/2261646377 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc2dc156d30 msgr2=0x7fc2dc1f2420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:10.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.266+0000 7fc2e315f640 1 --2- 172.21.15.72:0/2261646377 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc2dc156d30 0x7fc2dc1f2420 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fc2d400cbc0 tx=0x7fc2d400a430 comp rx=0 tx=0).stop 2024-04-15T15:17:10.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.266+0000 7fc2e315f640 1 -- 172.21.15.72:0/2261646377 shutdown_connections 2024-04-15T15:17:10.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.266+0000 7fc2e315f640 1 --2- 172.21.15.72:0/2261646377 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fc2c806c9b0 0x7fc2c806ee70 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.266+0000 7fc2e315f640 1 --2- 172.21.15.72:0/2261646377 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc2dc156d30 0x7fc2dc1f2420 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.266+0000 7fc2e315f640 1 --2- 172.21.15.72:0/2261646377 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fc2dc0cd5d0 0x7fc2dc1f1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.266+0000 7fc2e315f640 1 --2- 172.21.15.72:0/2261646377 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fc2dc0cd160 0x7fc2dc0b7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.266+0000 7fc2e315f640 1 -- 172.21.15.72:0/2261646377 >> 172.21.15.72:0/2261646377 conn(0x7fc2dc0b43c0 msgr2=0x7fc2dc0b3660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:10.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.266+0000 7fc2e315f640 1 -- 172.21.15.72:0/2261646377 shutdown_connections 2024-04-15T15:17:10.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:10.266+0000 7fc2e315f640 1 -- 172.21.15.72:0/2261646377 wait complete. 2024-04-15T15:17:10.279 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":112,"stamp":"2024-04-15T15:17:09.370868+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":221384,"kb_used_data":5224,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753336,"statfs":{"total":773094113280,"available":772867416064,"internally_reserved":0,"allocated":5349376,"data_stored":3007880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-133,"num_objects":-1,"num_object_clones":0,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-1,"num_write_kb":-1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.307984"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":214,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-15T15:17:01.072224+0000","last_change":"2024-04-15T15:14:40.494279+0000","last_active":"2024-04-15T15:17:01.072224+0000","last_peered":"2024-04-15T15:17:01.072224+0000","last_clean":"2024-04-15T15:17:01.072224+0000","last_became_active":"2024-04-15T15:14:40.493404+0000","last_became_peered":"2024-04-15T15:14:40.493404+0000","last_unstale":"2024-04-15T15:17:01.072224+0000","last_undegraded":"2024-04-15T15:17:01.072224+0000","last_fullsized":"2024-04-15T15:17:01.072224+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:39.418508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:39.418508+0000","last_clean_scrub_stamp":"2024-04-15T15:14:39.418508+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:01:51.789246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28108,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343732,"statfs":{"total":96636764160,"available":96607981568,"internally_reserved":0,"allocated":1114112,"data_stored":818761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27528,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344312,"statfs":{"total":96636764160,"available":96608575488,"internally_reserved":0,"allocated":520192,"data_stored":228393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27528,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344312,"statfs":{"total":96636764160,"available":96608575488,"internally_reserved":0,"allocated":520192,"data_stored":228393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27528,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344312,"statfs":{"total":96636764160,"available":96608575488,"internally_reserved":0,"allocated":520192,"data_stored":228393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27528,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344312,"statfs":{"total":96636764160,"available":96608575488,"internally_reserved":0,"allocated":520192,"data_stored":228393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27528,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344312,"statfs":{"total":96636764160,"available":96608575488,"internally_reserved":0,"allocated":520192,"data_stored":228393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27528,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344312,"statfs":{"total":96636764160,"available":96608575488,"internally_reserved":0,"allocated":520192,"data_stored":228393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28108,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343732,"statfs":{"total":96636764160,"available":96607981568,"internally_reserved":0,"allocated":1114112,"data_stored":818761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:17:10.281 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-15T15:14:39.418508+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=15, tm_hour=15, tm_min=14, tm_sec=39, tm_wday=0, tm_yday=106, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=15, tm_hour=15, tm_min=17, tm_sec=5, tm_wday=0, tm_yday=106, tm_isdst=0) 2024-04-15T15:17:10.281 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-15T15:17:21.129 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:17:21.133 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:17:30.282 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:17:30.397 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.396+0000 7fee365bd640 1 Processor -- start 2024-04-15T15:17:30.398 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.396+0000 7fee365bd640 1 -- start start 2024-04-15T15:17:30.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.397+0000 7fee365bd640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fee300af400 0x7fee300ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:30.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.397+0000 7fee365bd640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fee300af860 0x7fee300cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:30.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.397+0000 7fee365bd640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee3006f8f0 con 0x7fee3015eba0 2024-04-15T15:17:30.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.397+0000 7fee365bd640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fee3015ef80 con 0x7fee300af860 2024-04-15T15:17:30.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.397+0000 7fee365bd640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fee300ce2b0 con 0x7fee300af400 2024-04-15T15:17:30.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.397+0000 7fee34b5a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fee300af860 0x7fee300cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:30.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.397+0000 7fee34b5a640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fee300af860 0x7fee300cd0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55938/0 (socket says 172.21.15.72:55938) 2024-04-15T15:17:30.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.397+0000 7fee34b5a640 1 -- 172.21.15.72:0/1520119594 learned_addr learned my addr 172.21.15.72:0/1520119594 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:30.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.397+0000 7fee34b5a640 1 -- 172.21.15.72:0/1520119594 >> v2:172.21.15.175:3300/0 conn(0x7fee300af400 msgr2=0x7fee300ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:30.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.397+0000 7fee34b5a640 1 --2- 172.21.15.72:0/1520119594 >> v2:172.21.15.175:3300/0 conn(0x7fee300af400 0x7fee300ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:30.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.397+0000 7fee34b5a640 1 -- 172.21.15.72:0/1520119594 >> v1:172.21.15.175:6789/0 conn(0x7fee3015eba0 legacy=0x7fee3006f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:30.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.397+0000 7fee34b5a640 1 -- 172.21.15.72:0/1520119594 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee300ce590 con 0x7fee300af860 2024-04-15T15:17:30.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.398+0000 7fee34b5a640 1 --2- 172.21.15.72:0/1520119594 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee300af860 0x7fee300cd0d0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fee30075490 tx=0x7fee2802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d613d058855aef29 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:30.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.398+0000 7fee2effd640 1 -- 172.21.15.72:0/1520119594 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee2802c9f0 con 0x7fee300af860 2024-04-15T15:17:30.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.398+0000 7fee2effd640 1 -- 172.21.15.72:0/1520119594 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee2802cb90 con 0x7fee300af860 2024-04-15T15:17:30.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.398+0000 7fee2effd640 1 -- 172.21.15.72:0/1520119594 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee280343f0 con 0x7fee300af860 2024-04-15T15:17:30.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.398+0000 7fee365bd640 1 -- 172.21.15.72:0/1520119594 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee300af860 msgr2=0x7fee300cd0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:30.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.398+0000 7fee365bd640 1 --2- 172.21.15.72:0/1520119594 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee300af860 0x7fee300cd0d0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fee30075490 tx=0x7fee2802be50 comp rx=0 tx=0).stop 2024-04-15T15:17:30.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.399+0000 7fee365bd640 1 -- 172.21.15.72:0/1520119594 shutdown_connections 2024-04-15T15:17:30.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.399+0000 7fee365bd640 1 --2- 172.21.15.72:0/1520119594 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee300af860 0x7fee300cd0d0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:30.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.399+0000 7fee365bd640 1 --2- 172.21.15.72:0/1520119594 >> v2:172.21.15.175:3300/0 conn(0x7fee300af400 0x7fee300ccb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:30.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.399+0000 7fee365bd640 1 -- 172.21.15.72:0/1520119594 >> 172.21.15.72:0/1520119594 conn(0x7fee300b1dd0 msgr2=0x7fee300b21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:30.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.399+0000 7fee365bd640 1 -- 172.21.15.72:0/1520119594 shutdown_connections 2024-04-15T15:17:30.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.399+0000 7fee365bd640 1 -- 172.21.15.72:0/1520119594 wait complete. 2024-04-15T15:17:30.405 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.400+0000 7fee365bd640 1 Processor -- start 2024-04-15T15:17:30.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.400+0000 7fee365bd640 1 -- start start 2024-04-15T15:17:30.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.401+0000 7fee365bd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee300af400 0x7fee301f44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:30.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.401+0000 7fee365bd640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fee300af860 0x7fee30151f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:30.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.401+0000 7fee365bd640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fee3015eba0 0x7fee300c9450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:30.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.401+0000 7fee365bd640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fee300b2b80 con 0x7fee300af400 2024-04-15T15:17:30.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.401+0000 7fee365bd640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fee3006ecc0 con 0x7fee300af860 2024-04-15T15:17:30.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.401+0000 7fee365bd640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fee300ceb50 con 0x7fee3015eba0 2024-04-15T15:17:30.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.401+0000 7fee2ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee300af400 0x7fee301f44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:30.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.401+0000 7fee2ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee300af400 0x7fee301f44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55940/0 (socket says 172.21.15.72:55940) 2024-04-15T15:17:30.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.401+0000 7fee2ffff640 1 -- 172.21.15.72:0/1715635863 learned_addr learned my addr 172.21.15.72:0/1715635863 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:30.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.401+0000 7fee34b5a640 1 --2- 172.21.15.72:0/1715635863 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fee3015eba0 0x7fee300c9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:30.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.401+0000 7fee2f7fe640 1 --2- 172.21.15.72:0/1715635863 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fee300af860 0x7fee30151f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:30.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.401+0000 7fee2ffff640 1 -- 172.21.15.72:0/1715635863 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fee3015eba0 msgr2=0x7fee300c9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:30.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.401+0000 7fee2ffff640 1 --2- 172.21.15.72:0/1715635863 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fee3015eba0 0x7fee300c9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:30.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.401+0000 7fee2ffff640 1 -- 172.21.15.72:0/1715635863 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fee300af860 msgr2=0x7fee30151f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:30.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.401+0000 7fee2ffff640 1 --2- 172.21.15.72:0/1715635863 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fee300af860 0x7fee30151f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:30.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.401+0000 7fee2ffff640 1 -- 172.21.15.72:0/1715635863 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee301f7bd0 con 0x7fee300af400 2024-04-15T15:17:30.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.401+0000 7fee2ffff640 1 --2- 172.21.15.72:0/1715635863 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee300af400 0x7fee301f44f0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fee2000c410 tx=0x7fee2000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:30.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.401+0000 7fee2cff9640 1 -- 172.21.15.72:0/1715635863 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee20017070 con 0x7fee300af400 2024-04-15T15:17:30.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.401+0000 7fee2cff9640 1 -- 172.21.15.72:0/1715635863 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee200035f0 con 0x7fee300af400 2024-04-15T15:17:30.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.401+0000 7fee365bd640 1 -- 172.21.15.72:0/1715635863 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee30152480 con 0x7fee300af400 2024-04-15T15:17:30.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.401+0000 7fee2cff9640 1 -- 172.21.15.72:0/1715635863 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee20003a20 con 0x7fee300af400 2024-04-15T15:17:30.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.402+0000 7fee365bd640 1 -- 172.21.15.72:0/1715635863 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee301f4e30 con 0x7fee300af400 2024-04-15T15:17:30.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.403+0000 7fee365bd640 1 -- 172.21.15.72:0/1715635863 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee300cf0f0 con 0x7fee300af400 2024-04-15T15:17:30.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.403+0000 7fee2cff9640 1 -- 172.21.15.72:0/1715635863 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fee2001a070 con 0x7fee300af400 2024-04-15T15:17:30.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.403+0000 7fee2cff9640 1 --2- 172.21.15.72:0/1715635863 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fee1c06c980 0x7fee1c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:30.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.404+0000 7fee2f7fe640 1 --2- 172.21.15.72:0/1715635863 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fee1c06c980 0x7fee1c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:30.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.404+0000 7fee2cff9640 1 -- 172.21.15.72:0/1715635863 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5421+0+0 (secure 0 0 0) 0x7fee2008aec0 con 0x7fee300af400 2024-04-15T15:17:30.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.404+0000 7fee2f7fe640 1 --2- 172.21.15.72:0/1715635863 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fee1c06c980 0x7fee1c06ee40 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fee30155500 tx=0x7fee24008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:30.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.408+0000 7fee2cff9640 1 -- 172.21.15.72:0/1715635863 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fee300cf0f0 con 0x7fee300af400 2024-04-15T15:17:30.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.578+0000 7fee365bd640 1 -- 172.21.15.72:0/1715635863 --> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fee301f7da0 con 0x7fee1c06c980 2024-04-15T15:17:30.581 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.579+0000 7fee2cff9640 1 -- 172.21.15.72:0/1715635863 <== mgr.4101 v2:172.21.15.72:6832/3503807805 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13359 (secure 0 0 0) 0x7fee301f7da0 con 0x7fee1c06c980 2024-04-15T15:17:30.581 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:17:30.583 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-04-15T15:17:30.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.581+0000 7fee365bd640 1 -- 172.21.15.72:0/1715635863 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fee1c06c980 msgr2=0x7fee1c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:30.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.581+0000 7fee365bd640 1 --2- 172.21.15.72:0/1715635863 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fee1c06c980 0x7fee1c06ee40 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fee30155500 tx=0x7fee24008040 comp rx=0 tx=0).stop 2024-04-15T15:17:30.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.581+0000 7fee365bd640 1 -- 172.21.15.72:0/1715635863 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee300af400 msgr2=0x7fee301f44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:30.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.581+0000 7fee365bd640 1 --2- 172.21.15.72:0/1715635863 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee300af400 0x7fee301f44f0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fee2000c410 tx=0x7fee2000c8d0 comp rx=0 tx=0).stop 2024-04-15T15:17:30.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.581+0000 7fee365bd640 1 -- 172.21.15.72:0/1715635863 shutdown_connections 2024-04-15T15:17:30.583 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.581+0000 7fee365bd640 1 --2- 172.21.15.72:0/1715635863 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fee1c06c980 0x7fee1c06ee40 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:30.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.581+0000 7fee365bd640 1 --2- 172.21.15.72:0/1715635863 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fee3015eba0 0x7fee300c9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:30.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.581+0000 7fee365bd640 1 --2- 172.21.15.72:0/1715635863 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fee300af860 0x7fee30151f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:30.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.581+0000 7fee365bd640 1 --2- 172.21.15.72:0/1715635863 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fee300af400 0x7fee301f44f0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:30.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.581+0000 7fee365bd640 1 -- 172.21.15.72:0/1715635863 >> 172.21.15.72:0/1715635863 conn(0x7fee300b1dd0 msgr2=0x7fee300d01a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:30.584 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.581+0000 7fee365bd640 1 -- 172.21.15.72:0/1715635863 shutdown_connections 2024-04-15T15:17:30.585 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.581+0000 7fee365bd640 1 -- 172.21.15.72:0/1715635863 wait complete. 2024-04-15T15:17:30.593 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":122,"stamp":"2024-04-15T15:17:29.375000+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":221376,"kb_used_data":5224,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753344,"statfs":{"total":773094113280,"available":772867424256,"internally_reserved":0,"allocated":5349376,"data_stored":3007880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":221236785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002581"},"pg_stats":[{"pgid":"1.0","version":"44'119","reported_seq":225,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-15T15:17:10.282214+0000","last_change":"2024-04-15T15:17:10.282252+0000","last_active":"2024-04-15T15:17:10.282214+0000","last_peered":"2024-04-15T15:17:10.282214+0000","last_clean":"2024-04-15T15:17:10.282214+0000","last_became_active":"2024-04-15T15:14:40.493404+0000","last_became_peered":"2024-04-15T15:14:40.493404+0000","last_unstale":"2024-04-15T15:17:10.282214+0000","last_undegraded":"2024-04-15T15:17:10.282214+0000","last_fullsized":"2024-04-15T15:17:10.282214+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"44'119","last_scrub_stamp":"2024-04-15T15:17:10.282132+0000","last_deep_scrub":"44'119","last_deep_scrub_stamp":"2024-04-15T15:17:10.282132+0000","last_clean_scrub_stamp":"2024-04-15T15:17:10.282132+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:46:24.966723+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28104,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343736,"statfs":{"total":96636764160,"available":96607985664,"internally_reserved":0,"allocated":1114112,"data_stored":818761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574885,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27528,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344312,"statfs":{"total":96636764160,"available":96608575488,"internally_reserved":0,"allocated":520192,"data_stored":228393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27528,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344312,"statfs":{"total":96636764160,"available":96608575488,"internally_reserved":0,"allocated":520192,"data_stored":228393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27528,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344312,"statfs":{"total":96636764160,"available":96608575488,"internally_reserved":0,"allocated":520192,"data_stored":228393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574885,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27528,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344312,"statfs":{"total":96636764160,"available":96608575488,"internally_reserved":0,"allocated":520192,"data_stored":228393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574885,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27528,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344312,"statfs":{"total":96636764160,"available":96608575488,"internally_reserved":0,"allocated":520192,"data_stored":228393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574885,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27528,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344312,"statfs":{"total":96636764160,"available":96608575488,"internally_reserved":0,"allocated":520192,"data_stored":228393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28104,"kb_used_data":1088,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343736,"statfs":{"total":96636764160,"available":96607985664,"internally_reserved":0,"allocated":1114112,"data_stored":818761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:17:30.594 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-15T15:17:30.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.695+0000 7fa2d359e640 1 Processor -- start 2024-04-15T15:17:30.697 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.695+0000 7fa2d359e640 1 -- start start 2024-04-15T15:17:30.698 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.696+0000 7fa2d359e640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa2d40cf2e0 0x7fa2d415f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:30.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.696+0000 7fa2d359e640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7fa2d4162b50 0x7fa2d406df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:30.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.696+0000 7fa2d359e640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2d406e6b0 con 0x7fa2d4147f60 2024-04-15T15:17:30.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.696+0000 7fa2d359e640 1 -- --> v2:172.21.15.72:3300/0 -- mon_getmap magic: 0 -- 0x7fa2d406e480 con 0x7fa2d40cf2e0 2024-04-15T15:17:30.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.696+0000 7fa2d359e640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7fa2d4162100 con 0x7fa2d4162b50 2024-04-15T15:17:30.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.696+0000 7fa2d1d9b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa2d40cf2e0 0x7fa2d415f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:30.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.697+0000 7fa2d1d9b640 1 --2- >> v2:172.21.15.72:3300/0 conn(0x7fa2d40cf2e0 0x7fa2d415f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55954/0 (socket says 172.21.15.72:55954) 2024-04-15T15:17:30.699 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.697+0000 7fa2d1d9b640 1 -- 172.21.15.72:0/664881933 learned_addr learned my addr 172.21.15.72:0/664881933 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:30.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.697+0000 7fa2d2d9d640 1 --2- 172.21.15.72:0/664881933 >> v2:172.21.15.175:3300/0 conn(0x7fa2d4162b50 0x7fa2d406df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:30.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.697+0000 7fa2d1d9b640 1 -- 172.21.15.72:0/664881933 >> v2:172.21.15.175:3300/0 conn(0x7fa2d4162b50 msgr2=0x7fa2d406df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:30.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.697+0000 7fa2d1d9b640 1 --2- 172.21.15.72:0/664881933 >> v2:172.21.15.175:3300/0 conn(0x7fa2d4162b50 0x7fa2d406df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:30.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.697+0000 7fa2d1d9b640 1 -- 172.21.15.72:0/664881933 >> v1:172.21.15.175:6789/0 conn(0x7fa2d4147f60 legacy=0x7fa2d40cebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:30.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.697+0000 7fa2d1d9b640 1 -- 172.21.15.72:0/664881933 --> v2:172.21.15.72:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2d406e990 con 0x7fa2d40cf2e0 2024-04-15T15:17:30.700 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.697+0000 7fa2d2d9d640 1 --2- 172.21.15.72:0/664881933 >> v2:172.21.15.175:3300/0 conn(0x7fa2d4162b50 0x7fa2d406df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:30.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.698+0000 7fa2d1d9b640 1 --2- 172.21.15.72:0/664881933 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2d40cf2e0 0x7fa2d415f620 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fa2d40b3e70 tx=0x7fa2c802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba9f5c167e32ee9f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:30.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.698+0000 7fa2d159a640 1 -- 172.21.15.72:0/664881933 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c802e070 con 0x7fa2d40cf2e0 2024-04-15T15:17:30.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.698+0000 7fa2d159a640 1 -- 172.21.15.72:0/664881933 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2c8037070 con 0x7fa2d40cf2e0 2024-04-15T15:17:30.705 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.698+0000 7fa2d159a640 1 -- 172.21.15.72:0/664881933 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c8031a00 con 0x7fa2d40cf2e0 2024-04-15T15:17:30.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.698+0000 7fa2d359e640 1 -- 172.21.15.72:0/664881933 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2d40cf2e0 msgr2=0x7fa2d415f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:30.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.698+0000 7fa2d359e640 1 --2- 172.21.15.72:0/664881933 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2d40cf2e0 0x7fa2d415f620 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fa2d40b3e70 tx=0x7fa2c802b880 comp rx=0 tx=0).stop 2024-04-15T15:17:30.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.698+0000 7fa2d359e640 1 -- 172.21.15.72:0/664881933 shutdown_connections 2024-04-15T15:17:30.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.698+0000 7fa2d359e640 1 --2- 172.21.15.72:0/664881933 >> v2:172.21.15.175:3300/0 conn(0x7fa2d4162b50 0x7fa2d406df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:30.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.698+0000 7fa2d359e640 1 --2- 172.21.15.72:0/664881933 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2d40cf2e0 0x7fa2d415f620 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:30.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.698+0000 7fa2d359e640 1 -- 172.21.15.72:0/664881933 >> 172.21.15.72:0/664881933 conn(0x7fa2d40b3ec0 msgr2=0x7fa2d40b42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:30.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.698+0000 7fa2d359e640 1 -- 172.21.15.72:0/664881933 shutdown_connections 2024-04-15T15:17:30.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.698+0000 7fa2d359e640 1 -- 172.21.15.72:0/664881933 wait complete. 2024-04-15T15:17:30.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.700+0000 7fa2d359e640 1 Processor -- start 2024-04-15T15:17:30.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.700+0000 7fa2d359e640 1 -- start start 2024-04-15T15:17:30.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.700+0000 7fa2d359e640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa2d40cf2e0 0x7fa2d41f46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:30.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.700+0000 7fa2d359e640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2d4147f60 0x7fa2d415d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:30.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.700+0000 7fa2d359e640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa2d4162b50 0x7fa2d415fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:30.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.700+0000 7fa2d359e640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d4162f60 con 0x7fa2d4147f60 2024-04-15T15:17:30.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.700+0000 7fa2d359e640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d41626c0 con 0x7fa2d4162b50 2024-04-15T15:17:30.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.700+0000 7fa2d359e640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7fa2d40b2a70 con 0x7fa2d40cf2e0 2024-04-15T15:17:30.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.701+0000 7fa2d1d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2d4147f60 0x7fa2d415d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:30.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.701+0000 7fa2d1d9b640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2d4147f60 0x7fa2d415d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55970/0 (socket says 172.21.15.72:55970) 2024-04-15T15:17:30.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.701+0000 7fa2d1d9b640 1 -- 172.21.15.72:0/3591643991 learned_addr learned my addr 172.21.15.72:0/3591643991 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:30.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.701+0000 7fa2d1d9b640 1 -- 172.21.15.72:0/3591643991 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa2d40cf2e0 msgr2=0x7fa2d41f46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:30.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.701+0000 7fa2d1d9b640 1 --2- 172.21.15.72:0/3591643991 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa2d40cf2e0 0x7fa2d41f46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:30.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.701+0000 7fa2d2d9d640 1 --2- 172.21.15.72:0/3591643991 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa2d4162b50 0x7fa2d415fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:30.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.701+0000 7fa2d1d9b640 1 -- 172.21.15.72:0/3591643991 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa2d4162b50 msgr2=0x7fa2d415fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:30.708 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.701+0000 7fa2d1d9b640 1 --2- 172.21.15.72:0/3591643991 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa2d4162b50 0x7fa2d415fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:30.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.701+0000 7fa2d1d9b640 1 -- 172.21.15.72:0/3591643991 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2d4160570 con 0x7fa2d4147f60 2024-04-15T15:17:30.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.701+0000 7fa2d1d9b640 1 --2- 172.21.15.72:0/3591643991 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2d4147f60 0x7fa2d415d930 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8000c00 tx=0x7fa2c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:30.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.701+0000 7fa2c2ffd640 1 -- 172.21.15.72:0/3591643991 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c8032880 con 0x7fa2d4147f60 2024-04-15T15:17:30.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.701+0000 7fa2c2ffd640 1 -- 172.21.15.72:0/3591643991 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2c8032a40 con 0x7fa2d4147f60 2024-04-15T15:17:30.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.701+0000 7fa2d359e640 1 -- 172.21.15.72:0/3591643991 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2d4160850 con 0x7fa2d4147f60 2024-04-15T15:17:30.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.701+0000 7fa2c2ffd640 1 -- 172.21.15.72:0/3591643991 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c8032e30 con 0x7fa2d4147f60 2024-04-15T15:17:30.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.701+0000 7fa2d359e640 1 -- 172.21.15.72:0/3591643991 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2d41610f0 con 0x7fa2d4147f60 2024-04-15T15:17:30.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.702+0000 7fa2d359e640 1 -- 172.21.15.72:0/3591643991 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2d4160a20 con 0x7fa2d4147f60 2024-04-15T15:17:30.709 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.703+0000 7fa2c2ffd640 1 -- 172.21.15.72:0/3591643991 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fa2c804c020 con 0x7fa2d4147f60 2024-04-15T15:17:30.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.703+0000 7fa2c2ffd640 1 --2- 172.21.15.72:0/3591643991 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa2b406c980 0x7fa2b406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:30.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.703+0000 7fa2c2ffd640 1 -- 172.21.15.72:0/3591643991 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5421+0+0 (secure 0 0 0) 0x7fa2c80ad8c0 con 0x7fa2d4147f60 2024-04-15T15:17:30.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.703+0000 7fa2d259c640 1 --2- 172.21.15.72:0/3591643991 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa2b406c980 0x7fa2b406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:30.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.704+0000 7fa2d259c640 1 --2- 172.21.15.72:0/3591643991 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa2b406c980 0x7fa2b406ee40 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fa2c40012e0 tx=0x7fa2c4007d50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:30.710 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.707+0000 7fa2c2ffd640 1 -- 172.21.15.72:0/3591643991 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa2c8040070 con 0x7fa2d4147f60 2024-04-15T15:17:30.884 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.883+0000 7fa2d359e640 1 -- 172.21.15.72:0/3591643991 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7fa2d40cdfb0 con 0x7fa2d4147f60 2024-04-15T15:17:30.887 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.886+0000 7fa2c2ffd640 1 -- 172.21.15.72:0/3591643991 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v16)=0 v16) ==== 123+0+0 (secure 0 0 0) 0x7fa2d40cdfb0 con 0x7fa2d4147f60 2024-04-15T15:17:30.888 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.887+0000 7fa2c2ffd640 1 -- 172.21.15.72:0/3591643991 <== mon.0 v2:172.21.15.72:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fa2c8037020 con 0x7fa2d4147f60 2024-04-15T15:17:30.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.888+0000 7fa2d359e640 1 -- 172.21.15.72:0/3591643991 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa2b406c980 msgr2=0x7fa2b406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:30.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.888+0000 7fa2d359e640 1 --2- 172.21.15.72:0/3591643991 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa2b406c980 0x7fa2b406ee40 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fa2c40012e0 tx=0x7fa2c4007d50 comp rx=0 tx=0).stop 2024-04-15T15:17:30.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.888+0000 7fa2d359e640 1 -- 172.21.15.72:0/3591643991 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2d4147f60 msgr2=0x7fa2d415d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:30.890 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.888+0000 7fa2d359e640 1 --2- 172.21.15.72:0/3591643991 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2d4147f60 0x7fa2d415d930 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8000c00 tx=0x7fa2c80046c0 comp rx=0 tx=0).stop 2024-04-15T15:17:30.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.888+0000 7fa2d359e640 1 -- 172.21.15.72:0/3591643991 shutdown_connections 2024-04-15T15:17:30.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.888+0000 7fa2d359e640 1 --2- 172.21.15.72:0/3591643991 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7fa2b406c980 0x7fa2b406ee40 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:30.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.888+0000 7fa2d359e640 1 --2- 172.21.15.72:0/3591643991 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7fa2d4162b50 0x7fa2d415fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:30.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.888+0000 7fa2d359e640 1 --2- 172.21.15.72:0/3591643991 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa2d4147f60 0x7fa2d415d930 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:30.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.888+0000 7fa2d359e640 1 --2- 172.21.15.72:0/3591643991 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7fa2d40cf2e0 0x7fa2d41f46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:30.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.888+0000 7fa2d359e640 1 -- 172.21.15.72:0/3591643991 >> 172.21.15.72:0/3591643991 conn(0x7fa2d40b3ec0 msgr2=0x7fa2d406f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:30.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.888+0000 7fa2d359e640 1 -- 172.21.15.72:0/3591643991 shutdown_connections 2024-04-15T15:17:30.891 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:30.889+0000 7fa2d359e640 1 -- 172.21.15.72:0/3591643991 wait complete. 2024-04-15T15:17:30.901 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-15T15:17:30.902 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:17:31.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.009+0000 7f7f81edf640 1 Processor -- start 2024-04-15T15:17:31.011 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.009+0000 7f7f81edf640 1 -- start start 2024-04-15T15:17:31.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.010+0000 7f7f81edf640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7f7c0af700 0x7f7f7c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:31.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.010+0000 7f7f81edf640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f7f7c06b960 0x7f7f7c06bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:31.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.010+0000 7f7f81edf640 1 -- --> v1:172.21.15.175:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f7c0cffd0 con 0x7f7f7c15c7c0 2024-04-15T15:17:31.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.010+0000 7f7f81edf640 1 -- --> v2:172.21.15.175:3300/0 -- mon_getmap magic: 0 -- 0x7f7f7c15cba0 con 0x7f7f7c06b960 2024-04-15T15:17:31.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.010+0000 7f7f81edf640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f7f7c06dc10 con 0x7f7f7c0af700 2024-04-15T15:17:31.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.010+0000 7f7f7bfff640 1 --2- >> v2:172.21.15.175:3300/0 conn(0x7f7f7c06b960 0x7f7f7c06bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:31.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.010+0000 7f7f7affd640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7f7c0af700 0x7f7f7c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:31.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.010+0000 7f7f7b7fe640 1 --1- >> v1:172.21.15.175:6789/0 conn(0x7f7f7c15c7c0 0x7f7f7c06e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.175:6789/0 says I am v1:172.21.15.72:44542/0 (socket says 172.21.15.72:44542) 2024-04-15T15:17:31.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.010+0000 7f7f7b7fe640 1 -- 172.21.15.72:0/3542933567 learned_addr learned my addr 172.21.15.72:0/3542933567 (peer_addr_for_me v1:172.21.15.72:0/0) 2024-04-15T15:17:31.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.011+0000 7f7f7bfff640 1 -- 172.21.15.72:0/3542933567 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7f7c0af700 msgr2=0x7f7f7c06eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:31.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.011+0000 7f7f7bfff640 1 --2- 172.21.15.72:0/3542933567 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7f7c0af700 0x7f7f7c06eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:31.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.011+0000 7f7f7bfff640 1 -- 172.21.15.72:0/3542933567 >> v1:172.21.15.175:6789/0 conn(0x7f7f7c15c7c0 legacy=0x7f7f7c06e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:31.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.011+0000 7f7f7bfff640 1 -- 172.21.15.72:0/3542933567 --> v2:172.21.15.175:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f7c06de60 con 0x7f7f7c06b960 2024-04-15T15:17:31.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.011+0000 7f7f7affd640 1 --2- 172.21.15.72:0/3542933567 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7f7c0af700 0x7f7f7c06eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:17:31.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.012+0000 7f7f7bfff640 1 --2- 172.21.15.72:0/3542933567 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7f7c06b960 0x7f7f7c06bdf0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7f740097a0 tx=0x7f7f7402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6168e14f539e39ff server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:31.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.012+0000 7f7f7a7fc640 1 -- 172.21.15.72:0/3542933567 <== mon.1 v2:172.21.15.175:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f74035020 con 0x7f7f7c06b960 2024-04-15T15:17:31.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.012+0000 7f7f7a7fc640 1 -- 172.21.15.72:0/3542933567 <== mon.1 v2:172.21.15.175:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f7f74031830 con 0x7f7f7c06b960 2024-04-15T15:17:31.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.012+0000 7f7f7a7fc640 1 -- 172.21.15.72:0/3542933567 <== mon.1 v2:172.21.15.175:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f74031c20 con 0x7f7f7c06b960 2024-04-15T15:17:31.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.012+0000 7f7f81edf640 1 -- 172.21.15.72:0/3542933567 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7f7c06b960 msgr2=0x7f7f7c06bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:31.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.012+0000 7f7f81edf640 1 --2- 172.21.15.72:0/3542933567 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7f7c06b960 0x7f7f7c06bdf0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7f740097a0 tx=0x7f7f7402b8d0 comp rx=0 tx=0).stop 2024-04-15T15:17:31.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.012+0000 7f7f81edf640 1 -- 172.21.15.72:0/3542933567 shutdown_connections 2024-04-15T15:17:31.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.012+0000 7f7f81edf640 1 --2- 172.21.15.72:0/3542933567 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7f7c06b960 0x7f7f7c06bdf0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:31.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.012+0000 7f7f81edf640 1 --2- 172.21.15.72:0/3542933567 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7f7c0af700 0x7f7f7c06eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:31.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.012+0000 7f7f81edf640 1 -- 172.21.15.72:0/3542933567 >> 172.21.15.72:0/3542933567 conn(0x7f7f7c0b1dd0 msgr2=0x7f7f7c0b21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:31.021 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.012+0000 7f7f81edf640 1 -- 172.21.15.72:0/3542933567 shutdown_connections 2024-04-15T15:17:31.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.012+0000 7f7f81edf640 1 -- 172.21.15.72:0/3542933567 wait complete. 2024-04-15T15:17:31.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.014+0000 7f7f81edf640 1 Processor -- start 2024-04-15T15:17:31.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.014+0000 7f7f81edf640 1 -- start start 2024-04-15T15:17:31.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.014+0000 7f7f81edf640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7f7c06b960 0x7f7f7c150c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:31.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.014+0000 7f7f81edf640 1 --2- >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7f7c0af700 0x7f7f7c0b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:31.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.014+0000 7f7f81edf640 1 --2- >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7f7c15c7c0 0x7f7f7c0b09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:31.022 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.014+0000 7f7f81edf640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 -- 0x7f7f7c0cfab0 con 0x7f7f7c06b960 2024-04-15T15:17:31.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.014+0000 7f7f81edf640 1 -- --> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] -- mon_getmap magic: 0 -- 0x7f7f7c0cf7d0 con 0x7f7f7c15c7c0 2024-04-15T15:17:31.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.014+0000 7f7f81edf640 1 -- --> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] -- mon_getmap magic: 0 -- 0x7f7f7c0cdc60 con 0x7f7f7c0af700 2024-04-15T15:17:31.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.015+0000 7f7f7b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7f7c06b960 0x7f7f7c150c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:31.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.015+0000 7f7f7b7fe640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7f7c06b960 0x7f7f7c150c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55972/0 (socket says 172.21.15.72:55972) 2024-04-15T15:17:31.023 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.015+0000 7f7f7b7fe640 1 -- 172.21.15.72:0/2454785938 learned_addr learned my addr 172.21.15.72:0/2454785938 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-04-15T15:17:31.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.015+0000 7f7f7bfff640 1 --2- 172.21.15.72:0/2454785938 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7f7c15c7c0 0x7f7f7c0b09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:31.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.015+0000 7f7f7affd640 1 --2- 172.21.15.72:0/2454785938 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7f7c0af700 0x7f7f7c0b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:31.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.015+0000 7f7f7b7fe640 1 -- 172.21.15.72:0/2454785938 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7f7c0af700 msgr2=0x7f7f7c0b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:31.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.015+0000 7f7f7b7fe640 1 --2- 172.21.15.72:0/2454785938 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7f7c0af700 0x7f7f7c0b04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:31.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.015+0000 7f7f7b7fe640 1 -- 172.21.15.72:0/2454785938 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7f7c15c7c0 msgr2=0x7f7f7c0b09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:31.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.015+0000 7f7f7b7fe640 1 --2- 172.21.15.72:0/2454785938 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7f7c15c7c0 0x7f7f7c0b09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:31.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.015+0000 7f7f7b7fe640 1 -- 172.21.15.72:0/2454785938 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f7c0c91f0 con 0x7f7f7c06b960 2024-04-15T15:17:31.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.015+0000 7f7f7affd640 1 --2- 172.21.15.72:0/2454785938 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7f7c0af700 0x7f7f7c0b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:31.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.015+0000 7f7f7b7fe640 1 --2- 172.21.15.72:0/2454785938 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7f7c06b960 0x7f7f7c150c70 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c00caa0 tx=0x7f7f6c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:31.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.015+0000 7f7f63fff640 1 -- 172.21.15.72:0/2454785938 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f6c017070 con 0x7f7f7c06b960 2024-04-15T15:17:31.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.015+0000 7f7f63fff640 1 -- 172.21.15.72:0/2454785938 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f7f6c00ac00 con 0x7f7f7c06b960 2024-04-15T15:17:31.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.015+0000 7f7f81edf640 1 -- 172.21.15.72:0/2454785938 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f7c0c5e50 con 0x7f7f7c06b960 2024-04-15T15:17:31.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.015+0000 7f7f63fff640 1 -- 172.21.15.72:0/2454785938 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f6c012400 con 0x7f7f7c06b960 2024-04-15T15:17:31.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.015+0000 7f7f81edf640 1 -- 172.21.15.72:0/2454785938 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f7c0c6410 con 0x7f7f7c06b960 2024-04-15T15:17:31.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.016+0000 7f7f81edf640 1 -- 172.21.15.72:0/2454785938 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f7c0c6610 con 0x7f7f7c06b960 2024-04-15T15:17:31.025 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.017+0000 7f7f63fff640 1 -- 172.21.15.72:0/2454785938 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f7f6c01a070 con 0x7f7f7c06b960 2024-04-15T15:17:31.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.017+0000 7f7f63fff640 1 --2- 172.21.15.72:0/2454785938 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7f6806c9b0 0x7f7f6806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:31.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.017+0000 7f7f63fff640 1 -- 172.21.15.72:0/2454785938 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5421+0+0 (secure 0 0 0) 0x7f7f6c08b710 con 0x7f7f7c06b960 2024-04-15T15:17:31.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.017+0000 7f7f7affd640 1 --2- 172.21.15.72:0/2454785938 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7f6806c9b0 0x7f7f6806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:31.026 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.018+0000 7f7f7affd640 1 --2- 172.21.15.72:0/2454785938 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7f6806c9b0 0x7f7f6806ee70 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c1511b0 tx=0x7f7f70008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:31.027 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.021+0000 7f7f63fff640 1 -- 172.21.15.72:0/2454785938 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7f6c059620 con 0x7f7f7c06b960 2024-04-15T15:17:31.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.235+0000 7f7f81edf640 1 -- 172.21.15.72:0/2454785938 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f7c0689e0 con 0x7f7f7c06b960 2024-04-15T15:17:31.237 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.236+0000 7f7f63fff640 1 -- 172.21.15.72:0/2454785938 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+3872 (secure 0 0 0) 0x7f7f6c05d920 con 0x7f7f7c06b960 2024-04-15T15:17:31.238 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-04-15T15:17:31.238 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":24,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4883,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6834/3363272836","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6832","nonce":3363272836},{"type":"v1","addr":"172.21.15.175:6834","nonce":3363272836}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5134,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.175:6835/1059187505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.175:6833","nonce":1059187505},{"type":"v1","addr":"172.21.15.175:6835","nonce":1059187505}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":5556,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6837/690132275","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6835","nonce":690132275},{"type":"v1","addr":"172.21.15.72:6837","nonce":690132275}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":6306,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.72:6836/3070626759","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6834","nonce":3070626759},{"type":"v1","addr":"172.21.15.72:6836","nonce":3070626759}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":24}],"filesystems":[]} 2024-04-15T15:17:31.240 INFO:teuthology.orchestra.run.smithi072.stderr:dumped fsmap epoch 24 2024-04-15T15:17:31.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.237+0000 7f7f81edf640 1 -- 172.21.15.72:0/2454785938 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7f6806c9b0 msgr2=0x7f7f6806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:31.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.237+0000 7f7f81edf640 1 --2- 172.21.15.72:0/2454785938 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7f6806c9b0 0x7f7f6806ee70 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c1511b0 tx=0x7f7f70008040 comp rx=0 tx=0).stop 2024-04-15T15:17:31.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.238+0000 7f7f81edf640 1 -- 172.21.15.72:0/2454785938 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7f7c06b960 msgr2=0x7f7f7c150c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:31.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.238+0000 7f7f81edf640 1 --2- 172.21.15.72:0/2454785938 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7f7c06b960 0x7f7f7c150c70 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c00caa0 tx=0x7f7f6c00cf60 comp rx=0 tx=0).stop 2024-04-15T15:17:31.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.238+0000 7f7f81edf640 1 -- 172.21.15.72:0/2454785938 shutdown_connections 2024-04-15T15:17:31.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.238+0000 7f7f81edf640 1 --2- 172.21.15.72:0/2454785938 >> [v2:172.21.15.72:6832/3503807805,v1:172.21.15.72:6833/3503807805] conn(0x7f7f6806c9b0 0x7f7f6806ee70 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:31.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.238+0000 7f7f81edf640 1 --2- 172.21.15.72:0/2454785938 >> [v2:172.21.15.175:3300/0,v1:172.21.15.175:6789/0] conn(0x7f7f7c15c7c0 0x7f7f7c0b09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:31.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.238+0000 7f7f81edf640 1 --2- 172.21.15.72:0/2454785938 >> [v2:172.21.15.175:3301/0,v1:172.21.15.175:6790/0] conn(0x7f7f7c0af700 0x7f7f7c0b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:31.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.238+0000 7f7f81edf640 1 --2- 172.21.15.72:0/2454785938 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7f7c06b960 0x7f7f7c150c70 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:31.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.238+0000 7f7f81edf640 1 -- 172.21.15.72:0/2454785938 >> 172.21.15.72:0/2454785938 conn(0x7f7f7c0b1dd0 msgr2=0x7f7f7c06e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:31.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.238+0000 7f7f81edf640 1 -- 172.21.15.72:0/2454785938 shutdown_connections 2024-04-15T15:17:31.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-04-15T15:17:31.238+0000 7f7f81edf640 1 -- 172.21.15.72:0/2454785938 wait complete. 2024-04-15T15:17:31.250 DEBUG:tasks.cephfs.filesystem:already dead... 2024-04-15T15:17:31.251 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-15T15:17:31.251 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-15T15:17:31.251 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:31.260 INFO:tasks.ceph.mds.a:Stopped 2024-04-15T15:17:31.261 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-15T15:17:31.261 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:31.310 INFO:tasks.ceph.mds.c:Stopped 2024-04-15T15:17:31.310 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-15T15:17:31.310 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:31.323 INFO:tasks.ceph.mds.b:Stopped 2024-04-15T15:17:31.323 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-15T15:17:31.323 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:31.379 INFO:tasks.ceph.mds.d:Stopped 2024-04-15T15:17:31.380 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-15T15:17:31.381 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-15T15:17:31.381 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:31.410 INFO:tasks.ceph.osd.0:Stopped 2024-04-15T15:17:31.410 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-15T15:17:31.411 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:31.474 INFO:tasks.ceph.osd.1:Stopped 2024-04-15T15:17:31.474 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-15T15:17:31.475 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:31.559 INFO:tasks.ceph.osd.2:Stopped 2024-04-15T15:17:31.559 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-15T15:17:31.560 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:31.598 INFO:tasks.ceph.osd.3:Stopped 2024-04-15T15:17:31.598 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-15T15:17:31.599 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:31.636 INFO:tasks.ceph.osd.4:Stopped 2024-04-15T15:17:31.636 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-15T15:17:31.636 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:31.710 INFO:tasks.ceph.osd.5:Stopped 2024-04-15T15:17:31.710 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-15T15:17:31.710 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:31.780 INFO:tasks.ceph.osd.6:Stopped 2024-04-15T15:17:31.780 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-15T15:17:31.781 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:31.846 INFO:tasks.ceph.osd.7:Stopped 2024-04-15T15:17:31.847 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-15T15:17:31.847 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-15T15:17:31.847 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:31.875 INFO:tasks.ceph.mgr.y:Stopped 2024-04-15T15:17:31.875 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-15T15:17:31.875 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:31.915 INFO:tasks.ceph.mgr.x:Stopped 2024-04-15T15:17:31.915 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-15T15:17:31.915 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-15T15:17:31.915 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:31.927 INFO:tasks.ceph.mon.a:Stopped 2024-04-15T15:17:31.927 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-15T15:17:31.927 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:31.967 INFO:tasks.ceph.mon.b:Stopped 2024-04-15T15:17:31.967 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-15T15:17:31.968 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:32.019 INFO:tasks.ceph.mon.c:Stopped 2024-04-15T15:17:32.020 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-15T15:17:32.020 DEBUG:teuthology.orchestra.run.smithi072:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow request' | head -n 1 2024-04-15T15:17:32.053 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi072.front.sepia.ceph.com 2024-04-15T15:17:32.053 DEBUG:teuthology.orchestra.run.smithi072:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-15T15:17:32.288 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi072.front.sepia.ceph.com 2024-04-15T15:17:32.289 DEBUG:teuthology.orchestra.run.smithi072:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-15T15:17:32.378 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi072.front.sepia.ceph.com 2024-04-15T15:17:32.379 DEBUG:teuthology.orchestra.run.smithi072:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-15T15:17:32.428 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi072.front.sepia.ceph.com 2024-04-15T15:17:32.429 DEBUG:teuthology.orchestra.run.smithi072:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-15T15:17:32.479 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi175.front.sepia.ceph.com 2024-04-15T15:17:32.479 DEBUG:teuthology.orchestra.run.smithi175:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-15T15:17:32.634 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi175.front.sepia.ceph.com 2024-04-15T15:17:32.634 DEBUG:teuthology.orchestra.run.smithi175:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-15T15:17:32.700 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi175.front.sepia.ceph.com 2024-04-15T15:17:32.701 DEBUG:teuthology.orchestra.run.smithi175:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-15T15:17:32.747 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi175.front.sepia.ceph.com 2024-04-15T15:17:32.748 DEBUG:teuthology.orchestra.run.smithi175:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-15T15:17:32.782 INFO:tasks.ceph:Archiving mon data... 2024-04-15T15:17:32.783 DEBUG:teuthology.misc:Transferring archived files from smithi072:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657120/data/mon.a.tgz 2024-04-15T15:17:32.783 DEBUG:teuthology.orchestra.run.smithi072:> mktemp 2024-04-15T15:17:32.800 INFO:teuthology.orchestra.run.smithi072.stdout:/tmp/tmp.gULY0WVwYz 2024-04-15T15:17:32.800 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.gULY0WVwYz 2024-04-15T15:17:32.947 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0666 /tmp/tmp.gULY0WVwYz 2024-04-15T15:17:33.037 DEBUG:teuthology.orchestra.remote:smithi072:/tmp/tmp.gULY0WVwYz is 350KB 2024-04-15T15:17:33.134 DEBUG:teuthology.orchestra.run.smithi072:> rm -fr /tmp/tmp.gULY0WVwYz 2024-04-15T15:17:33.149 DEBUG:teuthology.misc:Transferring archived files from smithi175:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657120/data/mon.b.tgz 2024-04-15T15:17:33.150 DEBUG:teuthology.orchestra.run.smithi175:> mktemp 2024-04-15T15:17:33.168 INFO:teuthology.orchestra.run.smithi175.stdout:/tmp/tmp.4sowcGdAhh 2024-04-15T15:17:33.168 DEBUG:teuthology.orchestra.run.smithi175:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.4sowcGdAhh 2024-04-15T15:17:33.334 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 0666 /tmp/tmp.4sowcGdAhh 2024-04-15T15:17:33.423 DEBUG:teuthology.orchestra.remote:smithi175:/tmp/tmp.4sowcGdAhh is 474KB 2024-04-15T15:17:33.522 DEBUG:teuthology.orchestra.run.smithi175:> rm -fr /tmp/tmp.4sowcGdAhh 2024-04-15T15:17:33.536 DEBUG:teuthology.misc:Transferring archived files from smithi175:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657120/data/mon.c.tgz 2024-04-15T15:17:33.537 DEBUG:teuthology.orchestra.run.smithi175:> mktemp 2024-04-15T15:17:33.592 INFO:teuthology.orchestra.run.smithi175.stdout:/tmp/tmp.aFGAwucwqM 2024-04-15T15:17:33.593 DEBUG:teuthology.orchestra.run.smithi175:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.aFGAwucwqM 2024-04-15T15:17:33.746 DEBUG:teuthology.orchestra.run.smithi175:> sudo chmod 0666 /tmp/tmp.aFGAwucwqM 2024-04-15T15:17:33.834 DEBUG:teuthology.orchestra.remote:smithi175:/tmp/tmp.aFGAwucwqM is 473KB 2024-04-15T15:17:33.932 DEBUG:teuthology.orchestra.run.smithi175:> rm -fr /tmp/tmp.aFGAwucwqM 2024-04-15T15:17:33.948 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-15T15:17:33.948 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-15T15:17:33.951 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-15T15:17:34.037 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-15T15:17:34.071 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-15T15:17:34.071 INFO:tasks.ceph:Archiving crash dumps... 2024-04-15T15:17:34.073 DEBUG:teuthology.misc:Transferring archived files from smithi072:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657120/remote/smithi072/crash 2024-04-15T15:17:34.074 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-15T15:17:34.107 DEBUG:teuthology.misc:Transferring archived files from smithi175:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657120/remote/smithi175/crash 2024-04-15T15:17:34.108 DEBUG:teuthology.orchestra.run.smithi175:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-15T15:17:34.139 INFO:tasks.ceph:Shutting down logrotate 2024-04-15T15:17:34.140 DEBUG:teuthology.orchestra.run.smithi072:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:17:34.152 DEBUG:teuthology.orchestra.run.smithi175:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:17:34.242 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-15T15:17:34.271 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-15T15:17:34.301 INFO:tasks.ceph:Compressing logs... 2024-04-15T15:17:34.302 DEBUG:teuthology.orchestra.run.smithi072:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-15T15:17:34.314 DEBUG:teuthology.orchestra.run.smithi175:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-15T15:17:34.339 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57191.log 2024-04-15T15:17:34.340 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47137.log 2024-04-15T15:17:34.340 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.57191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53394.log 2024-04-15T15:17:34.401 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-15T15:17:34.401 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37433.log 2024-04-15T15:17:34.401 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37393.log 2024-04-15T15:17:34.401 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.37433.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-15T15:17:34.401 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.37393.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37433.log.gz 2024-04-15T15:17:34.401 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-04-15T15:17:34.402 INFO:teuthology.orchestra.run.smithi175.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.37393.log.gz 2024-04-15T15:17:34.402 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.45005.log 2024-04-15T15:17:34.402 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-15T15:17:34.402 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.1.45005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.45499.log 2024-04-15T15:17:34.402 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-15T15:17:34.402 INFO:teuthology.orchestra.run.smithi175.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-04-15T15:17:34.402 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.1.45499.log: 88.8% -- replaced with /var/log/ceph/ceph-client.1.45005.log.gz 2024-04-15T15:17:34.403 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-15T15:17:34.403 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37353.log 2024-04-15T15:17:34.403 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37473.log 2024-04-15T15:17:34.403 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.37353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.45840.log 2024-04-15T15:17:34.403 INFO:teuthology.orchestra.run.smithi175.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37353.log.gz 2024-04-15T15:17:34.403 INFO:teuthology.orchestra.run.smithi175.stderr: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 91.2% -- replaced with /var/log/ceph/ceph-client.1.45499.log.gz 2024-04-15T15:17:34.403 INFO:teuthology.orchestra.run.smithi175.stderr: 2024-04-15T15:17:34.403 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.37473.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-15T15:17:34.403 INFO:teuthology.orchestra.run.smithi175.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37473.log.gz 2024-04-15T15:17:34.404 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.3.45840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.44300.log 2024-04-15T15:17:34.404 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37313.log 2024-04-15T15:17:34.404 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.3.44300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.43843.log 2024-04-15T15:17:34.404 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.37313.log: 91.0% -- replaced with /var/log/ceph/ceph-client.3.44300.log.gz 91.5% -- replaced with /var/log/ceph/ceph-client.3.45840.log.gz 2024-04-15T15:17:34.404 INFO:teuthology.orchestra.run.smithi175.stderr: 2024-04-15T15:17:34.405 INFO:teuthology.orchestra.run.smithi175.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37313.log.gz 2024-04-15T15:17:34.405 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-15T15:17:34.405 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.1.43843.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-15T15:17:34.405 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37593.log 2024-04-15T15:17:34.405 INFO:teuthology.orchestra.run.smithi175.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.1.43843.log.gz 2024-04-15T15:17:34.405 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37553.log 2024-04-15T15:17:34.405 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-client.admin.37593.log: 91.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-15T15:17:34.405 INFO:teuthology.orchestra.run.smithi175.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37593.log.gz 2024-04-15T15:17:34.406 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-04-15T15:17:34.406 INFO:teuthology.orchestra.run.smithi175.stderr: 92.9%/var/log/ceph/ceph-client.admin.37553.log: -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-15T15:17:34.406 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-15T15:17:34.406 INFO:teuthology.orchestra.run.smithi175.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37553.log.gz 2024-04-15T15:17:34.406 INFO:teuthology.orchestra.run.smithi175.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37513.log 2024-04-15T15:17:34.406 INFO:teuthology.orchestra.run.smithi175.stderr:/var/log/ceph/ceph-mds.d.log: /var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.37513.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37513.log.gz 2024-04-15T15:17:34.406 INFO:teuthology.orchestra.run.smithi175.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-04-15T15:17:34.498 INFO:teuthology.orchestra.run.smithi175.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-15T15:17:34.503 INFO:teuthology.orchestra.run.smithi175.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-15T15:17:34.531 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.47137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49676.log 2024-04-15T15:17:34.532 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57191.log.gz 2024-04-15T15:17:34.532 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53394.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47137.log.gz 2024-04-15T15:17:34.532 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46260.log 2024-04-15T15:17:34.532 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49676.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.53394.log.gz 2024-04-15T15:17:34.532 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43441.log 2024-04-15T15:17:34.532 INFO:teuthology.orchestra.run.smithi072.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49676.log.gz 2024-04-15T15:17:34.532 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.46260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47347.log 2024-04-15T15:17:34.533 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46260.log.gz 2024-04-15T15:17:34.533 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.43441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45625.log 2024-04-15T15:17:34.533 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.47347.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.31567.log 2024-04-15T15:17:34.533 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43441.log.gz 2024-04-15T15:17:34.533 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51509.log 2024-04-15T15:17:34.533 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47347.log.gz 2024-04-15T15:17:34.533 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45625.log.gz 2024-04-15T15:17:34.533 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph.tmp-client.admin.31567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57524.log 2024-04-15T15:17:34.533 INFO:teuthology.orchestra.run.smithi072.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.31567.log.gz 2024-04-15T15:17:34.534 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.51509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49466.log 2024-04-15T15:17:34.534 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51509.log.gz 2024-04-15T15:17:34.534 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.57524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43724.log 2024-04-15T15:17:34.534 INFO:teuthology.orchestra.run.smithi072.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57524.log.gz 2024-04-15T15:17:34.534 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49466.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-15T15:17:34.534 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.43724.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49466.log.gz 2024-04-15T15:17:34.534 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55134.log 2024-04-15T15:17:34.534 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-mon.a.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43724.log.gz 2024-04-15T15:17:34.535 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43399.log 2024-04-15T15:17:34.535 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.55134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43673.log 2024-04-15T15:17:34.535 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.43399.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55134.log.gz 2024-04-15T15:17:34.535 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50023.log 2024-04-15T15:17:34.535 INFO:teuthology.orchestra.run.smithi072.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43399.log.gz 2024-04-15T15:17:34.535 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.43673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55302.log 2024-04-15T15:17:34.535 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50023.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43673.log.gz 2024-04-15T15:17:34.535 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52854.log 2024-04-15T15:17:34.535 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.55302.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50023.log.gz 2024-04-15T15:17:34.536 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53999.log 2024-04-15T15:17:34.536 INFO:teuthology.orchestra.run.smithi072.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55302.log.gz 2024-04-15T15:17:34.536 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.52854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58344.log 2024-04-15T15:17:34.536 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46265.log 89.1% 2024-04-15T15:17:34.536 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52854.log.gz 2024-04-15T15:17:34.536 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.58344.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53999.log.gz 2024-04-15T15:17:34.536 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58098.log 2024-04-15T15:17:34.536 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58344.log.gz 2024-04-15T15:17:34.536 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.46265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46465.log 2024-04-15T15:17:34.537 INFO:teuthology.orchestra.run.smithi072.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46265.log.gz 2024-04-15T15:17:34.537 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.58098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58260.log 2024-04-15T15:17:34.537 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58098.log.gz 2024-04-15T15:17:34.537 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.46465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59483.log 2024-04-15T15:17:34.537 INFO:teuthology.orchestra.run.smithi072.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46465.log.gz 2024-04-15T15:17:34.537 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.58260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45933.log 2024-04-15T15:17:34.537 INFO:teuthology.orchestra.run.smithi072.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58260.log.gz 2024-04-15T15:17:34.537 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.59483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45927.log 2024-04-15T15:17:34.537 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55422.log 2024-04-15T15:17:34.538 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59483.log.gz 2024-04-15T15:17:34.538 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45927.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45933.log.gz 2024-04-15T15:17:34.538 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51881.log 2024-04-15T15:17:34.538 INFO:teuthology.orchestra.run.smithi072.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45927.log.gz 2024-04-15T15:17:34.538 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.55422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49844.log 2024-04-15T15:17:34.538 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.51881.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.55422.log.gz 2024-04-15T15:17:34.538 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50957.log 2024-04-15T15:17:34.538 INFO:teuthology.orchestra.run.smithi072.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51881.log.gz 2024-04-15T15:17:34.539 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47095.log 2024-04-15T15:17:34.539 INFO:teuthology.orchestra.run.smithi072.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49844.log.gz 2024-04-15T15:17:34.539 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52625.log 2024-04-15T15:17:34.539 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.47095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57896.log 2024-04-15T15:17:34.539 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50957.log.gz 2024-04-15T15:17:34.539 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47095.log.gz 2024-04-15T15:17:34.539 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.52625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53915.log 2024-04-15T15:17:34.539 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.57896.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52625.log.gz 2024-04-15T15:17:34.539 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60113.log 2024-04-15T15:17:34.540 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53915.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.57896.log.gz 2024-04-15T15:17:34.540 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48878.log 2024-04-15T15:17:34.540 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53915.log.gz 2024-04-15T15:17:34.540 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.60113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52583.log 2024-04-15T15:17:34.540 INFO:teuthology.orchestra.run.smithi072.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60113.log.gz 2024-04-15T15:17:34.540 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.48878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47221.log 2024-04-15T15:17:34.540 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.52583.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48878.log.gz 2024-04-15T15:17:34.540 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45211.log 2024-04-15T15:17:34.540 INFO:teuthology.orchestra.run.smithi072.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52583.log.gz 2024-04-15T15:17:34.541 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.47221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57854.log 2024-04-15T15:17:34.541 INFO:teuthology.orchestra.run.smithi072.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47221.log.gz 2024-04-15T15:17:34.541 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44024.log 2024-04-15T15:17:34.541 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.57854.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45211.log.gz 2024-04-15T15:17:34.541 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45331.log 2024-04-15T15:17:34.541 INFO:teuthology.orchestra.run.smithi072.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57854.log.gz 2024-04-15T15:17:34.541 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.44024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53099.log 2024-04-15T15:17:34.541 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44024.log.gz 2024-04-15T15:17:34.541 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59035.log 2024-04-15T15:17:34.542 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53099.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45331.log.gz 2024-04-15T15:17:34.542 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37943.log 2024-04-15T15:17:34.542 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53099.log.gz 2024-04-15T15:17:34.542 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.59035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47389.log 2024-04-15T15:17:34.542 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.37943.log: gzip 89.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.59035.log.gz --verbose 2024-04-15T15:17:34.542 INFO:teuthology.orchestra.run.smithi072.stderr: -- /var/log/ceph/ceph-client.admin.49760.log 2024-04-15T15:17:34.542 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.47389.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37943.log.gz 2024-04-15T15:17:34.542 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54832.log 2024-04-15T15:17:34.542 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47389.log.gz 2024-04-15T15:17:34.543 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56717.log 2024-04-15T15:17:34.543 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.54832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58470.log 89.4% 2024-04-15T15:17:34.543 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49760.log.gz 2024-04-15T15:17:34.543 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.56717.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54832.log.gz 2024-04-15T15:17:34.543 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58218.log 2024-04-15T15:17:34.543 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.58470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53268.log 2024-04-15T15:17:34.543 INFO:teuthology.orchestra.run.smithi072.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.58470.log.gz 2024-04-15T15:17:34.543 INFO:teuthology.orchestra.run.smithi072.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.56717.log.gz 2024-04-15T15:17:34.544 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.58218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53659.log 2024-04-15T15:17:34.544 INFO:teuthology.orchestra.run.smithi072.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58218.log.gz 2024-04-15T15:17:34.544 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54233.log 2024-04-15T15:17:34.544 INFO:teuthology.orchestra.run.smithi072.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53268.log.gz 2024-04-15T15:17:34.544 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55076.log 2024-04-15T15:17:34.544 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1%/var/log/ceph/ceph-client.admin.54233.log: -- replaced with /var/log/ceph/ceph-client.admin.53659.log.gz 2024-04-15T15:17:34.544 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46801.log 2024-04-15T15:17:34.544 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.55076.log: 89.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.54233.log.gz 2024-04-15T15:17:34.544 INFO:teuthology.orchestra.run.smithi072.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47263.log 2024-04-15T15:17:34.545 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.46801.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55076.log.gz 2024-04-15T15:17:34.545 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51425.log 2024-04-15T15:17:34.545 INFO:teuthology.orchestra.run.smithi072.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46801.log.gz 2024-04-15T15:17:34.545 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.47263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43898.log 2024-04-15T15:17:34.545 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.51425.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47263.log.gz 2024-04-15T15:17:34.545 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49256.log 2024-04-15T15:17:34.545 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.43898.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51425.log.gz 2024-04-15T15:17:34.545 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54167.log 2024-04-15T15:17:34.546 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43898.log.gz 2024-04-15T15:17:34.546 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50459.log/var/log/ceph/ceph-client.admin.49256.log: 2024-04-15T15:17:34.546 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.54167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45253.log 2024-04-15T15:17:34.546 INFO:teuthology.orchestra.run.smithi072.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49256.log.gz 2024-04-15T15:17:34.546 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51157.log 2024-04-15T15:17:34.546 INFO:teuthology.orchestra.run.smithi072.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50459.log.gz 2024-04-15T15:17:34.546 INFO:teuthology.orchestra.run.smithi072.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.54167.log.gz 2024-04-15T15:17:34.546 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57650.log 2024-04-15T15:17:34.546 INFO:teuthology.orchestra.run.smithi072.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45253.log.gz 2024-04-15T15:17:34.547 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.51157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45127.log 2024-04-15T15:17:34.547 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51157.log.gz 2024-04-15T15:17:34.547 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.57650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55464.log 2024-04-15T15:17:34.547 INFO:teuthology.orchestra.run.smithi072.stderr: 88.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.57650.log.gz -5 2024-04-15T15:17:34.547 INFO:teuthology.orchestra.run.smithi072.stderr: --verbose -- /var/log/ceph/ceph-client.admin.56845.log 2024-04-15T15:17:34.547 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45127.log: /var/log/ceph/ceph-client.admin.55464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44859.log 2024-04-15T15:17:34.547 INFO:teuthology.orchestra.run.smithi072.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45127.log.gz 2024-04-15T15:17:34.547 INFO:teuthology.orchestra.run.smithi072.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55464.log.gz 2024-04-15T15:17:34.547 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.56845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55674.log 2024-04-15T15:17:34.548 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.44859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46264.log 2024-04-15T15:17:34.548 INFO:teuthology.orchestra.run.smithi072.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44859.log.gz 2024-04-15T15:17:34.548 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.55674.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.56845.log.gz 2024-04-15T15:17:34.548 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57482.log 2024-04-15T15:17:34.548 INFO:teuthology.orchestra.run.smithi072.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55674.log.gz 2024-04-15T15:17:34.548 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.46264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46267.log 2024-04-15T15:17:34.548 INFO:teuthology.orchestra.run.smithi072.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46264.log.gz 2024-04-15T15:17:34.548 INFO:teuthology.orchestra.run.smithi072.stderr:gzip/var/log/ceph/ceph-client.admin.57482.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.56640.log 2024-04-15T15:17:34.549 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.46267.log: gzip -5 --verbose 88.6% -- /var/log/ceph/ceph-client.admin.51325.log -- replaced with /var/log/ceph/ceph-client.admin.57482.log.gz 2024-04-15T15:17:34.549 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-04-15T15:17:34.549 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46267.log.gz 2024-04-15T15:17:34.549 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.56640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50417.log 2024-04-15T15:17:34.549 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56640.log.gz 2024-04-15T15:17:34.549 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.51325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54776.log 2024-04-15T15:17:34.549 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50417.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51325.log.gz 2024-04-15T15:17:34.549 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49214.log 2024-04-15T15:17:34.549 INFO:teuthology.orchestra.run.smithi072.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50417.log.gz 2024-04-15T15:17:34.550 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.54776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46843.log 2024-04-15T15:17:34.550 INFO:teuthology.orchestra.run.smithi072.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54776.log.gz 2024-04-15T15:17:34.550 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55758.log 2024-04-15T15:17:34.550 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.46843.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49214.log.gz 2024-04-15T15:17:34.550 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50501.log 2024-04-15T15:17:34.550 INFO:teuthology.orchestra.run.smithi072.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46843.log.gz 2024-04-15T15:17:34.550 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.55758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50191.log 2024-04-15T15:17:34.550 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50501.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55758.log.gz 2024-04-15T15:17:34.550 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52029.log 2024-04-15T15:17:34.551 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50191.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50501.log.gz 2024-04-15T15:17:34.551 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52247.log 2024-04-15T15:17:34.551 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.52029.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50191.log.gz 2024-04-15T15:17:34.551 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53142.log 2024-04-15T15:17:34.551 INFO:teuthology.orchestra.run.smithi072.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52029.log.gz 2024-04-15T15:17:34.551 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.52247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.56093.log 2024-04-15T15:17:34.551 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1%/var/log/ceph/ceph-client.admin.53142.log: -- replaced with /var/log/ceph/ceph-client.admin.52247.log.gz 2024-04-15T15:17:34.551 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59819.log 2024-04-15T15:17:34.552 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53142.log.gz 2024-04-15T15:17:34.552 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.0.56093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45929.log 2024-04-15T15:17:34.552 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.59819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37583.log 2024-04-15T15:17:34.552 INFO:teuthology.orchestra.run.smithi072.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.59819.log.gz 2024-04-15T15:17:34.552 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59077.log 2024-04-15T15:17:34.552 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50107.log 2024-04-15T15:17:34.552 INFO:teuthology.orchestra.run.smithi072.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45929.log.gz 2024-04-15T15:17:34.552 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.37583.log: /var/log/ceph/ceph-client.admin.59077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47179.log 2024-04-15T15:17:34.552 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37583.log.gz 2024-04-15T15:17:34.553 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43940.log 2024-04-15T15:17:34.553 INFO:teuthology.orchestra.run.smithi072.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59077.log.gz 2024-04-15T15:17:34.553 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50107.log: /var/log/ceph/ceph-client.admin.47179.log: 91.8% -- replaced with /var/log/ceph/ceph-client.0.56093.log.gz 2024-04-15T15:17:34.553 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45930.log 2024-04-15T15:17:34.553 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50107.log.gz 2024-04-15T15:17:34.553 INFO:teuthology.orchestra.run.smithi072.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47179.log.gz 2024-04-15T15:17:34.553 INFO:teuthology.orchestra.run.smithi072.stderr:gzip/var/log/ceph/ceph-client.admin.43940.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.45709.log 2024-04-15T15:17:34.553 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51839.log 2024-04-15T15:17:34.554 INFO:teuthology.orchestra.run.smithi072.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43940.log.gz 2024-04-15T15:17:34.554 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54125.log 2024-04-15T15:17:34.554 INFO:teuthology.orchestra.run.smithi072.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45930.log.gz 2024-04-15T15:17:34.554 INFO:teuthology.orchestra.run.smithi072.stderr: 88.5%/var/log/ceph/ceph-client.admin.51839.log: -- replaced with /var/log/ceph/ceph-client.admin.45709.log.gz 2024-04-15T15:17:34.554 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59861.log 2024-04-15T15:17:34.554 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.54125.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51839.log.gz 2024-04-15T15:17:34.554 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48726.log 2024-04-15T15:17:34.554 INFO:teuthology.orchestra.run.smithi072.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54125.log.gz 2024-04-15T15:17:34.554 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.59861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49718.log 2024-04-15T15:17:34.555 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.48726.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.59861.log.gz 2024-04-15T15:17:34.555 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58302.log 2024-04-15T15:17:34.555 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49718.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48726.log.gz 2024-04-15T15:17:34.555 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58806.log 2024-04-15T15:17:34.555 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49718.log.gz 2024-04-15T15:17:34.555 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.58302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59567.log 2024-04-15T15:17:34.555 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.58806.log: gzip -5 87.5% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.58302.log.gz /var/log/ceph/ceph-client.admin.58993.log 2024-04-15T15:17:34.555 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-04-15T15:17:34.555 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.59567.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58806.log.gz 2024-04-15T15:17:34.556 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57247.log 2024-04-15T15:17:34.556 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59567.log.gz 2024-04-15T15:17:34.556 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.58993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58951.log 2024-04-15T15:17:34.556 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.58993.log.gz 2024-04-15T15:17:34.556 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37836.log 2024-04-15T15:17:34.556 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.57247.log: /var/log/ceph/ceph-client.admin.58951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45667.log 2024-04-15T15:17:34.556 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57247.log.gz 2024-04-15T15:17:34.556 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58951.log.gz 2024-04-15T15:17:34.557 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.37836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43045.log 2024-04-15T15:17:34.557 INFO:teuthology.orchestra.run.smithi072.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37836.log.gz 2024-04-15T15:17:34.557 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50149.log 2024-04-15T15:17:34.557 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.43045.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45667.log.gz 2024-04-15T15:17:34.557 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53310.log 2024-04-15T15:17:34.557 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43045.log.gz 2024-04-15T15:17:34.557 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43314.log 2024-04-15T15:17:34.557 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50149.log.gz 2024-04-15T15:17:34.557 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55260.log 2024-04-15T15:17:34.558 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.43314.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53310.log.gz 2024-04-15T15:17:34.558 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60155.log 2024-04-15T15:17:34.558 INFO:teuthology.orchestra.run.smithi072.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43314.log.gz 2024-04-15T15:17:34.558 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.55260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54692.log 2024-04-15T15:17:34.558 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55260.log.gz 2024-04-15T15:17:34.558 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.60155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58848.log 2024-04-15T15:17:34.558 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1%/var/log/ceph/ceph-client.admin.54692.log: -- replaced with /var/log/ceph/ceph-client.admin.60155.log.gz 2024-04-15T15:17:34.558 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43814.log 2024-04-15T15:17:34.558 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.58848.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54692.log.gz 2024-04-15T15:17:34.559 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60255.log 2024-04-15T15:17:34.559 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58848.log.gz 2024-04-15T15:17:34.559 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.43814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46969.log 2024-04-15T15:17:34.559 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43814.log.gz 2024-04-15T15:17:34.559 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.60255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44584.log 2024-04-15T15:17:34.559 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.46969.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60255.log.gz 2024-04-15T15:17:34.559 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50831.log 2024-04-15T15:17:34.559 INFO:teuthology.orchestra.run.smithi072.stderr: 89.0%/var/log/ceph/ceph-client.admin.44584.log: -- replaced with /var/log/ceph/ceph-client.admin.46969.log.gz 2024-04-15T15:17:34.559 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51923.log 2024-04-15T15:17:34.560 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44985.log 2024-04-15T15:17:34.560 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.51923.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50831.log.gz 2024-04-15T15:17:34.560 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52163.log 2024-04-15T15:17:34.560 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- 88.1% /var/log/ceph/ceph-client.admin.45085.log 2024-04-15T15:17:34.560 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51923.log.gz 2024-04-15T15:17:34.560 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.44985.log: /var/log/ceph/ceph-client.admin.52163.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44985.log.gz 2024-04-15T15:17:34.560 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58140.log 2024-04-15T15:17:34.560 INFO:teuthology.orchestra.run.smithi072.stderr: 92.7% 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44584.log.gz -- replaced with /var/log/ceph/ceph-client.admin.52163.log.gz 2024-04-15T15:17:34.560 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-04-15T15:17:34.561 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45931.log 2024-04-15T15:17:34.561 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45085.log: /var/log/ceph/ceph-client.admin.58140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58638.log 2024-04-15T15:17:34.561 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45085.log.gz 2024-04-15T15:17:34.561 INFO:teuthology.orchestra.run.smithi072.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.58140.log.gz 2024-04-15T15:17:34.561 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52770.log 2024-04-15T15:17:34.561 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.58638.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45931.log.gz 2024-04-15T15:17:34.561 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52541.log 2024-04-15T15:17:34.561 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.52770.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.58638.log.gz 2024-04-15T15:17:34.561 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59441.log 2024-04-15T15:17:34.562 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52770.log.gz 2024-04-15T15:17:34.562 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.52541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44108.log 2024-04-15T15:17:34.562 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.59441.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52541.log.gz 2024-04-15T15:17:34.562 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-15T15:17:34.562 INFO:teuthology.orchestra.run.smithi072.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.59441.log.gz 2024-04-15T15:17:34.562 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.44108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56783.log 2024-04-15T15:17:34.562 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60297.log 2024-04-15T15:17:34.562 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.56783.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.44108.log.gz 2024-04-15T15:17:34.563 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51671.log 2024-04-15T15:17:34.563 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56783.log.gz 2024-04-15T15:17:34.563 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.60297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49088.log 2024-04-15T15:17:34.563 INFO:teuthology.orchestra.run.smithi072.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60297.log.gz 2024-04-15T15:17:34.563 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48175.log 2024-04-15T15:17:34.563 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.51671.log: /var/log/ceph/ceph-client.admin.49088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60071.log 2024-04-15T15:17:34.563 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51671.log.gz 2024-04-15T15:17:34.563 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.48175.log: 89.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-15T15:17:34.563 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49088.log.gz 2024-04-15T15:17:34.564 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44174.log 2024-04-15T15:17:34.564 INFO:teuthology.orchestra.run.smithi072.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48175.log.gz 2024-04-15T15:17:34.564 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53831.log 2024-04-15T15:17:34.564 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.60071.log: /var/log/ceph/ceph-client.admin.44174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47643.log 2024-04-15T15:17:34.564 INFO:teuthology.orchestra.run.smithi072.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60071.log.gz 2024-04-15T15:17:34.564 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53831.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.44174.log.gz 2024-04-15T15:17:34.564 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43233.log 2024-04-15T15:17:34.564 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.47643.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53831.log.gz 2024-04-15T15:17:34.565 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46268.log 2024-04-15T15:17:34.565 INFO:teuthology.orchestra.run.smithi072.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47643.log.gz 2024-04-15T15:17:34.565 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.43233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49298.log 2024-04-15T15:17:34.565 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43233.log.gz 2024-04-15T15:17:34.565 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.46268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50375.log 2024-04-15T15:17:34.565 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49298.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46268.log.gz 2024-04-15T15:17:34.565 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51797.log 2024-04-15T15:17:34.565 INFO:teuthology.orchestra.run.smithi072.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49298.log.gz 2024-04-15T15:17:34.565 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44066.log 2024-04-15T15:17:34.566 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50375.log.gz 2024-04-15T15:17:34.566 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.51797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59777.log 2024-04-15T15:17:34.566 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.44066.log: gzip 88.6% -5 -- replaced with /var/log/ceph/ceph-client.admin.51797.log.gz 2024-04-15T15:17:34.566 INFO:teuthology.orchestra.run.smithi072.stderr: --verbose -- /var/log/ceph/ceph-client.admin.44817.log 2024-04-15T15:17:34.566 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.59777.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44066.log.gz 2024-04-15T15:17:34.566 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51241.log 2024-04-15T15:17:34.566 INFO:teuthology.orchestra.run.smithi072.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59777.log.gz 2024-04-15T15:17:34.566 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.44817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45928.log 2024-04-15T15:17:34.566 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.51241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59399.log 88.3% 2024-04-15T15:17:34.567 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44817.log.gz 2024-04-15T15:17:34.567 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45928.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51241.log.gz 2024-04-15T15:17:34.567 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55632.log 2024-04-15T15:17:34.567 INFO:teuthology.orchestra.run.smithi072.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45928.log.gz 2024-04-15T15:17:34.567 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.59399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51383.log 2024-04-15T15:17:34.567 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.55632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43770.log 2024-04-15T15:17:34.567 INFO:teuthology.orchestra.run.smithi072.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59399.log.gz 2024-04-15T15:17:34.567 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.51383.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55632.log.gz 2024-04-15T15:17:34.567 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53957.log 2024-04-15T15:17:34.568 INFO:teuthology.orchestra.run.smithi072.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51383.log.gz 2024-04-15T15:17:34.568 INFO:teuthology.orchestra.run.smithi072.stderr:gzip/var/log/ceph/ceph-client.admin.43770.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.56258.log 2024-04-15T15:17:34.568 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49172.log 2024-04-15T15:17:34.568 INFO:teuthology.orchestra.run.smithi072.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43770.log.gz 2024-04-15T15:17:34.568 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.56258.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53957.log.gz 2024-04-15T15:17:34.568 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49004.log 2024-04-15T15:17:34.568 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56258.log.gz 2024-04-15T15:17:34.568 INFO:teuthology.orchestra.run.smithi072.stderr:gzip/var/log/ceph/ceph-client.admin.49172.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.48920.log 2024-04-15T15:17:34.568 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51593.log 2024-04-15T15:17:34.569 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49172.log.gz 2024-04-15T15:17:34.569 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.48920.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49004.log.gz 2024-04-15T15:17:34.569 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53575.log 2024-04-15T15:17:34.569 INFO:teuthology.orchestra.run.smithi072.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48920.log.gz 2024-04-15T15:17:34.569 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.51593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49634.log 2024-04-15T15:17:34.569 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53575.log: gzip -5 --verbose -- 88.7% /var/log/ceph/ceph-client.admin.45457.log 2024-04-15T15:17:34.569 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51593.log.gz 2024-04-15T15:17:34.569 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49634.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53575.log.gz 2024-04-15T15:17:34.569 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47473.log 2024-04-15T15:17:34.570 INFO:teuthology.orchestra.run.smithi072.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49634.log.gz 2024-04-15T15:17:34.570 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43137.log 2024-04-15T15:17:34.570 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.47473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55176.log 2024-04-15T15:17:34.570 INFO:teuthology.orchestra.run.smithi072.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45457.log.gz 2024-04-15T15:17:34.570 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.43137.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47473.log.gz 2024-04-15T15:17:34.570 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46261.log 2024-04-15T15:17:34.570 INFO:teuthology.orchestra.run.smithi072.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43137.log.gz 2024-04-15T15:17:34.570 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.55176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58056.log 2024-04-15T15:17:34.570 INFO:teuthology.orchestra.run.smithi072.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55176.log.gz 2024-04-15T15:17:34.571 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.46261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45924.log 2024-04-15T15:17:34.571 INFO:teuthology.orchestra.run.smithi072.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46261.log.gz 2024-04-15T15:17:34.571 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.58056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45373.log 2024-04-15T15:17:34.571 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49928.log 2024-04-15T15:17:34.571 INFO:teuthology.orchestra.run.smithi072.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.58056.log.gz 2024-04-15T15:17:34.571 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53057.log 2024-04-15T15:17:34.571 INFO:teuthology.orchestra.run.smithi072.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45924.log.gz 2024-04-15T15:17:34.571 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49928.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45373.log.gz 2024-04-15T15:17:34.571 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-15T15:17:34.572 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49928.log.gz 2024-04-15T15:17:34.572 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50333.log 2024-04-15T15:17:34.572 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46927.log 2024-04-15T15:17:34.572 INFO:teuthology.orchestra.run.smithi072.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53057.log.gz 2024-04-15T15:17:34.572 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56216.log 2024-04-15T15:17:34.572 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50333.log.gz 2024-04-15T15:17:34.572 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.46927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54908.log 2024-04-15T15:17:34.572 INFO:teuthology.orchestra.run.smithi072.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46927.log.gz 2024-04-15T15:17:34.572 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53789.log 2024-04-15T15:17:34.573 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.56216.log: /var/log/ceph/ceph-client.admin.54908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49508.log 2024-04-15T15:17:34.573 INFO:teuthology.orchestra.run.smithi072.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.56216.log.gz 2024-04-15T15:17:34.573 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54908.log.gz 2024-04-15T15:17:34.573 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58386.log 2024-04-15T15:17:34.573 INFO:teuthology.orchestra.run.smithi072.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53789.log.gz 2024-04-15T15:17:34.573 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59945.log 2024-04-15T15:17:34.573 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53352.log 2024-04-15T15:17:34.573 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49508.log.gz 2024-04-15T15:17:34.574 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.58386.log: /var/log/ceph/ceph-client.admin.59945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51984.log 2024-04-15T15:17:34.574 INFO:teuthology.orchestra.run.smithi072.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.58386.log.gz 2024-04-15T15:17:34.574 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.59945.log.gz 2024-04-15T15:17:34.574 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55716.log 2024-04-15T15:17:34.574 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.51984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45751.log 2024-04-15T15:17:34.574 INFO:teuthology.orchestra.run.smithi072.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53352.log.gz 2024-04-15T15:17:34.574 INFO:teuthology.orchestra.run.smithi072.stderr: 89.4%/var/log/ceph/ceph-client.admin.55716.log: -- replaced with /var/log/ceph/ceph-client.admin.51984.log.gz 2024-04-15T15:17:34.574 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54992.log 2024-04-15T15:17:34.574 INFO:teuthology.orchestra.run.smithi072.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.55716.log.gz 2024-04-15T15:17:34.575 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52415.log 2024-04-15T15:17:34.575 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45751.log: /var/log/ceph/ceph-client.admin.54992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49550.log 2024-04-15T15:17:34.575 INFO:teuthology.orchestra.run.smithi072.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45751.log.gz 2024-04-15T15:17:34.575 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54992.log.gz 2024-04-15T15:17:34.575 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.52415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49802.log 2024-04-15T15:17:34.575 INFO:teuthology.orchestra.run.smithi072.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52415.log.gz 2024-04-15T15:17:34.575 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.54493.log 2024-04-15T15:17:34.575 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49550.log: /var/log/ceph/ceph-client.admin.49802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45043.log 2024-04-15T15:17:34.576 INFO:teuthology.orchestra.run.smithi072.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49550.log.gz 2024-04-15T15:17:34.576 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.0.54493.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49802.log.gz 2024-04-15T15:17:34.576 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54041.log 2024-04-15T15:17:34.576 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60029.log 2024-04-15T15:17:34.576 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.54041.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.45043.log.gz 2024-04-15T15:17:34.576 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57379.log 2024-04-15T15:17:34.576 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.0.54493.log.gz 2024-04-15T15:17:34.576 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.60029.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54041.log.gz 2024-04-15T15:17:34.576 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59161.log 2024-04-15T15:17:34.577 INFO:teuthology.orchestra.run.smithi072.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.60029.log.gz 2024-04-15T15:17:34.577 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.57379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48836.log 2024-04-15T15:17:34.577 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45169.log 2024-04-15T15:17:34.577 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.59161.log: /var/log/ceph/ceph-client.admin.48836.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.57379.log.gz 2024-04-15T15:17:34.577 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- 87.8% /var/log/ceph/ceph-client.admin.50579.log 2024-04-15T15:17:34.577 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59161.log.gz 2024-04-15T15:17:34.577 INFO:teuthology.orchestra.run.smithi072.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48836.log.gz 2024-04-15T15:17:34.577 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48962.log 2024-04-15T15:17:34.578 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45169.log.gz 2024-04-15T15:17:34.578 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43525.log 2024-04-15T15:17:34.578 INFO:teuthology.orchestra.run.smithi072.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50579.log.gz 2024-04-15T15:17:34.578 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57608.log 2024-04-15T15:17:34.578 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.48962.log: /var/log/ceph/ceph-client.admin.43525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53015.log 2024-04-15T15:17:34.578 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48962.log.gz 2024-04-15T15:17:34.578 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43525.log.gz 2024-04-15T15:17:34.578 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.57608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57566.log 2024-04-15T15:17:34.578 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53015.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57608.log.gz 2024-04-15T15:17:34.579 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59315.log 2024-04-15T15:17:34.579 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53015.log.gz 2024-04-15T15:17:34.579 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.57566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52934.log 2024-04-15T15:17:34.579 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57566.log.gz 2024-04-15T15:17:34.579 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49382.log 2024-04-15T15:17:34.579 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.59315.log: /var/log/ceph/ceph-client.admin.52934.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-15T15:17:34.579 INFO:teuthology.orchestra.run.smithi072.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59315.log.gz 2024-04-15T15:17:34.579 INFO:teuthology.orchestra.run.smithi072.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52934.log.gz 2024-04-15T15:17:34.580 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55034.log 2024-04-15T15:17:34.580 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58428.log 2024-04-15T15:17:34.580 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2%/var/log/ceph/ceph-osd.3.log: -- replaced with /var/log/ceph/ceph-client.admin.49382.log.gz 2024-04-15T15:17:34.580 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.55034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55506.log 2024-04-15T15:17:34.580 INFO:teuthology.orchestra.run.smithi072.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55034.log.gz 2024-04-15T15:17:34.580 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.58428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57728.log 2024-04-15T15:17:34.580 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.55506.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58428.log.gz 2024-04-15T15:17:34.580 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53705.log 2024-04-15T15:17:34.581 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55506.log.gz 2024-04-15T15:17:34.581 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.57728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46602.log 2024-04-15T15:17:34.581 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55833.log 2024-04-15T15:17:34.581 INFO:teuthology.orchestra.run.smithi072.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57728.log.gz 2024-04-15T15:17:34.581 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53705.log.gz 2024-04-15T15:17:34.581 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.46602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57289.log 2024-04-15T15:17:34.581 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.55833.log: gzip -5 --verbose 88.5% -- /var/log/ceph/ceph-client.admin.46596.log -- replaced with /var/log/ceph/ceph-client.admin.46602.log.gz 2024-04-15T15:17:34.581 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-04-15T15:17:34.581 INFO:teuthology.orchestra.run.smithi072.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55833.log.gz 2024-04-15T15:17:34.582 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.57289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58764.log 2024-04-15T15:17:34.582 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57289.log.gz 2024-04-15T15:17:34.582 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.46596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53436.log 2024-04-15T15:17:34.582 INFO:teuthology.orchestra.run.smithi072.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46596.log.gz 2024-04-15T15:17:34.582 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.58764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53480.log 2024-04-15T15:17:34.582 INFO:teuthology.orchestra.run.smithi072.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.58764.log.gz 2024-04-15T15:17:34.582 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50747.log 2024-04-15T15:17:34.582 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53480.log: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59609.log 2024-04-15T15:17:34.583 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53436.log.gz 2024-04-15T15:17:34.583 INFO:teuthology.orchestra.run.smithi072.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53480.log.gz 2024-04-15T15:17:34.583 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49886.log 2024-04-15T15:17:34.583 INFO:teuthology.orchestra.run.smithi072.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50747.log.gz 2024-04-15T15:17:34.583 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.59609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55380.log 2024-04-15T15:17:34.583 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45932.log 2024-04-15T15:17:34.583 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49886.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59609.log.gz 2024-04-15T15:17:34.583 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.55380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50621.log 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49886.log.gz 2024-04-15T15:17:34.583 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-04-15T15:17:34.584 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45932.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55380.log.gz 2024-04-15T15:17:34.584 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50873.log 2024-04-15T15:17:34.584 INFO:teuthology.orchestra.run.smithi072.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45932.log.gz 2024-04-15T15:17:34.584 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43982.log 2024-04-15T15:17:34.584 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50873.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.50621.log.gz 2024-04-15T15:17:34.584 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44775.log 2024-04-15T15:17:34.584 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50873.log.gz 2024-04-15T15:17:34.584 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.43982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58908.log 2024-04-15T15:17:34.585 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.44775.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43982.log.gz 2024-04-15T15:17:34.585 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45499.log 2024-04-15T15:17:34.585 INFO:teuthology.orchestra.run.smithi072.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44775.log.gz 2024-04-15T15:17:34.585 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.58908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50705.log 2024-04-15T15:17:34.585 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45499.log: gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58908.log.gz 2024-04-15T15:17:34.585 INFO:teuthology.orchestra.run.smithi072.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.59357.log 2024-04-15T15:17:34.585 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45499.log.gz 2024-04-15T15:17:34.585 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51041.log 2024-04-15T15:17:34.585 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.59357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46604.log 2024-04-15T15:17:34.586 INFO:teuthology.orchestra.run.smithi072.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50705.log.gz 2024-04-15T15:17:34.586 INFO:teuthology.orchestra.run.smithi072.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59357.log.gz 2024-04-15T15:17:34.586 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.51041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53873.log 2024-04-15T15:17:34.586 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.46604.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51041.log.gz 2024-04-15T15:17:34.586 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55590.log 2024-04-15T15:17:34.586 INFO:teuthology.orchestra.run.smithi072.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46604.log.gz 2024-04-15T15:17:34.586 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45415.log 2024-04-15T15:17:34.586 INFO:teuthology.orchestra.run.smithi072.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53873.log.gz 2024-04-15T15:17:34.586 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.55590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49424.log 2024-04-15T15:17:34.587 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51115.log 2024-04-15T15:17:34.587 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55590.log.gz 2024-04-15T15:17:34.587 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45415.log: /var/log/ceph/ceph-client.admin.49424.log: gzip -5 --verbose -- 87.6% /var/log/ceph/ceph-client.admin.47053.log 2024-04-15T15:17:34.587 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45415.log.gz 2024-04-15T15:17:34.587 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49424.log.gz 2024-04-15T15:17:34.587 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.51115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58512.log 2024-04-15T15:17:34.587 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59903.log 2024-04-15T15:17:34.587 INFO:teuthology.orchestra.run.smithi072.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51115.log.gz 2024-04-15T15:17:34.588 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.47053.log: /var/log/ceph/ceph-client.admin.58512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59987.log 2024-04-15T15:17:34.588 INFO:teuthology.orchestra.run.smithi072.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47053.log.gz 2024-04-15T15:17:34.588 INFO:teuthology.orchestra.run.smithi072.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.58512.log.gz 2024-04-15T15:17:34.588 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.59903.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-04-15T15:17:34.588 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53226.log 2024-04-15T15:17:34.588 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59903.log.gz 2024-04-15T15:17:34.588 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.59987.log: /var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52205.log 2024-04-15T15:17:34.588 INFO:teuthology.orchestra.run.smithi072.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59987.log.gz 2024-04-15T15:17:34.588 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52457.log 2024-04-15T15:17:34.589 INFO:teuthology.orchestra.run.smithi072.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53226.log.gz 2024-04-15T15:17:34.589 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.52205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54734.log 2024-04-15T15:17:34.589 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.52457.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52205.log.gz 2024-04-15T15:17:34.589 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47515.log 2024-04-15T15:17:34.589 INFO:teuthology.orchestra.run.smithi072.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52457.log.gz 2024-04-15T15:17:34.589 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.54734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46600.log 2024-04-15T15:17:34.589 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.47515.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54734.log.gz 2024-04-15T15:17:34.589 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46254.log 2024-04-15T15:17:34.590 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.46600.log: gzip -5 --verbose -- 91.4% -- replaced with /var/log/ceph/ceph-client.admin.47515.log.gz 2024-04-15T15:17:34.590 INFO:teuthology.orchestra.run.smithi072.stderr: /var/log/ceph/ceph-client.admin.49130.log 2024-04-15T15:17:34.590 INFO:teuthology.orchestra.run.smithi072.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46600.log.gz 2024-04-15T15:17:34.590 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.46254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51199.log 2024-04-15T15:17:34.590 INFO:teuthology.orchestra.run.smithi072.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.46254.log.gz 2024-04-15T15:17:34.590 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50233.log 2024-04-15T15:17:34.590 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.51199.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49130.log.gz 2024-04-15T15:17:34.590 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.56518.log 2024-04-15T15:17:34.590 INFO:teuthology.orchestra.run.smithi072.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51199.log.gz 2024-04-15T15:17:34.591 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46605.log 2024-04-15T15:17:34.591 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.2.56518.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50233.log.gz 2024-04-15T15:17:34.591 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49340.log 2024-04-15T15:17:34.591 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.46605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47011.log 2024-04-15T15:17:34.591 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46605.log.gz 2024-04-15T15:17:34.591 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46598.log 2024-04-15T15:17:34.591 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49340.log.gz 2024-04-15T15:17:34.591 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58722.log 2024-04-15T15:17:34.591 INFO:teuthology.orchestra.run.smithi072.stderr: 91.4%/var/log/ceph/ceph-client.admin.47011.log: -- replaced with /var/log/ceph/ceph-client.2.56518.log.gz 2024-04-15T15:17:34.592 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.46598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52727.log 2024-04-15T15:17:34.592 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.58722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50065.log 2024-04-15T15:17:34.592 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47011.log.gz 2024-04-15T15:17:34.592 INFO:teuthology.orchestra.run.smithi072.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46598.log.gz 2024-04-15T15:17:34.592 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58722.log.gz 2024-04-15T15:17:34.592 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.52727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44943.log 2024-04-15T15:17:34.592 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52727.log.gz 2024-04-15T15:17:34.592 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.48052.log 2024-04-15T15:17:34.592 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7%/var/log/ceph/ceph-client.admin.44943.log: -- replaced with /var/log/ceph/ceph-client.admin.50065.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45583.log 2024-04-15T15:17:34.593 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-04-15T15:17:34.593 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.0.48052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59735.log 2024-04-15T15:17:34.593 INFO:teuthology.orchestra.run.smithi072.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44943.log.gz 2024-04-15T15:17:34.593 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.45583.log: -- /var/log/ceph/ceph-client.admin.51551.log 2024-04-15T15:17:34.593 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.59735.log: 90.1% -- replaced with /var/log/ceph/ceph-client.0.48052.log.gz 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45583.log.gz 2024-04-15T15:17:34.593 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-04-15T15:17:34.593 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52812.log 2024-04-15T15:17:34.593 INFO:teuthology.orchestra.run.smithi072.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59735.log.gz 2024-04-15T15:17:34.593 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54650.log 2024-04-15T15:17:34.594 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.51551.log: /var/log/ceph/ceph-client.admin.52812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49592.log 2024-04-15T15:17:34.594 INFO:teuthology.orchestra.run.smithi072.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51551.log.gz 2024-04-15T15:17:34.594 INFO:teuthology.orchestra.run.smithi072.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52812.log.gz 2024-04-15T15:17:34.594 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43356.log 2024-04-15T15:17:34.594 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.54650.log: /var/log/ceph/ceph-client.admin.49592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50999.log 2024-04-15T15:17:34.594 INFO:teuthology.orchestra.run.smithi072.stderr: 88.0%/var/log/ceph/ceph-client.admin.43356.log: -- replaced with /var/log/ceph/ceph-client.admin.49592.log.gzgzip 87.9% -5 -- replaced with /var/log/ceph/ceph-client.admin.54650.log.gz 2024-04-15T15:17:34.594 INFO:teuthology.orchestra.run.smithi072.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47305.log 2024-04-15T15:17:34.594 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-04-15T15:17:34.595 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43356.log.gz 2024-04-15T15:17:34.595 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59651.log 2024-04-15T15:17:34.595 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50999.log: /var/log/ceph/ceph-client.admin.47305.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-15T15:17:34.595 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8%/var/log/ceph/ceph-client.admin.59651.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50999.log.gz 2024-04-15T15:17:34.595 INFO:teuthology.orchestra.run.smithi072.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47305.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55548.log 2024-04-15T15:17:34.595 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-04-15T15:17:34.595 INFO:teuthology.orchestra.run.smithi072.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.59651.log.gz 2024-04-15T15:17:34.595 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53747.log 2024-04-15T15:17:34.595 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.55548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47431.log 2024-04-15T15:17:34.596 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55548.log.gz 2024-04-15T15:17:34.596 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57337.log 2024-04-15T15:17:34.596 INFO:teuthology.orchestra.run.smithi072.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53747.log.gz 2024-04-15T15:17:34.596 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.47431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55218.log 2024-04-15T15:17:34.596 INFO:teuthology.orchestra.run.smithi072.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47431.log.gz 2024-04-15T15:17:34.596 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.57337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50291.log 2024-04-15T15:17:34.596 INFO:teuthology.orchestra.run.smithi072.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57337.log.gz/var/log/ceph/ceph-client.admin.55218.log: 2024-04-15T15:17:34.596 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58680.log 2024-04-15T15:17:34.596 INFO:teuthology.orchestra.run.smithi072.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55218.log.gz 2024-04-15T15:17:34.597 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51283.log 2024-04-15T15:17:34.597 INFO:teuthology.orchestra.run.smithi072.stderr: 88.7%/var/log/ceph/ceph-client.admin.58680.log: -- replaced with /var/log/ceph/ceph-client.admin.50291.log.gz 2024-04-15T15:17:34.597 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46885.log 2024-04-15T15:17:34.597 INFO:teuthology.orchestra.run.smithi072.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58680.log.gz 2024-04-15T15:17:34.597 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.51283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57770.log 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51283.log.gz 2024-04-15T15:17:34.597 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-04-15T15:17:34.597 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.46885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46601.log 2024-04-15T15:17:34.597 INFO:teuthology.orchestra.run.smithi072.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46885.log.gz/var/log/ceph/ceph-client.admin.57770.log: 2024-04-15T15:17:34.597 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54950.log 2024-04-15T15:17:34.598 INFO:teuthology.orchestra.run.smithi072.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57770.log.gz 2024-04-15T15:17:34.598 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.46601.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54083.log 2024-04-15T15:17:34.598 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46601.log.gz 2024-04-15T15:17:34.598 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51467.log 2024-04-15T15:17:34.598 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.54950.log: /var/log/ceph/ceph-client.admin.54083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43856.log 2024-04-15T15:17:34.598 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.51467.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54083.log.gz 2024-04-15T15:17:34.598 INFO:teuthology.orchestra.run.smithi072.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54950.log.gz 2024-04-15T15:17:34.598 INFO:teuthology.orchestra.run.smithi072.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51467.log.gz 2024-04-15T15:17:34.598 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-15T15:17:34.599 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.43856.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-15T15:17:34.599 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43856.log.gz 2024-04-15T15:17:34.599 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59119.log 2024-04-15T15:17:34.599 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57131.log 2024-04-15T15:17:34.599 INFO:teuthology.orchestra.run.smithi072.stderr: 84.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-15T15:17:34.599 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.59119.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-04-15T15:17:34.599 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.57131.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59119.log.gz 2024-04-15T15:17:34.599 INFO:teuthology.orchestra.run.smithi072.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57131.log.gz 2024-04-15T15:17:34.599 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52289.log 2024-04-15T15:17:34.600 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46269.log 2024-04-15T15:17:34.600 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-mds.c.log: /var/log/ceph/ceph-client.admin.52289.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52289.log.gz 2024-04-15T15:17:34.600 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52120.log 2024-04-15T15:17:34.600 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.46269.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46269.log.gz 2024-04-15T15:17:34.600 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.48562.log 2024-04-15T15:17:34.600 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.52120.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52120.log.gz 2024-04-15T15:17:34.600 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53184.log 2024-04-15T15:17:34.600 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.2.48562.log: 91.3% -- replaced with /var/log/ceph/ceph-client.2.48562.log.gz 2024-04-15T15:17:34.600 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53617.log 2024-04-15T15:17:34.601 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53184.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53184.log.gz 2024-04-15T15:17:34.601 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52667.log 2024-04-15T15:17:34.601 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53617.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53617.log.gz 2024-04-15T15:17:34.601 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58554.log 2024-04-15T15:17:34.601 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.52667.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52667.log.gz 2024-04-15T15:17:34.601 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53524.log 2024-04-15T15:17:34.601 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.58554.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.58554.log.gz 2024-04-15T15:17:34.601 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49046.log 2024-04-15T15:17:34.601 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.53524.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53524.log.gz 2024-04-15T15:17:34.602 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43567.log 2024-04-15T15:17:34.602 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.49046.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49046.log.gz 2024-04-15T15:17:34.602 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50789.log 2024-04-15T15:17:34.602 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.43567.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43567.log.gz 2024-04-15T15:17:34.602 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52499.log 2024-04-15T15:17:34.602 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50789.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50789.log.gz 2024-04-15T15:17:34.602 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57954.log 2024-04-15T15:17:34.602 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.52499.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52499.log.gz 2024-04-15T15:17:34.603 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52331.log 2024-04-15T15:17:34.603 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.57954.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57954.log.gz 2024-04-15T15:17:34.603 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51713.log 2024-04-15T15:17:34.603 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.52331.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52331.log.gz 2024-04-15T15:17:34.603 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57812.log 2024-04-15T15:17:34.603 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.51713.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51713.log.gz 2024-04-15T15:17:34.603 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50663.log 2024-04-15T15:17:34.603 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.57812.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57812.log.gz 2024-04-15T15:17:34.603 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45541.log 2024-04-15T15:17:34.604 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50663.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50663.log.gz 2024-04-15T15:17:34.604 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44901.log 2024-04-15T15:17:34.604 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.45541.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45541.log.gz 2024-04-15T15:17:34.604 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59525.log 2024-04-15T15:17:34.604 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.44901.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44901.log.gz 2024-04-15T15:17:34.604 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51755.log 2024-04-15T15:17:34.604 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.59525.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59525.log.gz 2024-04-15T15:17:34.604 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59693.log 2024-04-15T15:17:34.604 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.51755.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51755.log.gz 2024-04-15T15:17:34.605 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-04-15T15:17:34.605 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.59693.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59693.log.gz 2024-04-15T15:17:34.605 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50915.log 2024-04-15T15:17:34.605 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-mds.a.log: 91.9% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-04-15T15:17:34.605 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43611.log 2024-04-15T15:17:34.605 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.50915.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50915.log.gz 2024-04-15T15:17:34.605 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60215.log 2024-04-15T15:17:34.605 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.43611.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43611.log.gz 2024-04-15T15:17:34.605 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48684.log 2024-04-15T15:17:34.606 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.60215.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60215.log.gz 2024-04-15T15:17:34.606 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43483.log 2024-04-15T15:17:34.606 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.48684.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48684.log.gz 2024-04-15T15:17:34.606 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58596.log 2024-04-15T15:17:34.606 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.43483.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.43483.log.gz 2024-04-15T15:17:34.606 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52078.log 2024-04-15T15:17:34.606 INFO:teuthology.orchestra.run.smithi072.stderr: 91.8%/var/log/ceph/ceph-client.admin.58596.log: -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-04-15T15:17:34.606 INFO:teuthology.orchestra.run.smithi072.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58596.log.gz 2024-04-15T15:17:34.606 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58014.log 2024-04-15T15:17:34.607 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.52078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52373.log 2024-04-15T15:17:34.607 INFO:teuthology.orchestra.run.smithi072.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52078.log.gz 2024-04-15T15:17:34.607 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.58014.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58014.log.gz 2024-04-15T15:17:34.607 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.52373.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52373.log.gz 2024-04-15T15:17:34.607 INFO:teuthology.orchestra.run.smithi072.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-04-15T15:17:34.650 INFO:teuthology.orchestra.run.smithi072.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-15T15:17:35.016 INFO:teuthology.orchestra.run.smithi175.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-15T15:17:35.024 INFO:teuthology.orchestra.run.smithi175.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-15T15:17:35.031 INFO:teuthology.orchestra.run.smithi175.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-15T15:17:35.054 INFO:teuthology.orchestra.run.smithi175.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-04-15T15:17:35.056 INFO:teuthology.orchestra.run.smithi175.stderr: 2024-04-15T15:17:35.056 INFO:teuthology.orchestra.run.smithi175.stderr:real 0m0.694s 2024-04-15T15:17:35.056 INFO:teuthology.orchestra.run.smithi175.stderr:user 0m2.783s 2024-04-15T15:17:35.056 INFO:teuthology.orchestra.run.smithi175.stderr:sys 0m0.118s 2024-04-15T15:17:35.061 INFO:teuthology.orchestra.run.smithi072.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-15T15:17:35.088 INFO:teuthology.orchestra.run.smithi072.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-15T15:17:35.097 INFO:teuthology.orchestra.run.smithi072.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-15T15:17:35.102 INFO:teuthology.orchestra.run.smithi072.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-15T15:17:35.104 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-04-15T15:17:35.104 INFO:teuthology.orchestra.run.smithi072.stderr:real 0m0.778s 2024-04-15T15:17:35.105 INFO:teuthology.orchestra.run.smithi072.stderr:user 0m3.233s 2024-04-15T15:17:35.105 INFO:teuthology.orchestra.run.smithi072.stderr:sys 0m0.322s 2024-04-15T15:17:35.105 INFO:tasks.ceph:Archiving logs... 2024-04-15T15:17:35.105 DEBUG:teuthology.misc:Transferring archived files from smithi072:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657120/remote/smithi072/log 2024-04-15T15:17:35.106 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-15T15:17:35.883 DEBUG:teuthology.misc:Transferring archived files from smithi175:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657120/remote/smithi175/log 2024-04-15T15:17:35.883 DEBUG:teuthology.orchestra.run.smithi175:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-15T15:17:36.324 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-15T15:17:36.335 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-15T15:17:36.336 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-15T15:17:36.339 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-15T15:17:36.424 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-15T15:17:36.433 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-15T15:17:36.433 DEBUG:teuthology.orchestra.run.smithi072:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T15:17:36.436 DEBUG:teuthology.orchestra.run.smithi175:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T15:17:36.450 INFO:teuthology.orchestra.run.smithi072.stderr:bash: line 1: ntpq: command not found 2024-04-15T15:17:36.453 INFO:teuthology.orchestra.run.smithi175.stderr:bash: line 1: ntpq: command not found 2024-04-15T15:17:36.455 INFO:teuthology.orchestra.run.smithi072.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T15:17:36.455 INFO:teuthology.orchestra.run.smithi072.stdout:=============================================================================== 2024-04-15T15:17:36.455 INFO:teuthology.orchestra.run.smithi072.stdout:^* hv01.front.sepia.ceph.com 3 6 377 26 +1567us[+1568us] +/- 50ms 2024-04-15T15:17:36.456 INFO:teuthology.orchestra.run.smithi072.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 26 -1816us[-1816us] +/- 76ms 2024-04-15T15:17:36.456 INFO:teuthology.orchestra.run.smithi072.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 27 -653us[ -652us] +/- 73ms 2024-04-15T15:17:36.456 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:17:36.457 INFO:teuthology.orchestra.run.smithi175.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T15:17:36.457 INFO:teuthology.orchestra.run.smithi175.stdout:=============================================================================== 2024-04-15T15:17:36.457 INFO:teuthology.orchestra.run.smithi175.stdout:^* hv01.front.sepia.ceph.com 3 6 377 26 +1581us[+1586us] +/- 50ms 2024-04-15T15:17:36.457 INFO:teuthology.orchestra.run.smithi175.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 26 -1806us[-1806us] +/- 76ms 2024-04-15T15:17:36.458 INFO:teuthology.orchestra.run.smithi175.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 25 -639us[ -639us] +/- 73ms 2024-04-15T15:17:36.458 INFO:teuthology.orchestra.run.smithi175.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:17:36.459 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-15T15:17:36.468 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-15T15:17:36.469 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-15T15:17:36.478 DEBUG:teuthology.orchestra.run.smithi072:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-15T15:17:36.579 DEBUG:teuthology.orchestra.run.smithi175:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-15T15:17:36.657 DEBUG:teuthology.orchestra.run.smithi072:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-15T15:17:36.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:17:36.691 DEBUG:teuthology.orchestra.run.smithi175:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-15T15:17:36.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:17:36.724 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-15T15:17:36.735 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-15T15:17:36.756 INFO:teuthology.task.internal:Duration was 617.397066 seconds 2024-04-15T15:17:36.756 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-15T15:17:36.767 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-15T15:17:36.767 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-15T15:17:36.770 DEBUG:teuthology.orchestra.run.smithi175:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-15T15:17:36.814 INFO:teuthology.orchestra.run.smithi072.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T15:17:36.817 INFO:teuthology.orchestra.run.smithi175.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T15:17:37.111 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-15T15:17:37.111 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi072.front.sepia.ceph.com 2024-04-15T15:17:37.111 DEBUG:teuthology.orchestra.run.smithi072:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-15T15:17:37.132 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi175.front.sepia.ceph.com 2024-04-15T15:17:37.132 DEBUG:teuthology.orchestra.run.smithi175:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-15T15:17:37.155 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-15T15:17:37.155 DEBUG:teuthology.orchestra.run.smithi072:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-15T15:17:37.175 DEBUG:teuthology.orchestra.run.smithi175:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-15T15:17:37.226 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-15T15:17:37.226 DEBUG:teuthology.orchestra.run.smithi072:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-15T15:17:37.244 DEBUG:teuthology.orchestra.run.smithi175:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-15T15:17:37.368 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-15T15:17:37.383 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-15T15:17:37.384 DEBUG:teuthology.orchestra.run.smithi072:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-15T15:17:37.413 DEBUG:teuthology.orchestra.run.smithi175:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-15T15:17:37.442 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-15T15:17:37.454 DEBUG:teuthology.orchestra.run.smithi072:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-15T15:17:37.457 DEBUG:teuthology.orchestra.run.smithi175:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-15T15:17:37.482 INFO:teuthology.orchestra.run.smithi072.stdout:kernel.core_pattern = core 2024-04-15T15:17:37.509 INFO:teuthology.orchestra.run.smithi175.stdout:kernel.core_pattern = core 2024-04-15T15:17:37.526 DEBUG:teuthology.orchestra.run.smithi072:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-15T15:17:37.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:17:37.555 DEBUG:teuthology.orchestra.run.smithi175:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-15T15:17:37.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:17:37.581 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-15T15:17:37.594 INFO:teuthology.task.internal:Transferring archived files... 2024-04-15T15:17:37.594 DEBUG:teuthology.misc:Transferring archived files from smithi072:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657120/remote/smithi072 2024-04-15T15:17:37.595 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-15T15:17:37.635 DEBUG:teuthology.misc:Transferring archived files from smithi175:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657120/remote/smithi175 2024-04-15T15:17:37.636 DEBUG:teuthology.orchestra.run.smithi175:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-15T15:17:37.673 INFO:teuthology.task.internal:Removing archive directory... 2024-04-15T15:17:37.673 DEBUG:teuthology.orchestra.run.smithi072:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-15T15:17:37.676 DEBUG:teuthology.orchestra.run.smithi175:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-15T15:17:37.723 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-15T15:17:37.735 INFO:teuthology.task.internal:Not uploading archives. 2024-04-15T15:17:37.735 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-15T15:17:37.747 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-15T15:17:37.748 DEBUG:teuthology.orchestra.run.smithi072:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-15T15:17:37.750 DEBUG:teuthology.orchestra.run.smithi175:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-15T15:17:37.763 INFO:teuthology.orchestra.run.smithi072.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 15 15:17 /home/ubuntu/cephtest 2024-04-15T15:17:37.778 INFO:teuthology.orchestra.run.smithi175.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 15 15:17 /home/ubuntu/cephtest 2024-04-15T15:17:37.779 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-15T15:17:37.797 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-15T15:17:37.830 INFO:teuthology.nuke:Checking targets against current locks 2024-04-15T15:17:37.857 DEBUG:teuthology.nuke:shortname: smithi072 2024-04-15T15:17:37.858 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-15T15:17:37.885 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi072.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657120', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 14:58:17.161213', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG+jvD3lC11LrXoo70wa+dUFmGv6veuLETc7MspcK85RTHe7zimRkbrxwOjw8TurdnsOI7szwqxlzsmbcgrpAUA='} 2024-04-15T15:17:37.886 DEBUG:teuthology.nuke:shortname: smithi175 2024-04-15T15:17:37.887 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-15T15:17:37.911 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi175.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657120', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 14:58:17.163025', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGpS+fS1n7SHFRiy96aTxyNYq1ubnETNBoHqo+uqNcC3llhjvSVSFpQM/r0JijXdeTsCN9v8KOD5Q2Fkkvx/hlQ='} 2024-04-15T15:17:37.951 INFO:teuthology.orchestra.console.smithi072:Power off 2024-04-15T15:17:37.952 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-15T15:17:37.978 DEBUG:teuthology.orchestra.console.smithi072:power off output: Chassis Power Control: Down/Off 2024-04-15T15:17:37.979 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T15:17:37.996 INFO:teuthology.orchestra.console.smithi175:Power off 2024-04-15T15:17:37.997 DEBUG:teuthology.orchestra.console.smithi175:pexpect command: ipmitool -H smithi175.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-15T15:17:38.010 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is on 2024-04-15T15:17:38.021 DEBUG:teuthology.orchestra.console.smithi175:power off output: Chassis Power Control: Down/Off 2024-04-15T15:17:38.022 DEBUG:teuthology.orchestra.console.smithi175:pexpect command: ipmitool -H smithi175.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T15:17:38.048 DEBUG:teuthology.orchestra.console.smithi175:check power output: Chassis Power is on 2024-04-15T15:17:42.011 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T15:17:42.049 DEBUG:teuthology.orchestra.console.smithi175:pexpect command: ipmitool -H smithi175.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T15:17:42.126 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is on 2024-04-15T15:17:42.165 DEBUG:teuthology.orchestra.console.smithi175:check power output: Chassis Power is on 2024-04-15T15:17:46.130 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T15:17:46.166 DEBUG:teuthology.orchestra.console.smithi175:pexpect command: ipmitool -H smithi175.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T15:17:46.241 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is off 2024-04-15T15:17:46.278 DEBUG:teuthology.orchestra.console.smithi175:check power output: Chassis Power is on 2024-04-15T15:17:46.343 INFO:teuthology.orchestra.console.smithi072:Power off completed 2024-04-15T15:17:50.282 DEBUG:teuthology.orchestra.console.smithi175:pexpect command: ipmitool -H smithi175.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T15:17:50.392 DEBUG:teuthology.orchestra.console.smithi175:check power output: Chassis Power is off 2024-04-15T15:17:50.493 INFO:teuthology.orchestra.console.smithi175:Power off completed 2024-04-15T15:17:50.598 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/quiesce} duration: 617.3970663547516 failure_reason: 'Test failure: test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce)' flavor: default owner: scheduled_leonidus@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=5fffb08842ea475c99a1ecd5351bc571 status: fail success: false 2024-04-15T15:17:50.599 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-15T15:17:50.686 INFO:teuthology.run:FAIL